1. UNDER THE HOOD
Mike Schonberg

2. Agenda
 Transports
 Queues
 Publishers
 Events
 Value Added Components
 Code
 Introduction to Bridges

3. Transports
 Establish connections connections between
publishers and subscribers
 Mechanism varies widely between middlewares
• NSD (NYSE Technologies Data Fabric)
• Broker based (Avis, AMQP, TIBCO Rendezvous)
• Multicast topic resolution (Informatica LBM)
 Global configuration and tuning
 Buffer sizes
 Network parameters
 Throttle rates

4. Queues
 Queue based threading model
 One dispatch thread per queue
 Several dispatch modes
• Single event: invoke one callback and return
• Timed: dispatch until empty or timeout
• Continuous: dispatch forever
 Callback based
 Advanced features
 Watermarks to monitor queue depth
 Enqueue callback to integrate with other event loops
 Custom callbacks

5. Publishers
 Send messages and requests
 One publisher per topic
 Associated with a specific transport
 Thinnest possible wrapper to native publishing
 Support throttled publishing
 Advanced publishing framework
 dqpublisher in source
 Stateful (market data publishing)

6. Subscriptions
 Subscription to a single topic
 Receive message from all publishers of topic
 onMsg() callback invoked for each message
 Creation throttled
 Optional wildcard support
 Basic vs. stateful (market data)
 Stateful receive initial state
 Stateful monitor sequence numbers

7. Inbox
 Receives replies through onMsg() callback
 Multiple replies possible
• From different subscribers
• From the same subscriber
 Parameter to publisher_sendFromInbox()
 Supports multiple requests

8. Timers
 Repeating timers
 Interval is floating point in seconds
 Resolution dependent on native middleware or OS
 onTimerCallback()
 Enqueued on most once per interval
 Queue depth makes non-deterministic
 OpenMAMA provides default implementation

9. IO Events
 Asynchronous IO support
 For non-blocking file descriptors and sockets
 Invokes callbacks for read and write readiness
 Support is optional
 Integrates asynchronous IO into OpenMAMA event
dispatching

10. OpenMAMA: Value-added Components

11. Other Components
 Stateful market data subscriptions/data
quality
 Advanced publishing framework
 Data dictionary
 Fault tolerance

12. Stateful Subscription
 Initial value
delivered point-to-
point
 Complex sequence
number logic
 Other potential
applications

13. Data Quality
 Essential for Stateful/Market Data Subscriptions
 Sequence number monitoring
 Request recap on gap detection
 Recap is an initial image
 Pre-initial spooling
 Service disruptions from native middleware
 Callbacks inform application of data quality events
 STALE -> data definitely lost
 POSSIBLY_STALE -> data may be lost

14. Advanced Publishing
 Framework for publishing to stateful/market data
subscribers
 Sequence number and status fields for data quality
 Initial value requests
 Recap requests
 Refresh messages
 Opitional

15. Data Dictionary
 OpenMAMA supports self describing messages
 Field type, name, id (integer) and value
 Sending names with every msg -> big messages
 Dictionary maps id’s to names, types and values
 From a server as a message
 Loaded from a file
 Built dynamically
 Allows integration of non-self describing payloads

16. Fault Tolerance
 Applications can run in fault tolerant groups
 Every instance has a unique weight
 Heart beat messages determine failure
 UDP multicast
 Bridge using OpenMAMA and native middleware
 Callbacks inform instances of transitions
 Application responsible for recovery logic

17. OpenMAMA in Action

18. Let’s write a program!
 Create a publisher
 Open a session
 Pack a payload
 Send message
 Create a subscriber
 Open interest in a topic
 Receive message callback
 Unpack message

19. OpenMAMA Publisher Example
 Taken from int main(void) {
mamapublisherc.c example initializeMama ();
createIntervalTimer ();
 Publishes a message at createInboundSubscription ();
regular interval createPublisher ();
 Subscribes to a request mama_start(gMamaBridge);
topic return 0;
}
 Sends reply on receiving a
request
 Key concepts
 Publishing
 Subscribing
 Replying to requests
 Creating messages
 Timers

20. Initialization
void initializeMama(void) {
 Load middleware mama_status status;
bridge status = mama_loadBridge(&gMamaBridge,
gMiddleware);
 By name “wmw”, if (status != MAMA_STATUS_OK) {
printf ("Error loading bridge: %sn",
“lbm”, “tibrv” or “avis” mamaStatus_stringForStatus (status));
exit (status);
}
 Initialize OpenMAMA status = mama_open();
 Create or get queue mama_getDefaultEventQueue(gMamaBridge,
&gMamaDefaultQueue);
 Create a transport status = mamaTransport_allocate(&gTransport);
status = mamaTransport_create(gTransport,
• Two step }
gTransportName, gMamaBridge);
1. Allocate
2. Create (initialize)

21. Create a Timer
static void createIntervalTimer(void){
 Initialize timer status = mamaTimer_create(&gTimer,
gMamaDefaultQueue,
 Default queue from timerCb,
1.0,
initialization NULL);
}
 1 second interval
static void MAMACALLTYPE
 Repeating timerCb(mamaTimer timer, void* closure){
publishMessage (NULL);
}
 Callback publishes
message

22. Create a Subscription
static void createInboundSubscription (void){
 Subscription for gInboundCb.onCreate = createCb;
requests gInboundCb.onError
gInboundCb.onMsg
= errorCb;
= msgCb;
gInboundCb.onQuality = NULL;
 Send reply for each gInboundCb.onGap = NULL;
gInboundCb.onRecapRequest = NULL;
request mamaSubscription_allocate(&gSubscription);
mamaSubscription_createBasic(gSubscription,
 Callbacks gTransport, gMamaDefaultQueue, gInboundCb,
“REQUEST_TOPIC”, NULL);
 No messages before }
static void MAMACALLTYPE
createCb inboundMsgCb(mamaSubscription subscription,
mamaMsg msg, void* closure, void* itemClosure){
 Two step create if (!mamaMsg_isFromInbox (msg)) {
return;
}
 msgCb for each }
publishMessage (msg);
message

23. Publishing
 Publisher requires transport and static void createPublisher(void){
mamaPublisher_create(&gPub, gTransport,
topic. “MAMA_TOPIC”, NULL, NULL);
 Optional source and root }
static void publishMessage(mamaMsg request){
static int msgNumber = 0;
 Sending Messages static mamaMsg msg = NULL;
 Static message reused
if (msg == NULL) {
mamaMsg_create(&msg);
}
else {
 Standard MAMA fields: Status mamaMsg_clear(msg);
and SeqNo }
mamaMsg_addI32(msg, MamaFieldMsgStatus.mName,
MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_OK);
 Custom field: Publisher Topic mamaMsg_addI32(msg, MamaFieldSeqNum.mName,
MamaFieldSeqNum.mFid, msgNumber++);
mamaMsg_addString (msg, "PubTopic", 10002, “MAMA_TOPIC”);
 request != NULL: called from if (request){
msgCb(). Point-to-point reply mamaPublisher_sendReplyToInbox(gPublisher, request, msg);
}
else {
 Request == NULL: called from }
mamaPublisher_send(gPublisher, msg);
timerCb(). Multicast message. }

24. What Does it Do?
 Timer callback fired every second
 Sends a message on “MAMA_TOPIC”
 Listens for messages on “REQUEST_TOPIC”
 Message callback fired
• If inbound message is a request then send reply

25. Middleware Bridges - Overview

26. Bridges
 Payload and Message Bridges
 Directly reflect top-level API
• Messages, subscriptions, queues, timers, etc.
 One-to-one function mapping
 Virtual function table
 Dynamic loading and static linking
 Multiple bridges simultaneously
 OpenMAMA macros build/load table
• Bridge implementations just provide methods

27. mschonberg@nyx.com