2. List of Some Message Queues (MQ)
I. RabbitMQ
II. CloudAMQP (RabbitMQ as a service)
III. Amazon SNS
IV. Stormmq (MQ as a service)
V. Apache ActiveMQ
VI. SwiftMQ
VII. Sparrow
VIII.Starling
IX. Kestrel
X. Kafka (used by linkedin) 7/4/2016 2
3. I- RabbitMQ
• Robust messaging for applications
• Easy to use
• Runs on all major operating systems
• Supports a huge number of developer platforms
• Open source and commercially supported
• Languages: Java | .NET | Ruby | Python | PHP | Delphi/Free Pascal RabbitMQ Client | more...
• Cloud: Cloud Foundry | EC2 | Google Compute Engine | more...
7/4/2016 3
5. II- CloudAMQP (RabbitMQ as a
service)
• CloudAMQP are managed RabbitMQ servers in the cloud
• Hosted message queues that lets you pass messages between processes and other systems.
• Messages are published to a queue by a producer, the consumers can then get the messages off the
queue when the consumer wants to handle the messages. In-between, it can route, buffer, and
persist the messages according to rules you give it.
• Messages can be sent cross languages, platforms and OS, this way of handling
messages decouple your processes and creates a highly scalable system.
7/4/2016 5
9. III- Amazon Simple Notification
Service (Amazon SNS)
• Send push messages, Deliver messages instantly to applications or users and eliminate polling in your apps.
• Scale as your needs grow, Start fast with three simpleAPIs, and deliver an unlimited number of messages.
• Use direct-addressing to send messages to individual devices or broadcast to multiple destinations at-once.
• Use your choice of platforms, Directly access SNS using iOS, Android, Java, Python, PHP, Node.js, or .NET SDKs from AWS.
• Deliver worldwide and across multiple protocols, Send notifications to Apple iOS, Android and other mobile devices, and
destinations like Amazon SQS queues, Lambda functions, email addresses, SMS, and HTTP endpoints.
• Easily connect with other AWS services, Use SNS as a message bus to send messages, alarms, and notifications from your AWS
services such asAmazon RDS,CloudWatch, and S3 to other AWS services such as SQS and Lambda.
• Get message delivery analytics, Get delivery status information viaAmazon CloudWatch on success rates, failure rates, and dwell
times for mobile push messages as well as deliveries to SMS, SQS, HTTP, and Lambda destinations.
• Usage-based pricing,You pay only for what you use with SNS. $0.50 per million publishes.
7/4/2016 9
10. III- How SNS works
There are two types of clients: publishers and subscribers.
• Publishers communicate asynchronously with
subscribers by producing and sending a message to a
topic, which is a logical access point and
communication channel.
• Subscribers (i.e., web servers, email addresses,
Amazon SQS queues, AWS Lambda functions)
consume or receive the message or notification over
one of the supported protocols (i.e.,Amazon SQS,
HTTP/S, email, SMS, Lambda) when they are
subscribed to the topic.
7/4/2016 10
11. IV- Stormmq (MQ as a service)
• Free. As In Beer: Send and receive messages absolutely free for up to 5 connected clients. No sign up charges. No
subscriptions. Unless you want more.
• Unlimited Monthly Messages: Send. Store. Receive. Any size. Any number: 1 or 1,000,000,000. Just pay for the
storage you need. Consume messages quickly?Then storage is free.
• Internet Scale: broker scales well beyond the C10K limit of traditional MQ technology, with no sync locks, and up
to 850,000 messages / sec for ultimate users. Designed for today’s 48 core hydras.
• PostgreSQL Admin Interface
• Store & Forward
• Never lose a message. Acknowledgments, partial re-delivery, the works.
• Follows Publish & Subscribe
• Secure by Default
7/4/2016 11
12. V- Apache ActiveMQ (Apollo)
• Multi-protocol broker and supports STOMP, AMQP, MQTT, Openwire, SSL, andWebSockets.
• Queue Browsers
• Durable Subscriptions onTopics
• Mirrored Queues
• Reliable Messaging
• Message Expiration, Message Swapping, Message Selectors, Message Groups
• JAAS Authentication
• ACL based Authorization
• SSL/TLS Support and Certificate based Authentication
• REST Management API
7/4/2016 12
13. VI- SwiftMQ
• HighAvailability JMS Router with ACTIVE/STANDBY deployment and active replication.
• Transparent JMS and JNDI failover, no extra reconnect code required.
• Automatic and transparent resolve of all open transactions in transit, includingXA transactions, as well as any non-transacted
acknowledge mode.
• Full inbound and outbound duplicate message detection. Messages are never delivered twice.
• No persistent messages lost during failover. Message sequence is guaranteed.
• Choice of 3 different store types: Replicated File Store, Shared File Store, Shared JDBC Store.
• Transparent reconnect from SwiftMQ Explorer, CLI, CLIAdminAPI.
• Easy conversion to HighAvailability by HAWizard.Turns any existing configuration into HighAvailability.
• HighAvailability test suite included.Test your JMS applications under a continuous failover scenario.
• Additional Swiftlets included at no extra cost: JMS XA/ASF Swiftlet, Routing/UnlimtedSwiftlet, Standard File Store Swiftlet, JDBC
Store Swiftlet.
7/4/2016 13
14. VII- Sparrow
• fast lightweight queue written in Ruby that speaks memcache.That means you can use Sparrow with
any memcached client library (Ruby or otherwise).
• keeps messages in memory, but persists them to disk, using Sqlite, when the queue is shutdown.
• Basic tests shows that Sparrow processes messages at a rate of around 1660 per second.The load
Sparrow can cope with increases exponentially as you add to the cluster.
• Sparrow's speed greatly increases when using the included client.
• comes with built in support for daemonization and clustering.
• If the Sparrow servers aren't available, queuing can fallback to Amazon SQS
7/4/2016 14
15. VIII- Starling
• Starling is a powerful but simple messaging server that enables reliable distributed queuing with an
absolutely minimal overhead.
• It speaks the MemCache protocol for maximum cross-platform compatibility.
• Any language that speaks MemCache can take advantage of Starling's queue facilities.
• Only tested with Memcache client 1.7.x and with SystemTimer (systemtimer.rubyforge.org/)
• Starling is “slow” as far as messaging systems are concerned. In practice, it's fast enough.
7/4/2016 15
16. IX- Kestrel
• Fast, It runs on the JVM so it can take advantage of the hard work people have put into java performance.
• Small, currently about 2500 lines of scala, because it relies on Netty (a rough equivalent of Danger's ziggurat or
Ruby's EventMachine) -- and because Scala is extremely expressive.
• Durable, Queues are stored in memory for speed, but logged into a journal on disk so that servers can be
shutdown or moved without losing any data.
• Reliable, A client can ask to "tentatively" fetch an item from a queue, and if that client disconnects from kestrel
before confirming ownership of the item, the item is handed to another client. In this way, crashing clients don't
cause lost messages.
• It is not strongly ordered, While each queue is strongly ordered on each machine, a cluster will appear "loosely
ordered" because clients pick a machine at random for each operation.The end result should be "mostly fair".
• It is notTransactional, This is not a database. Item ownership is transferred with acknowledgement, but kestrel
does not support grouping multiple operations into an atomic unit.
7/4/2016 16
17. X- Kafka
• It is a distributed publish-subscribe messaging system.
• Current usage of Kafka at LinkedIn includes:
• Collecting log data (e.g., frontend activity events and service metrics) and loading them into Hadoop/DWH
for offline reporting and ad hoc analysis.
• Feeding log data directly to online applications, such as security, firehose (feeding Signal and LinkedIn
Today), wotc, and realtime service dashboard.
• Deploying derived data (e.g., customized emails and connection strength scores for Cloud) from Hadoop to
production.
• Automatic load balancing
• Asynchronous send
• Semantic partitioning
7/4/2016 17
