A quick intro into message brokers and NATS Streaming compared to Apache Kafka

1. NATS Streaming - an
alternative to Apache Kafka?
Anton Zadorozhniy

2. Agenda
• Designs of message brokers
• NATS project
• NATS Streaming concepts
• Limitations compared to Kafka
• Advantages
• Demo

3. What’s a message broker and how it’s used
• Data management tool to communicate between two or more
applications:
• Pub/Sub, Point-to-point, Request/Response, Streams
• Common uses:
• Application integration
• Data ingestion
• Stream processing / low latency analytics

4. What’s inside message broker: two designs
Classic design
• ActiveMQ, Qpid, RabbitMQ,
NATS
• Features:
• Routing/filtering
• Re-queing
• TTL/deadletter
Commit log design
• Kafka, Pulsar, NATS Streaming
• Features:
• Stream processing
• Multiple consumers for the same
stream
• Persistence*

5. Classic design
• Think of ArrayList
• Broker tracks progress of each message, destroys message if it was
consumed
• Broker can re-que message if consumer failed to acknowledge
consumption
• Broker could implement TTL and other routing/filtering logic

6. Commit log design
• Think of LinkedList
• Only add messages to the head of list / log
• Broker always stores messages, even if it was consumed multiple
times (provides “time machine” capability)
• Multiple consumers are cheap (no data duplication)

7. NATS project
• Message broker written in Go with goals to be simple and fast
• Maintained by Cloud Native Compute Foundation
• Use cases: mobile apps, microservices and cloud native, IoT
• https://nats.io
• Initially in-memory message broker
• Seriously fast, on good machine provides 20 mln msg/s
• NATS Streaming is an extension for stream processing
• Also checkout Liftbridge

8. NATS performance (a bit old)

9. NATS Streaming: concepts
• Channels: persistent queues,
similar to Kafka topics
• Each channel is a ring buffer
• Messages are written to channels
even if there’s no subscription
• Subscriptions: progress entity,
similar to Kafka consumer groups
• Messages are written to channels
even if there’re no subscriptions
• Sequence: position in ring buffer,
similar to Kafka offset
• Also allows positioning on timestamp

10. NATS Streaming: system architecture
• Clustering: raft clusters with replicated state, only one machine is
active
• Fault tolerance: ability to use shared state between multiple NATS
Streaming services
• Partitioning: ability to assign different channels (“distribute
channels”) across multiple NATS Streaming nodes to distribute load
• Clustering and Partitioning/FT are incompatible!

11. Limitations comparing to Kafka
• Replication is only available in clustering mode, with no scalability
• No scalability for single channel
• Instead of log controlled by retention time, NATS is using ring buffer –
provides very little control on how data is deleted
• Kafka persistence is actually really great (low latency), esp for large
messages
• Rather small ecosystem with little integrations available:
• Nothing similar to Kafka Connect
• No support from things like Flink or Spark Streaming

12. Advantages of NATS Streaming
• Simplicity
• Small footprint (12 MB statically linked binary, 20 MB for NATS
Streaming)
• Ability to use existing NATS clusters

13. Demo