My slides for ApacheCon North America 2014.

1. SCALING MQTT WITH
KAFKA
Tim Kellogg
April 7, 2014
@kellogh

2. • MQTT broker
• Protocol onboarding
• Cloud environment (we’re a startup)

4. • Standard
• Lightweight
o >= 2 byte overhead per message
• Easy to parse
o Length prefixed strings
• Requires very little resources on client side
o Broker keeps track of state

5. • Reliable
o QoS 1 & 2
o Last Will & Testament messages
• Secure
o Username + Password
o Tunnel over TLS

6. Publish / Subscribe
Pub
Pub
Pub
Broker
Sub
Topic/A
Topic/B
Topic/C
Topic/B
SubTopic/C
SubTopic/A

7. Topics
• foo/bar/baz
• com.example/device/17/thermo
• Patterns
• com.example/device/+/thermo
• com.example/device/#

8. Scaling Goals
• More than 2 Million
connected publishers
• More than 65,000
msg/s
• Single subscriber

9. Scaling Goals
• Amazon’s EC2
• Horizontal scaling
o Reduce cost
o Plan for the future
o Less impact from
downtime

10. Problems with Scaling MQTT

11. Load Balancing
• Which broker to connect to?
o DNS load balancing
• HAProxy
• QoS 1-2 messages stored in
Cassandra
o Consistent hash ring

12. Single Subscriber
Pub
Pub
Pub
Broker
Sub
Topic/A
Topic/B
Topic/C
Topic/#

13. Single Subscriber
Pub
Pub
Pub
Broker
Sub
Topic/A
Topic/B
Topic/C
Topic/#
Broker
Broker
LoadBalancing

14. Single Subscriber
Pub
Pub
Pub
Broker
Sub
Topic/A
Topic/B
Topic/C
Topic/#
Broker
Broker
LoadBalancing
LoadBalancing

15. Single Subscriber
Broker
Subscriber
Topic/#
Broker
Broker

16. Using HTTP

17. POST From The Broker
Pub
Pub
Pub
Broker
Topic/A
Topic/B
Topic/C Broker
Broker HTTP POST
LoadBalancing
Server
HTTP POST Server
HTTP POST Server
LoadBalancing

18. Benefits
• Easy to load balance
• Well known & well
supported

19. Drawbacks
• HTTP is heavy
• Headers
• Creating & destroying
TCP connections
• Subscriber servers must
be available
• Retry logic to guarantee
delivery

20. Apache Kafka

21. •
• Distributed log
aggregation framework
• Server to server
• “Smart” clients
• Apache ZooKeeper

22. • Append-only files per topic
o Client keeps track of what messages it’s processed
• No topic wildcards
• Key is used for out of band data
• device/42/thermo è topic: device-thermo key: 42

23. Subscriber Group
Pub
Pub
Pub
Broker

24. Subscriber Group
Pub
Pub
Pub
Broker
Broker
Broker
LoadBalancing
Kafka

25. Results

26. • Linear scaling for fire hose subscriber
• At least 2 million clients
• At least 65,000 msg/s

27. Wish List
• Security
• Configuration

28. Open Source IoT

29. The Book: Mastering The Internet of Things
Questions?
@kellogh