RabbitMQ vs Apache Kafka - Part 1

1. RabbitMQ vs Apache Kafka Comparing two giants of the messaging space Part 1 – Core Concepts and Topologies

2. Background • Jack Vanlightly • Software Architect of SII Concatel, Barcelona • Event-Driven Architectures • Messaging Systems • Data Integrations • https://jack-vanlightly.com

3. The Log (Apache Kafka) The Queue (RabbitMQ) VS

4. The Log (Apache Kafka) The Queue (RabbitMQ) The Queue • First In First Out data structure • Optimized for efficient write and read operations from either end of the sequence • Stored data is transient in nature • Not shared between applications 7 6 5 4 3 2 1Write Read VS The Log • Also known as Write-Ahead Logs, Transaction Logs and Commit Logs. • Append-only. Ordered By Time. • Unique log sequence numbers. • Stored data is persistent in nature • Shared between applications 1 2 3 4 5 6 7 Reader Reader

5. RabbitMQ The Building Blocks Routing Fanout Exchange Direct Exchange Topic Exchange Header Exchange Consistent Hashing Exchange Random Exchange Sharding Queues and Consumers Queues Competing Consumers Non-Competing Consumers Ephemeral Queues Lazy Queues Priority Queues More Deadletter Exchange Alternate Exchange Virtual Hosts

6. Exchanges, Queues and Bindings • Publishers send messages to Exchanges • Exchanges route messages to Queues and even other Exchanges. • Consumers read from Queues • Bindings link Exchanges to Queues and even Exchanges to Exchanges Publisher Exchange Queue Consumer binding RabbitMQ Building Blocks Traditional queues on steroids

7. Publish – Subscribe Publisher Fanout Exchange Queue B Consumer Fanout Simplest Publish-Subscribe pattern. Each consumer receives every message Queue A Queue C Consumer Consumer

8. Publish – Subscribe with Routing Direct Exchange Routing by Routing Key Exact match routing key = binding key Publisher Direct Exchange Queue B Consumer Queue A Queue C Consumer Consumer created modified cancelled Routing Key Binding Key: created Binding Key: modified Binding Key: cancelled

9. Publish – Subscribe with Routing Topic Exchange Wildcard routing by Routing Key * = match 1 word # = match multiple words booking.baggage.added booking.baggage.removed booking.passenger.added booking.passenger.removed payment.received Publisher Topic Exchange Queue B Consumer Queue A Queue C Consumer Consumer Routing Key booking.# *.passenger.* payment.received

10. Publish – Subscribe with Routing Header Exchange Routing by message headers Publisher Header Exchange Queue B Consumer Queue A Queue C Consumer Consumer VIP=true, Country=UK, All VIP=False VIP: true Country: UK VIP=true, Country=UK, Any

11. Point-to-Point Messaging Default Exchange Direct exchange type All queues have implicit binding to the default exchange. Routes messages by Routing Key to Queue Name Publisher Default Exchange Orders Consumer Notifications Shipping Consumer Consumer Routing Key

12. Protecting Against Routing Failures Alternate Exchange Not an exchange type but a configured relationship between two exchanges. Route unrouteable messages to an alternate exchange Publisher Topic Exchange Queue Consumer Queue Unrouteable messages Consumer Consumer created modified Fanout Exchangeunrouteable

13. Scaling Out Consumers Fanout Exchange Shipping Shipping Non-Competing Consumers Each consumer receives every message Scale Out with Competing Consumers Each Shipping consumer receives a subset of the messages. Notifications Billing Billing Notifier Sales Publisher Fanout Exchange Shipping Shipping Shipping Shipping Notifications Notifier Billing Billing Sales Publisher

14. Scaling Out Queues Random Exchange Load balance messages across queues randomly Publisher Random Exchange Shipping.2 Consumer Shipping.1 Shipping.3 Consumer Consumer

15. Scaling Out Queues With Data Locality Consistent Hashing Exchange Partition messages across queues via hashing function over routing key, message header or message property Publisher Hashing Exchange Shipping.2 Consumer Shipping.1 Shipping.3 Consumer Consumer Routing Key 1/3 hash space 1/3 hash space 1/3 hash space

16. Scaling Out Queues With Data Locality Sharding Plugin and Modulus Exchange Partition messages across queues over multiple hosts via hashing function on the routing key. Publisher Modulus Exchange Host 1 Shard 2 Consumer Host 1 Shard 1 Host 1 Shard 3 Consumer Consumer Routing Key Publisher Modulus Exchange Host 2 Shard 2 Consumer Host 2 Shard 1 Host 2 Shard 3 Consumer Consumer Routing Key

17. Apply limits to queues • Length • Size • Time limits (TTL) Eject messages from a queue when: • A queue size/length limit reached. • A message has spent longer than the TTL time limit in the queue Route to a deadletter exchange to avoid message loss. Queue Limits and Deadletter Exchanges 7 6 5 4 3 2 1 X Queue Exchange Consumer Queue

18. Ephemeral • Auto-Delete Queue • Queue Expiration (TTL) • Exclusive Queues • Auto-Delete Exchanges Lazy Queues • Memory optimized queues. Priority Queues • No longer FIFO. • Publisher sets priority on messages • Priority Queue moves higher priority messages ahead of lower • Drawbacks – blocked low priority messages, low priority can eject high priority when queue is full Ephemeral Exchanges and Queues Lazy Queues Priority Queues

19. • Allows Multi-Tenant architecture • Isolation of groups of exchanges, queues and users • No routing between two virtual hosts Virtual Hosts

20. Apache Kafka The Building Blocks Topics Partitions Consumer Groups

21. • Distributed because Kafka is deployed as a cluster of nodes, for both fault tolerance and scale • Partitioned because each topic can be sharded into multiple partitions, with each partition storing a disjoint set of the data. • Replicated because partitions are usually replicated across multiple nodes (servers). • Commit Log because messages are stored in an append only fashion. Apache Kafka Core Concepts The Rise of the Distributed, Partitioned, Replicated, Commit Log

22. • Kafka has a simple Publish Subscribe model • Topics • Each Topic has one or more partitions • Each partition has a replication factor of one or more Topics, Partitions and Consumer Groups Producer ConsumerTopic Producer ConsumerPartition 2 Partition 1 Partition 3

23. Consumer Group Topics, Partitions and Consumer Groups Producer P 1 P 2 P 3 C1 C2 C3 A single consumer group of three consumers reads from a topic of three partitions. The unit of parallelism of a topic is the partition.

24. Topics, Partitions and Consumer Groups Consumer Group Producer P 1 P 2 P 3 C1 C2 C3 C4 A single consumer group of four consumers reads from a topic of three partitions. One consumer sits idle.

25. Consumer Group Topics, Partitions and Consumer Groups Producer P 1 P 2 P 5 C1 C2 C3 P 3 P 4 A single consumer group of three consumers reads from a topic of five partitions. Two consumers read from two partitions.

26. Consumer Group Topics, Partitions and Consumer Groups Producer P 1 P 2 P 3 C1 C2 C3 Consumer Group C4 C5 C6 Two consumer groups of three consumers read from a topic of three partitions

27. • Producers append messages to the end of partitions. • The partition is chosen by the producer (round robin or partition key). • Messages are removed according to the data retention policy. • Consumers keep track their position in the partitions they consume from – called the consumer offset Topics, Partitions and Consumer Groups 1 2 3 4 5 6 7 Producer Consumer 1 Write Read at offsetData retention policyX

28. Topics, Partitions and Consumer Groups 1 2 3 4 5 6 7 Producer Consumer 1 CG ARead at offset Consumer 4 CG BRead at offset Partition 1 1 2 3 4 5 6 7 Consumer 2 CG ARead at offset Consumer 5 CG B Read at offset Partition 28 1 2 3 4 5 6 Consumer 3 CG ARead at offset Consumer 6 CG BRead at offset Partition 3

29. Topics, Partitions and Consumer Groups Consumer Group Producer P 1 P 2 P 3 C1 C2 C3 Batch Batch Batch Batch

30. Log Compaction • Clean up old versions of a record (by record key). Keep latest version only. • Periodic process • Kafka as an event store Apache Kafka Log Compaction A X G U D W J X E A E O P X A X G U D W J X E A E O D X G U W J A E O D X

31. Avro Schema Registry • Send messages in a compact binary format – Avro • Schema Registry service automates sharing of Avro schemas between producers and consumers • Schema Registry to safely control evolution of schemas Apache Kafka Schema Registry Producer ConsumerTopic Schema Registry Send schema Get schema

32. Data Replication / Event Sourcing • Kafka as a single source of truth • Sits at the center of your architecture • Kafka Connect Apache Kafka Data Replication Event Sourcing Kafka Topics

33. Apache Kafka Example Topologies RabbitMQ Example Topologies VS

34. Example Scenario eCommerce Site Sales and Inventory Billing Shipping PlaceOrder OrderPlaced OrderBilledOrderPlaced eCommerce Site Sales and Inventory Billing Shipping ModifyOrder OrderModified ModificationBilledOrderModified eCommerce Site Sales and Inventory Billing Shipping CancelOrder OrderCancelled OrderCancelled

35. Example Scenario – RabbitMQ Fanout Exchanges eCommerce Site PlaceOrder SalesSales OrderPlaced BillingBilling Shipping Shipping OrderRefunded Fanout Exchange per Event, Single Queue per Application ModifyOrder CancelOrder OrderModified OrderCancelled OrderBilled Modification Billed

36. Example Scenario – RabbitMQ Fanout Exchanges eCommerce Site PlaceOrder Sales Scaling our Single Queue, Maintaining Relative Ordering ModifyOrder CancelOrder Sales.1 Sales Hashing Exchange Sales.2 Sales.3 Sales.4 Sales.5 Sales Sales Sales Sales

37. Example Scenario – RabbitMQ Topic Exchanges eCommerce Site amq.topic SalesSales ShippingShipping place.order modify.order cancel.order place.order modify.order cancel.order Billing Billing order.placed order.modified order.cancelled order.billed modification.billed order.placed order.modified order.cancelled order.billed order.modification.billed order.refunded Single Topic Exchange (amq.topic) order.placed order.modified order.cancelled

38. Example Scenario – Kafka Topics eCommerce Site SalesOrder Requests Topic Orders Topic Related Events in One Topic Billing Shipping OrderPlaced Event OrderModified Event OrderCancelled EventOrderBilled Event ModificationBilled Event OrderRefunded Event OrderShipped Event OrderShippingCancelled Event PlaceOrder Request ModifyOrder Request CancelOrder Request

39. Example Scenario – Kafka Topics Related Events in a Few Topics eCommerce Site SalesOrder Requests Topic Orders Topic Billing Shipping OrderPlaced Event OrderModified Event OrderCancelled Event OrderBilled Event ModificationBilled Event OrderRefunded Event OrderShipped Event OrderShippingCancelled Event PlaceOrder Request ModifyOrder Request CancelOrder Request Billing Topic Shipments Topic

40. Example Scenario – Kafka Topics One Topic per Event eCommerce Site SalesModify Order Requests Topic OrdersModified Topic Billing Shipping OrderPlaced Event OrderModified Event OrderCancelled Event OrderShipped Event OrderShippingCancelled Event PlaceOrder Request ModifyOrder Request CancelOrder Request Order Refunded Topic Shipments Topic Order Billed Topic ModificationBilled Topic OrdersPlaced Topic OrdersCancelled Topic CancelledShipments Topic OrderRefunded Event OrderBilled Event ModificationBilled Event Place Order Requests Topic Cancel Order Requests Topic

41. Modifying the Scenario – Class of Service eCommerce Site Sales and Inventory Billing Shipping PlaceOrder-Priority PlaceOrder-Standard OrderPlaced-Priority OrderPlaced-Standard OrderBilled-Priority OrderBilled-Standard OrderPlaced-Priority OrderPlaced-Standard eCommerce Site Sales and Inventory Billing Shipping ModifyOrder-Priority ModifyOrder-Standard OrderModified-Priority OrderModified-Standard ModificationBilled-Priority ModificationBilled-Standard OrderModified-Priority OrderModified-Standard

42. Scenario #2 – RabbitMQ Topic Exchanges eCommerce Site PlaceOrder Sales Sales-Standard OrderPlaced Billing-Priority Billing Shipping Shipping-Priority OrderRefunded Direct Exchange per Event, Routing Key with Class of Service, Queue per Class of Service ModifyOrder CancelOrder OrderModified OrderCancelled OrderBilled Modification Billed Sales-Priority Billing-Standard Shipping-Standard priority standard

43. Scenario #2 – RabbitMQ Two Layered Topic Exchanges eCommerce Site amq.topic Sales Sales-Standard Shipping-Priority place.order.standard modify.order.standard cancel.order.standard Billing-Priority Billing order.placed.priority order.modified.priority order.cancelled.priority Single Topic Exchange (amq.topic) Routing Key with event name Class of Service as Message Header Sales-Priority Billing-Standard order.placed.* order.modified.* order.cancelled.* Shipping-Standard order.placed.* order.modified.* order.cancelled.* order.billed.* modification.billed.* Shipping Sales (Topic) place.order.* modify.order.* cancel.order.* Billing (Topic) Shipping (Topic) order.billed.standard modification.billed.standard order.refunded.standard order.billed.priority modification.billed.priority order.refunded.priority #.priority #.standard #.priority #.priority #.standard #.standard place.order.priority modify.order.priority cancel.order.priority order.placed.standard order.modified.standard order.cancelled.standard

44. Scenario #2 – RabbitMQ Topic and Header Exchanges eCommerce Site amq.topic Sales Sales-Standard Shipping-Priority place.order modify.order cancel.order Billing-Priority Billing order.placed order.modified order.cancelled Single Topic Exchange (amq.topic) Routing Key with event name Class of Service as Message Header Sales-Priority Billing-Standard order.placed order.modified order.cancelled Shipping-Standard order.placed order.modified order.cancelled, order.billed modification.billed Shipping Sales (Header) place.order modify.order cancel.order Billing (Header) Shipping (Header)order.billed modification.billed order.refunded

45. Scenario #2 – RabbitMQ Topic Exchanges Priority VHost Replicate preferred routing topology without Class of Service, in two separate virtual hosts Standard VHost

46. Scenario #2 – Kafka Topics eCommerce Site Sales-Priority Order Requests Topic Orders Topic Related Events in One Topic – Dedicated Application Instances Billing-Priority Shipping-Standard OrderPlaced Event OrderModified Event OrderCancelled Event OrderBilled Event ModificationBilled Event OrderRefunded Event OrderShipped Event OrderShippingCancelled Event PlaceOrder Request ModifyOrder Request CancelOrder Request Sales-Standard Billing-Standard Shipping-Priority

47. Scenario #2 – Kafka Topics eCommerce Site Priority Orders Topic Related Events in Class of Service Based Topics – Dedicated Instances Billing-Priority Shipping-Standard OrderPlaced Event OrderModified Event OrderCancelled Event OrderBilled Event, ModificationBilled Event, OrderRefunded Event PlaceOrder Request ModifyOrder Request CancelOrder Request Sales-Priority Shipping-Priority Billing-Standard Standard Orders Topic OrderShipped Event, OrderShippingCancelled Event OrderShipped Event, OrderShippingCancelled Event OrderBilled Event, ModificationBilled Event, OrderRefunded Event Priority Order Requests Topic Sales-Standard Standard Order Requests Topic OrderPlaced Event OrderModified Event OrderCancelled Event

48. Scenario #2 – Replicate Events to Other Systems eCommerce Site Sales-Priority Order Requests Topic Orders Topic Kafka Topic as an Event Store and Data Replication Source Billing-Priority Shipping-Standard Sales-Standard Billing-Standard Shipping-Priority

49. Decoupling of publishers from subscribers Endpoint Decoupling RabbitMQ: The endpoint we publish to is decoupled from the endpoint we consume from. Kafka: The endpoint we publish to is the same as we subscribe from. Temporal Decoupling RabbitMQ: Consumers read a message once, most likely close to the time of publishing. Kafka: Consumers are decoupled temporally from the publisher. Consumers can read messages multiple times, and go back in history to read old messages.

50. RabbitMQ KafkaSimple Pub Sub Easily Evolvable Topologies Transient Messages Persistent Messages AMQP Time Travel Data Replication Event Store Simple Scaling Flexible Message Routing

51. Thank you! Questions? Jack Vanlightly 12 NOVEMBER 2018 London, UK CALL FOR TALKS close 1 June EARLY BIRD TICKETS available now!