Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

RabbitMQ vs Apache Kafka - Part 1

RabbitMQ vs Kafka

Messaging is at the core of many architectures and two giants in the messaging space are RabbitMQ and Apache Kafka. In this webinar we'll take a look at RabbitMQ and Kafka within the context of real-time event-driven architectures.
In this session we’re joined by guest speaker Jack Vanlightly who will explore what RabbitMQ and Apache Kafka are and their approach to messaging. Each technology has made very different decisions regarding every aspect of their design, each with strengths and weaknesses, enabling different architectural patterns.


WEBINAR LIVE DATE: Wednesday 23 May 2018 | 17:30 CEST / 16:30 BST / 11:30 EDT / 08:30 PDT

Link to video: https://www.youtube.com/watch?v=sjDnqrnnYNM

———————————————————————

SPEAKER CONTACT DETAILS

JACK VANLIGHTLY - Jack Vanlightly is a software architect based in Barcelona specialising in event-driven architectures, data processing pipelines and data stores both relational and non-relational.

Twitter: https://twitter.com/vanlightly


———————————————————————

COMPANY CONTACT DETAILS

ERLANG SOLUTIONS
- Website: https://www.erlang-solutions.com
- Twitter: https://www.twitter.com/ErlangSolutions
- LinkedIn: http://www.linkedin.com/company/erlan…
- GitHub: https://github.com/esl

  • Inicia sesión para ver los comentarios

RabbitMQ vs Apache Kafka - Part 1

  1. 1. RabbitMQ vs Apache Kafka Comparing two giants of the messaging space Part 1 – Core Concepts and Topologies
  2. 2. Background • Jack Vanlightly • Software Architect of SII Concatel, Barcelona • Event-Driven Architectures • Messaging Systems • Data Integrations • https://jack-vanlightly.com
  3. 3. The Log (Apache Kafka) The Queue (RabbitMQ) VS
  4. 4. The Log (Apache Kafka) The Queue (RabbitMQ) The Queue • First In First Out data structure • Optimized for efficient write and read operations from either end of the sequence • Stored data is transient in nature • Not shared between applications 7 6 5 4 3 2 1Write Read VS The Log • Also known as Write-Ahead Logs, Transaction Logs and Commit Logs. • Append-only. Ordered By Time. • Unique log sequence numbers. • Stored data is persistent in nature • Shared between applications 1 2 3 4 5 6 7 Reader Reader
  5. 5. RabbitMQ The Building Blocks Routing Fanout Exchange Direct Exchange Topic Exchange Header Exchange Consistent Hashing Exchange Random Exchange Sharding Queues and Consumers Queues Competing Consumers Non-Competing Consumers Ephemeral Queues Lazy Queues Priority Queues More Deadletter Exchange Alternate Exchange Virtual Hosts
  6. 6. Exchanges, Queues and Bindings • Publishers send messages to Exchanges • Exchanges route messages to Queues and even other Exchanges. • Consumers read from Queues • Bindings link Exchanges to Queues and even Exchanges to Exchanges Publisher Exchange Queue Consumer binding RabbitMQ Building Blocks Traditional queues on steroids
  7. 7. Publish – Subscribe Publisher Fanout Exchange Queue B Consumer Fanout Simplest Publish-Subscribe pattern. Each consumer receives every message Queue A Queue C Consumer Consumer
  8. 8. Publish – Subscribe with Routing Direct Exchange Routing by Routing Key Exact match routing key = binding key Publisher Direct Exchange Queue B Consumer Queue A Queue C Consumer Consumer created modified cancelled Routing Key Binding Key: created Binding Key: modified Binding Key: cancelled
  9. 9. Publish – Subscribe with Routing Topic Exchange Wildcard routing by Routing Key * = match 1 word # = match multiple words booking.baggage.added booking.baggage.removed booking.passenger.added booking.passenger.removed payment.received Publisher Topic Exchange Queue B Consumer Queue A Queue C Consumer Consumer Routing Key booking.# *.passenger.* payment.received
  10. 10. Publish – Subscribe with Routing Header Exchange Routing by message headers Publisher Header Exchange Queue B Consumer Queue A Queue C Consumer Consumer VIP=true, Country=UK, All VIP=False VIP: true Country: UK VIP=true, Country=UK, Any
  11. 11. Point-to-Point Messaging Default Exchange Direct exchange type All queues have implicit binding to the default exchange. Routes messages by Routing Key to Queue Name Publisher Default Exchange Orders Consumer Notifications Shipping Consumer Consumer Routing Key
  12. 12. Protecting Against Routing Failures Alternate Exchange Not an exchange type but a configured relationship between two exchanges. Route unrouteable messages to an alternate exchange Publisher Topic Exchange Queue Consumer Queue Unrouteable messages Consumer Consumer created modified Fanout Exchangeunrouteable
  13. 13. Scaling Out Consumers Fanout Exchange Shipping Shipping Non-Competing Consumers Each consumer receives every message Scale Out with Competing Consumers Each Shipping consumer receives a subset of the messages. Notifications Billing Billing Notifier Sales Publisher Fanout Exchange Shipping Shipping Shipping Shipping Notifications Notifier Billing Billing Sales Publisher
  14. 14. Scaling Out Queues Random Exchange Load balance messages across queues randomly Publisher Random Exchange Shipping.2 Consumer Shipping.1 Shipping.3 Consumer Consumer
  15. 15. Scaling Out Queues With Data Locality Consistent Hashing Exchange Partition messages across queues via hashing function over routing key, message header or message property Publisher Hashing Exchange Shipping.2 Consumer Shipping.1 Shipping.3 Consumer Consumer Routing Key 1/3 hash space 1/3 hash space 1/3 hash space
  16. 16. Scaling Out Queues With Data Locality Sharding Plugin and Modulus Exchange Partition messages across queues over multiple hosts via hashing function on the routing key. Publisher Modulus Exchange Host 1 Shard 2 Consumer Host 1 Shard 1 Host 1 Shard 3 Consumer Consumer Routing Key Publisher Modulus Exchange Host 2 Shard 2 Consumer Host 2 Shard 1 Host 2 Shard 3 Consumer Consumer Routing Key
  17. 17. Apply limits to queues • Length • Size • Time limits (TTL) Eject messages from a queue when: • A queue size/length limit reached. • A message has spent longer than the TTL time limit in the queue Route to a deadletter exchange to avoid message loss. Queue Limits and Deadletter Exchanges 7 6 5 4 3 2 1 X Queue Exchange Consumer Queue
  18. 18. Ephemeral • Auto-Delete Queue • Queue Expiration (TTL) • Exclusive Queues • Auto-Delete Exchanges Lazy Queues • Memory optimized queues. Priority Queues • No longer FIFO. • Publisher sets priority on messages • Priority Queue moves higher priority messages ahead of lower • Drawbacks – blocked low priority messages, low priority can eject high priority when queue is full Ephemeral Exchanges and Queues Lazy Queues Priority Queues
  19. 19. • Allows Multi-Tenant architecture • Isolation of groups of exchanges, queues and users • No routing between two virtual hosts Virtual Hosts
  20. 20. Apache Kafka The Building Blocks Topics Partitions Consumer Groups
  21. 21. • Distributed because Kafka is deployed as a cluster of nodes, for both fault tolerance and scale • Partitioned because each topic can be sharded into multiple partitions, with each partition storing a disjoint set of the data. • Replicated because partitions are usually replicated across multiple nodes (servers). • Commit Log because messages are stored in an append only fashion. Apache Kafka Core Concepts The Rise of the Distributed, Partitioned, Replicated, Commit Log
  22. 22. • Kafka has a simple Publish Subscribe model • Topics • Each Topic has one or more partitions • Each partition has a replication factor of one or more Topics, Partitions and Consumer Groups Producer ConsumerTopic Producer ConsumerPartition 2 Partition 1 Partition 3
  23. 23. Consumer Group Topics, Partitions and Consumer Groups Producer P 1 P 2 P 3 C1 C2 C3 A single consumer group of three consumers reads from a topic of three partitions. The unit of parallelism of a topic is the partition.
  24. 24. Topics, Partitions and Consumer Groups Consumer Group Producer P 1 P 2 P 3 C1 C2 C3 C4 A single consumer group of four consumers reads from a topic of three partitions. One consumer sits idle.
  25. 25. Consumer Group Topics, Partitions and Consumer Groups Producer P 1 P 2 P 5 C1 C2 C3 P 3 P 4 A single consumer group of three consumers reads from a topic of five partitions. Two consumers read from two partitions.
  26. 26. Consumer Group Topics, Partitions and Consumer Groups Producer P 1 P 2 P 3 C1 C2 C3 Consumer Group C4 C5 C6 Two consumer groups of three consumers read from a topic of three partitions
  27. 27. • Producers append messages to the end of partitions. • The partition is chosen by the producer (round robin or partition key). • Messages are removed according to the data retention policy. • Consumers keep track their position in the partitions they consume from – called the consumer offset Topics, Partitions and Consumer Groups 1 2 3 4 5 6 7 Producer Consumer 1 Write Read at offsetData retention policyX
  28. 28. Topics, Partitions and Consumer Groups 1 2 3 4 5 6 7 Producer Consumer 1 CG ARead at offset Consumer 4 CG BRead at offset Partition 1 1 2 3 4 5 6 7 Consumer 2 CG ARead at offset Consumer 5 CG B Read at offset Partition 28 1 2 3 4 5 6 Consumer 3 CG ARead at offset Consumer 6 CG BRead at offset Partition 3
  29. 29. Topics, Partitions and Consumer Groups Consumer Group Producer P 1 P 2 P 3 C1 C2 C3 Batch Batch Batch Batch
  30. 30. Log Compaction • Clean up old versions of a record (by record key). Keep latest version only. • Periodic process • Kafka as an event store Apache Kafka Log Compaction A X G U D W J X E A E O P X A X G U D W J X E A E O D X G U W J A E O D X
  31. 31. Avro Schema Registry • Send messages in a compact binary format – Avro • Schema Registry service automates sharing of Avro schemas between producers and consumers • Schema Registry to safely control evolution of schemas Apache Kafka Schema Registry Producer ConsumerTopic Schema Registry Send schema Get schema
  32. 32. Data Replication / Event Sourcing • Kafka as a single source of truth • Sits at the center of your architecture • Kafka Connect Apache Kafka Data Replication Event Sourcing Kafka Topics
  33. 33. Apache Kafka Example Topologies RabbitMQ Example Topologies VS
  34. 34. Example Scenario eCommerce Site Sales and Inventory Billing Shipping PlaceOrder OrderPlaced OrderBilledOrderPlaced eCommerce Site Sales and Inventory Billing Shipping ModifyOrder OrderModified ModificationBilledOrderModified eCommerce Site Sales and Inventory Billing Shipping CancelOrder OrderCancelled OrderCancelled
  35. 35. Example Scenario – RabbitMQ Fanout Exchanges eCommerce Site PlaceOrder SalesSales OrderPlaced BillingBilling Shipping Shipping OrderRefunded Fanout Exchange per Event, Single Queue per Application ModifyOrder CancelOrder OrderModified OrderCancelled OrderBilled Modification Billed
  36. 36. Example Scenario – RabbitMQ Fanout Exchanges eCommerce Site PlaceOrder Sales Scaling our Single Queue, Maintaining Relative Ordering ModifyOrder CancelOrder Sales.1 Sales Hashing Exchange Sales.2 Sales.3 Sales.4 Sales.5 Sales Sales Sales Sales
  37. 37. Example Scenario – RabbitMQ Topic Exchanges eCommerce Site amq.topic SalesSales ShippingShipping place.order modify.order cancel.order place.order modify.order cancel.order Billing Billing order.placed order.modified order.cancelled order.billed modification.billed order.placed order.modified order.cancelled order.billed order.modification.billed order.refunded Single Topic Exchange (amq.topic) order.placed order.modified order.cancelled
  38. 38. Example Scenario – Kafka Topics eCommerce Site SalesOrder Requests Topic Orders Topic Related Events in One Topic Billing Shipping OrderPlaced Event OrderModified Event OrderCancelled EventOrderBilled Event ModificationBilled Event OrderRefunded Event OrderShipped Event OrderShippingCancelled Event PlaceOrder Request ModifyOrder Request CancelOrder Request
  39. 39. Example Scenario – Kafka Topics Related Events in a Few Topics eCommerce Site SalesOrder Requests Topic Orders Topic Billing Shipping OrderPlaced Event OrderModified Event OrderCancelled Event OrderBilled Event ModificationBilled Event OrderRefunded Event OrderShipped Event OrderShippingCancelled Event PlaceOrder Request ModifyOrder Request CancelOrder Request Billing Topic Shipments Topic
  40. 40. Example Scenario – Kafka Topics One Topic per Event eCommerce Site SalesModify Order Requests Topic OrdersModified Topic Billing Shipping OrderPlaced Event OrderModified Event OrderCancelled Event OrderShipped Event OrderShippingCancelled Event PlaceOrder Request ModifyOrder Request CancelOrder Request Order Refunded Topic Shipments Topic Order Billed Topic ModificationBilled Topic OrdersPlaced Topic OrdersCancelled Topic CancelledShipments Topic OrderRefunded Event OrderBilled Event ModificationBilled Event Place Order Requests Topic Cancel Order Requests Topic
  41. 41. Modifying the Scenario – Class of Service eCommerce Site Sales and Inventory Billing Shipping PlaceOrder-Priority PlaceOrder-Standard OrderPlaced-Priority OrderPlaced-Standard OrderBilled-Priority OrderBilled-Standard OrderPlaced-Priority OrderPlaced-Standard eCommerce Site Sales and Inventory Billing Shipping ModifyOrder-Priority ModifyOrder-Standard OrderModified-Priority OrderModified-Standard ModificationBilled-Priority ModificationBilled-Standard OrderModified-Priority OrderModified-Standard
  42. 42. Scenario #2 – RabbitMQ Topic Exchanges eCommerce Site PlaceOrder Sales Sales-Standard OrderPlaced Billing-Priority Billing Shipping Shipping-Priority OrderRefunded Direct Exchange per Event, Routing Key with Class of Service, Queue per Class of Service ModifyOrder CancelOrder OrderModified OrderCancelled OrderBilled Modification Billed Sales-Priority Billing-Standard Shipping-Standard priority standard
  43. 43. Scenario #2 – RabbitMQ Two Layered Topic Exchanges eCommerce Site amq.topic Sales Sales-Standard Shipping-Priority place.order.standard modify.order.standard cancel.order.standard Billing-Priority Billing order.placed.priority order.modified.priority order.cancelled.priority Single Topic Exchange (amq.topic) Routing Key with event name Class of Service as Message Header Sales-Priority Billing-Standard order.placed.* order.modified.* order.cancelled.* Shipping-Standard order.placed.* order.modified.* order.cancelled.* order.billed.* modification.billed.* Shipping Sales (Topic) place.order.* modify.order.* cancel.order.* Billing (Topic) Shipping (Topic) order.billed.standard modification.billed.standard order.refunded.standard order.billed.priority modification.billed.priority order.refunded.priority #.priority #.standard #.priority #.priority #.standard #.standard place.order.priority modify.order.priority cancel.order.priority order.placed.standard order.modified.standard order.cancelled.standard
  44. 44. Scenario #2 – RabbitMQ Topic and Header Exchanges eCommerce Site amq.topic Sales Sales-Standard Shipping-Priority place.order modify.order cancel.order Billing-Priority Billing order.placed order.modified order.cancelled Single Topic Exchange (amq.topic) Routing Key with event name Class of Service as Message Header Sales-Priority Billing-Standard order.placed order.modified order.cancelled Shipping-Standard order.placed order.modified order.cancelled, order.billed modification.billed Shipping Sales (Header) place.order modify.order cancel.order Billing (Header) Shipping (Header)order.billed modification.billed order.refunded
  45. 45. Scenario #2 – RabbitMQ Topic Exchanges Priority VHost Replicate preferred routing topology without Class of Service, in two separate virtual hosts Standard VHost
  46. 46. Scenario #2 – Kafka Topics eCommerce Site Sales-Priority Order Requests Topic Orders Topic Related Events in One Topic – Dedicated Application Instances Billing-Priority Shipping-Standard OrderPlaced Event OrderModified Event OrderCancelled Event OrderBilled Event ModificationBilled Event OrderRefunded Event OrderShipped Event OrderShippingCancelled Event PlaceOrder Request ModifyOrder Request CancelOrder Request Sales-Standard Billing-Standard Shipping-Priority
  47. 47. Scenario #2 – Kafka Topics eCommerce Site Priority Orders Topic Related Events in Class of Service Based Topics – Dedicated Instances Billing-Priority Shipping-Standard OrderPlaced Event OrderModified Event OrderCancelled Event OrderBilled Event, ModificationBilled Event, OrderRefunded Event PlaceOrder Request ModifyOrder Request CancelOrder Request Sales-Priority Shipping-Priority Billing-Standard Standard Orders Topic OrderShipped Event, OrderShippingCancelled Event OrderShipped Event, OrderShippingCancelled Event OrderBilled Event, ModificationBilled Event, OrderRefunded Event Priority Order Requests Topic Sales-Standard Standard Order Requests Topic OrderPlaced Event OrderModified Event OrderCancelled Event
  48. 48. Scenario #2 – Replicate Events to Other Systems eCommerce Site Sales-Priority Order Requests Topic Orders Topic Kafka Topic as an Event Store and Data Replication Source Billing-Priority Shipping-Standard Sales-Standard Billing-Standard Shipping-Priority
  49. 49. Decoupling of publishers from subscribers Endpoint Decoupling RabbitMQ: The endpoint we publish to is decoupled from the endpoint we consume from. Kafka: The endpoint we publish to is the same as we subscribe from. Temporal Decoupling RabbitMQ: Consumers read a message once, most likely close to the time of publishing. Kafka: Consumers are decoupled temporally from the publisher. Consumers can read messages multiple times, and go back in history to read old messages.
  50. 50. RabbitMQ KafkaSimple Pub Sub Easily Evolvable Topologies Transient Messages Persistent Messages AMQP Time Travel Data Replication Event Store Simple Scaling Flexible Message Routing
  51. 51. Thank you! Questions? Jack Vanlightly 12 NOVEMBER 2018 London, UK CALL FOR TALKS close 1 June EARLY BIRD TICKETS available now!

×