Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Demystifying Kafka

521 visualizaciones

Publicado el

Kafka is an open-source distributed commit log addressing low latency, high throughput, scalability, fault-tolerance, and disk-based retention. It can be used to build tracking systems, messaging systems, high performance streaming platforms, real-time analysis, audit log….you name it. In our case, it’s been used to build a scalable event-store and messaging platform that stores billions of messages.

In this talk, we’re taking a closer look at essential Kafka concepts such as partition rebalancing, offset management, replication, producer/broker request fetching, file segments, etc. to understand what makes Kafka so scalable, resilient, performant and fault tolerant. We will also touch upon Kafka transactions to know what they are and how to leverage them. Last but not the least we will highlight some potential pitfalls that one should watch out when going to production with Kafka.

Publicado en: Ingeniería
  • Sé el primero en comentar

Demystifying Kafka

  1. 1. Demystifying Kafka By : Nakul Mishra “If you are not afraid of going deep, you will have no fear of heights either” #DevoxxBE @nklmish
  2. 2. Do you Like Lego?
  3. 3. Simplicity is the Key to complexity #DevoxxPL @nklmish#DevoxxBE @nklmish
  4. 4. #DevoxxPL @nklmish You build complex system #DevoxxBE @nklmish
  5. 5. Starting with a simple idea #DevoxxPL @nklmish#DevoxxBE @nklmish
  6. 6. Traditional Messaging + Distributed System ≠ Best Idea #DevoxxPL @nklmish#DevoxxBE @nklmish
  7. 7. #DevoxxPL @nklmish Three challenges #DevoxxBE @nklmish
  8. 8. Scalability #DevoxxPL @nklmish#DevoxxBE @nklmish
  9. 9. Throughput #DevoxxPL @nklmish#DevoxxBE @nklmish
  10. 10. Resiliency #DevoxxPL @nklmish#DevoxxBE @nklmish
  11. 11. Why Kafka? #DevoxxPL @nklmish#DevoxxBE @nklmish
  12. 12. #DevoxxPL @nklmish#DevoxxBE @nklmish • Support both Point-to-point & publish-subscribe (consumer group generalises this concept). Kafka - more than a message queue
  13. 13. #DevoxxPL @nklmish Each topic in Kafka has its own journal #DevoxxBE @nklmish Producer Support both Point-to-point & publish-subscribe Partition-0 Partition-1 Partition-2
  14. 14. #DevoxxPL @nklmish#DevoxxBE @nklmish Support both Point-to-point & publish-subscribe Rolling log files Producer Partition-0 Partition-1 Partition-2
  15. 15. #DevoxxPL @nklmish#DevoxxBE @nklmish Support both Point-to-point & publish-subscribe
  16. 16. #DevoxxPL @nklmish#DevoxxBE @nklmish • Support both Point-to-point & publish-subscribe (consumer group generalises this concept). • Highly Scalable, available & durable. • Kafka Connect interface - pull & push for data. • Architecture inherits more from storage system like HBase, Cassandra, HDFS
 vs traditional message system. • Stronger ordering guarantees than a traditional messaging system Kafka - more than a message queue [1]: New Relic 15 million messages/sec [2]: LinkedIn 1.1 Trillion messages/day [3]: Netflix 2 Trillion messages/day at Peak
  17. 17. #DevoxxPL @nklmish#DevoxxBE @nklmish Ordering guarantees, traditional messaging system R0 R1 R2 R3 … C0 C1 C2 Parallelconsumption Server side: Queue retains records in-order on the server Async delivery R0, t=1 R2, t=2 R1, t=0 M essaging system solves this using “exclusive consum er”
  18. 18. Kafka is better than MOM #DevoxxPL @nklmish#DevoxxBE @nklmish
  19. 19. I didn’t mean my Mom ;) #DevoxxPL @nklmish#DevoxxBE @nklmish
  20. 20. #DevoxxPL @nklmish#DevoxxBE @nklmish MOM vs. Kafka Broker Centric Approach Client Centric Approach Index structures (Btree or Hash Tables) Log structured Retention impacts performance Designed for Retention Outrage: Significant slow down Outrage: won’t cause infrastructure to 
 slow down significantly [1]: Large Queue Depth & Performance Problem
  21. 21. How Kafka? #DevoxxPL @nklmish#DevoxxBE @nklmish
  22. 22. “Immutability changes everything” - Pat Helland #DevoxxPL @nklmish#DevoxxBE @nklmish
  23. 23. “If simplicity is the soul of efficiency than Immutability must be the body” #DevoxxPL @nklmish#DevoxxBE @nklmish
  24. 24. #DevoxxPL @nklmish#DevoxxBE @nklmish Log Structure Log (append only journal) Writes (Append only) Reads (single seek & scan)
  25. 25. #DevoxxBE @nklmish
  26. 26. #DevoxxPL @nklmish Scalability - comes from Log #DevoxxBE @nklmish
  27. 27. C1 #DevoxxPL @nklmish Parallel Consumption - comes from Log #DevoxxBE @nklmish Partition-0 Partition-1 Payment topic Payment-consumer-group
  28. 28. C1 #DevoxxPL @nklmish#DevoxxBE @nklmish Partition-0 Partition-1 Payment topic Payment-consumer-group C2 Parallel Consumption - comes from Log
  29. 29. C1 #DevoxxPL @nklmish#DevoxxBE @nklmish Partition-0 Partition-1 Payment topic Payment-consumer-group C2 Partition-2 Parallel Consumption - comes from Log
  30. 30. #DevoxxPL @nklmish#DevoxxBE @nklmish • Doesn’t : • Even import them in JVM • Buffer messages to user space • Kernel level IO, copies directly from disk buffer to socket Java#transferTo Massive throughput - comes from Log
  31. 31. #DevoxxPL @nklmish#DevoxxBE @nklmish • OS read-ahead cache is impressive. • Stays nice & warm even if the service is restarted. Caching - comes from OS
  32. 32. Exactly Once Processing, comes from Kafka Transaction BankTransfer
 Service X Consume Y Produce #DevoxxPL @nklmish Exactly once delivery, definitive design #DevoxxBE @nklmish KIP-101 KIP-98
  33. 33. #DevoxxPL @nklmish Duplication Source? #DevoxxBE @nklmish Producer Send
  34. 34. #DevoxxPL @nklmish Duplication Source? #DevoxxBE @nklmish Producer Send Network problem (Acknowledgment error)
  35. 35. #DevoxxPL @nklmish Duplication Source? #DevoxxBE @nklmish Idempotent producer , solves this ProducerResend dups Send
  36. 36. #DevoxxPL @nklmish Duplication Source? #DevoxxBE @nklmish Consumer Fetch ProducerResend dups Send
  37. 37. #DevoxxPL @nklmish Duplication Source? #DevoxxBE @nklmish Consumer Fetch ProducerResend dups Send Consumer crash, (Error committing offset)
  38. 38. #DevoxxPL @nklmish Duplication Source? #DevoxxBE @nklmish Consumer Fetch ProducerResend dups Send Fetch dups
  39. 39. #DevoxxPL @nklmish Chandy & Lampot, marker messages #DevoxxBE @nklmish Kafka Transaction Producer Transaction Coordinator
  40. 40. #DevoxxPL @nklmish A Chandy & Lampot, marker messages #DevoxxBE @nklmish Kafka Transaction Producer Transaction Coordinator
  41. 41. #DevoxxPL @nklmish A B Tx-log Chandy & Lampot, marker messages #DevoxxBE @nklmish Kafka Transaction Producer Transaction Coordinator (Internal-Kafka-topic), RF =3 init(t.id)
  42. 42. #DevoxxPL @nklmish A B Tx-log Chandy & Lampot, marker messages #DevoxxBE @nklmish Kafka Transaction Producer Transaction Coordinator (Internal-Kafka-topic), RF =3 init(t.id) beginTransaction()
  43. 43. #DevoxxPL @nklmish A Data logs B C Tx-log Chandy & Lampot, marker messages #DevoxxBE @nklmish Kafka Transaction m0 m1 m2 m3 m4 m5 m6 Producer Transaction Coordinator init(t.id) t.id -> ongoing (Internal-Kafka-topic), RF =3
  44. 44. #DevoxxPL @nklmish A D Data logs B C Tx-log Chandy & Lampot, marker messages #DevoxxBE @nklmish Kafka Transaction m0 m1 C m2 m3 m4 C m5 m6 C Producer Transaction Coordinator init(t.id) t.id -> ongoing t.id -> prepare t.id -> committed (Internal-Kafka-topic), RF =3
  45. 45. #DevoxxPL @nklmish A D Data logs B C Tx-log Chandy & Lampor, marker messages #DevoxxBE @nklmish Kafka Transaction m0 m1 C m2 m3 m4 C m5 m6 C Producer Transaction Coordinator init(t.id) t.id -> ongoing t.id -> prepare t.id -> committed Consum er: isolation.level = read_com m itted (Internal-Kafka-topic), RF =3
  46. 46. #DevoxxPL @nklmish#DevoxxBE @nklmish • Rock solid replication protocol and leader election process. • Relies on Replication to avoid sync calls!! Resiliency - comes from Replication [1]: PacificA
  47. 47. #DevoxxPL @nklmish Filesystem writes 
 “foo” to disk Hard disk #DevoxxBE @nklmish Write path
  48. 48. #DevoxxPL @nklmish Buffer Cache Written to Hard disk #DevoxxBE @nklmish Filesystem 
 writes “Foo” to disk Caches Write path
  49. 49. #DevoxxPL @nklmish Buffer Cache Filesystem 
 writes “Foo” to disk Written to Disk drive controller cache Moves to Caches Hard disk #DevoxxBE @nklmish Write path
  50. 50. #DevoxxPL @nklmish Buffer Cache Filesystem 
 writes “Foo” to disk Written to Disk drive controller cache Moves to Caches Disk Caches Moves to Hard disk #DevoxxBE @nklmish Write path
  51. 51. #DevoxxPL @nklmish Buffer Cache Filesystem 
 writes “Foo” to disk Written to Disk drive controller cache Moves to Caches Hardware-level cache 
 (write-through or write-back) Disk Caches Hard disk Moves to #DevoxxBE @nklmish Write path
  52. 52. #DevoxxPL @nklmish Buffer Cache Filesystem 
 writes “Foo” to disk Written to Disk drive controller cache Moves to Caches Hardware-level cache 
 (write-through or write-back) Disk Caches Moves to Finally written to disk Hard disk #DevoxxBE @nklmish Write path
  53. 53. Replication P0 P0 P0 Kafka cluster Leader A B C Kafka partition == Replicated Log #DevoxxBE @nklmish Producer P0
  54. 54. Replication P0 P0 P0 Kafka cluster Leader A B C Kafka partition == Replicated Log Partition: P0
 Leader: A ISR: [B, C] #DevoxxBE @nklmish Producer P0
  55. 55. Replication P0 P0 P0 Kafka cluster Leader A B C Kafka partition == Replicated Log Partition: P0
 Leader: A ISR: [B, C] #DevoxxBE @nklmish Producer P0 Slow
  56. 56. Replication P0 P0 P0 Kafka cluster Leader A B C Kafka partition == Replicated Log Partition: P0
 Leader: A ISR: [B] #DevoxxBE @nklmish Producer P0 Slow
  57. 57. Replication P0 P1 P0 P1 P0 P1 Kafka cluster Leader Broker-0 Broker-1 Broker-2 Kafka partition == Replicated Log Partition: P0
 Leader: Broker-0 #DevoxxBE @nklmish Producer P1
  58. 58. Replication P0 P1 P0 P1 P0 P1 Kafka cluster Leader Broker-0 Broker-1 Broker-2 Kafka partition == Replicated Log Partition: P0
 Leader: Broker-0 #DevoxxBE @nklmish Producer Partition: P1
 Leader: Broker-1
  59. 59. Replication P0 P1 P2 P0 P1 P2 P0 P1 P2 Kafka cluster Leader Broker-0 Broker-1 Broker-2 Kafka partition == Replicated Log Partition: P0
 Leader: Broker-0 #DevoxxBE @nklmish Producer Partition: P1
 Leader: Broker-1 Partition: P2
 Leader: Broker-2
  60. 60. #DevoxxPL @nklmish Kafka & Event sourcing? #DevoxxBE @nklmish
  61. 61. #DevoxxPL @nklmish#DevoxxBE @nklmish Events ordering - send to same partition (aggregate Identifier). Events replay - Log (replayable) Data Retention & Durability Event store & projections - let’s find out Kafka - for event sourcing ?
  62. 62. Event store ? #DevoxxPL @nklmish#DevoxxBE @nklmish
  63. 63. Event Store #DevoxxPL @nklmish Time #DevoxxBE @nklmish
  64. 64. Event Store #DevoxxPL @nklmish Time #DevoxxBE @nklmish
  65. 65. Event Store #DevoxxPL @nklmish Time #DevoxxBE @nklmish
  66. 66. Event Store #DevoxxPL @nklmish Time #DevoxxBE @nklmish
  67. 67. Event Store #DevoxxPL @nklmish Time #DevoxxBE @nklmish
  68. 68. Event Store #DevoxxPL @nklmish validate sequence number Time #DevoxxBE @nklmish
  69. 69. Event Store - Load aggregate #DevoxxPL @nklmish Time #DevoxxBE @nklmish
  70. 70. #DevoxxPL @nklmish Time Event Store - Load aggregate #DevoxxBE @nklmish
  71. 71. #DevoxxPL @nklmish Event Store - aggregate loaded #DevoxxBE @nklmish
  72. 72. What’s missing #DevoxxPL @nklmish • Identity & Concurrency Control • Load efficiently events for specific aggregates #DevoxxBE @nklmish }No builtin support in Kafka, but…
  73. 73. #DevoxxPL @nklmish Kafka, a database inside out Kafka Commit Log Stream Processor KSQ L Kafka Indexes Commit log Query engine Caching Traditional database #DevoxxBE @nklmish
  74. 74. #DevoxxPL @nklmish#DevoxxBE @nklmish • Following options: • Optimistic concurrency control • Single write principle Identity & concurrency control
  75. 75. #DevoxxPL @nklmish#DevoxxBE @nklmish • Kafka Streams Aggregates & Projections
  76. 76. Business logic Kafka Stream WalletService Kafka Streams #DevoxxBE @nklmish Bob:+100 Doe:-100 Bob:+200 Bob:+400 Lee:+400 Bob:-200 Jen:+200 Doe:+200 Jen:+100 1. Default state store, rockDB
 2. Local writes to event store -> pushed back to Kafka State store
  77. 77. #DevoxxPL @nklmish#DevoxxBE @nklmish • Be conscious, when setting deletion policy -> (Integer.MAX_VALUE) is not good ;) • Shared storage, remember neighbours can be chatty. • Avoid network throttling & instability, enforce Quotas. • Adding new broker != automatic data partitions assignment for existing topic, use bin/kafka-reassign-partitions.sh • Aggregate (Stateful processing)-> use compacted topic -> avoid loading whole versioned history. • Slow controlled shutdown, don’t use kill command -> upgrade to Kafka 1.1.0 :) What we learned in the process?
  78. 78. #DevoxxPL @nklmish#DevoxxBE @nklmish • Kafka is always on • Designed from ground up for distributed world • Kafka - a database inside out • Its fun to play lego How Kafka helped us go through the 3 challenges?
  79. 79. #DevoxxPL @nklmish#DevoxxBE @nklmish

×