Kafka streams at Scale Preview
Walmart.com generates millions of events per second. At WalmartLabs, I’m working in a team called the Customer Backbone (CBB), where we wanted to upgrade to a platform capable of processing this event volume in real-time and store the state/knowledge of possibly all the Walmart Customers generated by the processing. Kafka streams’ event-driven architecture seemed like the only obvious choice. However, there are a few challenges w.r.t. Walmart’s scale: • the clusters need to be large and the problems thereof. • infinite retention of changelog topics, wasting valuable disk. • slow stand-by task recovery in case of a node failure (changelog topics have GBs of data) • no repartitioning in Kafka Streams. As part of the event-driven development and addressing the challenges above, I’m going to talk about some bold new ideas we developed as features/patches to Kafka Streams to deal with the scale required at Walmart. • Cold Bootstrap: Where in case of a Kafka Streams node failure, how instead of recovering from the change-log topic, we bootstrap the standby from active’s RocksDB using JSch and zero event loss by careful offset management. • Dynamic Repartitioning: We added support for repartitioning in Kafka Streams where state is distributed among the new partitions. We can now elastically scale to any number of partitions and any number of nodes. • Cloud/Rack/AZ aware task assignment: No active and standby tasks of the same partition are assigned to the same rack. • Decreased Partition Assignment Size: With large clusters like ours (>400 nodes and 3 stream threads per node), the size of Partition Assignment of the KS cluster being few 100MBs, it takes a lot of time to settle a rebalance. Key Takeaways: • Basic understanding of Kafka Streams. • Productionizing Kafka Streams at scale. • Using Kafka Streams as Distributed NoSQL DB
Recomendados
Recomendados
Más contenido relacionado
Último
Último (20)
Destacado
Destacado (20)
Kafka streams at Scale Preview
- 1. Processing multi-million events per second Walmart
- 2. Kafka Streams at Scale Deepak Goyal Customer Backbone Walmart Labs
- 4. Event Flow kafka cluster app cluster app-0 app-0’ stand-by app-0’’ stand-by change-log topic partition 0’’ partition 0’ partition 0 input topic partition 0’’ partition 0’ partition 0 rocks db rocks db active rocks db Event Flow @walmartlabs
- 5. Challenges 1. Fault Recovery 2. Horizontal Scalability 3. Cloud Readiness 4. Restricted RocksDB 5. Large Clusters @walmartlabs
- 6. keep streaming . . . . . . . . . . . . . . . . . . .
Notas del editor
- Stateful Processing: like adding new transaction to the list of transactions for a customer. Millions of interactions on the website to be processed in real-time. Talk about non-streaming architecture alternatives like Dynomite(Netflix) and how Kakfa Streams was the right choice.
- Kafka streams seemed to be the only obvious choice but it came with its bag of challenges. Forked over an year ago, made number of commits (142).
- Single process with embedded RocksDB and akka server to serve keys from DB.
- N partitions App partition zero maps to kafka topic’s zero’th partition No apostrophe -> Active One apostrophe -> Stand-by Each broker can hold multiple partitions Each app instance can hold multiple tasks