Balance Kafka Cluster with Zero Data Movement with Haochen Li & Yaodong Yang
Balance Kafka Cluster with Zero Data Movement with Haochen Li & Yaodong Yang
"Load balancing is a key factor in achieving high performance and cost efficiency for Kafka clusters. It helps on saving over-provisioned resources caused by skewed brokers, either CPU, memory, or disk storage. However, it’s not easy to maintain load balancing for a cluster due to the variable topic traffic pattern and capacity expansion. Data ingestion use cases have several unique characteristics: 1) There is no order requirement for data events; 2) All partitions from a topic are produced and consumed evenly; 3) Kafka producers and consumers are transparent about the partition count increase. Therefore, each partition from the same topic has the same requirements of Kafka broker resources. With these observations, we proposed a new partition placement strategy that is able to achieve the optimal cluster load balancing with Zero Data movement. The strategy will place partitions and leaders from the same topic evenly among all brokers for cluster operations, such as new topic creation, topic parallelism change, and Kafka cluster expansion. It does not need costly data partitions movement among the brokers in the runtime. Leveraging this approach, all of our Kafka clusters are balanced very well, which has resulted in measurable storage cost savings without any performance degradation."
Recomendados
Más contenido relacionado
Similar a Balance Kafka Cluster with Zero Data Movement with Haochen Li & Yaodong Yang
Similar a Balance Kafka Cluster with Zero Data Movement with Haochen Li & Yaodong Yang(20)
Más de HostedbyConfluent
Más de HostedbyConfluent(20)
Último
Último(20)
Balance Kafka Cluster with Zero Data Movement with Haochen Li & Yaodong Yang
- 1. Balance Kafka Cluster with Zero Data Movement Yaodong Yang (Apple), Haochen Li (Apple)
- 2. Yaodong Yang, Apple Inc. May, 2023 Haochen Li, Apple Inc. NOT A CONTRIBUTION Balance Kafka Cluster with Zero Data Movement
- 3. Kafka Cluster Load Balancing • Bene fi ts • High Performance • Cost E ffi ciency • Determining Factors • Kafka Partition Placement • Kafka Partition Access Pattern • Challenges • Kafka Partitions are Heterogenous • Storage Retention Requirement • Produce & Consume Tra ffi c Pattern
- 4. Current Solution • Continuously rebalance Kafka cluster based on Load Metrics • collect the load metrics from Kafka • generate the cluster load model • compute the optimization proposal • execute the proposal • Overhead • data movement between di ff erent brokers • negative impact for producers and consumers • long time to fi nish (hours or even days) • infra cost
- 5. Data Ingestion Use Case • Workload Pattern • Data events are randomly assigned to partitions from the kafka topic • All partitions from one topic are consumed evenly • Kafka producers and consumers don’t have strict requirement for Kafka Partition Count • Kafka Partitions from the same topic • Same data volumes produced, consumed and retained
- 6. Kafka Partition Replica Placement • Partition Replica Placement Strategy • Partition Count • scale_number: Number of Leader Replica per broker for a topic • partition_count = scale_number * broker_count • Partition Replica Placement • For every Kafka Topic, the number of Replicas in each broker should be the same. • For every Kafka Topic, the number of Leader Replicas in each broker should be the same. • Same load on individual Kafka Brokers. • Same hardware utilization on individual Kafka brokers • CPU • Storage Volume • Network
- 7. Scenarios • New Topic Creation • Generate the Replica Assignment for the new topic • Create the topic in the Kafka cluster with the above Replica Assignment
- 8. Scenarios • Increase Partition Count: scale_number increase • Generate the Replica Assignment for the new partitions • Create partitions in the Kafka cluster with the above Replica Assignment
- 9. Scenarios • Add more brokers • Generate the Replica Assignment for partitions in new brokers • Create partitions in the Kafka cluster with the above Replica Assignment
- 10. Scenarios • Ingestion tra ffi c volume and retention changes • no impact on the load balance of Kafka Cluster • Remove some brokers • data movement is unavoidable • avoid it with cluster migration if possible • Cluster Migration & Merge • rebalance the cluster: • partition reassignment • scale_number increase
- 11. Implementation • Current • Implemented as a Topic Operator • Deployed in production • Plan • Open a KIP in Apache Kafka Project • Contribute back to upstream
- 12. Take Away • Partition Placement Strategy can greatly improve the Load Balance of Kafka Clusters
- 13. Thank you!