How much Kafka? Gwen Shapira discusses the art and science of capacity planning.

1. 1
How Much Kafka?
The Art and Science of Capacity Planning

2. 2
About Me
● Gwen Shapira
● Principal Data Architect
● At Confluent
● Committer to Apache Kafka
● Tweets. A lot. @gwenshap

3. 3
In Which We Answer Numeric
Questions About Kafka

4. 4
Factors
● Performance Requirements
● Availability Requirements
● Stability concerns
● Organizational concerns
● Operations “comfort zone”
● Kafka limitations
● Costs
● Rate of upgrades

5. 5
Choose 2:
Easy to
Operate
Cheap
Fast

6. 6
Choose 2:

7. 7
It is tradeoffs all the way down
● Retention - disk size
● Throughput - network, CPU
● Producer performance - disk IO
● Consumer performance - CPU, memory
Just performance
requirements!

8. 8
How Many Clusters?

9. 9
Separate clusters for...
● DR
● Geographical distribution
● Writing vs Reading
● Real-time vs Batch
● Dev / Test
● High throughput
● Highly reliable
● Security

10. 10
You will have many clusters.
Plan for it.

11. 11
How Many Brokers?

12. 12
Kafka is built to scale horizontally
● Largest cluster: 200+ nodes
● Lots of work on improving controller in 1.1, 2.0
● Larger / more loaded brokers mean longer restarts and recovery.
● Larger brokers require tuning to take full advantage.

13. 13
Disks depends on version
- Before 1.1: RAID 10 recommended
- 1.1 and up:
- KIP 112 - Broker will survive loss of disk
- KIP 113 - Can assign replicas to specific disks

14. 14
How Many Zookeepers?

15. 15
Right now: 3 or 5

16. 16
Future: 0

17. 17
How Many Topics?

18. 18
From capacity perspective
Topics don’t exist.

19. 19
How Many Partitions?

20. 20
The 25K question
1. Read Jun Rao blog on topic
2. More partitions == more scale
3. More partitions == more throughput
4. More partitions != more speed
5. Controller improvements in 1.1 mean more
partitions per broker

21. 21
Do not design for partition
per user or device!

22. 22
How Much Memory?

23. 23
# of partitions
*
Max Fetch Size
+
Compaction Buffer
+
“Extra”

24. 24
How Many clients?

25. 25
Lots. 50K has been done.

26. 26
Consumer groups have been tested
at 50+ consumers per group

27. 27
Not all clients are same
1. Producers have very high throughput
2. Especially when tuned
3. EOS / Order require single-writer-for-entity
4. Many consumer groups is where Kafka shines

28. 28
How Much Throughput?

29. 29
Benchmarks
● Kafka ships with performance tools
● And your fav language has tools
● Your own workload (or similar)
● Your own configuration
● Your own failure scenarios

30. 30
Tuning
● Don’t fly blind
● Why is it slow?
● Where is the bottleneck?
● Version control for all configuration
● Automate the
“change->test->observe” loop

31. 31
My broker is slow 101
● Are all brokers working?
● Did you saturate network capacity?
● Is CPU utilization high?
● Are you running an old version?
● Do you have HUGE messages?
● Is it really the broker?

32. 32
Capture as many metrics as possible
Alert / Dashboard select few

33. 33
Do you enjoy tuning, capacity
planning, monitoring and all that?
If yes - we are hiring.
If not - check out Confluent Cloud

34. 34
Questions?
https://slackpass.io/confluentcommunity
https://www.confluent.io/blog
https://confluent.cloud
@gwenshap
@confluentinc

35. 35
Resources
http://tutorials.jenkov.com/java-performance/jmh.html
http://notes.stephenholiday.com/Kafka.pdf
https://www.slideshare.net/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600
https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster

36. 36