When it comes to choosing a distributed streaming platform for real-time data pipelines, everyone knows the answer - Apache Kafka! And when it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. However, with all great things, the devil is truly in the details. While Kubernetes does provide all the building blocks that are needed, a lot of thought is required to truly create an enterprise-grade Kafka platform that can be used in production. In this technical deep dive, Michael and Viktor will go through challenges and pitfalls of managing Kafka on Kubernetes as well as the goals and lessons learned from the development of the Confluent Operator for Kubernetes. NOTE: This talk together with Michael Ng from Confluent

1. @gamussa | #KafkaSummit | @ConfluentINc
Streaming on Kubernetes:
Does it have to be the hard
way?
Fall, 2019 / San Francisco, Ca 2019
@gamussa | #kafkasummit | @ConfluentINc

2. @gamussa | #KafkaSummit | @ConfluentINc
2

3. @gamussa | #KafkaSummit | @ConfluentINc
3
I build highly scalable
Hello World
apps

4. @gamussa | #KafkaSummit | @ConfluentINc
4
Evolution of #devkafkaops
Scripts
Ansible
Containers Cloud

6. @gamussa | #KafkaSummit | @ConfluentINc
6

7. @gamussa | #KafkaSummit | @ConfluentINc
7
https://twitter.com/claesmogren/status/1108456688175079424

8. @gamussa | #KafkaSummit | @ConfluentINc
8
https://twitter.com/claesmogren/status/1108456688175079424
https://twitter.com/QuinnyPig/status/1150927901782499330

9. @gamussa | #KafkaSummit | @ConfluentINc
9
🙋

10. @gamussa | #KafkaSummit | @ConfluentINc
10
Who run stateless apps in
Kubernetes?
Who thinks it’s a good
idea?
Who run stateful apps in
Kubernetes?
Who thinks it’s a good
idea?
🙋

11. @gamussa | #KafkaSummit | @ConfluentINc
11
Well, it’s tricky ©
Translating an existing architecture to Kubernetes
External access to brokers and other components
Persistent Storage options on prem and clouds
Security Configuration and Upgrades
#devkafkaops

12. @gamussa | #KafkaSummit | @ConfluentINc
12
kafkaesque world of Kafka on
Kubernetes

13. @gamussa | #KafkaSummit | @ConfluentINc
13
Dude, you said
«it doesn’t have
to be hard way»

14. @gamussa | #KafkaSummit | @ConfluentINc
14
@Micha8LNg

15. @gamussa | #KafkaSummit | @ConfluentINc
15
DO KAFKA ON KUBERNETES DEMO
AND EVERYONE LOOSES THEIR MINDS

16. Tweet, yeah 🚀
Follow @gamussa @confluentinc
📸 🖼 % 😂
Tag @gamussa
With #KafkaSummit

17. @gamussa | #KafkaSummit | @ConfluentINc
17
Kubernetes
Fundamentals

18. @gamussa | #KafkaSummit | @ConfluentINc
18

19. @gamussa | #KafkaSummit | @ConfluentINc
19
Kubernetes
●Schedules and allocates resources
●Networking between Pods
●Storage
●Service Discovery

20. @gamussa | #KafkaSummit | @ConfluentINc
20
Kubernetes Architecture
https://thenewstack.io/kubernetes-an-overview/
kubectl

21. @gamussa | #KafkaSummit | @ConfluentINc
21

22. @gamussa | #KafkaSummit | @ConfluentINc
22
Pod
• Basic Unit of Deployment in
Kubernetes
• A collection of containers sharing:
• Namespace
• Network
• Volumes

23. @gamussa | #KafkaSummit | @ConfluentINc
23
Storage
• Persistent Volume (PV) & Persistent
Volume Claim (PVC)
• PV is a piece of storage that is
provisioned dynamic or static of
any individual pod that uses the
PV

24. @gamussa | #KafkaSummit | @ConfluentINc
24
Kubernetes
Storage classes
Fast Slow
Admin

25. @gamussa | #KafkaSummit | @ConfluentINc
25
Kubernetes
Storage classes
Fast Slow
Claim
I feel the need
for speed ©
Admin
Developer

26. @gamussa | #KafkaSummit | @ConfluentINc
26
Kubernetes
Storage classes
Fast Slow
Claim
Pod
I feel the need
for speed ©
Fast PV
Admin
Developer

27. @gamussa | #KafkaSummit | @ConfluentINc
27
Kubernetes
Storage classes
Fast Slow
Claim
Pod
I feel the need
for speed ©
Claim
Fast PV
Admin
Developer

28. @gamussa | #KafkaSummit | @ConfluentINc
28
Provisioning storage in Kubernetes
Storage
System
Kubernetes
controller
Kubernetes
node

29. @gamussa | #KafkaSummit | @ConfluentINc
29
Provisioning storage in Kubernetes
The controller requests storage system
to attach volume
Storage
System
Kubernetes
controller
Kubernetes
node

30. @gamussa | #KafkaSummit | @ConfluentINc
30
Provisioning storage in Kubernetes
The storage system attaches the volume
Storage
System
Kubernetes
controller
Kubernetes
node

31. @gamussa | #KafkaSummit | @ConfluentINc
31
Provisioning storage in Kubernetes
Kubelet requests storage system to mount
Volume
Storage
System
Kubernetes
controller
Kubernetes
node

32. @gamussa | #KafkaSummit | @ConfluentINc
32
Provisioning storage in Kubernetes
Kubelet can now request container runtime
to bind mount the volume
into the requested path in the container.
Storage
System
Kubernetes
controller
Kubernetes
node
/var/lib

33. @gamussa | #KafkaSummit | @ConfluentINc
33
But I
just
want to
deploy
kafka

35. @gamussa | #KafkaSummit | @ConfluentINc
Confluent Operator:
Apache Kafka on
Kubernetes made simple
Run Apache Kafka and Confluent
Platform as a cloud-native
application on Kubernetes to
minimize operating complexity and
increase developer agility
35
On-Premises Cloud
Confluent Platform
Confluent Operator
Kubernetes
AWS Azure GCP
RH OpenShift Mesosphere Pivotal
Docker Images

36. Custom resource
controller

37. @gamussa | #KafkaSummit | @ConfluentINc
a custom Kubernetes
controller
37
● Nodes and pods are where
Applications run on Kubernetes
● Applications use objects like
StatefulSets, Configmaps, PVs
● Custom Controllers create
custom resources that provide
unique application functionality:
○ Upgrades, elasticity, Kafka Operational
Logic
PODS
StatefulSets
ConfigMaps
PVs
Custom
Resources
API Server
Scheduler
Controllers &
Custom
Controllers
Master Node Worker Node

38. @gamussa | #KafkaSummit | @ConfluentINc
38

39. @gamussa | #KafkaSummit | @ConfluentINc
39
Workloads Deployment

40. @gamussa | #KafkaSummit | @ConfluentINc
40
Usual suspects
Certified
container images
Operator Helm Charts

42. Rolling Upgrade
Kafka Broker Upgrades:
1. Stop the broker, upgrade Kafka
2. Wait for Partition Leader reassignment
3. Start the upgraded broker
4. Wait for zero under-replicated
partitions
5. Upgrade the next broker

43. @gamussa | #KafkaSummit | @ConfluentINc
43
Will it Scale
Spin up new brokers, connect workers
easily
Manual Rebalance required in v1.0
Determine balancing plan
Execute balancing plan
Monitor Resources

44. 44
One More Thing…

45. @gamussa | #KafkaSummit | @ConfluentINc
45
Usual suspects
Certified
container images

46. @gamussa | #KafkaSummit | @ConfluentINc
46
Kotlin DSL for Kubernetes
fun main() {
val client = DefaultKafkaClient().inNamespace("operator")
println(client.kafkaClusters().list())
client.kafkaClusters().create(
newKafkaCluster {
metadata {
name = "kafka"
}
spec {
replicas = 3
}
}
)
println(client.kafkaClusters().list())
}

47. @gamussa | #KafkaSummit | @ConfluentINc
47
Kotlin DSL for Kubernetes
fun main() {
val client = DefaultKafkaClient().inNamespace("operator")
println(client.kafkaClusters().list())
client.kafkaClusters().create(
newKafkaCluster {
metadata {
name = "kafka"
}
spec {
replicas = 3
}
}
)
println(client.kafkaClusters().list())
}

48. @gamussa | #KafkaSummit | @ConfluentINc
48

49. @@gamussa | #KafkaSummit | @ConfluentINc
Thanks!
@gamussa
viktor@confluent.io
@Micha8LNg
michael.ng@confluent.io
https://slackpass.io/confluentcommunity
#kubernetes

50. 50