Hellmar Becker, Senior Sales Engineer, Confluent Title of Talk: Orchestrate Apache Kafka® on Kubernetes Abstract: Many organisations have adopted micro- services architectures to evolve their systems away from the slow and unresponsive shared-state architectures of the past. Apache Kafka is often chosen to be the backbone for these event driven services. While the main drivers for adopting micro-services architectures are to improve speed and agility for the development of applications, it also inherently increases operational complexity due to its distributed nature. To deal with that, many have encountered the need for systems that help automate and manage their micro-services deployment; adopting container technologies like Docker and orchestration frameworks like Kubernetes along the way. https://www.meetup.com/Apache-Kafka-Germany-Munich/events/265114406/

1. Orchestrate Apache
Kafka in the Cloud
Hellmar Becker
October 29, 2019

2. 2
About Me
● Hellmar Becker
● Systems Engineer @ Confluent
● Formerly at Hortonworks, ING Bank (Amsterdam),
IBM
● Lives near Munich
● Github: github.org/hellmarbecker
● Twitter: @Hellmar_Becker

3. @@Hellmar_Becker | @confluentinc
#devkafkaops

4. @Hellmar_Becker | @confluentinc

5. @@Hellmar_Becker | @confluentinc
https://twitter.com/kelseyhightower/status/963413508300812295

6. @@Hellmar_Becker | @confluentinc
https://twitter.com/kelseyhightower/status/963414038603427840

7. @@Hellmar_Becker | @confluentinc
Don’t despair…
“… not even over the fact that you don't
despair.
Just when everything seems over with,
new forces come marching up, and
precisely that means that you are alive”
Franz Kafka

8. Kafka Streaming Architecture
Fundamentals

9. @@Hellmar_Becker | @confluentinc

10. @@Hellmar_Becker | @confluentinc

11. @@Hellmar_Becker | @confluentinc
Event Streaming Platform Architecture
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka Streams
Kafka Brokers
KSQL
Kafka Streams

12. Kubernetes Fundamentals

13. @@Hellmar_Becker | @confluentinc
Monolith
Microservices
Docker
Kubernetes

14. @@Hellmar_Becker | @confluentinc
https://twitter.com/sahrizv/status/1018184792611827712

15. @@Hellmar_Becker | @confluentinc

16. @@Hellmar_Becker | @confluentinc
Orchestration
Compute
Networking
Storage
Service Discovery

17. @@Hellmar_Becker | @confluentinc
Kubernetes
Schedules and allocates resources
Networking between Pods
Storage
Service Discovery

18. @@Hellmar_Becker | @confluentinc
Refresher - Kubernetes Architecture
https://thenewstack.io/kubernetes-an-overview/
kubectl

19. @@Hellmar_Becker | @confluentinc
Pod
Basic Unit of Deployment in
Kubernetes
A collection of containers
sharing:
⦁ Namespace
⦁ Network
⦁ Volumes

20. @@Hellmar_Becker | @confluentinc
Storage
Persistent Volume (PV) & Persistent
Volume Claim (PVC)
⦁ Both PV and PVC are
‘resources’

21. @@Hellmar_Becker | @confluentinc
Storage
Persistent Volume (PV) &
Persistent Volume Claim (PVC)
⦁ PV is a piece of storage that
is provisioned dynamic or
static of any individual pod
that uses the PV

22. @@Hellmar_Becker | @confluentinc
Storage
Persistent Volume (PV) & Persistent
Volume Claim (PVC)
⦁ PVC is a request for storage by
a User

23. @@Hellmar_Becker | @confluentinc
Storage
Persistent Volume (PV) & Persistent
Volume Claim (PVC)
⦁ PVCs consume PV

24. Stateful Workloads

25. @@Hellmar_Becker | @confluentinc
StatefulSet
Rely on Headless Service to
provide network identity
Ideal for highly available
stateful workloads
Headless Service
Pod-0
Containers
Volumes
Pod-1
Containers
Volumes
Pod-2
Containers
Volumes

26. @@Hellmar_Becker | @confluentinc
StatefulSet
Rely on Headless
Service to provide
network identity
Headless Service
Pod-0
Containers
Volumes
Pod-1
Containers
Volumes
Pod-2
Containers
Volumes

27. @@Hellmar_Becker | @confluentinc
StatefulSet
Ideal for highly
available stateful
workloads
Headless Service
Pod-0
Containers
Volumes
Pod-1
Containers
Volumes
Pod-2
Containers
Volumes

28. @gamussa #DSGTech @confluentinc
Workloads Deployment

29. @@Hellmar_Becker | @confluentinc
Helm Charts

30. @@Hellmar_Becker | @confluentinc
Helm Charts

31. @@Hellmar_Becker | @confluentinc
Helm Charts

32. @@Hellmar_Becker | @confluentinc
Helm Charts
Package Manager
⦁ Package multiple K8s
resources into one
deployment unit:
Chart

33. @@Hellmar_Becker | @confluentinc
Kafka deployment checklist
PVC for Storage
StatefulSet for 3-node zk
Optional Pod Anti-Affinity to spread the ZK
ensemble across nodes
Headless Service
ConfigMap for Prometheus JMX exporter
Uses ZK Headless Svc
PVC for Storage
StatefulSet for n-node Kafka
A group of NodePort Services for external
traffic
ConfigMap for Prometheus JMX exporter

34. @gamussa #DSGTech @confluentinc
Basic components are not enough

35. @@Hellmar_Becker | @confluentinc
Meet Kubernetes Operator

36. @@Hellmar_Becker | @confluentinc
Kubernetes Operator
Embedded with operational
knowledge of both data software
and Kubernetes
⦁ Backup/restore
⦁ Scale up/down
⦁ Rebalance data
⦁ Regular health checks

37. @@Hellmar_Becker | @confluentinc
Controller
Brain behind Kubernetes
resources
⦁ e.g. replication
controller, namespace
controller etc.

38. @@Hellmar_Becker | @confluentinc
Custom Resource Definition(CRD)
Extend existing
Kubernetes API
StatefulSet ReplicaSet CRD...API
StatefulSet
Controller
ReplicaSet
Controller
Custom
Controller
Controller
StatefulSet ReplicaSet
Custom
Resource
Instance
...
...

39. @@Hellmar_Becker | @confluentinc
Custom Resource Definition(CRD)
Usually works together
Custom Controller StatefulSet ReplicaSet CRD...API
StatefulSet
Controller
ReplicaSet
Controller
Custom
Controller
Controller
StatefulSet ReplicaSet
Custom
Resource
Instance
...
...

40. @@Hellmar_Becker | @confluentinc
Custom Resource Definition(CRD)
Users can create and
access Customer
Resources with kubectl,
just as they do for built-
in resources like pods.
StatefulSet ReplicaSet CRD...API
StatefulSet
Controller
ReplicaSet
Controller
Custom
Controller
Controller
StatefulSet ReplicaSet
Custom
Resource
Instance
...
...

41. @@Hellmar_Becker | @confluentinc
Operator
Deploy and Manage your production
streaming platform with Confluent
Operator.
Automated Provisioning
Platform Operations
Resiliency
Monitoring

42. @@Hellmar_Becker | @confluentinc
Confluent Platform Reference Architecture
Each Confluent Platform
component has specific
characteristics:
Security (SSL certificates)
DNS names and zones
Host selection
Fault tolerance
Scaling
Kafka Connect
Zookeeper Nodes
Schema RegistryREST Proxy
Application
Load Balancer *
Application
Native Client
library
Application
Kafka Streams
Kafka Brokers

43. @@Hellmar_Becker | @confluentinc
Confluent Operator: Automated Provisioning
Kafka Pod Kafka Pod Kafka Pod
Storage
Load Balancer

44. @@Hellmar_Becker | @confluentinc
Confluent Operator: Scale Horizontally
Automate scaling:
Spin up new broker pod(s)
Distribute partitions to the new
broker(s)
Determine balancing plan
Execute balancing plan
Monitor resources

45. @@Hellmar_Becker | @confluentinc
Confluent Operator: Rolling Upgrade
Automated rolling upgrade with
no downtime for Kafka.
Stop broker
Wait for leader election to
complete
Start broker with new version
Wait for zero under-replicated-
partitions
Repeat

46. @Hellmar_Becker | @confluentinc
Will it fly? Let’s see

47. @@Hellmar_Becker | @confluentinc
Confluent Operator
Automate provisioning
Monitor SLAs through
Confluent Control Center
or Prometheus
Scale your Kafkas and CP
clusters elastically
Operate at scale with
enterprise support from
Confluent

48. @Hellmar_Becker | @confluentinc
Advanced use cases
vs.

49. @@Hellmar_Becker | @confluentinc
Don’t despair!

50. @@Hellmar_Becker | @confluentinc
Thanks!
@Hellmar_Becker
hellmar@confluent.io