1. Highly available Kafka Consumers and
Streams on Kubernetes
Adrian McCague, Zopa Bank

3. 0.10.0 → 3.X
3
• In Production since 2017
• Nearly 700 consumer groups
• Realtime, stream-processing applications
• Kafka Streams ‘early adopters’
• Kubernetes hosted
• … not been without pain …

4. Kafka Consumer
4
Consumer steady-state

5. Kafka Consumer
5
Consumer death
? session timeout

6. Kafka Consumer
6
Consumer join group
rebalance time

7. Kafka Consumer
7
Consumer partitions assigned

8. Kafka Consumer
8
recovery time = session timeout + rebalance time

9. Kafka Consumer
9
Consumer group steady-state

10. Kafka Consumer
10
Consumer group rebalance
rebalance time

11. Kafka Consumer
11
Consumer group (possible) transient steady-state

12. Kafka Consumer
12
Consumer group rebalance 2
rebalance time

13. Kafka Consumer
13
Consumer group new steady-state

14. High
availability
goals
14
• Minimise ‘stop-the-world’ situations
• Max availability in planned events
• Max availability in un-planned events

15. Planned
events
15
• Cluster maintenance
• Worker node maintenance
• New application deployment
• Application horizontal scaling

16. Un-planned
events
16
• Cluster auto-scaling
• Worker node termination
• Container termination
• Application horizontal scaling
• Transient network issues
• Pod pre-emption

17. Intermediate
goals
17
• Maximise replica redundancy
• Keep as many replicas up as possible
• Do not impede cluster operations

18. Kubernetes
18
Typical high-availability setup

19. Kubernetes
19
Schedule consumers (pods) evenly

20. Kubernetes
20
Schedule consumers (pods) evenly

21. Kubernetes
21
Schedule consumers (pods) evenly

22. kind: Deployment
apiVersion: apps/v1
metadata:
name: consumer-app
...
spec:
...
template:
metadata:
labels:
app.kubernetes.io/name: consumer-app
spec:
...
containers:
- ...
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- consumer-app
topologyKey: topology.kubernetes.io/zone
- weight: 1
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- consumer-app
topologyKey: kubernetes.io/hostname
Kubernetes
22
Schedule consumers (pods) evenly
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- consumer-app
topologyKey: topology.kubernetes.io/zone
- weight: 1
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- consumer-app
topologyKey: kubernetes.io/hostname

23. kind: Deployment
apiVersion: apps/v1
metadata:
name: consumer-app
...
spec:
...
template:
metadata:
labels:
app.kubernetes.io/name: consumer-app
spec:
...
containers:
- ...
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway / DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/name: consumer-app
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway / DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/name: consumer-app
Kubernetes
23
Schedule consumers (pods) evenly - in Kubernetes 1.24
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/name: consumer-app
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/name: consumer-app

24. Kubernetes
24
Ideal outcome

25. Intermediate
goals
25
• Maximise replica redundancy
• Keep as many replicas up as possible
• Do not impede cluster operations

26. kind: Deployment
apiVersion: apps/v1
metadata:
name: consumer-app
...
spec:
...
template:
metadata:
labels:
app.kubernetes.io/name: consumer-app
spec:
...
containers:
- ...
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
scheme: HTTP
timeoutSeconds: 20
periodSeconds: 15
successThreshold: 1
failureThreshold: 2
----------
kind: PodDisruptionBudget
apiVersion: policy/v1
metadata:
name: consumer-app
spec:
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/name: consumer-app
Make Kubernetes aware of consumer state
26
Configure readiness probes & Pod Disruption Budgets
containers:
- ...
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
scheme: HTTP
timeoutSeconds: 20
periodSeconds: 15
successThreshold: 1
failureThreshold: 2
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/name: consumer-app

27. Make Kubernetes aware of consumer state
27
Tie-in readiness to rebalancing state
Kafka Consumer
class org.apache.kafka.clients.consumer.KafkaConsumer<K,V>
void subscribe(Collection<String> topics, ConsumerRebalanceListener callback)
public interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener
void onPartitionsRevoked(Collection<TopicPartition> partitions)
void onPartitionsAssigned(Collection<TopicPartition> partitions)

28. Make Kubernetes aware of consumer state
28
Tie-in readiness to rebalancing state
Kafka Streams
class org.apache.kafka.streams.KafkaStreams
public void setStateListener(KafkaStreams.StateListener listener)
interface org.apache.kafka.streams.KafkaStreams.StateListener
void onChange(KafkaStreams.State newState, KafkaStreams.State
oldState)
Ready: oldState == State.REBALANCING && newState == State.RUNNING
Unready: newState != State.RUNNING

29. 29
https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java

30. Kubernetes good citizens
30
Factor out your applications carefully
• Try to stick to a single API type per application
• Kafka Streams: Interactive Queries can be an exception
• Keep your microservices ‘micro’
• Probes should be efficient and not depend on external services

31. Kubernetes will respect the disruption budget
31
Multiple scenarios supported
• Cluster maintenance
• Cluster autoscaling
• Rolling deployment*
• Involuntary disruptions*
* Counts towards the budget only

32. kind: Deployment
apiVersion: apps/v1
metadata:
name: consumer-app
...
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
...
template:
metadata:
labels:
app.kubernetes.io/name: consumer-app
spec:
...
containers:
- ...
Kubernetes rolling deployment
32
Minimise the fallout from a rollout
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
The default strategy is generally unfriendly for consumers

33. Kubernetes rolling deployment
33
Steady state

34. Kubernetes rolling deployment
34
First Pod

35. Kubernetes rolling deployment
35
First Pod Complete

36. Kubernetes rolling deployment
36
Second Pod

37. Kubernetes rolling deployment
37
Last Pod

38. Kubernetes rolling deployment
38
Rolling update complete

39. Intermediate
goals
39
• Maximise replica redundancy
• Keep as many replicas up as possible
• Do not impede cluster operations

40. High
availability
goals
40
• Minimise ‘stop-the-world’ situations
• Max availability in planned events
• Max availability in un-planned events

41. Cooperative incremental rebalancing
41
Eager rebalancing (default)

42. Cooperative incremental rebalancing
42
Localise the rebalance to where it’s required

43. Cooperative incremental rebalancing
43
Read the documentation!
Kafka Consumer
partition.assignment.strategy =
[org.apache.kafka.clients.consumer.CooperativeStickyAssignor]
Before 3.0 default: [RangeAssignor]
After 3.0 default: [RangeAssignor, CooperativeStickyAssignor]
Opting in

44. Cooperative incremental rebalancing
44
Opting in
Read the documentation!
Kafka Streams
Built into the StreamPartitionAssignor, for
free™
Available from Kafka clients 2.4
https://kafka.apache.org/34/documentation/streams/upgrade-guide

45. Almost there..
45
Minimise the rebalance time, but still wait for session timeout
recovery time = session timeout + rebalance time

46. Static consumer group membership
46
Making static membership work with well with Kubernetes
“group.instance.id”
CommonClientConfigs.GROUP_INSTANCE_ID_C
ONFIG
Works together session.timeout.ms
Increased to 45s in 3.0 (from 10s)
KIP-345: Introduce static membership protocol to reduce consumer rebalances (2.4)

47. Use the Kubernetes ‘downward API’
47
Making the pod name available to the application
kind: Deployment
apiVersion: apps/v1
metadata:
name: consumer-app
...
spec:
...
template:
...
spec:
...
containers:
- ...
env:
- name: K8S_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
...
env:
- name: K8S_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name

48. Static consumer group membership
48
Making static membership work with well with Kubernetes

49. Get a static name with StatefulSets
49
Constant set of pod names
* Bonus: no surges, pod by pod rolling deployments

50. Autoscaling?
50
Watch out for static membership
• Static membership prevents consumers from sending leave group requests
• session.timeout.ms could be set low, at the risk of more rebalances
• Dynamic membership?
• Could use an operator to scale down
interface org.apache.kafka.clients.admin.Admin
RemoveMembersFromConsumerGroupResult
removeMembersFromConsumerGroup(
String groupId, RemoveMembersFromConsumerGroupOptions options);

51. In review
51
Optimised pod layout
• Topology spread constraints
Made Kubernetes aware of replica state
• Health checks + Pod Disruption Budgets
Minimised rebalances + stop-the-world
• Incremental Co-operative Rebalancing
Minimised session timeout impact
• Static membership
recovery time = session timeout + rebalance time

52. Bonus:
Kafka Streams
52
recovery time = session timeout + rebalance time
+ task assignment time + state restore time

53. Kafka Streams State Stores
53
Stores are typically written to Kafka for durability

54. Kafka Streams State Stores
54
Allows for restoration to another replica
* Only stopping the world with streams client Kafka < 2.4

55. Kafka Streams State Stores
55
Service can resume

56. Kafka Streams State Stores
56
Standby replicas
“num.standby.replicas” =
1

57. Kafka Streams State Stores
57
Standby replicas – faster restoration of the delta
“num.standby.replicas” =
1
--

58. Kafka Streams State Stores
58
Standby replicas – after restoration
“num.standby.replicas” =
1

59. Consider persistent volumes
59
Making the pod name available to the application
kind: StatefulSet
apiVersion: apps/v1
metadata:
name: consumer-app
...
spec:
...
template:
...
spec:
...
containers:
- ...
volumeMounts:
- name: statefulstorage
mountPath: /data
volumeClaimTemplates:
- metadata:
name: statefulstorage
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp2-retain
resources:
requests:
storage: 2Gi
containers:
- ...
volumeMounts:
- name: statefulstorage
mountPath: /data
volumeClaimTemplates:
- metadata:
name: statefulstorage
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp2-retain
resources:
requests:
storage: 2Gi
state.dir = “/data”

60. Kafka Streams:
to be aware of
60
• KIP-441: Smooth scaling out of Kafka Streams (2.6)
• KIP-535: Allow IQ reads during rebalance (2.5)
• KAFKA-13439: Eager rebalancing of Kafka Streams
deprecated (3.1)
• KIP-708: Rack aware Kafka Streams apps (3.2)

61. Future
developments
61
• KIP-848: The Next Generation of the Consumer
Rebalance Protocol
• KAFKA-10199: Separate state restoration into separate
threads (3.5?)

62. Questions
Strictly Private & Confidential