Kafka Retry and DLQ work presented for Bay Area Apache Kafka Meetup.

1. Reliable Messaging Systems with Kafka at Uber
George Teo
Software Engineer @ Streaming Data
June 19, 2018

2. Agenda
● Kafka @ Uber
● What is Messaging?
● High-level design
● Future work

3. Stream
Processing
- Driver-Rider Match
- ETA
App Views
Vehicle Information
KAFKA
Real-time Driver-Rider Matching

4. A bunch more...
● Fraud Detection
● Share My ETA
● Driver & Rider Signups
● Etc.

5. Apache Kafka - Use Cases
● Inter-service messaging
● Stream processing
○ AthenaX - self-serve platform (Samza, Flink)
● Database changelog transport
○ Schemaless, Cassandra, MySQL
● Data Lake Ingestion
○ HDFS, S3
● Log Aggregation

6. PRODUCERS
CONSUMERS
Real-time
Analytics, Alerts,
Dashboards
Samza / Flink
Applications
Data Science
Analytics
Reporting
Kafka
Vertica / Hive
Rider App
Driver App
API / Services
Etc.
Ad-hoc Exploration
ELK
Kafka @ Uber
Debugging
Hadoop
Surge Mobile App
Cassandra
Schemaless
MySQL
DATABASES
AWS S3
(Internal) Services

7. Apache Kafka - Use Cases
● Inter-service messaging
● Stream processing
○ AthenaX - self-serve platform (Samza, Flink)
● Database changelog transport
○ Schemaless, Cassandra, MySQL
● Ingestion
○ HDFS, S3
● Logging

8. Trillions+ ~PBs
Messages/Day
Data Volume
Scale
excluding replication
Tens of
Thousands
Topics
10+ clusters
100+ broker per
cluster

9. Agenda
● Kafka @ Uber
● What is Messaging?
● High-level Design
● Future Work

10. Messaging
● Streaming
● Queuing

11. Messaging
● Streaming
○ Ordered

12. Streaming - Ordered
Partition
0 1 2 3 4 5 6 7 8 9 10 11 12
Commit Offset 7
Consumer
Application
78

13. Messaging
● Streaming
○ Ordered
○ Commit up to

14. Streaming - Commit Up To
Partition
0 1 2 3 4 5 6 7 8 9 10 11 12
Commit Offset 9
Consumer
Application
789

15. Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned

16. Streaming - Partitioned
Kafka Topic
Partition 0
0 1 2 3 4 5 6 7 8 9 10 11 12
Partition 1
0 1 2 3 4 5 6 7 8 9 10 11 12
Partition 2
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Instance 2
Consumer
Instance 1
Consumer Application

17. Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned Consumer
● Queuing*
○ Unordered

18. Queueing- Unordered
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application
6
Queue
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application

19. Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned Consumer
● Queuing*
○ Unordered
○ Ack

20. Queueing- Ack
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application
6
Queue
0 1 2 3 4 5 6 7 8 9 10 11 12
Ack 6
Consumer
Application

21. Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned Consumer
● Queuing*
○ Unordered
○ Ack/Nack

22. Queueing- Nack
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application
Queue
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application
Nack 4
5
Ack 5
4

23. Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned Consumer
● Queuing*
○ Unordered
○ Ack/Nack
○ Redelivery/DLQ

24. Queueing- Redelivery/DLQ
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application
Queue
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application4
Nack 4

25. Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned Consumer
● Queuing*
○ Unordered
○ Ack/Nack
○ Redelivery/DLQ
○ Competing Consumer

26. Queuing - Competing
Consumer
Instance 2
Consumer
Instance 1
Consumer Application
0 1 2 3 4 5 6 7 8 9 10 11 12
Queue
0 1 2 3 4 5 6 7 8 9 10 11 12

27. Why Streaming?
● Data Analytics
● Event-driven Architecture/Event Sourcing
● Log Aggregation
● Scalability*

28. Why Queuing?
● More friendly acknowledgement semantics
● Non-blocking retry
● Elastic Consumer scalability
● High consumer throughput

29. Agenda
● Kafka @ Uber
● Motivation
● High-level design
● Future work

30. Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter failover
● Golang, Java, Python client

31. Why Kafka?
● Other options:
○ ___MQ
○ Apache Pulsar
○ Build yet another messaging queue from ground up
● Why Kafka?
○ Reliability
○ Performance
○ Cost efficiency
■ Operational, engineering costs

32. Native Kafka Consumer
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process

33. Consumer
Process
Competing Consumer
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1

34. Competing Consumer
Kafka Topic
Partition 0
0 1 2 3 4 5 6 7 8 9 10 11 12
Partition 1
0 1 2 3 4 5 6 7 8 9 10 11 12
Partition 2
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Instance 2
Consumer
Instance 1
Consumer Application
Worker 1
Worker 2
Worker 3
Worker 4

35. Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client

36. Message Flow - Incoming Message
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
Consumer
Worker1Consumer
Worker1Consumer
Worker1
1
56 4
3

37. Message Flow - Inflight Message Tracker
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1
2

38. Message Flow - Passed to worker
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1
2
3
4
5

39. Consumer Application
In-Flight Offset tracker
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed

40. Consumer Application
In-flight Offset Tracker
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
3

41. Consumer Application
In-flight Offset Tracker
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
3 6 4 5

42. Consumer Application
In-flight Offset Tracker - Ack
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
3 6 4

43. Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client

44. Consumer Application
In-flight Offset Tracker
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
3 6 4 7

45. Consumer Application
In-flight Offset Tracker
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
3 6 8 7

46. Consumer Application
In-flight Offset Tracker - Nack
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
6 8 7

47. topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 6 7 8
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 (3)
3
Nack

48. Consumer Application
In-flight Offset Tracker - Nack
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
6 8 7

49. Ack
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
6 7 8 9
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 (3)
4
Commit up to 5

50. Consumer Application
In-flight Offset Tracker - Nack
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
6 8 7

51. Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client

52. topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 (3)
5
Redelivery
6 7 8 9
0 (3)

53. Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client

54. Delay
Delay Between Retries
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 1
5
6 7 8 9

55. Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client

56. topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 1 2 3 4 5 6
topic__cg__dlq
Partition 0
0 1 2 3 4 5 6
Dead Letter Queue

57. Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client

58. topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 1 2 3 4 5 6
topic__cg__dlq
Partition 0
0 1 2 3 4 5 6
Purge
DLQ - Purge

59. topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 1 2 3 4 5 6
topic__cg__dlq
Partition 0
0 1 2 3 4 5 6
Merge
DLQ - Merge

60. Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client

61. DC2
DC1
Regional
Kafka
Regional
Kafka
Aggregate
Kafka
uReplicator
Offset Sync Service
Aggregate
Kafka
uReplicator
Kafka Setup @ Uber

62. DC2
DC1
Consumer
Application
Active
Consumer
Application
Passive
Multi-DC Consumer
Regional
Kafka
Regional
Kafka
Aggregate
Kafka
uReplicator
Offset Sync Service
Aggregate
Kafka
uReplicator

63. Multi-DC Consumer
DC2
DC1
Consumer
Application
Active
Consumer
Application
Passive
Regional
Kafka
Regional
Kafka
Aggregate
Kafka
uReplicator
Offset Sync Service
Aggregate
Kafka
uReplicator

64. Multi-DC Consumer
DC2
DC1
Consumer
Application
Active
Consumer
Application
Passive
Regional
Kafka
Regional
Kafka
Aggregate
Kafka
uReplicator
Offset Sync Service
Aggregate
Kafka
uReplicator
Offset
sync

65. Multi-DC Consumer
DC2
DC1
Consumer
Application
Active
Consumer
Application
Passive
Regional
Kafka
Regional
Kafka
Aggregate
Kafka
uReplicator
Offset Sync Service
Aggregate
Kafka
uReplicator

66. Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter failover
● Golang, Java, Python client

67. API in Golang
handlerConsumer, _ := consumer.NewHandlerConsumer(consumer.Params{
Subscriptions: []Subscription{
{
Handler: func(context.Context, []byte) error {
// Do stuff
return nil
},
TopicConfig: NewTopicConfig("topic",
AllActiveStrategyActivePassive)
.WithConcurrency(100)
.WithTimeout(time.Second),
},
}
})
handlerConsumer.Start()
defer handlerConsumer.Stop()

68. Agenda
● Use cases & scale
● Motivation
● High-level design
● Future work

69. Future Work
● Open source
● Async RPC
● Runtime offset reset
● Active-Active MultiDC Consumer
● Federated multi-cluster Kafka

70. Links
● https://github.com/uber-go/kafka-client
● https://eng.uber.com/reliable-reprocessing/
● https://github.com/uber/uReplicator

71. Thank you
Proprietary and confidential © 2016 Uber Technologies, Inc. All rights reserved. No part of this document may be
reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any
information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use
of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise
exempt from disclosure under applicable law. All recipients of this document are notified that the information contained
herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any
way disclose this document or any of the enclosed information to any person other than employees of addressee to the
extent necessary for consultations with authorized personnel of Uber.
More open-source projects at eng.uber.com

72. Proprietary and confidential © 2018 Uber Technologies, Inc. All rights reserved. No part of this
document may be reproduced or utilized in any form or by any means, electronic or mechanical,
including photocopying, recording, or by any information storage or retrieval systems, without
permission in writing from Uber. This document is intended only for the use of the individual or entity
to whom it is addressed and contains information that is privileged, confidential or otherwise exempt
from disclosure under applicable law. All recipients of this document are notified that the information
contained herein includes proprietary and confidential information of Uber, and recipient may not
make use of, disseminate, or in any way disclose this document or any of the enclosed information
to any person other than employees of addressee to the extent necessary for consultations with
authorized personnel of Uber.