Enviar búsqueda
Cargar
Kafka Retry and DLQ
•
8 recomendaciones
•
6,313 vistas
G
George Teo
Seguir
Kafka Retry and DLQ work presented for Bay Area Apache Kafka Meetup.
Leer menos
Leer más
Tecnología
Denunciar
Compartir
Denunciar
Compartir
1 de 72
Recomendados
Implementing Domain Events with Kafka
Implementing Domain Events with Kafka
Andrei Rugina
Introduction to Apache Kafka
Introduction to Apache Kafka
AIMDek Technologies
Envoy and Kafka
Envoy and Kafka
Adam Kotwasinski
Kafka presentation
Kafka presentation
Mohammed Fazuluddin
Producer Performance Tuning for Apache Kafka
Producer Performance Tuning for Apache Kafka
Jiangjie Qin
Introduction to Apache Kafka
Introduction to Apache Kafka
Jeff Holoman
How is Kafka so Fast?
How is Kafka so Fast?
Ricardo Paiva
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?
confluent
Más contenido relacionado
La actualidad más candente
Grafana Loki: like Prometheus, but for Logs
Grafana Loki: like Prometheus, but for Logs
Marco Pracucci
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connect
confluent
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explained
confluent
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and Grafana
Arvind Kumar G.S
Kafka 101
Kafka 101
Clement Demonchy
Loki - like prometheus, but for logs
Loki - like prometheus, but for logs
Juraj Hantak
Getting Started with Confluent Schema Registry
Getting Started with Confluent Schema Registry
confluent
kafka
kafka
Amikam Snir
A visual introduction to Apache Kafka
A visual introduction to Apache Kafka
Paul Brebner
Introduction to Kafka Streams
Introduction to Kafka Streams
Guozhang Wang
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practices
confluent
Apache Kafka - Martin Podval
Apache Kafka - Martin Podval
Martin Podval
Uber: Kafka Consumer Proxy
Uber: Kafka Consumer Proxy
confluent
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Kai Wähner
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producer
confluent
Building High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in Kafka
confluent
How to Lock Down Apache Kafka and Keep Your Streams Safe
How to Lock Down Apache Kafka and Keep Your Streams Safe
confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
HostedbyConfluent
Redis cluster
Redis cluster
iammutex
Introduction to Prometheus
Introduction to Prometheus
Julien Pivotto
La actualidad más candente
(20)
Grafana Loki: like Prometheus, but for Logs
Grafana Loki: like Prometheus, but for Logs
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connect
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explained
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and Grafana
Kafka 101
Kafka 101
Loki - like prometheus, but for logs
Loki - like prometheus, but for logs
Getting Started with Confluent Schema Registry
Getting Started with Confluent Schema Registry
kafka
kafka
A visual introduction to Apache Kafka
A visual introduction to Apache Kafka
Introduction to Kafka Streams
Introduction to Kafka Streams
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practices
Apache Kafka - Martin Podval
Apache Kafka - Martin Podval
Uber: Kafka Consumer Proxy
Uber: Kafka Consumer Proxy
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producer
Building High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in Kafka
How to Lock Down Apache Kafka and Keep Your Streams Safe
How to Lock Down Apache Kafka and Keep Your Streams Safe
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Redis cluster
Redis cluster
Introduction to Prometheus
Introduction to Prometheus
Similar a Kafka Retry and DLQ
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA
Introduction to Apache Kafka
Introduction to Apache Kafka
Ricardo Bravo
Build real time stream processing applications using Apache Kafka
Build real time stream processing applications using Apache Kafka
Hotstar
Kafka Practices @ Uber - Seattle Apache Kafka meetup
Kafka Practices @ Uber - Seattle Apache Kafka meetup
Mingmin Chen
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®
confluent
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Christo Kutrovsky
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
confluent
Akka gRPC quick-guide
Akka gRPC quick-guide
Knoldus Inc.
Akka gRPC quick-guide
Akka gRPC quick-guide
Knoldus Inc.
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah Watkins
Ceph Community
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
HostedbyConfluent
Query and audit logging in cassandra
Query and audit logging in cassandra
Vinay Kumar Chella
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward
Storing State Forever: Why It Can Be Good For Your Analytics
Storing State Forever: Why It Can Be Good For Your Analytics
Yaroslav Tkachenko
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
AWSKRUG - AWS한국사용자모임
Lisbon Mulesoft Meetup - Logging Aggregation & Visualization
Lisbon Mulesoft Meetup - Logging Aggregation & Visualization
Steve Michael Fernandes
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
confluent
Building a Dynamic Rules Engine with Kafka Streams
Building a Dynamic Rules Engine with Kafka Streams
HostedbyConfluent
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
confluent
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
Lagopus SDN/OpenFlow switch
Similar a Kafka Retry and DLQ
(20)
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Introduction to Apache Kafka
Introduction to Apache Kafka
Build real time stream processing applications using Apache Kafka
Build real time stream processing applications using Apache Kafka
Kafka Practices @ Uber - Seattle Apache Kafka meetup
Kafka Practices @ Uber - Seattle Apache Kafka meetup
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
Akka gRPC quick-guide
Akka gRPC quick-guide
Akka gRPC quick-guide
Akka gRPC quick-guide
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah Watkins
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
Query and audit logging in cassandra
Query and audit logging in cassandra
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Storing State Forever: Why It Can Be Good For Your Analytics
Storing State Forever: Why It Can Be Good For Your Analytics
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
Lisbon Mulesoft Meetup - Logging Aggregation & Visualization
Lisbon Mulesoft Meetup - Logging Aggregation & Visualization
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Building a Dynamic Rules Engine with Kafka Streams
Building a Dynamic Rules Engine with Kafka Streams
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
Último
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
DianaGray10
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
codyslingerland1
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
IES VE
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Neo4j
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
nooralam814309
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
DianaGray10
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
Safe Software
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
Francesco Corti
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
Knoldus Inc.
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Tejal81
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
DanBrown980551
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
Eric D. Schabell
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
Neo4j
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
KaustubhBhavsar6
March Patch Tuesday
March Patch Tuesday
Ivanti
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
DianaGray10
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
IPLOOK Networks
Último
(20)
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
March Patch Tuesday
March Patch Tuesday
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
Kafka Retry and DLQ
1.
Reliable Messaging Systems
with Kafka at Uber George Teo Software Engineer @ Streaming Data June 19, 2018
2.
Agenda ● Kafka @
Uber ● What is Messaging? ● High-level design ● Future work
3.
Stream Processing - Driver-Rider Match -
ETA App Views Vehicle Information KAFKA Real-time Driver-Rider Matching
4.
A bunch more... ●
Fraud Detection ● Share My ETA ● Driver & Rider Signups ● Etc.
5.
Apache Kafka -
Use Cases ● Inter-service messaging ● Stream processing ○ AthenaX - self-serve platform (Samza, Flink) ● Database changelog transport ○ Schemaless, Cassandra, MySQL ● Data Lake Ingestion ○ HDFS, S3 ● Log Aggregation
6.
PRODUCERS CONSUMERS Real-time Analytics, Alerts, Dashboards Samza /
Flink Applications Data Science Analytics Reporting Kafka Vertica / Hive Rider App Driver App API / Services Etc. Ad-hoc Exploration ELK Kafka @ Uber Debugging Hadoop Surge Mobile App Cassandra Schemaless MySQL DATABASES AWS S3 (Internal) Services
7.
Apache Kafka -
Use Cases ● Inter-service messaging ● Stream processing ○ AthenaX - self-serve platform (Samza, Flink) ● Database changelog transport ○ Schemaless, Cassandra, MySQL ● Ingestion ○ HDFS, S3 ● Logging
8.
Trillions+ ~PBs Messages/Day Data Volume Scale excluding
replication Tens of Thousands Topics 10+ clusters 100+ broker per cluster
9.
Agenda ● Kafka @
Uber ● What is Messaging? ● High-level Design ● Future Work
10.
Messaging ● Streaming ● Queuing
11.
Messaging ● Streaming ○ Ordered
12.
Streaming - Ordered Partition 0
1 2 3 4 5 6 7 8 9 10 11 12 Commit Offset 7 Consumer Application 78
13.
Messaging ● Streaming ○ Ordered ○
Commit up to
14.
Streaming - Commit
Up To Partition 0 1 2 3 4 5 6 7 8 9 10 11 12 Commit Offset 9 Consumer Application 789
15.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned
16.
Streaming - Partitioned Kafka
Topic Partition 0 0 1 2 3 4 5 6 7 8 9 10 11 12 Partition 1 0 1 2 3 4 5 6 7 8 9 10 11 12 Partition 2 0 1 2 3 4 5 6 7 8 9 10 11 12 Consumer Instance 2 Consumer Instance 1 Consumer Application
17.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned Consumer ● Queuing* ○ Unordered
18.
Queueing- Unordered 0 1
2 3 4 5 6 7 8 9 10 11 12 Consumer Application 6 Queue 0 1 2 3 4 5 6 7 8 9 10 11 12 Consumer Application
19.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned Consumer ● Queuing* ○ Unordered ○ Ack
20.
Queueing- Ack 0 1
2 3 4 5 6 7 8 9 10 11 12 Consumer Application 6 Queue 0 1 2 3 4 5 6 7 8 9 10 11 12 Ack 6 Consumer Application
21.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned Consumer ● Queuing* ○ Unordered ○ Ack/Nack
22.
Queueing- Nack 0 1
2 3 4 5 6 7 8 9 10 11 12 Consumer Application Queue 0 1 2 3 4 5 6 7 8 9 10 11 12 Consumer Application Nack 4 5 Ack 5 4
23.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned Consumer ● Queuing* ○ Unordered ○ Ack/Nack ○ Redelivery/DLQ
24.
Queueing- Redelivery/DLQ 0 1
2 3 4 5 6 7 8 9 10 11 12 Consumer Application Queue 0 1 2 3 4 5 6 7 8 9 10 11 12 Consumer Application4 Nack 4
25.
Messaging ● Streaming ○ Ordered ○
Commit up to ○ Partitioned Consumer ● Queuing* ○ Unordered ○ Ack/Nack ○ Redelivery/DLQ ○ Competing Consumer
26.
Queuing - Competing Consumer Instance
2 Consumer Instance 1 Consumer Application 0 1 2 3 4 5 6 7 8 9 10 11 12 Queue 0 1 2 3 4 5 6 7 8 9 10 11 12
27.
Why Streaming? ● Data
Analytics ● Event-driven Architecture/Event Sourcing ● Log Aggregation ● Scalability*
28.
Why Queuing? ● More
friendly acknowledgement semantics ● Non-blocking retry ● Elastic Consumer scalability ● High consumer throughput
29.
Agenda ● Kafka @
Uber ● Motivation ● High-level design ● Future work
30.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter failover ● Golang, Java, Python client
31.
Why Kafka? ● Other
options: ○ ___MQ ○ Apache Pulsar ○ Build yet another messaging queue from ground up ● Why Kafka? ○ Reliability ○ Performance ○ Cost efficiency ■ Operational, engineering costs
32.
Native Kafka Consumer topic Partition
0 0 1 2 3 4 5 6 Consumer Process
33.
Consumer Process Competing Consumer topic Partition 0 0
1 2 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1
34.
Competing Consumer Kafka Topic Partition
0 0 1 2 3 4 5 6 7 8 9 10 11 12 Partition 1 0 1 2 3 4 5 6 7 8 9 10 11 12 Partition 2 0 1 2 3 4 5 6 7 8 9 10 11 12 Consumer Instance 2 Consumer Instance 1 Consumer Application Worker 1 Worker 2 Worker 3 Worker 4
35.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
36.
Message Flow -
Incoming Message topic Partition 0 0 1 2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 Consumer Worker1Consumer Worker1Consumer Worker1 1 56 4 3
37.
Message Flow -
Inflight Message Tracker topic Partition 0 0 1 2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1 2
38.
Message Flow -
Passed to worker topic Partition 0 0 1 2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1 2 3 4 5
39.
Consumer Application In-Flight Offset
tracker 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed
40.
Consumer Application In-flight Offset
Tracker 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 3
41.
Consumer Application In-flight Offset
Tracker 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 3 6 4 5
42.
Consumer Application In-flight Offset
Tracker - Ack 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 3 6 4
43.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
44.
Consumer Application In-flight Offset
Tracker 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 3 6 4 7
45.
Consumer Application In-flight Offset
Tracker 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 3 6 8 7
46.
Consumer Application In-flight Offset
Tracker - Nack 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 6 8 7
47.
topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 6 7 8 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 (3) 3 Nack
48.
Consumer Application In-flight Offset
Tracker - Nack 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 6 8 7
49.
Ack topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 6 7 8 9 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 (3) 4 Commit up to 5
50.
Consumer Application In-flight Offset
Tracker - Nack 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Topic Partition 0 Consumer Worker1 Consumer Worker2 Consumer Worker3 Consumer Worker4 Committed Read 6 8 7
51.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
52.
topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 (3) 5 Redelivery 6 7 8 9 0 (3)
53.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
54.
Delay Delay Between Retries topic Partition
0 0 1 2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 1 5 6 7 8 9
55.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
56.
topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 1 2 3 4 5 6 topic__cg__dlq Partition 0 0 1 2 3 4 5 6 Dead Letter Queue
57.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
58.
topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 1 2 3 4 5 6 topic__cg__dlq Partition 0 0 1 2 3 4 5 6 Purge DLQ - Purge
59.
topic Partition 0 0 1
2 3 4 5 6 Consumer Process In-flight offset tracker Partition 0 3 4 5 6 Consumer Worker1Consumer Worker1Consumer Worker1 topic__cg__retry Partition 0 0 1 2 3 4 5 6 topic__cg__dlq Partition 0 0 1 2 3 4 5 6 Merge DLQ - Merge
60.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter Consumer ● Golang, Java, Python client
61.
DC2 DC1 Regional Kafka Regional Kafka Aggregate Kafka uReplicator Offset Sync Service Aggregate Kafka uReplicator Kafka
Setup @ Uber
62.
DC2 DC1 Consumer Application Active Consumer Application Passive Multi-DC Consumer Regional Kafka Regional Kafka Aggregate Kafka uReplicator Offset Sync
Service Aggregate Kafka uReplicator
63.
Multi-DC Consumer DC2 DC1 Consumer Application Active Consumer Application Passive Regional Kafka Regional Kafka Aggregate Kafka uReplicator Offset Sync
Service Aggregate Kafka uReplicator
64.
Multi-DC Consumer DC2 DC1 Consumer Application Active Consumer Application Passive Regional Kafka Regional Kafka Aggregate Kafka uReplicator Offset Sync
Service Aggregate Kafka uReplicator Offset sync
65.
Multi-DC Consumer DC2 DC1 Consumer Application Active Consumer Application Passive Regional Kafka Regional Kafka Aggregate Kafka uReplicator Offset Sync
Service Aggregate Kafka uReplicator
66.
Requirements ● Ack /
Nack ● Redelivery ● Delay between retries ● Dead letter queue ● DLQ - purge / merge ● (Limited) Competing Consumer ● Multi-datacenter failover ● Golang, Java, Python client
67.
API in Golang handlerConsumer,
_ := consumer.NewHandlerConsumer(consumer.Params{ Subscriptions: []Subscription{ { Handler: func(context.Context, []byte) error { // Do stuff return nil }, TopicConfig: NewTopicConfig("topic", AllActiveStrategyActivePassive) .WithConcurrency(100) .WithTimeout(time.Second), }, } }) handlerConsumer.Start() defer handlerConsumer.Stop()
68.
Agenda ● Use cases
& scale ● Motivation ● High-level design ● Future work
69.
Future Work ● Open
source ● Async RPC ● Runtime offset reset ● Active-Active MultiDC Consumer ● Federated multi-cluster Kafka
70.
Links ● https://github.com/uber-go/kafka-client ● https://eng.uber.com/reliable-reprocessing/ ●
https://github.com/uber/uReplicator
71.
Thank you Proprietary and
confidential © 2016 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber. More open-source projects at eng.uber.com
72.
Proprietary and confidential
© 2018 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber.