SlideShare una empresa de Scribd logo
1 de 72
Reliable Messaging Systems with Kafka at Uber
George Teo
Software Engineer @ Streaming Data
June 19, 2018
Agenda
● Kafka @ Uber
● What is Messaging?
● High-level design
● Future work
Stream
Processing
- Driver-Rider Match
- ETA
App Views
Vehicle Information
KAFKA
Real-time Driver-Rider Matching
A bunch more...
● Fraud Detection
● Share My ETA
● Driver & Rider Signups
● Etc.
Apache Kafka - Use Cases
● Inter-service messaging
● Stream processing
○ AthenaX - self-serve platform (Samza, Flink)
● Database changelog transport
○ Schemaless, Cassandra, MySQL
● Data Lake Ingestion
○ HDFS, S3
● Log Aggregation
PRODUCERS
CONSUMERS
Real-time
Analytics, Alerts,
Dashboards
Samza / Flink
Applications
Data Science
Analytics
Reporting
Kafka
Vertica / Hive
Rider App
Driver App
API / Services
Etc.
Ad-hoc Exploration
ELK
Kafka @ Uber
Debugging
Hadoop
Surge Mobile App
Cassandra
Schemaless
MySQL
DATABASES
AWS S3
(Internal) Services
Apache Kafka - Use Cases
● Inter-service messaging
● Stream processing
○ AthenaX - self-serve platform (Samza, Flink)
● Database changelog transport
○ Schemaless, Cassandra, MySQL
● Ingestion
○ HDFS, S3
● Logging
Trillions+ ~PBs
Messages/Day
Data Volume
Scale
excluding replication
Tens of
Thousands
Topics
10+ clusters
100+ broker per
cluster
Agenda
● Kafka @ Uber
● What is Messaging?
● High-level Design
● Future Work
Messaging
● Streaming
● Queuing
Messaging
● Streaming
○ Ordered
Streaming - Ordered
Partition
0 1 2 3 4 5 6 7 8 9 10 11 12
Commit Offset 7
Consumer
Application
78
Messaging
● Streaming
○ Ordered
○ Commit up to
Streaming - Commit Up To
Partition
0 1 2 3 4 5 6 7 8 9 10 11 12
Commit Offset 9
Consumer
Application
789
Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned
Streaming - Partitioned
Kafka Topic
Partition 0
0 1 2 3 4 5 6 7 8 9 10 11 12
Partition 1
0 1 2 3 4 5 6 7 8 9 10 11 12
Partition 2
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Instance 2
Consumer
Instance 1
Consumer Application
Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned Consumer
● Queuing*
○ Unordered
Queueing- Unordered
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application
6
Queue
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application
Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned Consumer
● Queuing*
○ Unordered
○ Ack
Queueing- Ack
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application
6
Queue
0 1 2 3 4 5 6 7 8 9 10 11 12
Ack 6
Consumer
Application
Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned Consumer
● Queuing*
○ Unordered
○ Ack/Nack
Queueing- Nack
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application
Queue
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application
Nack 4
5
Ack 5
4
Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned Consumer
● Queuing*
○ Unordered
○ Ack/Nack
○ Redelivery/DLQ
Queueing- Redelivery/DLQ
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application
Queue
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Application4
Nack 4
Messaging
● Streaming
○ Ordered
○ Commit up to
○ Partitioned Consumer
● Queuing*
○ Unordered
○ Ack/Nack
○ Redelivery/DLQ
○ Competing Consumer
Queuing - Competing
Consumer
Instance 2
Consumer
Instance 1
Consumer Application
0 1 2 3 4 5 6 7 8 9 10 11 12
Queue
0 1 2 3 4 5 6 7 8 9 10 11 12
Why Streaming?
● Data Analytics
● Event-driven Architecture/Event Sourcing
● Log Aggregation
● Scalability*
Why Queuing?
● More friendly acknowledgement semantics
● Non-blocking retry
● Elastic Consumer scalability
● High consumer throughput
Agenda
● Kafka @ Uber
● Motivation
● High-level design
● Future work
Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter failover
● Golang, Java, Python client
Why Kafka?
● Other options:
○ ___MQ
○ Apache Pulsar
○ Build yet another messaging queue from ground up
● Why Kafka?
○ Reliability
○ Performance
○ Cost efficiency
■ Operational, engineering costs
Native Kafka Consumer
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
Consumer
Process
Competing Consumer
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1
Competing Consumer
Kafka Topic
Partition 0
0 1 2 3 4 5 6 7 8 9 10 11 12
Partition 1
0 1 2 3 4 5 6 7 8 9 10 11 12
Partition 2
0 1 2 3 4 5 6 7 8 9 10 11 12
Consumer
Instance 2
Consumer
Instance 1
Consumer Application
Worker 1
Worker 2
Worker 3
Worker 4
Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client
Message Flow - Incoming Message
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
Consumer
Worker1Consumer
Worker1Consumer
Worker1
1
56 4
3
Message Flow - Inflight Message Tracker
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1
2
Message Flow - Passed to worker
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1
2
3
4
5
Consumer Application
In-Flight Offset tracker
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Consumer Application
In-flight Offset Tracker
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
3
Consumer Application
In-flight Offset Tracker
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
3 6 4 5
Consumer Application
In-flight Offset Tracker - Ack
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
3 6 4
Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client
Consumer Application
In-flight Offset Tracker
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
3 6 4 7
Consumer Application
In-flight Offset Tracker
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
3 6 8 7
Consumer Application
In-flight Offset Tracker - Nack
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
6 8 7
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 6 7 8
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 (3)
3
Nack
Consumer Application
In-flight Offset Tracker - Nack
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
6 8 7
Ack
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
6 7 8 9
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 (3)
4
Commit up to 5
Consumer Application
In-flight Offset Tracker - Nack
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Topic Partition 0
Consumer
Worker1
Consumer
Worker2
Consumer
Worker3
Consumer
Worker4
Committed
Read
6 8 7
Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 (3)
5
Redelivery
6 7 8 9
0 (3)
Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client
Delay
Delay Between Retries
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 1
5
6 7 8 9
Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 1 2 3 4 5 6
topic__cg__dlq
Partition 0
0 1 2 3 4 5 6
Dead Letter Queue
Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 1 2 3 4 5 6
topic__cg__dlq
Partition 0
0 1 2 3 4 5 6
Purge
DLQ - Purge
topic
Partition 0
0 1 2 3 4 5 6
Consumer
Process
In-flight offset tracker
Partition 0
3 4 5 6
Consumer
Worker1Consumer
Worker1Consumer
Worker1
topic__cg__retry
Partition 0
0 1 2 3 4 5 6
topic__cg__dlq
Partition 0
0 1 2 3 4 5 6
Merge
DLQ - Merge
Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter Consumer
● Golang, Java, Python client
DC2
DC1
Regional
Kafka
Regional
Kafka
Aggregate
Kafka
uReplicator
Offset Sync Service
Aggregate
Kafka
uReplicator
Kafka Setup @ Uber
DC2
DC1
Consumer
Application
Active
Consumer
Application
Passive
Multi-DC Consumer
Regional
Kafka
Regional
Kafka
Aggregate
Kafka
uReplicator
Offset Sync Service
Aggregate
Kafka
uReplicator
Multi-DC Consumer
DC2
DC1
Consumer
Application
Active
Consumer
Application
Passive
Regional
Kafka
Regional
Kafka
Aggregate
Kafka
uReplicator
Offset Sync Service
Aggregate
Kafka
uReplicator
Multi-DC Consumer
DC2
DC1
Consumer
Application
Active
Consumer
Application
Passive
Regional
Kafka
Regional
Kafka
Aggregate
Kafka
uReplicator
Offset Sync Service
Aggregate
Kafka
uReplicator
Offset
sync
Multi-DC Consumer
DC2
DC1
Consumer
Application
Active
Consumer
Application
Passive
Regional
Kafka
Regional
Kafka
Aggregate
Kafka
uReplicator
Offset Sync Service
Aggregate
Kafka
uReplicator
Requirements
● Ack / Nack
● Redelivery
● Delay between retries
● Dead letter queue
● DLQ - purge / merge
● (Limited) Competing Consumer
● Multi-datacenter failover
● Golang, Java, Python client
API in Golang
handlerConsumer, _ := consumer.NewHandlerConsumer(consumer.Params{
Subscriptions: []Subscription{
{
Handler: func(context.Context, []byte) error {
// Do stuff
return nil
},
TopicConfig: NewTopicConfig("topic",
AllActiveStrategyActivePassive)
.WithConcurrency(100)
.WithTimeout(time.Second),
},
}
})
handlerConsumer.Start()
defer handlerConsumer.Stop()
Agenda
● Use cases & scale
● Motivation
● High-level design
● Future work
Future Work
● Open source
● Async RPC
● Runtime offset reset
● Active-Active MultiDC Consumer
● Federated multi-cluster Kafka
Links
● https://github.com/uber-go/kafka-client
● https://eng.uber.com/reliable-reprocessing/
● https://github.com/uber/uReplicator
Thank you
Proprietary and confidential © 2016 Uber Technologies, Inc. All rights reserved. No part of this document may be
reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any
information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use
of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise
exempt from disclosure under applicable law. All recipients of this document are notified that the information contained
herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any
way disclose this document or any of the enclosed information to any person other than employees of addressee to the
extent necessary for consultations with authorized personnel of Uber.
More open-source projects at eng.uber.com
Proprietary and confidential © 2018 Uber Technologies, Inc. All rights reserved. No part of this
document may be reproduced or utilized in any form or by any means, electronic or mechanical,
including photocopying, recording, or by any information storage or retrieval systems, without
permission in writing from Uber. This document is intended only for the use of the individual or entity
to whom it is addressed and contains information that is privileged, confidential or otherwise exempt
from disclosure under applicable law. All recipients of this document are notified that the information
contained herein includes proprietary and confidential information of Uber, and recipient may not
make use of, disseminate, or in any way disclose this document or any of the enclosed information
to any person other than employees of addressee to the extent necessary for consultations with
authorized personnel of Uber.

Más contenido relacionado

La actualidad más candente

Grafana Loki: like Prometheus, but for Logs
Grafana Loki: like Prometheus, but for LogsGrafana Loki: like Prometheus, but for Logs
Grafana Loki: like Prometheus, but for LogsMarco Pracucci
 
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka ConnectFrom Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connectconfluent
 
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals ExplainedApache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explainedconfluent
 
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaMonitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaArvind Kumar G.S
 
Loki - like prometheus, but for logs
Loki - like prometheus, but for logsLoki - like prometheus, but for logs
Loki - like prometheus, but for logsJuraj Hantak
 
Getting Started with Confluent Schema Registry
Getting Started with Confluent Schema RegistryGetting Started with Confluent Schema Registry
Getting Started with Confluent Schema Registryconfluent
 
A visual introduction to Apache Kafka
A visual introduction to Apache KafkaA visual introduction to Apache Kafka
A visual introduction to Apache KafkaPaul Brebner
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka StreamsGuozhang Wang
 
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best PracticesKafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practicesconfluent
 
Apache Kafka - Martin Podval
Apache Kafka - Martin PodvalApache Kafka - Martin Podval
Apache Kafka - Martin PodvalMartin Podval
 
Uber: Kafka Consumer Proxy
Uber: Kafka Consumer ProxyUber: Kafka Consumer Proxy
Uber: Kafka Consumer Proxyconfluent
 
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Kai Wähner
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producerconfluent
 
Building High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in KafkaBuilding High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in Kafkaconfluent
 
How to Lock Down Apache Kafka and Keep Your Streams Safe
How to Lock Down Apache Kafka and Keep Your Streams SafeHow to Lock Down Apache Kafka and Keep Your Streams Safe
How to Lock Down Apache Kafka and Keep Your Streams Safeconfluent
 
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, ConfluentKafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, ConfluentHostedbyConfluent
 
Redis cluster
Redis clusterRedis cluster
Redis clusteriammutex
 
Introduction to Prometheus
Introduction to PrometheusIntroduction to Prometheus
Introduction to PrometheusJulien Pivotto
 

La actualidad más candente (20)

Grafana Loki: like Prometheus, but for Logs
Grafana Loki: like Prometheus, but for LogsGrafana Loki: like Prometheus, but for Logs
Grafana Loki: like Prometheus, but for Logs
 
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka ConnectFrom Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connect
 
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals ExplainedApache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explained
 
Monitoring using Prometheus and Grafana
Monitoring using Prometheus and GrafanaMonitoring using Prometheus and Grafana
Monitoring using Prometheus and Grafana
 
Kafka 101
Kafka 101Kafka 101
Kafka 101
 
Loki - like prometheus, but for logs
Loki - like prometheus, but for logsLoki - like prometheus, but for logs
Loki - like prometheus, but for logs
 
Getting Started with Confluent Schema Registry
Getting Started with Confluent Schema RegistryGetting Started with Confluent Schema Registry
Getting Started with Confluent Schema Registry
 
kafka
kafkakafka
kafka
 
A visual introduction to Apache Kafka
A visual introduction to Apache KafkaA visual introduction to Apache Kafka
A visual introduction to Apache Kafka
 
Introduction to Kafka Streams
Introduction to Kafka StreamsIntroduction to Kafka Streams
Introduction to Kafka Streams
 
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best PracticesKafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practices
 
Apache Kafka - Martin Podval
Apache Kafka - Martin PodvalApache Kafka - Martin Podval
Apache Kafka - Martin Podval
 
Uber: Kafka Consumer Proxy
Uber: Kafka Consumer ProxyUber: Kafka Consumer Proxy
Uber: Kafka Consumer Proxy
 
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
Confluent REST Proxy and Schema Registry (Concepts, Architecture, Features)
 
Common issues with Apache Kafka® Producer
Common issues with Apache Kafka® ProducerCommon issues with Apache Kafka® Producer
Common issues with Apache Kafka® Producer
 
Building High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in KafkaBuilding High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in Kafka
 
How to Lock Down Apache Kafka and Keep Your Streams Safe
How to Lock Down Apache Kafka and Keep Your Streams SafeHow to Lock Down Apache Kafka and Keep Your Streams Safe
How to Lock Down Apache Kafka and Keep Your Streams Safe
 
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, ConfluentKafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
 
Redis cluster
Redis clusterRedis cluster
Redis cluster
 
Introduction to Prometheus
Introduction to PrometheusIntroduction to Prometheus
Introduction to Prometheus
 

Similar a Kafka Retry and DLQ

Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...Data Con LA
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache KafkaRicardo Bravo
 
Build real time stream processing applications using Apache Kafka
Build real time stream processing applications using Apache KafkaBuild real time stream processing applications using Apache Kafka
Build real time stream processing applications using Apache KafkaHotstar
 
Kafka Practices @ Uber - Seattle Apache Kafka meetup
Kafka Practices @ Uber - Seattle Apache Kafka meetupKafka Practices @ Uber - Seattle Apache Kafka meetup
Kafka Practices @ Uber - Seattle Apache Kafka meetupMingmin Chen
 
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®confluent
 
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel QueriesChristo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel QueriesChristo Kutrovsky
 
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and PerformanceCiti Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performanceconfluent
 
Akka gRPC quick-guide
Akka gRPC quick-guideAkka gRPC quick-guide
Akka gRPC quick-guideKnoldus Inc.
 
Akka gRPC quick-guide
Akka gRPC quick-guideAkka gRPC quick-guide
Akka gRPC quick-guideKnoldus Inc.
 
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsExperiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsCeph Community
 
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...HostedbyConfluent
 
Query and audit logging in cassandra
Query and audit logging in cassandraQuery and audit logging in cassandra
Query and audit logging in cassandraVinay Kumar Chella
 
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...Flink Forward
 
Storing State Forever: Why It Can Be Good For Your Analytics
Storing State Forever: Why It Can Be Good For Your AnalyticsStoring State Forever: Why It Can Be Good For Your Analytics
Storing State Forever: Why It Can Be Good For Your AnalyticsYaroslav Tkachenko
 
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집AWSKRUG - AWS한국사용자모임
 
Lisbon Mulesoft Meetup - Logging Aggregation & Visualization
Lisbon Mulesoft Meetup - Logging Aggregation & VisualizationLisbon Mulesoft Meetup - Logging Aggregation & Visualization
Lisbon Mulesoft Meetup - Logging Aggregation & VisualizationSteve Michael Fernandes
 
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at UberDisaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uberconfluent
 
Building a Dynamic Rules Engine with Kafka Streams
Building a Dynamic Rules Engine with Kafka StreamsBuilding a Dynamic Rules Engine with Kafka Streams
Building a Dynamic Rules Engine with Kafka StreamsHostedbyConfluent
 
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...confluent
 
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
DPDK summit 2015: It's kind of fun  to do the impossible with DPDKDPDK summit 2015: It's kind of fun  to do the impossible with DPDK
DPDK summit 2015: It's kind of fun to do the impossible with DPDKLagopus SDN/OpenFlow switch
 

Similar a Kafka Retry and DLQ (20)

Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
Data Con LA 2019 - Unifying streaming and message queue with Apache Kafka by ...
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Build real time stream processing applications using Apache Kafka
Build real time stream processing applications using Apache KafkaBuild real time stream processing applications using Apache Kafka
Build real time stream processing applications using Apache Kafka
 
Kafka Practices @ Uber - Seattle Apache Kafka meetup
Kafka Practices @ Uber - Seattle Apache Kafka meetupKafka Practices @ Uber - Seattle Apache Kafka meetup
Kafka Practices @ Uber - Seattle Apache Kafka meetup
 
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®
 
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel QueriesChristo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
Christo Kutrovsky - Maximize Data Warehouse Performance with Parallel Queries
 
Citi Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and PerformanceCiti Tech Talk: Monitoring and Performance
Citi Tech Talk: Monitoring and Performance
 
Akka gRPC quick-guide
Akka gRPC quick-guideAkka gRPC quick-guide
Akka gRPC quick-guide
 
Akka gRPC quick-guide
Akka gRPC quick-guideAkka gRPC quick-guide
Akka gRPC quick-guide
 
Experiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah WatkinsExperiences building a distributed shared log on RADOS - Noah Watkins
Experiences building a distributed shared log on RADOS - Noah Watkins
 
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot | Yupeng ...
 
Query and audit logging in cassandra
Query and audit logging in cassandraQuery and audit logging in cassandra
Query and audit logging in cassandra
 
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
 
Storing State Forever: Why It Can Be Good For Your Analytics
Storing State Forever: Why It Can Be Good For Your AnalyticsStoring State Forever: Why It Can Be Good For Your Analytics
Storing State Forever: Why It Can Be Good For Your Analytics
 
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
Aurora Serverless, 서버리스 RDB의 서막 - 트랙2, Community Day 2018 re:Invent 특집
 
Lisbon Mulesoft Meetup - Logging Aggregation & Visualization
Lisbon Mulesoft Meetup - Logging Aggregation & VisualizationLisbon Mulesoft Meetup - Logging Aggregation & Visualization
Lisbon Mulesoft Meetup - Logging Aggregation & Visualization
 
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at UberDisaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
Disaster Recovery for Multi-Region Apache Kafka Ecosystems at Uber
 
Building a Dynamic Rules Engine with Kafka Streams
Building a Dynamic Rules Engine with Kafka StreamsBuilding a Dynamic Rules Engine with Kafka Streams
Building a Dynamic Rules Engine with Kafka Streams
 
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
Kafka Cluster Federation at Uber (Yupeng Fui & Xiaoman Dong, Uber) Kafka Summ...
 
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
DPDK summit 2015: It's kind of fun  to do the impossible with DPDKDPDK summit 2015: It's kind of fun  to do the impossible with DPDK
DPDK summit 2015: It's kind of fun to do the impossible with DPDK
 

Último

UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingFrancesco Corti
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNeo4j
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxKaustubhBhavsar6
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch TuesdayIvanti
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...DianaGray10
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 

Último (20)

UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
 
Where developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is goingWhere developers are challenged, what developers want and where DevEx is going
Where developers are challenged, what developers want and where DevEx is going
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
Novo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4jNovo Nordisk's journey in developing an open-source application on Neo4j
Novo Nordisk's journey in developing an open-source application on Neo4j
 
How to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptxHow to become a GDSC Lead GDSC MI AOE.pptx
How to become a GDSC Lead GDSC MI AOE.pptx
 
March Patch Tuesday
March Patch TuesdayMarch Patch Tuesday
March Patch Tuesday
 
Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...Explore the UiPath Community and ways you can benefit on your journey to auto...
Explore the UiPath Community and ways you can benefit on your journey to auto...
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 

Kafka Retry and DLQ