Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020

•

0 recomendaciones•2,367 vistas

What does a Kafka administrator need to do if they have a user who demands that message delivery be guaranteed, fast, and low cost? In this talk we walk through the architecture we created to deliver for such users. Learn around the alternatives we considered and the pros and cons around what we came up with. In this talk, we’ll be forced to dive into broker restart and failure scenarios and things we need to do to prevent leader elections from slowing down incoming requests. We’ll need to take care of the consumers as well to ensure that they don’t process the same request twice. We also plan to describe our architecture by showing a demo of simulated requests being produced into Kafka clusters and consumers processing them in lieu of us aggressively causing failures on the Kafka clusters. We hope the audience walks away with a deeper understanding of what it takes to build robust Kafka clients and how to tune them to accomplish stringent delivery guarantees.

Tecnología

Presentation title
ANDREY FALKO and CAN CECEN - August 2020
Can Apache Kafka®
Handle a Lyft Ride?

Agenda
Who Are We?
Rides Without Kafka
Rides With PubSub Architecture + Demo
Hardened PubSub Architecture + Demo
Future Work

Who Are We?
Andrey Falko
● Launched Lyft’s internal Kafka platform
● Enjoys hiking, skiing, and kayaking
● Discovered that he is a cat person
Can Cecen
● Launched MSK and now taking Lyft’s
Kafka platform to the next level
● Outside coding, uses fingers to play
bass guitar
● Has a dog, but she’s more like a cat.
Dogcat person?

Persist ride request quickly
Transition states correctly
Get feedback to user quickly
Eﬃciency
Requirements Summary

Queueing vs. Publish/Subscribe
1 2 3 4 5 6 7
Queue Consumer 1
Consumer 2
Consumer 3

Queueing vs. Publish/Subscribe
1 2 3 4 5 6
Queue Consumer 1 7
Consumer 2
Consumer 3

Queueing vs. Publish/Subscribe
1 2 3 4 5
Queue Consumer 1 7
Consumer 2 6
Consumer 3

Queueing vs. Publish/Subscribe
1 2 3 4
Queue Consumer 1 7
Consumer 2 6
Consumer 3 5

Queueing vs. Publish/Subscribe
1 2 3 4 5 6 7
PubSub - Immutable Log Consumer Group 1
Consumer Group 2
Consumer Group 3

Queueing vs. Publish/Subscribe
1 2 3 4 5 6 7
PubSub - Immutable Log Consumer Group 1 7
Consumer Group 2 7
Consumer Group 3 7

Queueing vs. Publish/Subscribe
1 2 3 4 5 6 7
PubSub - Immutable Log Consumer Group 1 5,6,7
Consumer Group 2 7
Consumer Group 3 7
Oﬀset for Consumer Group 1

Queueing vs. Publish/Subscribe
Queue PubSub
Single Subscriber Multi-subscriber
Non-replayable Replayable

Languages
Python
Golang
Java
Ruby
Onboarding time
How can we simplify?
Non-homogenous Kafka Clients

Simplify conﬁguration
Inherit defaults
Homogenous metrics
Wrapper per programming language
Metrics namespace
Better Client Experience

Security Updates
Need to update Linux Kernel
Network, Host, and Operator Errors
Kafka Version Upgrades
Disaster Recovery
Kafka Operations - Restarts

Make switch from Kafka’s ZNodes
Reuse remediator <https://lft.to/305FyTg>
Fast Cluster Switch

Envoy as Kafka Proxy
KIP-559 - Make it easier to intercept Kafka protocol traﬃc
Envoy gives us:
Observability
Fault injection and a framework to implement health
checks
Service Discovery

Distributed logs are good for state machines
Take higher p50 to gain lower p99 latency
Invest in automation to support multi-cluster Kafka
Takeaways

Thank You
Thank you!
github.com/afalko
Andrey Falko <afalko@lyft.com>
linkedin.com/in/andrey-falko/
github.com/cancecen
Can Cecen <ccecen@lyft.com>
linkedin.com/in/yilmazcancecen
https://bit.ly/32ufowH

Kafka on K8s with Envoy?
Stateless web service model
Svc 1 Svc 2
Kubelet 1 Kubelet 2
Service Discovery
Envoy Envoy

Kafka on K8s with Envoy?
Stateless web service model
Svc 1 Svc 2
Kubelet 1 Kubelet 2
Service Discovery
Envoy Envoy
I’m Svc 1 at 10.1.1.10 I’m Svc 2 at 10.1.1.20

Kafka on K8s with Envoy?
Stateless web service model
Svc 1 Svc 2
Kubelet 1 Kubelet 2
Service Discovery
Envoy Envoy
localhost:svc-port

Kafka on K8s with Envoy?
Stateless web service model
Svc 1 Svc 2
Kubelet 1 Kubelet 2
Service Discovery
Envoy Envoy
localhost:svc-port
Svc 2 at 10.1.1.20

Kafka on K8s with Envoy?
Stateless web service model
Svc 1 Svc 2
Kubelet 1 Kubelet 2
Service Discovery
Envoy Envoy
localhost:svc-port
10.1.1.20:svc-port

What about Kafka with Envoy?
Client-side can handle bootstrap
Client Kafka
Kubelet 1
Kubelets
Service Discovery
Envoy Envoy

What about Kafka with Envoy?
Client-side can handle bootstrap
Client Kafka
Kubelet 1
Kubelets
Service Discovery
Envoy Envoy
localhost:9092
Kafka at 10.1.1.20,
10.1.1.21, 10.1.123
10.1.1.21:9092

What about Kafka with Envoy?
Client-side can handle bootstrap
Client Kafka
Kubelet 1
Kubelets
Service Discovery
Envoy Envoy
Cluster Metadata
Kafka at 10.1.1.20,
10.1.1.21, 10.1.123
Cluster Metadata
Cluster
Metadata

What about Kafka with Envoy?
Model breaks: client-side Envoy bypassed
Client Kafka
Kubelet 1
Kubelets
Service Discovery
Envoy Envoy
Write to topic
example-0 w/ leader:
10.1.1.23:9092

What about Kafka with Envoy?
Server-side 2x connections
Kafka
Kubelets
EnvoyWrite to topic
example-0 w/ leader:
10.1.1.23:9092
Client
Envoy

Más contenido relacionado

La actualidad más candente

Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL

confluent

As a Cloud Service Provider, Clever Cloud makes heavy use of messaging in its core architecture. Initially using other messaging solutions, we faced a number of different challenges. Then we discovered Apache Pulsar, which held the promise to solve these challenges. This session will dive into the adoption of Apache Pulsar at Clever Cloud, including how we worked with StreamNative to develop KoP (Kafka on Pulsar), and how we're using Apache Pulsar as the foundation for our cloud backbone.

Apache Pulsar: A Foundation Backbone for Clever Cloud - Pulsar Virtual Summit...

StreamNative

Cloud migration: it's practically a rite of passage for anyone who's built infrastructure on bare metal. When we migrated our 5-year-old Kafka deployment from the datacenter to GCP, we were faced with the task of making our highly mutable server infrastructure more cloud-friendly. This led to a surprising decision: we chose to run our Kafka cluster on Kubernetes. I'll share war stories from our Kafka migration journey, explain why we chose Kubernetes over arguably simpler options like GCP VMs, and present the lessons we learned while making our way toward a stable and self-healing Kubernetes deployment. I'll also go through some improvements in the more recent Kafka releases that make upgrades crucial for any Kafka deployment on immutable and ephemeral infrastructure. You'll learn what happens when you try to run one complex distributed system on top of another, and come away with some handy tricks for automating cloud cluster management, plus some migration pitfalls to avoid. And if you're not sure whether running Kafka on Kubernetes is right for you, our experiences should provide some extra data points that you can use as you make that decision.

Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019

confluent

Introducing Exactly Once Semantics To Apache Kafka

Apurva Mehta

Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture

confluent

When it comes to choosing a distributed streaming platform for real-time data pipelines, everyone knows the answer - Apache Kafka! And when it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. However, with all great things, the devil is truly in the details. While Kubernetes does provide all the building blocks that are needed, a lot of thought is required to truly create an enterprise-grade Kafka platform that can be used in production. In this technical deep dive, Michael and Viktor will go through challenges and pitfalls of managing Kafka on Kubernetes as well as the goals and lessons learned from the development of the Confluent Operator for Kubernetes. NOTE: This talk together with Michael Ng from Confluent

Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov,...

confluent

Dual writes are a common source of issues in distributed event-driven applications. A dual write occurs when an application has to change data in two different systems - for instance, when an application needs to persist data in the database and send a Kafka message to notify other systems. If one of these two operations fail, you might end up with inconsistent data which can be hard to detect and fix. OpenShift Streams for Apache Kafka is Red Hat's fully hosted and managed Apache Kafka service targeting development teams that want to incorporate streaming data and scalable messaging in their applications, without the burden of setting up and maintaining a Kafka cluster infrastructure. Debezium is an open source distributed platform for change data capture. Built on top of Apache Kafka, it allows applications to react to inserts, updates, and deletes in your databases. In this session you will learn how you can leverage OpenShift Streams for Apache Kafka and Debezium to avoid the dual write issue in an event-driven application using the outbox pattern. More specifically, we will show you how to: Provision a Kafka cluster on OpenShift Streams for Apache Kafka. Deploy and configure Debezium to use OpenShift Streams for Apache Kafka. Refactor an application to leverage Debezium and OpenShift Streams for Apache Kafka to avoid the dual write problem.

Friends don't let friends do dual writes: Outbox pattern with OpenShift Strea...

Red Hat Developers

Over the years at Yelp, we have relied on Kafka to build many complex applications and stream processing data-pipelines that solve a multitude of use cases, including powering our product experimentation workflow, search indexing, asynchronous task processing and more. Today, Kafka is at the core of our infrastructure. These applications use different versions of Kafka clients and different programming languages.To fulfill the requirements of these diverse use cases, we run several specialized Kafka clusters for high-availability, consistency, exactly-once and infinite retention. We endeavor to keep our clusters up-to-date with newer Kafka versions that bring with them several critical bug fixes and exciting features like dynamic broker configuration, exactly-once semantics, kafka offset management and improved tooling. Our journey with Kafka started with version 0.8.2.0. Upgrading Kafka while ensuring client compatibility, zero-downtime, negligible performance degradation across our ever-growing multi-regional cluster deployment exposed us to a plethora of unique challenges. This session will focus on the challenges we encountered and how we evolved our infrastructure tooling and upgrade strategy to overcome them. I will be talking about: -- How we rolled out new features such as kafka offset storage, message timestamp, reassignment auto-throttling, etc. -- Core technical issues discovered during upgrades such as failure of log cleaners due to large offsets while upgrading. -- The in-house test-suite that we built in order to: validate new kafka versions against our existing tooling and client-libraries, exercise the upgrade and rollback process and benchmark performance. -- The automation we built for safe and fast rolling upgrades and broker configuration deployment.

Running large scale Kafka upgrades at Yelp (Manpreet Singh,Yelp) Kafka Summit...

confluent

Westpac Bank Tech Talk 1: Dive into Apache Kafka

confluent

Kafka aws

Ariel Moskovich

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly. While Kafka and Cassandra underpins the data layer of the stack providing capability to stream, disseminate, store and retrieve data at very low latency, Kubernetes is a container orchestration technology that helps in automated application deployment and scaling of application clusters. In this presentation, we will reveal how we architected a massive scale deployment of a streaming data pipeline with Kafka and Cassandra to cater to an example Anomaly detection application running on a Kubernetes cluster and generating and processing massive amount of events. Anomaly detection is a method used to detect unusual events in an event stream. It is widely used in a range of applications such as financial fraud detection, security, threat detection, website user analytics, sensors, IoT, system health monitoring, etc. When such applications operate at massive scale generating millions or billions of events, they impose significant computational, performance and scalability challenges to anomaly detection algorithms and data layer technologies. We will demonstrate the scalability, performance and cost effectiveness of Apache Kafka, Cassandra and Kubernetes, with results from our experiments allowing the Anomaly detection application to scale to 19 Billion anomaly checks per day.

ApacheCon2019 Talk: Kafka, Cassandra and Kubernetesat Scale – Real-time Ano...

Paul Brebner

Introduction to Apache Kafka

Shiao-An Yuan

Apache Kafka - Martin Podval

Martin Podval

Apache Kafka is the most popular open-source stream-processing software for collecting, processing, storing, and analyzing data at scale. Most known for its excellent performance, low latency, fault tolerance, and high throughput, it's capable of handling thousands of messages per second. For mission-critical applications, how do you ensure that the performance delivered is the performance required? This is especially important as Kafka is written in Java and Scala and runs on the JVM. The JVM is a fantastic platform that delivers on an internet scale. In this session, we'll explore how making changes to the JVM design can eliminate the problems of garbage collection pauses and raise the throughput of applications. For cloud-based Kafka applications, this can deliver both lower latency and reduced infrastructure costs. All without changing a line of code!

Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul

HostedbyConfluent

Confluent Cloud runs a modified version of Apache Kafka - redesigned to be cloud-native and deliver a serverless user experience. In this talk, we will discuss key improvements we've made to Kafka and how they contribute to Confluent Cloud availability, elasticity, and multi-tenancy. You'll learn about innovations that you can use on-prem, and everything you need to make the most of Confluent Cloud.

Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...

HostedbyConfluent

Netflix Data Pipeline With Kafka

Steven Wu

Upgrades suck. We get it. They are risky and time consuming and you have better things to do. In this talk we'll present good reasons to upgrade anyway and give suggestions on how to de-risk your upgrades. Straight from the team that upgrades Kafka almost every week. We'll review all the releases in the past year - major, minor and bug-fixes. We'll explain the differences between those and what can you expect from each. We'll go into the most important features and most critical fixes and improvements, so you'll have ample ammunition when you explain to your boss why you really have to upgrade Kafka. Then we'll discuss how we validate new releases and suggest a safe upgrade process - because we know that uneventful upgrades are a key to the next upgrade.

Please Upgrade Apache Kafka. Now. (Gwen Shapira, Confluent) Kafka Summit SF 2019

confluent

Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter

HostedbyConfluent

Apache Kafka is a key part of the Big Data infrastructure at Salesforce, enabling publish/subscribe and data transport in near real-time at enterprise scale handling trillions of messages per day. In this session, hear from the teams at Salesforce that manage Kafka as a service, running over a hundred clusters across on-premise and public cloud environments with over 99.9% availability. Hear about best practices and innovations, including: * How to manage multi-tenant clusters in a hybrid environment * High volume data pipelines with Mirus replicating data to Kafka and blob storage * Kafka Fault Injection Framework built on Trogdor and Kibosh * Automated recovery without data loss * Using Envoy as an SNI-routing Kafka gateway We hope the audience will have practical takeaways for building, deploying, operating, and managing Kafka at scale in the enterprise.

Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...

HostedbyConfluent

If your data platform is powered only by batch data processing, you know you are always trailing your customer. Your databases aren’t always up to date. Your inability to have a synchronized data flow across systems leads to operational inefficiencies. And, your dreams of running advanced real-time AI and ML applications can’t be fulfilled. However, you might be wary of the implications of turning your product into an event-driven one. In this presentation we’ll share our experience transforming our CDP-based marketing orchestration engine to be both real-time and highly scalable with the Kafka ecosystem. We will look into how we saved resources with Connect when ingesting and syncing data with NoSQL databases, data warehouses and third-party platforms. What we did to turn ksqlDB into our data transformation, aggregation and querying hub, reducing latency and costs. How Streams helps us activate multiple real-time applications such as building identity graphs, updating materialized views in high frequency for efficient real-time lookups and inferencing machine learning models. Finally, we will look at how Confluent Cloud solved our pre-rollout sizing and scaling questions, significantly reducing time-to-market.

Learnings from the Field. Lessons from Working with Dozens of Small & Large D...

HostedbyConfluent

La actualidad más candente (20)

Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL

Apache Pulsar: A Foundation Backbone for Clever Cloud - Pulsar Virtual Summit...

Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019

Introducing Exactly Once Semantics To Apache Kafka

Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture

Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov,...

Friends don't let friends do dual writes: Outbox pattern with OpenShift Strea...

Running large scale Kafka upgrades at Yelp (Manpreet Singh,Yelp) Kafka Summit...

Westpac Bank Tech Talk 1: Dive into Apache Kafka

Kafka aws

ApacheCon2019 Talk: Kafka, Cassandra and Kubernetesat Scale – Real-time Ano...

Introduction to Apache Kafka

Apache Kafka - Martin Podval

Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul

Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...

Netflix Data Pipeline With Kafka

Please Upgrade Apache Kafka. Now. (Gwen Shapira, Confluent) Kafka Summit SF 2019

Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter

Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...

Learnings from the Field. Lessons from Working with Dozens of Small & Large D...

Similar a Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020

At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker. This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters. In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner. You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant. You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC. When to use a client library, and when to deploy an external pod (DaemonSet, StatefulSet) or even deploy a sidecar.

Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...

Natan Silnitsky

Spark Streaming has supported Kafka since it's inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, with a similar design to the 0.8 Direct DStream approach. However, there are notable differences in usage, and many exciting new features. In this talk, we will cover what are the main differences between this new integration and the previous one (for Kafka 0.8), and why Direct DStreams have replaced Receivers for good. We will also see how to achieve different semantics (at least one, at most one, exactly once) with code examples. Finally, we will briefly introduce the usage of this integration in Billy Mobile to ingest and process the continuous stream of events from our AdNetwork.

Spark streaming + kafka 0.10

Joan Viladrosa Riera

Kafka Explainaton

NguyenChiHoangMinh

[Big Data Spain] Apache Spark Streaming + Kafka 0.10: an Integration Story

Joan Viladrosa Riera

Trivadis TechEvent 2016 Apache Kafka - Scalable Massage Processing and more! ...

Trivadis

After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL. Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go. Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.

Apache Kafka - Scalable Message Processing and more!

Guido Schmutz

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...

Natan Silnitsky

How to build 1000 microservices with Kafka and thrive

Natan Silnitsky

Kafka is the bedrock of Wix’s distributed Mega Microservices system. Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices. In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These Include: * How to increase dev velocity of event-driven style code. * How to optimize working with Kafka in polyglot setting * How to migrate from request-reply to event-driven * How to tackle multiple DCs environment.

10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA

Natan Silnitsky

Apache Kafka sits at the core of the modern scalable event driven architecture. It’s no longer used only as logging infrastructure, but as a core component in thousands of companies around the world. It has the unique capability to provide low-latency, fault-tolerant pipeline at scale that is very important for today’s world of big data. During this talk we’ll see what makes Apache Kafka perfect for the job. We’ll explore how to optimize it for throughput or for durability. And we’ll also go over the messaging semantics it provides. Last but not least, we’ll see how Apache Kafka can help us solve some everyday problems that we face when we build large scale systems in an elegant way.

Scaling big with Apache Kafka

Nikolay Stoitsev

Apache Kafka - A modern Stream Processing Platform

Guido Schmutz

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...

Natan Silnitsky

Spark Streaming + Kafka 0.10: an integration story by Joan Viladrosa Riera at...

Big Data Spain

Event-driven application architectures are becoming increasingly common as a large number of users demand more interactive, real-time, and intelligent responses. Yet it can be challenging to decide how to capture and perform real-time data analysis and deliver differentiating experiences. Join experts from Confluent and AWS to learn how to build Apache Kafka®-based streaming applications backed by machine learning models. Adopting the recommendations will help you establish repeatable patterns for high performing event-based apps.

Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...

HostedbyConfluent

Kafka Connect & Kafka Streams/KSQL - the ecosystem around Kafka

Guido Schmutz

[Spark Summit EU 2017] Apache spark streaming + kafka 0.10 an integration story

Joan Viladrosa Riera

Spark Streaming has supported Kafka since it’s inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, with a similar design to the 0.8 Direct DStream approach. However, there are notable differences in usage, and many exciting new features. In this talk, we will cover what are the main differences between this new integration and the previous one (for Kafka 0.8), and why Direct DStreams have replaced Receivers for good. We will also see how to achieve different semantics (at least one, at most one, exactly once) with code examples. Finally, we will briefly introduce the usage of this integration in Billy Mobile to ingest and process the continuous stream of events from our AdNetwork.

Apache Spark Streaming + Kafka 0.10 with Joan Viladrosariera

Spark Summit

Kafka is the bedrock of Wix's distributed microservices system. For the last 5 years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 microservices. We’ve managed to achieve higher decoupling and independence for our various services and dev teams that have very different use-cases while maintaining a single uniform infrastructure in place. In these slides you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These include: * How to increase dev velocity of event driven style code. * How to optimize working with Kafka in polyglot setting * How to support growing amount of traffic and developers. * How to tackle multiple DCs environment.

10 Lessons Learned from using Kafka with 1000 microservices - java global summit

Natan Silnitsky

Kонструируем микросервисную архитектуру на Symfony и связываем с Kong

Alex Demchenko

Netflix Keystone—Cloud scale event processing pipeline

Monal Daxini

Similar a Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020 (20)