When it comes to choosing a distributed streaming platform for real-time data pipelines, everyone knows the answer: Apache Kafka! And when it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. However, with all great things, the devil is truly in the details. While Kubernetes does provide all the building blocks that are needed, a lot of thought is required to truly create an enterprise-grade Kafka platform that can be used in production. In this technical deep dive, Michael and Viktor will go through challenges and pitfalls of managing Kafka on Kubernetes as well as the goals and lessons learned from the development of the Confluent Operator for Kubernetes. NOTE: This talk will be delivered with Michael Ng, product manager, Confluent
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov and Michael Ng, Confluent) Kafka Summit NYC 2019
1. @gamussa | #kafkasummit | @ConfluentINc
Kafka on Kubernetes:
Does it really have to be
«The Hard Way»?
April, 2019 / New York, 2019
@gamussa | #kafkasummit | @ConfluentINc
6. @gamussa | #kafkasummit | @ConfluentINc
6
Who run stateless
workloads in Kubernetes?
Who thinks it’s a good
idea?
Who run stateful
workloads in Kubernetes?
Who thinks it’s a good
idea?
🙋
17. @gamussa | #kafkasummit | @ConfluentINc
17
DO KAFKA ON KUBERNETES DEMO
AND EVERYONE LOOSES THEIR MIND
18. @gamussa | #kafkasummit | @ConfluentINc
18
What just happened?
ZK and Kafka deployed
Security with TLS is configured
External access is configured
Monitoring is enabled
19. @gamussa | #kafkasummit | @ConfluentINc
19
Confluent Operator - Automated
Security Configuration
SASL PLAIN and Mutual TLS Authentication
Automate configuration of truststores and
keystores with secret objects
Automate configuration of Kafka and all
Confluent Platform Components
20. @gamussa | #kafkasummit | @ConfluentINc
20
Confluent Operator - Scale
Automate Scaling:
Spin up new brokers, connect workers easily
Distribute partitions to new brokers:
Determine balancing plan
Execute balancing plan
Monitor Resources
24. @gamussa | #kafkasummit | @ConfluentINc
24
GA Plans● We are in private Preview
Release now
● 24 customers testing the
Operator in Preview:
● Global customers
● Banks, Fin Tech,
Retailers, Consumer Tech
● We are in the final
stages of Preview and
about to launch soon