Event Streaming CTO Roundtable for Cloud-native Kafka Architectures

Event Streaming CTO Roundtable
Real-World Use Cases for Data in Motion with Cloud-native Architectures
Kai Waehner
Field CTO
kai.waehner@confluent.io
linkedin.com/in/kaiwaehner
@KaiWaehner
confluent.io
kai-waehner.de

@KaiWaehner - www.kai-waehner.de – Cloud-native Event Streaming CTO Roundtable
Agenda
• Data in Motion with Event Streaming
• Streaming ETL Pipelines
• IT Modernisation and Hybrid Multi-Cloud
• Customer Experience and Customer 360
• IoT and Big Data Processing
• Machine Learning and Analytics

This is a fundamental paradigm shift...
4
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud-
Native
Event
Streaming

Real-time Data beats Slow Data.
Logistics
Real-time sensor
diagnostics
Delivery planning
ETA updates
Payment
Fraud detection
Risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Cybersecurity
Threat detection
Intrusion detection
Incident response
Fraud detection

Apache Kafka is the Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
6

Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.

Lambda Architecture
Option 1: Unified serving layer
9
Data
Source
Real-Time Layer
(Data Processing in Motion)
Batch Layer
(Data Processing at Rest)
Serving Layer
Real-Time App
Batch App
ms
min/hr

10
Data
Source
Real-Time Layer
Batch Layer
Real-time Query
Mixed Query
ms
min/hr
Speed
View
Batch
View
Batch Query
Lambda Architecture
Option 2: Separate serving layers

11
Data
Source
Real-Time Layer
Real-Time App
Storage
Batch App
Storage
ms
min/hr
Storage
Kappa Architecture
One pipeline for real-time and batch consumers

Kappa @ Uber
12

Kappa @ Shopify
13
Kappa Building Blocks
The Log (Kafka)
Durability with Topic Compaction and Tiered Storage
Consistency via Exactly-Once Semantics (EOS)
Data Integration via Kafka Connect
Elasticity via dynamic Kafka clusters
Streaming Framework (Kafka Streams / Flink)
Reliability and scalability
Fault tolerance
State management
Sinks
Update/Upsert for simplified design:
RDBMS, NoSQL, Compacted Kafka Topics
Append-only: Regular Kafka Topics, Time Series

Kappa @ Disney
14
www.kai-waehner.de | @KaiWaehner | Streaming Machine Learning without a Data Lake

Mainframe Offloading
Journey from Mainframe
to Hybrid* and Cloud
PHASE 3
Hybrid
Replication
Mainframe
Replacement
PHASE 2
PHASE 1
* with or without the mainframe

Strangler Design Pattern - A Big Bang will FAIL !!!
https://paulhammant.com/2013/07/14/legacy-application-strangulation-case-studies/

Year 0: Direct Communication between Mainframe and App
Application
1) Direct Legacy Mainframe Communication to App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Core Banking ‘1970’
(Mainframe)

Year 1: Kafka for Decoupling between Mainframe and App
Application
2) Kafka for Decoupling between Mainframe and App
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party CDC Tool

Year 2 to 4: New Projects and Applications
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
3) New Projects and Applications
External
Solution
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
(Mainframe)
Mainframe Integration
- Change Data Capture (IIDR)
- Kafka Connect (JMS, MQ, JDBC)
- REST Proxy
- Kafka Client
- 3rd Party CDC Tool

Year 5: Mainframe Replacement
Application
Microservices
Agile, Lightweight
(but Scalable, Robust)
Applications
Big Data Project (Elastic,
Spark,
AWS Services, …)
3) New Projects and Applications
4) Mainframe Replacement
External
Solution
(Modern Technology)
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14

Integration Platform
for legacy and modern technologies

Global Event Streaming
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO=0
Stream Data Globally with
Replication and Cluster Linking
23

Domain
Data
Product
Focus on Business and Data Products with Decoupled Microservices
Data
Mesh
Mesh is a logical view,
not physical!

Context-specific Customer 360
26
Electrical retailer
Hyper-personalized online retail experience,
turning each customer visit into a one-on-one
marketing opportunity
Correlation of historical customer data with real-
time digital signals
Maximize customer satisfaction and revenue
growth, increased customer conversions
https://www.confluent.io/customers/ao/

Innovative Business Modell
• Clothing rental subscription service
• Very different from a typical e-commerce model
• Need for a real-time event driven architecture
Benefits of serverless Confluent Cloud
• Cut launch time from over a year to 6 months
• Stable production ops set up in 1 week vs. 6 months
• Administrative overhead reduced by 10
27
https://www.confluent.io/customers/nuuly/

‘My Porsche’
A digital service platform for customers, fans, and enthusiasts
28
https://medium.com/porschedev

Omnichannel Retail
Time
P
C3 C2
C1
Sales Talk on site in
Car Dealership
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Car Configurator
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago

Omnichannel Retail
Time
P
C3 C2
C1
Machine Learning
Context-specific
Recommendations
Location-based
Customer Action
Customer 360
(Business Intelligence, Machine Learning)
Machine Learning
Train Recommendation Engine
Reporting
All Customer Interactions

CRM
Real-Time Location System
(RTLS) for Asset Tracking
Customer data
Advanced Planning
and Scheduling (APS)
Manager
Get report
API
Customer Customer
Customer
data
Truck
schedule
Payment
data
Route
details
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Customer
data
Train
schedule
Payment
data
Loyalty
information
Wavelength
Cloud VPC
Carrier #1 5G
Wavelength
Carrier #2 5G
32
Hybrid Streaming Data Exchange

Event Topics Storage Partitions
Events / sec Kafka Servers
10,000,000 25,000 1,000,000 1,500
250,000 500 25,000 25
100 5 300 3
Kafka Scales with Your Business.

BMW Group
Mission-critical workloads at the edge and in the cloud
• Why Kafka? Decoupling. Transparency. Innovation.
• Why Confluent? Stability is key in manufacturing
• Decoupling between logistics and production systems
• Provide edge platform (self-managed) + Azure Cloud (fully-managed) + bidirectional
integration
• Use case
• Logistics and supply chain in global plants
• Right stock in place (physically and in ERP systems like SAP)
• Just in time, just in sequence
• Lot of critical applications
34
Jay Kreps, Confluent CEO
Felix Böhm, BMW Plant Digitalization and Cloud Transformation
Keynote at Kafka Summit Eurpoe 2021:
https://www.youtube.com/watch?v=3cG2ud7TRs4
(My Notes from the BMW Keynote at Kafka Summit EU 2021)

Apache Kafka as Infrastructure for ML

Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams/
ksqlDB
Kafka Connect
Confluent REST Proxy
Confluent Schema Registry
Go/.NET/Python
Kafka Producer
ksqlDB
Python
Client

Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model B
Model A
Producer
Distributed
Commit Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
38
Model X
(at a later time)

Confluent Tiered Storage for Kafka
39

“CREATE STREAM AnomalyDetection AS
SELECT sensor_id, detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)
Model Deployment with
Apache Kafka, ksqlDB
and TensorFlow
41

Fraud Detection @ Grab
GrabDefence SaaS service build with Confluent Cloud, Kafka Streams and ML for stateful stream processing
Billions of fraud and safety detections performed daily for millions of transactions (1.6% is lost in fraud in Southeast Asia)

Kai Waehner
Field CTO
kai.waehner@confluent.io
@KaiWaehner
kai-waehner.de
confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Event Streaming CTO Roundtable for Cloud-native Kafka Architectures

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Event Streaming CTO Roundtable for Cloud-native Kafka Architectures

Similar a Event Streaming CTO Roundtable for Cloud-native Kafka Architectures (20)

Más de Kai Wähner

Más de Kai Wähner (19)

Último

Último (20)

Event Streaming CTO Roundtable for Cloud-native Kafka Architectures