Use Cases, Architectures, and Real-World Examples for data in motion and real-time event streaming powered by Apache Kafka across the supply chain and logistics. Case studies and deployments include Baader, Walmart, Migros, Albertsons, Domino's Pizza, Instacart, Grab, Royal Caribbean, and more.
Reassessing the Bedrock of Clinical Function Models: An Examination of Large ...
Apache Kafka for Real-time Supply Chainin the Food and Retail Industry
1. The Rise of Data in Motion
in the Food and Retail Industry
Use Cases, Architectures and Real-World Examples powered by Apache Kafka
Kai Waehner
Field CTO
kai.waehner@confluent.io
linkedin.com/in/kaiwaehner
confluent.io
kai-waehner.de
@KaiWaehner
2. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
The Food Industry
2
3. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
The world is changing.
4. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Stores become Software
5. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Payments become Software
7. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Partnering becomes Software
2
m
in
8. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for
growth
“Good enough” to run on
yesterday’s data
9. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
The Food Industry
9
The Real-Time Food
and Retail Supply Chain
10. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
This transformation is
happening everywhere
11. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Food and Retail
12. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Resorts World Las Vegas + Grubhub
15
13. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
What enables this
transformation?
14. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Real-time Data beats Slow Data.
Logistics
Real-time sensor
diagnostics
Delivery planning
ETA updates
Payment
Fraud detection
Risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS
reporting
Personalization
Sales
Real-time
recommendations
Personalized
coupon feed
Pay by walking out
15. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Chick-fil-A Restaurant IoT
• One of the largest American fast food
restaurant chains
• Chick-Fil-A IoT platform RIoT (Restaurant
IoT)
• Kubernetes cluster in each of their 2000+
restaurants for real-time analytics at the
edge without an internet connection
• Thousands of units of industrial kitchen
equipment, food safety sensors and
point-of-sale devices are sending millions
of streaming messages per second
18
https://www.linkedin.com/pulse/why-did-chicken-need-iot-imran-bashir/
https://www.infoq.com/presentations/chick-fil-a-k8-clusters/
https://medium.com/@cfatechblog/edge-computing-at-chick-fil-a-7d67242675e2
16. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
This is a fundamental paradigm shift...
19
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming
17. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
What is Event Streaming?
18. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
‘Event’ is what happens in your business
Transportation
TPMS sensor in Carol’s car detected low tire-pressure at 5:11am.
Kafka
Payment
Alice sent $250 to Bob on Friday at 7:34pm.
Kafka
Retail
Sabine’s order of a Fujifilm camera was shipped at 9:10am.
Kafka
19. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Data in Motion in the Food and Retail Industry
Your Business as Streams of Events, powered by Kafka
Inventory
Shipping
Reporting
Orders
Frontend
Retail Example
20. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
24
Increase Revenue
→ Customer Experience, Loyalty
Decrease Costs
→ Increase Operational Efficiency
Mitigate Risks
→ Regulatory Compliance
Reporting & Analytics
Stock inventory management
Legacy IT Replacement
(e.g. Middleware replacement)
Cyber Security
(incl. SIEM)
Fraud Prevention
(Anomalous credit card transactions)
Legacy IT Modernization
(e.g. Mainframe off-load / augmentation)
Example Retail Solutions for Event Streaming
Buyer journey - personalized offers
(Contextualized Next Best Action /
Targeted Offers)
Real-time Inventory Management
(across web, store, …)
Customer 360
(Call Center - Know Your Customer - KYC)
RT mobile applications
(clothing, food and beverage ordering, …)
Migration to the Cloud
(Hybrid on-prem / Cloud. Also Hybrid
Public Cloud vendors)
Microservices Architecture
Data Infrastructure layer
Business Application layer - the use cases
Data Pipelines Messaging
Microservice/
Event Sourcing Stream Processing Data Integration Streaming ETL Log Aggregation
21. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Event Streaming
in the Food Industry?
22. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Food Value Chain
IoT-Based and Data-Driven
Single source of truth
across the food value chain
(in the factories, and across regions)
Business critical operations
(tracking, calculations, alerts, …)
https://www.confluent.io/blog/creating-iot-based-data-driven-food-value-chain-with-confluent-cloud/
23. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Migros
Switzerland's largest retail company, largest supermarket
chain and largest employer
Real-time transportation information visualization with MQTT
and Kafka
Supply chain optimization with a single streaming pipeline
(including replay whole day of events)
Forecast the truck arrival time; plan and reschedule truck tours
27
https://www.meetup.com/Messaging-Streaming-Switzerland/events/278115421
24. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
“Walmart is a $500 billion in revenue
company, so every second is worth
millions of dollars. Having Confluent
as our partner has been invaluable.
Kafka and Confluent are the
backbone of our digital
omnichannel transformation
and success at Walmart.”
VP of Walmart Cloud
25. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Real-Time Inventory System
https://www.confluent.io/blog/walmart-real-time-inventory-management-using-kafka/
https://www.confluent.io/kafka-summit-san-francisco-2019/when-kafka-meets-the-scaling-and-reliability-needs-of-worlds-largest-retailer-a-walmart-story/
● Investment in Kafka and Confluent has helped topline
company growth
● 8,500 nodes processing 11 billion events per day
● Deliver an omnichannel experience so every
customer can shop the way they want to
26. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Real-time Replenishment System
https://www.confluent.io/events/kafka-summit-americas-2021/speed-without-accidents-kafka-exemplified/
● Caters millions of its online and walk in customers
● Ensures optimal availability of needed assortment
and timely delivery on online fulfillment
● 4+ billion messages in 3 hours generate an order
plan for the entire network of Walmart stores with
great accuracy
● ~470M ordering decisions made daily
● Kafka as data hub and for real-time processing
● Spark for micro-batches
27. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Simplifying Omnichannel Retail at Scale
31
https://kafka-summit.org/sessions/simplifying-omni-channel-retail-scale/
28. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Domino’s Pizza
• Multinational pizza restaurant chain with ~17,000 stores
• Transformation from a traditional pizza company to an e-commerce driven organization
• Data-first approach and relentless customer focus
• Real-time operation views to franchise owners (order volume by channel, store efficiency metrics, …) across different
ordering channels
Benefits
Improve store operational real-time analytics
Support global expansion goals via legacy IT modernization
Implement more personalized marketing campaigns
Real-time single pane of glass
https://www.confluent.io/customers/dominos/
29. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
FREE NOW
Stateful stream processing with Confluent Cloud, Kafka Connect, Kafka Streams, Schema Registry
Cloud-native application elasticity and scalability leveraging Kafka and Kubernetes capabilities
Use cases: Dynamic pricing, fraud detection, real-time analytics for marketing campaigns, etc.
Various information about the trip, location and business performance
35
30. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Fraud Detection @ Grab
GrabDefence SaaS service build with Confluent Cloud, Kafka Streams and ML for stateful stream processing
Billions of fraud and safety detections performed daily for millions of transactions (1.6% is lost in fraud in Southeast Asia)
Data science and engineering platform to search for anomalous and suspicious transactions and identifying
high-risk individuals
Example: An individual who pretends to be both the driver and passenger, and makes cashless payments to
get promotions
31. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
instacart
Grocery delivery and pick-up service in the United States
and Canada.
Order groceries from participating retailers with the
shopping being done by a personal shopper
Elastic scale and fast, agile internal adoption of real-time
data processing with Confluent Cloud
https://www.confluent.io/events/kafka-summit-americas-2021/driving-innovation-with-data-in-motion-at-instacart/
Covid Pandemic:
“Handle 10 years' worth
of growth in six week”
32. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Context-specific Customer 360
39
Electrical retailer
Hyper-personalized online
retail experience, turning
each customer visit into a
one-on-one marketing
opportunity
Correlation of historical
customer data with real-
time digital signals
Maximize customer
satisfaction and revenue
growth, increased
customer conversions
https://www.confluent.io/customers/ao/
33. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Live commerce with real-time data correlation
including integration of CRM, loyalty, inventory, chatbots, location-based services, etc.
40
34. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Live Commerce with Data in Motion
Your Business as Streams of Events, powered by Kafka
Influencer Show
on Veggie Food
Payment
Shipping
Sales
Platform
Social
Platform
35. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Live Commerce with Data in Motion
Interact with the customer during the show.
Recommend products that need to be sold.
Provide context-specific pricing.
All automated. In real-time. At scale.
Data Consolidation
Streaming ETL
(Kafka Connect + Kafka Streams)
Mobile App
(Swift on iOS)
(4) Receive
product recommendation
and context-specific price
Order Fulfillment &
Payment Service
Buy Now Pay Later (BNPL)
(C++)
Chatbot
External API
(REST API)
(1) Click on
food
(5) Order
product
(2) Q&A with chatbot
(5) Fulfill order
CRM / Loyalty
Salesforce
(Kafka Connect)
Source: channelize.io
Inventory
Live monitoring
(ksqlDB)
(3) Context-specific
pricing engine
36. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
@KaiWaehner - www.kai-waehner.de
Cross-Company Stream Data Exchange
Streaming Replication and API Management
Cluster Linking
Food
Supplier
Delivery
Service
Streaming integration
between companies
API Management
(REST et al) is not appropriate
for
streaming data
Restaurant
37. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
What is
Apache Kafka?
38. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Kafka: The Trinity of Event Streaming
01
Publish & Subscribe
to Streams of Events
02
Store
your Event Streams
03
Process & Analyze
your Events Streams
39. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Apache Kafka is a Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
46
40. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Kafka Makes Your Business Real-time.
CREATE STREAM payments (user VARCHAR, amount INT)
WITH (kafka_topic = 'all_payments', value_format = 'avro');
CREDIT
SERVICE
ksqlDB
CREATE TABLE credit_scores AS
SELECT user, updateScore(p.amount) AS credit_score
FROM payments AS p
GROUP BY user
EMIT CHANGES;
RISK
SERVICE
ksqlDB
41. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Risk Management
with Stream Processing
51
British multinational groceries and general merchandise retailer
One of Europe's largest retailers
Risk-management platform built to detect and prevent attacks
https://www.confluent.io/resources/kafka-summit-2020
/risk-management-in-retail-with-stream-processing/
42. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Central Nervous System
Your Business as Streams of Events, powered by Kafka
Inventory
Event streams are stored for
reuse and with high
availability.
Shipping
Events are processed in real-
time as soon as they happen.
Frontend
Reporting
Add new use cases easily by
tapping into existing streams.
Orders
Event-driven apps and services
communicate through streams.
44. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Databases
Messaging
ETL / Data Integration
Data Warehouse
Why can’t I do this with my
existing data platforms?
45. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Enterprise Data Platform Requirements Are Shifting
1 3 4
2
Scalable for
Transactional Data
Transient Raw data
Built for
Historical Data
Built for Real-
Time Events
Scalable for
ALL data
Persistent +
Durable
Enriched
data
● Value: Trigger real-
time workflows (i.e.
real-time order
management)
● Value: Scale across
the enterprise (i.e.
customer 360)
● Value: Build
mission-critical
apps with zero data
loss (i.e. instant
payments)
● Value: Add context &
situational awareness
(i.e. ride sharing ETA)
59
46. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Only Event Streaming Has All 4 Requirements
Messaging
Databases
Event Streaming
Data Warehouse
BUILT FOR REAL-
TIME EVENTS
SCALABLE
FOR ALL DATA
PERSISTENT &
DURABLE
CAPABLE OF
ENRICHMENT
60
Good for transactional applications
Good for ultra low-latency, fire-and-forget use cases
Good for batch data integration
Good for historical analytics and reporting
Platform for Event-Driven Transformation
(Scalable Messaging + Real-Time Data Integration + Stream Processing)
ETL/Data Integration
47. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Kafka
Connect
Kafka Cluster
Data Warehouse
Integration
Domain-Driven Design and Decoupled Microservices
POS Interface Mobile Backend
Kafka Broker
in the Store
Kafka-native Client
(Java, Python C++, KSQL, etc.)
Data
Governance
Event Streaming Platform
Analytics Domain Store Domain Customer Domain
Security
48. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Event Streaming in the Cloud
62
Focus on
Business Outcomes
• Real-time
• Serverless
• Elastic
49. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Architectures for
Apache Kafka in Retail?
50. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Real-time Inventory System
C2
Time
P
C3
C1
C2
C3
Asset / People Tracking
Real-Time Alerting
Geofencing for
Distance Enforcement
Continuous
Real-Time Aggregation
Plan Future Location
Requirements
Batch Analytics
on Historical Events
51. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
for RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking
68
52. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
‘My Porsche’
A digital service platform for customers, fans, and enthusiasts
70
https://medium.com/porschedev
53. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Omnichannel Retail
Time
P
C3 C2
C1
Sales Talk on site in
Car Dealership
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Car Configurator
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago
54. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Omnichannel Retail
Time
P
C3 C2
C1
Machine Learning
Context-specific
Recommendations
Location-based
Customer Action
Customer 360
(Business Intelligence, Machine Learning)
Machine Learning
Train Recommendation Engine
Reporting
All Customer Interactions
55. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Hybrid Retail Architecture
56. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Point of Sale
(POS) Loyalty
System
Local Inventory
Management
Payment Discount
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Global Inventory
Management
Event Streaming at the Edge
in the Smart Retail Store
Item Availability
57. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Disconnected Edge
Time
P
C3 C2
C1
Context-specific
Advertisement
Real-time
(Milliseconds)
Location-based
Customer Action
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
Payment Processing
Near Real-time
(Seconds)
Replication to Cloud
Batch
(Depending on Network Bandwidth)
59. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Event Streaming Is The Future Of Data
82
Infrastructure
as code
Data as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming
61. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
84
62. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
85
63. Apache Kafka in Retail – @KaiWaehner - www.kai-waehner.de
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.