Not all workloads allow cloud computing. Low latency, cybersecurity, and cost-efficiency require a suitable combination of edge computing and cloud integration.
This session explores architectures and design patterns for software and hardware considerations to deploy hybrid data streaming with Apache Kafka anywhere. A live demo shows data synchronization from the edge to the public cloud across continents with Kafka on Hivecell and Confluent Cloud.
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
Kafka for Real-Time Replication between Edge and Hybrid Cloud
1. How to Deploy a Kafka Cluster in
Production onto Your Desk or Anywhere!
IoT Use Cases, Architectures, and Live Demo for Data in Motion powered by Apache Kafka
Kai Waehner
Field CTO
kai.waehner@confluent.io
linkedin.com/in/kaiwaehner
confluent.io
kai-waehner.de
@KaiWaehner
Dominik Pilat
VP of Customer Success & Field CTO
d.pilat@hivecell.com
linkedin.com/in/dpilat
hivecell.com
2. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Agenda
• Data in Motion at the Edge and in Hybrid Architectures
• Live Demo – Hybrid Data Streaming with Confluent on
Hivecell and Confluent Cloud
• Designing an Edge Solution
• Next steps for Kafka at the Edge
3. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Agenda
• Data in Motion at the Edge and in Hybrid Architectures
• Live Demo – Hybrid Data Streaming with Confluent on
Hivecell and Confluent Cloud
• Designing an Edge Solution
• Next steps for Kafka at the Edge
4. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Real-time Data beats Slow Data.
Manufacturing
Sensor diagnostics
MES/ERP Integration
Reporting
Edge Computing
Condition Monitoring
Predictive Maintenance
Quality Assurance
Logistics
Supply Chain
Inventory management
Track & Trace
Context-specific routing
Cybersecurity
Threat detection
Intrusion detection
Incident response
Fraud detection
5. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
AWS Cloud Outage hit Disney World Visitors…
https://www.cnet.com/tech/services-and-software/disney-parks-were-already-facing-heat-from-fans-then-an-aws-outage-came-along/
6. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Apache Kafka is the Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data
warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
6
7. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
MQTT
Data
Historian
STREAM
PROCESSING
CONNECTORS
Example Architecture for Data in Motion with Event
Streaming
ksqlDB
KStreams
Bi-directional real-time integration between the OT and IT worlds
APP
Oracle
DB
ORACLE CDC
SOURCE
CONNECTOR
APP
8. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
for RPO=0 and RTO~0
Stream Data Globally with
Replication and Cluster Linking
8
9. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
What is the “Edge” for Kafka?
• Edge is NOT a data center
• Kafka clients AND the Kafka broker(s)
• Offline business continuity
• Often 100+ locations
• Low-footprint and low-touch
• Hybrid integration
Example:
Single broker, 1 GB Ram
100 MB/sec
10. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Data Processing
at the Edge
Time
P
C1
C2
C3
Know-your-customer
Loyalty app, predictive behavior, …
Estimated
time of arrival
Connect to the
gaming server
for kids
Play games, earn rewards, communicate with
other kids in the train, …
Always on (even “offline”)
Replayability
Cost-efficiency
Low latency
11. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Shipping Industry
Marine, Oil Transport, Vessel Fleet, Shipping Line, Drones
Real-time Operations, Logistics, Predictive Maintenance, Security
Customer Data
Crew, Cargo
Vessel Data
Fuel Consumption, Speed,
Planned Maintenance
Automatic Identification System (AIS)
Unique Identification,
Position, Course, Weather, Draft
Drone Data
Deliveries,
Survey/Inspection
of Assets such as Oil Rigs,
Pipelines, Offshore Turbines
Edge Analytics
Bidirectional Edge to Cloud Integration
Data Ingestion
Stream
Processing
Data
Integration
Logistics
Track&Trace
Routing
Monitoring
Alerting
Command&Control
Batch Analytics
Reporting
Machine Learning
Backend Systems
Oracle, SAP,
OSIsoft PI, etc.
X = Event Streaming
X = Other Technologies
Bi-Directional Hybrid Cloud
Replication
12. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Royal Caribbean - Offline Edge for Swimming Retail Stores
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/
13. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Devon Energy
Oil & Gas Industry
Improve drilling and well completion operations
Edge stream processing/analytics + closed-loop control ready
Vendor agnostic (pumping, wireline, coil, offset wells, drilling
operations, producing wells)
Replication to the cloud in real-time at scale
Cloud agnostic (AWS, GCP, Azure)
Source: Energy in Data - Powered by AAPG, SEG & SPE: energyindata.org
14. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
CRM
Real-Time Location System
(RTLS) for Asset Tracking
Customer data
Advanced Planning
and Scheduling (APS)
Manager
Get report
API
Customer Customer
Customer
data
Truck
schedule
Payment
data
Route
details
Streams of real time events
Event Streaming in Hybrid Industrial
IoT
Wavelength
Public Cloud VPC
Campus 5G Telco Carrier 5G
Smart Factory Edge Computing
15. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Analytics
Database
Data Lake
CRM
Kafka as Data Historian
Continuous real time data ingestion, processing and monitoring 24/7 at scale
MQTT Modbus Firewall
Siemens S7
Kafka Connect
Integration | Storage | Processing | Access | Cloud | Security
Storage
Kafka Streams
/
ksqlDB
OPC-UA
Stateless + Stateful
Kafka
Connect
Kafka Clients
Java, C++. Python, Go, …
REST
Proxy
SCADA
DCS
ERP
MES
Syslog
File
X
Y
Z
A
B
C
Cloud
Factory
16. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Condition Monitoring and Predictive Maintenance
Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams
/ ksqlDB)
8 9
1
0
1
1
1
2
1
3
1
4
1
5
1
6
3 4 5 6 7
2
1
Time
Sensor Events
17. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Condition Monitoring and Predictive Maintenance
Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams
/ ksqlDB)
8 9
1
0
1
1
1
2
1
3
1
4
1
5
3 4 5 6 7
2
1
Time
Sensor Events
1
6
Condition Monitoring
(Temperature Spikes)
Stateless Filter Above-Threshold Events
Streams
builder
.stream(”temperature-sensor")
.filter((key, sensor-data) ->
sensor-data.temperature > 100)
.to(”temperature-spikes");
18. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Condition Monitoring and Predictive Maintenance
Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams
/ ksqlDB)
8 9
1
0
1
1
1
2
1
3
1
4
1
5
1
6
3 4 5 6 7
2
1
Time
Sensor Events
Predictive Maintenance
(Continuous Anomaly Detection)
Stateful Correlation of Events
CREATE TABLE anomaly_detection AS
SELECT temperature_spike_id, COUNT(*) AS total_spikes,
AVG(temperature) AS avg_temperature
FROM sensor-data
WINDOW TUMBLING (SIZE 1 HOUR)
GROUP BY temperature_spike_id
EMIT CHANGES;
19. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Condition Monitoring and Predictive Maintenance
Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams
/ ksqlDB)
8 9
1
0
1
1
1
2
1
3
1
4
1
5
1
6
3 4 5 6 7
2
1
Time
Sensor Events
Predictive Maintenance
(Continuous Anomaly Detection)
Real-time Machine Learning
CREATE STREAM anomaly_detection AS
SELECT sensor_id, detect_anomaly(sensor_values)
FROM machine;
TensorFlow model embedded in User Defined
Function (UDF)
20. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Agenda
• Data in Motion at the Edge and in Hybrid Architectures
• Live Demo – Hybrid Data Streaming with Confluent on
Hivecell and Confluent Cloud
• Designing an Edge Solution
• Next steps for Kafka at the Edge
21. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Edge
Cloud
Texas Bavaria
Hybrid Data Streaming: Edge and Cloud
22. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Texas
Texas data
Streams of real time events
Texas data
Streams of real time events
Bavaria data
Bavaria
Hybrid Data Streaming: Edge and Cloud
23. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Agenda
• Data in Motion at the Edge and in Hybrid Architectures
• Live Demo – Hybrid Data Streaming with Confluent on
Hivecell and Confluent Cloud
• Designing an Edge Solution
• Next steps for Kafka at the Edge
24. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Conditions of edge environments
• No data closet
• Or data closet is full, overheating and under powered
• Limited or unreliable power supply
• Limited number of ports
• No IT staff on site
• Hundreds or thousands of locations
• Limited network bandwidth
• Demand for compute power always increasing
25. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Requirements for an edge solution
No special equipment for power, air conditioning or networking
No technicians required onsite to install and configure
Start with smallest footprint possible to demonstrate ROI
Easily add more compute power as workload expands
Run distributed software for containers, messaging and machine learning
Monitor, manage and upgrade centrally, even when behind a firewall
26. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Hivecell
✓ No special equipment
✓ No technicians required
✓ Start with smallest footprint
✓ Easily add more compute power
✓ Run distributed software
✓ Monitor, manage and update remotely
27. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Agenda
• Data in Motion at the Edge and in Hybrid Architectures
• Live Demo – Hybrid Data Streaming with Confluent on
Hivecell and Confluent Cloud
• Designing an Edge Solution
• Next steps for Kafka at the Edge
28. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
The Rise of Data in Motion
for Edge and Hybrid Scenarios
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
29
2017
29. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver 1
New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid,
Multi-Region)
Build Contextual
Event-Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
30
30. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native. Everywhere.
31. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Container orchestration at the
edge
Enterprise streaming at the
edge
HIVECELL = PAAS AT THE EDGE
Hyperconverged
infrastructure at the edge
HCI
32. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
EDGE AS A SERVICE.
EVERYTHING
YOU NEED IS
INCLUDED.
• Kubernetes and Confluent as a service
• Framework licenses/support/updates
• Hive Control - Provision/monitor system
• Operating system (OS) support/updates
• Hardware support/replacements
• Evergreen Hardware: upgrades every 3 years
User just uses Kafka. At the edge.
33. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
34
Hivecell and Confluent enable Kafka at the Edge.
Reliable. Scalable. Simple.
Conclusion
34. Data in Motion Everywhere with Apache Kafka in Hybrid Edge and Cloud Infrastructures
Stay in touch!
35
Confluent Cloud
cnfl.io/confluent-cloud
1 hour free edge consulting
session with Dominik Pilat
https://hivecell.com/contact/