More Related Content Similar to Low-latency real-time data processing at giga-scale with Kafka | John DesJardins, Hazelcast (20) More from HostedbyConfluent (20) Low-latency real-time data processing at giga-scale with Kafka | John DesJardins, Hazelcast1. © HAZELCAST | CONFIDENTIAL | 1
© HAZELCAST | CONFIDENTIAL | 1
Low-latency data
processing at Giga-scale
John DesJardins – CTO, Hazelcast
2. © HAZELCAST | CONFIDENTIAL | 2
© HAZELCAST | CONFIDENTIAL | 2
Topics to Cover
✦ Why Latency Matters
✦ Enemies of Performance
✦ Anti-Patterns
✦ Architecture - Low-latency Processing with Kafka and Hazelcast
✦ Benefits
✦ Demonstration
✦ Learn More
3. © HAZELCAST | CONFIDENTIAL | 3
© HAZELCAST | CONFIDENTIAL | 3
Why Does Latency Matter?
Latency Windows
✦ 300 milliseconds – Blink of an Eye
✦ 50 milliseconds – Maximum Latency Window for Credit Card Authorization & Fraud Checks
✦ Other Latency-Critical Moments:
• Time to Respond to Medical Device Alerts in ER
• Time to Respond to Industrial Equipment Failure
• Time Window to Prevent Shopping Cart Abandonment
4. © HAZELCAST | CONFIDENTIAL | 4
© HAZELCAST | CONFIDENTIAL | 4
Enemies of Performance & Scalability
✦ Network Hops
✦ Storage I/O
✦ Processor – “Context Switching”
✦ Excessive or Unnecessary “Staged” Elements to Architecture
✦ Data Enrichment Requiring Added Data Retrieval
5. © HAZELCAST | CONFIDENTIAL | 5
© HAZELCAST | CONFIDENTIAL | 5
Anti-Pattern for Low Latency – Lambda Architecture
Web Server
Web Client
Real-time
Events
Ingest
& Filter
Aggregated
Store
Kafka
✦ Excess Ping-Ponging of Data
✦ Data Stored and Then Aggregated
✦ Data Often Stored Again for Querying
✦ Doesn’t Lend Itself to Ad-Hoc Queries
✦ Operational Data Views Add Latency
✦ Reference Data Not Available for Enrichment
Real-time
Feed
Aggregate
& Store
6. © HAZELCAST | CONFIDENTIAL | 6
© HAZELCAST | CONFIDENTIAL | 6
Anti-Pattern for Low Latency – “Delta” Architecture
✦ Repeated Persistence
✦ Repeated Network Hops
✦ Many Changes to Data
✦ Longest Lag from Data Birth to Value Creation
Ingest
“Raw”
Refine &
Enrich
Cleanse
d
Real-time
Analytics &
Machine
Learning
Data Lake
Batch Data
Data Is
Born
Action
Taken,
Value
Created
7. © HAZELCAST | CONFIDENTIAL | 7
© HAZELCAST | CONFIDENTIAL | 7
Continuous Query with Drilldown
Process Flow
✦ A query kicks off a data ingest job
✦ Data is aggregated and indexed
into the in-memory store
✦ End users can drilldown into
trade details
via the web UI
Business Benefits
✦ Near-real-time visibility
✦ Cost-effective
✦ Reliable, high uptime
✦ Cloud-native for when move to
public cloud is desired
Technical Benefits
✦ No database to maintain, no storage of data copies
✦ Easily extensible architecture
Hazelcast Cluster
Web Server
Web Client
Real-time Events Data Store
Ingest
Continuous
Query 1
(load and agg)
Index
Query
Submit query
Cancel query
Data Source
(e.g., Kafka, Kinesis,
Log Files)
8. © HAZELCAST | CONFIDENTIAL | 8
© HAZELCAST | CONFIDENTIAL | 8
Kafka Topic
Kafka Topic
Kafka Topic
Data Processing
Live Events
Analytics
Live Events
Analytics
Live Events
Analytics
Live Events
Analytics
Web Client
Architecture – Scale Ingest/Query Processing Together
Streaming Ingest Data Store Query
Streaming Ingest Data Store Query
Streaming Ingest Data Store Query
Streaming Ingest Data Store Query
Event Stream Input
9. © HAZELCAST | CONFIDENTIAL | 9
© HAZELCAST | CONFIDENTIAL | 9
Data-in-Motion
“Events”
Streaming Analytics
Data-at-Rest
“Traditional Data”
Databases
UNIFIED
Why Is This Different?
10. © HAZELCAST | CONFIDENTIAL | 10
© HAZELCAST | CONFIDENTIAL | 10
Advantages
✦ Partitioning of Both Compute & Data = Maximizing Parallel & Distributed Architecture
✦ Data-Aware Processing = Data Locality – Less Movement of Data
✦ Collaborative Worksharing = Simplifies Scalability
✦ In-Memory Optimized = Further Drive Down Latency
11. © HAZELCAST | CONFIDENTIAL | 11
© HAZELCAST | CONFIDENTIAL | 11
Summary of Benchmark Stats
▪ 1 BILLION TPS with 99% under 30ms latency
▪ 45 nodes
▪ Linear scaling with predictable latency
12. © HAZELCAST | CONFIDENTIAL | 12
© HAZELCAST | CONFIDENTIAL | 12
Continuous
Business
Events
Pipeline
MQ
Kafka
IoT
Custom
Connector
Enterprise
Applications
Hazelcast
File Watcher
Socket
Database
Events
Sources Sinks
MQ
Kafka
Alerts
Interactive
Analytics
Enterprise
Applications
Hazelcast
Databases
HDFS, S3,
NoSQL
Files
Actionable
Context
Hazelcast Platform
The Hazelcast Platform
Transform Combine Stream ML Inference
Stream and Batch
Processing Engine
Compute
In-Memory
Data Store
Storage
Persistence
13. © HAZELCAST | CONFIDENTIAL | 13
© HAZELCAST | CONFIDENTIAL | 13
From Streaming Events to Data Applications
▪ Modular approach: streaming jobs become
building blocks for rich data applications,
connected by either operational data store or
messaging - Simplifies development, deployment
& scale
Hazelcast turns high-volume streams into data applications
▪ Built-in Operational Store Continuously serves
data to users
▪ Event-driven (push) or request-driven (pull)
access
▪ Indexed for concurrent, low-latency queries
Source Results
Operational
Data
Web Client
Sink
Enrichment Group & Filter
Aggregate
& Analyze
14. © HAZELCAST | CONFIDENTIAL | 14
© HAZELCAST | CONFIDENTIAL | 14
Kafka Topic
Kafka Topic
Kafka Topic
Data Processing
Live Events
Analytics
Live Events
Analytics
Live Events
Analytics
Live Events
Analytics
Web Client
Architecture – Scale Ingest/Query Processing Together
Streaming Ingest Machine Learning
Streaming Ingest Machine Learning
Streaming Ingest Machine Learning
Streaming Ingest Machine Learning
Event Stream Input
15. © HAZELCAST | CONFIDENTIAL | 15
© HAZELCAST | CONFIDENTIAL | 15
Demonstration – Real-
time Trade Processing
16. © HAZELCAST | CONFIDENTIAL | 16
© HAZELCAST | CONFIDENTIAL | 16
Give It a Try or Learn More
✦ Giga-scale Benchmark Details
https://hazelcast.com/blog/billion-events-per-second-with-millisecond-latency-streaming-analytics-at-giga-scale/
✦ Trade Monitoring Solution Info
https://hazelcast.com/use-cases/risk-compliance-trade-monitoring/
Hazelcast Solution Demos Github Repo:
https://github.com/hazelcast/hazelcast-platform-demos
✦ Try Hazelcast 5.0 for free
https://hazelcast.com/get-started/
https://cloud.hazelcast.com/sign-up
✦ Learn More
Docs: https://docs.hazelcast.com/home/index.html
Training: https://training.hazelcast.com/
Blogs: https://hazelcast.com/blog/
17. © HAZELCAST | CONFIDENTIAL | 17
© HAZELCAST | CONFIDENTIAL | 17
Thank You
john.desjardins@hazelcast.com
@johnmdesjardins