Apache Kafka and Apache Flink together are a winning stack for data analytics that is used by many companies across industries. The two projects complement each other perfectly: Kafka offers a world-class log for event stream storage and transport, while Flink is a powerful system for analytics and applications on top of those event streams. This talk will demonstrate how to use Kafka and Flink together for "unified analytics": Analytics that seamlessly combine processing of real-time data and historic data. Using SQL as the language for our sample applications, we will walk though various scenarios for unified analytics, such as - Running the same query for processing real-time data from Kafka and for batch-accelerated processing of the historic data stored in Kafka. - Writing queries that combine data in Kafka with tables in external systems (like S3) - Switching between streams of historic data (from S3) and real-time streams in Kafka. The audience will learn how combining real-time and historic data is becoming convenient with the combination of Kafka and Flink.

1. © 2019 Ververica
Stephan Ewen
CTO @ Ververica, Apache Flink PMC
A Unified Analytics Platform with Apache Flink and
Apache Kafka
a.k.a. Unified Streaming & Batch Processing

2. © 2019 Ververica
2
Apache Kafka and Apache Flink

3. © 2019 Ververica
3
Apache Kafka and Apache Flink
Log Storage File Storage
DB/Table
s
JDB
C
Debezium
Analytics and Applications on
Streaming Data &
Data-at-Rest

4. © 2019 Ververica
4
Flink Runtime
Stateful Stream- and Batch Processing
DataStream API StateFun
SQL & Table API
more declarative
more explicit
control

5. © 2019 Ververica
5
How big can you go? – Alibaba Singles Day
Search Rec. Security
BI
Ads
incl. sub-second updates to the GMV dashboard
Real-time Data Applications
Infrastructure
>5K
nodes
Data Size
985PB
Throughput (Peak)
2.5B
events/sec
Latency
Sub-sec
State Size (Biggest)
100TB
>500K
CPU cores

6. © 2019 Ververica
6
How small can you go? - U-Hopper FogGuru
Cluster of 5 Raspberry Pi 3b+ Data volume: 800 events/sec
Docker Swarm + Flink + Mosquitto
“The Fridge”

7. © 2019 Ververica
Streaming & Batch SQL
= SQL!

8. © 2019 Ververica
8
Our Sample Setup
Real-time
Queries
real-time
events
historical data
ingest
Historical
Queries
Combined Queries

9. © 2019 Ververica
9
SQL – Static Data (Batch Case)
user cnt
Mary 2
Bob 1
Liz 1
SELECT
user,
COUNT(url) as cnt
FROM clicks
GROUP BY user
user cTime url
Mary 12:00:00 https://…
Bob 12:00:00 https://…
Mary 12:00:02 https://…
Liz 12:00:03 https://…

10. © 2019 Ververica
10
SQL – Static Data (Streaming Case)
user cTime url
user cnt
SELECT
user,
COUNT(url) as cnt
FROM clicks
GROUP BY user
Mary 12:00:00 https://…
Bob 12:00:00 https://…
Mary 12:00:02 https://…
Liz 12:00:03 https://…
Bob 1
Liz 1
Mary 1
Mary 2

11. © 2019 Ververica
11
Demo Time

12. © 2019 Ververica
Stream to Batches and Back Again

13. © 2019 Ververica
13
ye olde stuff new events
2020-4-1
12:00 am
2016-4-1
1:00 am
2016-4-1
2:00 am
2016-4-7
11:00pm
2016-4-7
10:00pm
…
Inges
t

14. © 2019 Ververica
14
ye olde stuff new events
2020-4-1
12:00 am
2016-4-1
1:00 am
2016-4-1
2:00 am
2016-4-7
11:00pm
2016-4-7
10:00pm
…
Real-tim
e
Queries
Historical
Queries
Inges
t

15. © 2019 Ververica
15
Why combine data in Kafka and in Object Stores?
•Why not just longer Kafka retention?
•Sample Numbers from a petabyte-scale Flink + Kafka user
– Compressed columnar data avg. 5x smaller
– Object Store 3-4x cost cheaper than persistent volumes (EBS)
– Object Store already replicated (cross AZ), replaces broker replication (3x)
🡪 ~50x cheaper data storage
•Faster access to compressed columnar data for
– Higher read parallelism
– Read optimizations: Predicate pushdowns, projection pushdowns, partition pruning, etc.

16. © 2019 Ververica
16
ye olde stuff new events
2020-4-1
12:00 am
2016-4-1
1:00 am
2016-4-1
2:00 am
2016-4-7
11:00pm
2016-4-7
10:00pm
…
Inges
t
while we are at it…
…normalize our data on the way
and convert currencies.

17. © 2019 Ververica
17
Temporal Join
Symbol Exchange Rate Timestamp Order Timestamp Currency
EUR 1.0 12.00 A @ 17.00 $ 12:05 USD
USD 0.88 12:01 B @ 12.32 £ 12:11 GBP
USD 0.90 12:03
GBP 1.17 12:04
USD 0.89 12:12
GBP 1.20 12:13
USD 0.87 12:17
C @ 111.51 $ 12:01 USD
D @ 2.39 $ 12:02 USD
E @ 17.11 $ 12:20 USD
F @ 243.50 £ 12:15 GBP
G @ 3.49 $ 12:10 USD
H @ 0.99 £ 12:16 GBP

18. © 2019 Ververica
18
Temporal Join
Symbol Exchange Rate Timestamp Order Timestamp Currency
EUR 1.0 12.00 A @ 17.00 $ 12:05 USD
B @ 12.32 £ 12:11 GBP
GBP 1.17 12:04
USD 0.90 12:03 C @ 111.51 $ 12:01 USD
D @ 2.39 $ 12:02 USD
E @ 17.11 $ 12:20 USD
F @ 243.50 £ 12:15 GBP
G @ 3.49 $ 12:07 USD
H @ 0.99 £ 12:16 GBP
@ 12:07h
Symbol Exchange Rate Timestamp
EUR 1.0 12.00
GBP 1.20 12:13
USD 0.89 12:12
@ 12:15h

19. © 2019 Ververica
19
Demo Time

20. © 2019 Ververica
Connecting the Streams

21. © 2019 Ververica
21
ye olde stuff new events
2020-4-1
12:00 am
2016-4-1
1:00 am
2016-4-1
2:00 am
2016-4-7
11:00pm
2016-4-7
10:00pm
…
Bootstrap Streaming Queries
Inges
t

22. © 2019 Ververica
22
Demo Time

23. © 2019 Ververica
23
Event Time in
Kafka Topic
Event time
Data (orders / currency rates)
Temporal Join
Time bucketing
on file system
Time partition pruning
for queries
Stitching together
Stream again from
files and stream tail

24. © 2019 Ververica
EOF

25. © 2019 Ververica
25
Apache Kafka and Apache Flink

26. © 2019 Ververica
26
Everything is a Stream
Bounded & Unbounded
Batch Processing complements
Stream Processing
(Compressed Columnar) File
Storage complements Log Storage
make the most of your
(event) time
SQL generalizes well
across batch & streaming
a unified engine makes
things easier