Aljoscha Krettek is the PMC chair of Apache Flink and Apache Beam, and co-founder of data Artisans. Apache Flink is an open-source platform for distributed stream and batch data processing. It allows for stateful computations over data streams in real-time and historically. Flink supports batch and stream processing using APIs like DataSet and DataStream. Data Artisans originated Flink and provides an application platform powered by Flink and Kubernetes for building stateful stream processing applications.

1. Aljoscha Krettek – PMC of Apache Flink and Apache Beam, Co-Founder at data Artisans
Apache Flink® and what it is used for

2. © 2018 data Artisans2
About Data Artisans
Original Creators of
Apache Flink®
RealTime Stream Processing
Enterprise Ready
2008 2009 2011
Studying and working at IBM
Development on Stratosphere begins
I joinTU Berlin research group with other co-founders
2014
data Artisans is founded

3. © 2018 data Artisans3
What is Apache Flink?
Queries
Applications
Devices
etc.
Database
Stream
File / Object
Storage
Historic
Data
Streams
Application
Stateful computations over streams
real-time and historic
fast, scalable, fault tolerant, in-memory,
event time, large state, exactly-once

4. © 2018 data Artisans4
Overview of Flink Use Cases
Event-driven
applications
Stream
Processing
Batch
processing

5. © 2018 data Artisans5
Batch Processing
a.k.a. collect now, figure
out later

6. © 2018 data Artisans6
Batch Processing Technologies Supported by Flink
DataSet
API

7. © 2018 data Artisans7
Observation 1
The origin of data are streams

8. © 2018 data Artisans8
Stream Processing
we can build applications directly
on data streams

9. © 2018 data Artisans9
Stream Processing Technologies Supported by Flink
DataStream
API

10. © 2018 data Artisans10
Observation 2
Stream Processing changes the database-centric
architecture

11. © 2018 data Artisans11
Changing the Two-Tier Architecture
reads/writes across
tier boundary
asynchronous writes
of large blobs
all modifications
are local
Classic tiered architecture Streaming architectureClassic tiered architecture
database
layer
compute
layer
application working state
+ historic state
compute
+
stream storage
and
snapshot storage
(backup)
application state
Streaming architecture

12. © 2018 data Artisans12
Keystone Routing Pipeline at Netflix
(as presented at Flink Forward San Francisco, 2018)
@
 Athena X
 SQL to define metrics
 Thresholds and actions to trigger
 Blends analytics and
actions
Streams from
Hadoop, Kafka, etc
SQL, thresholds,
actions
Analytics
Alerts
Derived streams

13. © 2018 data Artisans13
Observation 3
Stream Processing is about building applications,
not platforms

14. © 2018 data Artisans14
Internal streaming data platforms
built with Apache Flink

15. © 2018 data Artisans15
KubernetesResource Manager
Logging
Metrics
CI / CD
Application Platforms
deploying new
applications
scaling
applications
Kubernetes

16. © 2018 data Artisans16
Kubernetes
Database
Kubernetes
• Example: Scaling down a replicated database
• 3 replicas, 4 node scale down
need to move or
reorganize data
before container
shutdown
Kubernetes & stateful applications What about stateful containers?

17. © 2018 data Artisans17
• consistent stateful upgrades
‒application evolution and bug fixes
• migration of application state
‒cluster migration, A/B testing
• re-processing and reinstatement
‒fix corrupt results, bootstrap new applications
• state evolution (schema evolution)
A B
Stateful Questions
Container-based
Resource Orchestration
Stateful Stream
Processing & Snapshots
Kubernetes Apache Flink
Container-based
platform for stateful
data-driven applications
dA Platform
Code, Resource, Config, and
Snapshot Management
Application
Manager

18. © 2018 data Artisans18
Architecture
Apache Flink
Stateful stream processing
Kubernetes
Container platform
Logging
Metrics
dA Application
Manager
Application
lifecycle
management
Storage
In Action
App
Manager
Kubernetes
Resource
Allocation
Job Control
Snapshot
Management
CI/CDWeb
interface

19. © 2018 data Artisans19
Powered by Apache Flink

20. © 2018 data Artisans20
Free Trial Download of data Artisans Platform
data-artisans.com/download

21. THANK YOU! WE ARE HIRING
data-artisans.com/careers
@aljoscha
@dataArtisans
@ApacheFlink

22. © 2018 data Artisans22
Backup Slides