Stream processing still evolves and changes at a speed that can make it hard to keep up with the developments. Being at the forefront of stream processing technology, the evolution of Apache Flink has mirrored many of these developments and continues to do so. We will take you on a journey through the major milestones of stream processing technology in past years, diving into the latest additions that Apache Flink and other communities introduced to the stream processing landscape, such as Streamng SQL, Time Versioned Tables, cluster-library-duality, language portability, etc. We will take a sneak peek into our crystal ball and present in what the Flink community is working on next.

1. Aljoscha Krettek, engineering manager & co-founder
Till Rohrmann, engineering manager & co-founder
The Past, Present, and
Future of Apache Flink®

2. © 2018 data Artisans2
Past

3. © 2018 data Artisans3
It all started in 2014
2009 - 2014 since 2014
● Batch processor on top of streaming runtime
● First Apache Flink 0.6.0 release August 2014

4. © 2018 data Artisans4
August 2014
Batch processing

5. © 2018 data Artisans5
Flink learns to stream in real time
DataStream API
Stream Processing
DataSet API
Batch Processing
Runtime
Distributed Streaming Data Flow

6. © 2018 data Artisans6
● Continuous & real-time
November 2014
Batch processing Stream processing

7. © 2018 data Artisans7
Flink learns to remember

8. © 2018 data Artisans8
Flink learns to remember

9. © 2018 data Artisans9
Flink learns to remember

10. © 2018 data Artisans10
Flink learns to remember
Remember where we left off

11. © 2018 data Artisans11
● Continuous & real-time
● Stateful & exactly once
June 2015
Batch processing Stream processing

12. © 2018 data Artisans12
Latency vs. Throughput?
high latency
low high throughput
Prevailing
belief
● 10s of millions of events/s
● Latency down to 1 ms
≠

13. © 2018 data Artisans13
Flink becomes event-time aware
Episode
IV
Episode
V
Episode
VI
Episode
I
Episode
II
Episode
III
Episode
VII
Episode
VIII
Processing
time
1977 1980 1983 1999 2002 2005 2015 2017
Flink becomes event-time aware

14. © 2018 data Artisans14
Flink becomes event-time aware
Episode
I
Episode
II
Episode
III
Episode
IV
Episode
V
Episode
VI
Episode
VII
Episode
VIII
Processing time
1999 2002 2005 1977 1980 1983 2015 2017
Processing time
Event time

15. © 2018 data Artisans15
● Continuous & real-time
● Stateful & exactly once
● High throughput & low
latency
● Event time
November 2015
Batch processing Stream processing

16. © 2018 data Artisans16
More than just analytics: ProcessFunction
class MyFunction extends ProcessFunction[MyEvent, Result] {
// declare state to use in the program
lazy val state: ValueState[CountWithTimestamp] = getRuntimeContext().getState(…)
def processElement(event: MyEvent, ctx: Context, out: Collector[Result]): Unit = {
// work with event and state and schedule timers
}
def onTimer(timestamp: Long, ctx: OnTimerContext, out: Collector[Result]): Unit = {
// handle callback when event-/processing- time instant is reached
}
}
● ProcessFunction gives access to state, time
and events
● Low level API
● Enables data-driven applications

17. © 2018 data Artisans17
● Continuous & real-time
● Stateful & exactly once
● High throughput & low
latency
● Event time
February 2017
Batch processing Stream processing
Data-driven
applications

18. © 2018 data Artisans18
Present & Future

19. © 2018 data Artisans19
Hardening
Faster network stack
Application level flow control
Resolving dependency hell
Present in a nutshell
Scaling
Incremental snapshots
Local recovery
Scalable timers
Interoperability
Resource elasticity
REST client-server interface
Container entrypoint
Stream SQL
SQL client
User-defined functions
More powerful joins
Misc
State TTL
Broadcast state
Kafka exactly-once producer

20. © 2018 data Artisans20
Large, larger, Flink
Time
State
Incremental
snapshots
● Snapshot only state diff
● Incremental snapshots allow to handle very large state

21. © 2018 data Artisans21
Faster failover is always better

22. © 2018 data Artisans22
Varying workloads
• Violating SLAs vs. wasting money
• Varying workloads require to adapt resources

23. © 2018 data Artisans23
Revamped distributed architecture
ResourceManager ClusterManager
TaskManagerJobManager
Dispatcher
Client
1. Submit job 2. Start job
3. Request slots
4. Allocate resources
5. Start TaskManager
6. Execute job
● Support for full resource elasticity
● Application parallelism can be dynamically changed

24. © 2018 data Artisans24
● Continuous & real-time
● Stateful & exactly once
● High throughput & low
latency
● Event time
● Applications as first
class citizens
Present & Future
Batch processing Stream processing
Data-driven
applications

25. © 2018 data Artisans25
Flink as a library (and still as a framework)
• Deploying Flink applications should be as easy as starting a process
• Bundle application code and Flink into a single image
• Process connects to other application processes and figures out its role
• Removing the cluster out of the equation
P2 P3
P1
P4
New process

26. © 2018 data Artisans26
How much control do I need?
Batch
processing
Continuous
processing
Real-time &
data-driven
applications
● Multiple short lived stages
● Different resource requirements
per stage
● Efficient execution requires
control over resources
● Flink allocates actively resources
● Continuously processing operators
● Constrained by external systems,
SLAs and application logic
● External system can assign
resources
● Flink reacts to available resources

27. © 2018 data Artisans27
Active vs. reactive mode
• Active mode
‒ Flink is aware of underlying cluster framework
‒ Flink allocate resources
‒ E.g. existing YARN and Mesos integration
• Reactive mode
‒ Flink is oblivious to its runtime environment
‒ External system allocates and releases resources
‒ Flink scales with respect to available resources
‒ Relevant for environments: Kubernetes, Docker, as a
library

28. © 2018 data Artisans28
Scaling automatically
• Latency
• Throughput
• Resource utilization
• Connector signals

29. © 2018 data Artisans29
How we create Flink Jobs
Flink APIs
Stream/Batch Processing
Runtime
Distributed Streaming Data Flow
Java/Scala

30. © 2018 data Artisans30
Flink SQL
Flink APIs
Stream/Batch Processing
Runtime
Distributed Streaming Data Flow
Java/Scala SQL
“NO CODING REQUIRED”
Source/Sink
definition in YAML
Configuration in
YAML
SQL commandline
User-defined
functions
Streaming and
Batch
Event time and
processing time
*since Flink 0.9.0 (June 2015)

31. © 2018 data Artisans31

32. © 2018 data Artisans32
“Join” me for some trading
buy buy sell buy
Join
$ 17
£ 42
12.5
buy sell
₪

33. © 2018 data Artisans33
Introducing Time-versioned Table Joins
buy buy sell buy
Join
buy sell
curr rate time
£ 42 3
£ 12 17
1453
31753
14
event time

34. © 2018 data Artisans34
SQL for pattern analysis?
SELECT * from ?

35. © 2018 data Artisans35
Introducing MATCH_RECOGNIZE
SELECT *
FROM TaxiRides
MATCH_RECOGNIZE (
PARTITION BY driverId
ORDER BY rideTime
MEASURES
S.rideId as sRideId
AFTER MATCH SKIP PAST LAST ROW
PATTERN (S M{2,} E)
DEFINE
S AS S.isStart = true,
M AS M.rideId <> S.rideId,
E AS E.isStart = false
AND E.rideId = S.rideId
)

36. © 2018 data Artisans36
Todays processing landscape
Streaming Batch

37. © 2018 data Artisans37
Batch/streaming unification

38. © 2018 data Artisans38
Into the Future
Big state
SQL

39. © 2018 data Artisans39
Thank
s!