Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
© 2019 Ververica
Aljoscha Krettek – Software Engineer, Flink PMC
Stephan Ewen – CTO, Flink PMC/Chair
Towards Flink 2.0: Re...
© 2019 Ververica2
This is joint work with many members of
the Apache Flink community
Timo, Dawid, Shaoxuan, Kurt, Guowei, ...
© 2019 Ververica3
Some of this presents work that is in
progress in the Flink community. Other
things are planned and/or h...
© 2019 Ververica
Batch and Streaming
© 2019 Ververica5
Everything Is a Stream
© 2019 Ververica6
What changes faster? Data or Query?
ad-hoc queries, data exploration,
ML training and
(hyper) parameter ...
© 2019 Ververica7
Latency vs. Completeness (for geeks)
1977 1980 1983 1999 2002 2005 2015
Processing Time
Episode
IV
Episo...
© 2019 Ververica8
Latency vs. Completeness (more formally)
*from the excellent Streaming 101 by Tyler Akidau:
https://www....
© 2019 Ververica9
Latency vs. Completeness
Bounded/
Batch
Unbounded/
Streaming
Data is as complete
as it gets within that
...
© 2019 Ververica10
What does this mean for processing?
older more recent
files,
message
bus, or …
© 2019 Ververica11
Stream-style processing
older more recent
watermark
unprocesse
d
© 2019 Ververica12
Batch-style processing
older more recent
watermark
unprocesse
d
© 2019 Ververica13
Batch and Streaming Processing Styles
S S S
M M M
R R R
S S S
S S S
M M M
R R R
S S S
more batch-y more...
© 2019 Ververica14
So this is the reason why we have
different APIs, different batch and
stream processing systems?
• Diff...
© 2019 Ververica
Current Stack
© 2019 Ververica16
Runtime
Tasks / JobGraph / Network Interconnect / Fault Tolerance
DataSet
“Operators“ and Drivers / Ope...
© 2019 Ververica17
What could be improved?
• Each API has its own internal graph representation  code duplication
• Multi...
© 2019 Ververica18
What does this mean for users?
• You have to decide between DataSet and DataStream when writing a job
–...
© 2019 Ververica
Future Stack
© 2019 Ververica20
DataSet
“Operators“ and Drivers / Operator
Graph / Visitors
DataStream
StreamOperator /
StreamTransform...
© 2019 Ververica21
Unifying the Batch and Streaming APIs
• DataStream API functionality is already a superset of DataSet A...
© 2019 Ververica22
A typical “unified” use case: Bootstrapping state
* See for example Bootstrapping State in Flink by Gre...
© 2019 Ververica23
Under-the-hood Changes
• StreamTransformation/StreamGraph
need to be beefed up to carry the
additional ...
© 2019 Ververica24
Selective Push Model Operator FLINK-11875
batch: pull-based operator (or Driver) streaming: push-based ...
© 2019 Ververica25
Current Source Interfaces
InputFormat
createInputSplits(): splits
openSplit(split)
assignInputSplit()
n...
© 2019 Ververica26
Batch InputFormat Processing
TaskManager TaskManager TaskManager
JobManager
(1) request split
(2) send ...
© 2019 Ververica27
Stream SourceFunction Processing
• Source have a run-loop that they manage completely on their own
• So...
© 2019 Ververica28
A New (unified) Source Interface
• This must support both batch and streaming use cases, allow Flink to...
© 2019 Ververica29
A New (unified) SourceInterface: Execution Style I
TaskManager TaskManager TaskManager
JobManager
(1) r...
© 2019 Ververica30
A New (unified) SourceInterface: Execution Style II
© 2019 Ververica31
Table API / SQL
• API: easy, it’s already unified ✅
• Translation and runtime (operators) need to be en...
© 2019 Ververica32
StreamTransformation DAG / StreamOperator
DataStream
“Physical“ Application API
Table API / SQL
Declara...
© 2019 Ververica
Closing
© 2019 Ververica34
Summary
• Semantics of unified processing are quite clear already
• For some things, work is ongoing an...
© 2019 Ververica
www.ververica.com @VervericaDataaljoscha@ververica.com
stephan@ververica.com
© 2019 Ververica
Thank you!
Questions?
Próxima SlideShare
Cargando en…5
×

Flink Forward San Francisco 2019: Towards Flink 2.0: Rethinking the stack and APIs to unify Batch & Stream - Stephan Ewen & Aljoscha Krettek

201 visualizaciones

Publicado el

Flink currently features different APIs for bounded/batch (DataSet) and streaming (DataStream) programs. And while the DataStream API can handle batch use cases, it is much less efficient in that compared to the DataSet API. The Table API was built as a unified API on top of both, to cover batch and streaming with the same API, and under the hood delegate to either DataSet or DataStream.

In this talk, we present the latest on the Flink community's efforts to rework the APIs and the stack for better unified batch & streaming experience. We will discuss:

- The future roles and interplay of DataSet, DataStream, and Table API

- The new Flink stack and the abstractions on which these APIs will build

- The new unified batch/streaming sources

- How batch and streaming optimizations differ in the runtime, and what the future interplay of batch and streaming execution could look like

Publicado en: Tecnología
  • Sé el primero en comentar

Flink Forward San Francisco 2019: Towards Flink 2.0: Rethinking the stack and APIs to unify Batch & Stream - Stephan Ewen & Aljoscha Krettek

  1. 1. © 2019 Ververica Aljoscha Krettek – Software Engineer, Flink PMC Stephan Ewen – CTO, Flink PMC/Chair Towards Flink 2.0: Rethinking the Stack and APIs to unify Batch & Streaming
  2. 2. © 2019 Ververica2 This is joint work with many members of the Apache Flink community Timo, Dawid, Shaoxuan, Kurt, Guowei, Becket, Jincheng, Fabian, Till, Andrey, Gary, Piotr, Stefan, etc. And many others …
  3. 3. © 2019 Ververica3 Some of this presents work that is in progress in the Flink community. Other things are planned and/or have design documents. Some were discussed at one point or another on the mailing lists or in person. This represents our understaning of the current state, this is not a fixed roadmap, Flink is an open-source Apache project.
  4. 4. © 2019 Ververica Batch and Streaming
  5. 5. © 2019 Ververica5 Everything Is a Stream
  6. 6. © 2019 Ververica6 What changes faster? Data or Query? ad-hoc queries, data exploration, ML training and (hyper) parameter tuning continuous applications, data pipelines, standing queries, anomaly detection, ML evaluation, … Data changes slowly compared to fast changing queries Data changes fast application logic is long-lived “batch” “streaming”
  7. 7. © 2019 Ververica7 Latency vs. Completeness (for geeks) 1977 1980 1983 1999 2002 2005 2015 Processing Time Episode IV Episode V Episode VI Episode I Episode II Episode III Episode VII Event Time 2016 Rogue One III.5 2017 Episode VIII
  8. 8. © 2019 Ververica8 Latency vs. Completeness (more formally) *from the excellent Streaming 101 by Tyler Akidau: https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-
  9. 9. © 2019 Ververica9 Latency vs. Completeness Bounded/ Batch Unbounded/ Streaming Data is as complete as it gets within that Batch Job No fine latency control Trade of latency versus completeness Processing-time timers for latency control
  10. 10. © 2019 Ververica10 What does this mean for processing? older more recent files, message bus, or …
  11. 11. © 2019 Ververica11 Stream-style processing older more recent watermark unprocesse d
  12. 12. © 2019 Ververica12 Batch-style processing older more recent watermark unprocesse d
  13. 13. © 2019 Ververica13 Batch and Streaming Processing Styles S S S M M M R R R S S S S S S M M M R R R S S S more batch-y more stream-y running not running can do things one-by-one everything is always-on running
  14. 14. © 2019 Ververica14 So this is the reason why we have different APIs, different batch and stream processing systems? • Different requirements • Optimization potential for batch and streaming • Also: historic developments and slow- changing organizations 💡
  15. 15. © 2019 Ververica Current Stack
  16. 16. © 2019 Ververica16 Runtime Tasks / JobGraph / Network Interconnect / Fault Tolerance DataSet “Operators“ and Drivers / Operator Graph / Visitors DataStream StreamOperators / StreamTransformation Graph* / Monolithic Translators Table API / SQL Logical Nodes* / Different translation paths
  17. 17. © 2019 Ververica17 What could be improved? • Each API has its own internal graph representation  code duplication • Multiple translation components between the different graphs  code duplication – DataStream API has an intermediate graph structure: StreamTransformation  StreamGraph  JobGraph • Separate (incompatible) operator implementations – DataStream API has StreamOperator, DataSet API has Drivers  two map operators, two flatMap operators – These are run by different lower-level Tasks – DataSet operators are optimized for different requirements than DataSet operators • Table API is translated to two distinct lower-level APIs  two different translation stacks – ”project operator” for DataStream and for DataSet • Connectors for each API are separate  a whole bunch of connectors all over the From a system design/code quality/architecture/development perspective
  18. 18. © 2019 Ververica18 What does this mean for users? • You have to decide between DataSet and DataStream when writing a job – Two (slightly) different APIs, with different capabilities – Different set of supported connectors: no Kafka DataSet connector, no HBase DataStream connector – Different performance characteristics – Different fault-tolerance behavior – Different scheduling logic • With Table API, you only have to learn one API – Still, the set of supported connectors depends on the underlying execution API – Feature set depends on whether there is an implementation for your underlying API • You cannot combine more batch-y with more stream-y sources/sinks • A “soft problem”: with two stacks of everything, less developer power will go into each one individual stack  less features, worse performance, more bugs that are fixed slower
  19. 19. © 2019 Ververica Future Stack
  20. 20. © 2019 Ververica20 DataSet “Operators“ and Drivers / Operator Graph / Visitors DataStream StreamOperator / StreamTransformation Graph* / Monolithic Translators Batch is a subset of streaming! Can’t we just? ✅❌ Done! 🥳
  21. 21. © 2019 Ververica21 Unifying the Batch and Streaming APIs • DataStream API functionality is already a superset of DataSet API functionality • We need to introduce BoundedStream to harness optimization potential, semantics are clear from earlier: –No processing-time timers –Watermark “jumps” from –Infinity to +Infinity at end of processing • DataStream translation and runtime (operators) need to be enhanced to use the added optimization potential • Streaming execution is the generic case that always works, “batch” enables additional “optimization rules”: bounded operators, different scheduling  we get feature parity automatically ✅ • Sources need to be unified as well  see later
  22. 22. © 2019 Ververica22 A typical “unified” use case: Bootstrapping state * See for example Bootstrapping State in Flink by Gregory Fee https://sf-2018.flink- “stream ” source “batch” source Stateful operatio n batch-y partstream-y part • We have a streaming use case • We want to bootstrap the state of some operations from a historical source • First execute bounded parts of the graph, then start the rest
  23. 23. © 2019 Ververica23 Under-the-hood Changes • StreamTransformation/StreamGraph need to be beefed up to carry the additional information about boundedness • Translation, scheduling, deployment, memory management and network stack needs to take this into account Graph Representation / DAG Operator / Task • StreamOperator needs to support batch-style execution  see next slide • Network stack must eventually support blocking inputs
  24. 24. © 2019 Ververica24 Selective Push Model Operator FLINK-11875 batch: pull-based operator (or Driver) streaming: push-based StreamOperator • StreamOperator needs additional API to tell the runtime which input to consume • Network stack/graph needs to be enhanced to deal with blocking inputs, 😱
  25. 25. © 2019 Ververica25 Current Source Interfaces InputFormat createInputSplits(): splits openSplit(split) assignInputSplit() nextRecord(): T closeCurrentSplit() SourceFunction run(OutputContext) close() batch streaming
  26. 26. © 2019 Ververica26 Batch InputFormat Processing TaskManager TaskManager TaskManager JobManager (1) request split (2) send split (3) process split • Splits are assigned to TaskManagers by the JobManager, which runs a copy of the InputFormat  Flink knows about splits and can be clever about scheduling, be reactive • Splits can be processed in arbitrary order • Split processing pulls records from the InputFormat • InputFormat knows nothing about watermarks, timestamps, checkpointing  bad for streaming
  27. 27. © 2019 Ververica27 Stream SourceFunction Processing • Source have a run-loop that they manage completely on their own • Sources have flexibility and can efficiently work with the source system: batch accesses, dealing with multiple topics from one consumer, threading model, etc… • Flink does not know what’s going on inside and can’t be clever about it • Sources have to implement their own per-partition watermarking, idleness tracking, what have you TaskManagerTaskManager TaskManager (1) do your thing
  28. 28. © 2019 Ververica28 A New (unified) Source Interface • This must support both batch and streaming use cases, allow Flink to be clever, be able to deal with event-time, watermarks, source idiosyncrasies, and enable snapshotting • This should enable new features: generic idleness detection, event-time alignment FLIP-27 Source createSplitEnumerator() createSplitReader SplitEnumerator discoverNewSplits() nextSplit() snapshotState() isDone() SplitReader addSplit() hasAvailable(): Future snapshotState() emitNext(Context): Status * FLINK-10886: Event-time alignment for sources; Jamie Grier (Lyft) contributed the first parts of this
  29. 29. © 2019 Ververica29 A New (unified) SourceInterface: Execution Style I TaskManager TaskManager TaskManager JobManager (1) request split (2) send split (3) process split • Splits are assigned to TaskManagers by the JobManager, which runs a copy of the SplitEnumerator  Flink knows about splits and can be clever about scheduling, be reactive • Splits can be processed in arbitrary order • Split processing is driven by the TaskManager working with SplitReader • SplitReader emits watermarks but Flink deals with idleness, per-split watermarking
  30. 30. © 2019 Ververica30 A New (unified) SourceInterface: Execution Style II
  31. 31. © 2019 Ververica31 Table API / SQL • API: easy, it’s already unified ✅ • Translation and runtime (operators) need to be enhanced to use the added optimization potential but use the StreamOperator for both batch and streaming style execution • Streaming execution is the generic case that always works, “batch” enables additional “optimization rules”: bounded operators, different scheduling  we get feature parity automatically ✅ • Sources will be unified from the unified source interface ✅ • This is already available in the Blink fork (by Alibaba), FLINK-11439 is the effort of getting that into Flink
  32. 32. © 2019 Ververica32 StreamTransformation DAG / StreamOperator DataStream “Physical“ Application API Table API / SQL Declarative API Runtime The Future Stack
  33. 33. © 2019 Ververica Closing
  34. 34. © 2019 Ververica34 Summary • Semantics of unified processing are quite clear already • For some things, work is ongoing and there are design documents (FLIPs) • Some things are farther in the future • This project requires changes in all components/layers of Flink: –API, deployment, network stack, scheduling, fault tolerance • You can follow all of this on the public mailing lists and FLIPs! • The Flink tech stack is going to be quite nice! 😎
  35. 35. © 2019 Ververica www.ververica.com @VervericaDataaljoscha@ververica.com stephan@ververica.com
  36. 36. © 2019 Ververica Thank you! Questions?

×