The world of big data involves an ever changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam (incubating) aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In this talk, I will: Cover briefly the capabilities of the Beam model for data processing and integration with IOs, as well as the current state of the Beam ecosystem. Discuss the benefits Beam provides regarding portability and ease-of-use. Demo the same Beam pipeline running on multiple runners in multiple deployment scenarios (e.g. Apache Flink on Google Cloud, Apache Spark on AWS, Apache Apex on-premise). Give a glimpse at some of the challenges Beam aims to address in the future.

1. 1
Realizing the promise of portability
with Apache Beam
https://s.apache.org/beam-portability-slides-jonthebeach
Tyler Akidau
Senior Staff Software Engineer at Google
Apache Beam PMC
@takidau
With many slides by Frances Perry (@francesjperry)
J On the Beach 2017

2. 2
Apache Beam: Open Source data processing APIs
Expresses data-parallel batch and streaming
algorithms using one unified API
Cleanly separates data processing logic from
runtime requirements
Supports execution on multiple distributed
processing runtime environments

3. 3
The evolution of Apache Beam
MapReduce Apache
Beam
Cloud
Dataflow
BigTable DremelColossus
FlumeMegastore Spanner
PubSub
Millwheel

4. 4
Table of Contents
01
02
03
04
Expressing data-parallel pipelines with the Beam model
The Beam vision for portability
Parallel and portable pipelines in practice
Getting Started with Apache Beam

5. 5
01 Expressing data-parallel pipelines
with the Beam Model
A unified model for batch and streaming

6. 6
Processing time vs. event time

7. 7
The Beam Model: asking the right questions
What results are calculated?
Where in event time are results calculated?
When in processing time are results materialized?
How do refinements of results relate?

8. 8
The Beam Model: What is being computed?
PCollection<KV<String, Integer>> input = IO.read(...)
.apply(ParDo.of(new ParseFn());
.apply(Sum.integersPerKey());

10. 10
The Beam Model: Where in event time?
PCollection<KV<String, Integer>> input = IO.read(...)
.apply(ParDo.of(new ParseFn());
.apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)))
.apply(Sum.integersPerKey());

12. 12
The Beam Model: When in processing time?
PCollection<KV<String, Integer>> input = IO.read(...)
.apply(ParDo.of(new ParseFn());
.apply(Window.into(FixedWindows.of(Duration.standardMinutes(2))
.triggering(AtWatermark())
.apply(Sum.integersPerKey());

14. 14
The Beam Model: How do refinements relate?
PCollection<KV<String, Integer>> input = IO.read(...)
.apply(ParDo.of(new ParseFn());
.apply(Window.into(FixedWindows.of(Duration.standardMinutes(2))
.triggering(AtWatermark()
.withEarlyFirings(
AtPeriod(Duration.standardMinutes(1)))
.withLateFirings(AtCount(1)))
.accumulatingFiredPanes())
.apply(Sum.integersPerKey());

16. 16
Customizing What/Where/When/How
3. Streaming 4. Streaming + Accumulation
1. Classic Batch 2. Windowed Batch

17. 17
02 The Beam vision for portability
“Write once, run anywhere”

18. 18
Beam Vision: mix and match SDKs and runtimes
● The Beam Model: the abstractions
at the core of Apache BeamLanguage A
SDK
Language C
SDK
Runner 1 Runner 3Runner 2
● Choice of SDK: Users write their
pipelines in a language that’s
familiar and integrated with their
other tooling
● Choice of Runners: Users choose
the right runtime for their current
needs -- on-prem / cloud, open
source / not, fully managed / not
● Scalability for Developers: Clean
APIs allow developers to contribute
modules independently
The Beam Model
Language A Language CLanguage B
The Beam Model
Language B
SDK

19. 19
Beam Vision: as of May 2017
First stable release: Beam 2.0.0
Beam’s Java SDK runs on multiple
runtime environments, including:
Apache Apex
Apache Flink
Apache Spark
Google Cloud Dataflow
[in development] Apache Gearpump
Cross-language infrastructure is in
progress.
Beam’s Python SDK currently runs
on Google Cloud Dataflow
Beam Model: Fn Runners
Apache
Spark
Cloud
Dataflow
Beam Model: Pipeline Construction
Apache
Flink
JavaPython
Apache
Apex
Apache
Gearpump
Python Java

20. 20
Example Beam Runners
Apache Spark
● Open-source
cluster-computing
framework
● Large ecosystem of
APIs and tools
● Runs on premise or in
the cloud
Apache Flink
● Open-source
distributed data
processing engine
● High-throughput and
low-latency stream
processing
● Runs on premise or in
the cloud
Google Cloud Dataflow
● Fully-managed service
for batch and stream
data processing
● Provides dynamic
auto-scaling,
monitoring tools, and
tight integration with
Google Cloud
Platform

21. 21
How do you build an abstraction layer?
Apache
Spark
Cloud
Dataflow
Apache
Flink
????????
????????

22. 22
Beam: the intersection of runner functionality?

23. 23
Beam: the union of runner functionality?

24. 24
Beam: the future!

25. 25
Categorizing Runner Capabilities
https://s.apache.org/beam-capability-matrix

26. 26
03 Parallel and portable pipelines
in practice
Demo time

27. 27
Demo!
(sort of)

43. 43
04 Getting started with Apache Beam
Beaming into the future

44. 44
Learn more!
Apache Beam
beam.apache.org
Demo code
github.com/davorbonaci/beam-portability-demo
The World Beyond Batch: Streaming 101 and 102
www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
www.oreilly.com/ideas/the-world-beyond-batch-streaming-102
The DataflowBeam Model paper, VLDB 2015
vldb.org/pvldb/vol8/p1792-Akidau.pdf
Streaming Systems book
www.streamingsystems.net
@takidau on Twitter

45. 45
05 Demo Screenshots
Because if I make them, I won’t need them (famous last words)