David Yan offers an overview of Apache Apex, a stream processing engine used in production by several large companies for real-time data analytics. Apache Apex uses a programming paradigm based on a directed acyclic graph (DAG). Each node in the DAG represents an operator, which can be data input, data output, or data transformation. Each directed edge in the DAG represents a stream, which is the flow of data from one operator to another. As part of Apex, the Malhar library provides a suite of connector operators so that Apex applications can read from or write to various data sources. It also includes utility operators that are commonly used in streaming applications, such as parsers, deduplicators and join, and generic building blocks that facilitate scalable state management and checkpointing. In addition to processing based on ingression time and processing time, Apex supports event-time windows and session windows. It also supports windowing, watermarks, allowed lateness, accumulation mode, triggering, and retraction detailed by Apache Beam as well as feedback loops in the DAG for iterative processing and at-least-once and “end-to-end” exactly-once processing guarantees. Apex provides various ways to fine-tune applications, such as operator partitioning, locality, and affinity. Apex is integrated with several open source projects, including Apache Beam, Apache Samoa (distributed machine learning), and Apache Calcite (SQL-based application specification). Users can choose Apex as the backend engine when running their application model based on these projects. David explains how to develop fault-tolerant streaming applications with low latency and high throughput using Apex, presenting the programming model with examples and demonstrating how custom business logic can be integrated using both the declarative high-level API and the compositional DAG-level API.

1. Developing Streaming Applications with
Apache Apex
David Yan <davidyan@apache.org>
PMC Member, Apache Apex
Software Engineer, Google
Strata+Hadoop World, San Jose, CA
March 16, 2017

2. Agenda
● Technical Overview of Apache Apex as a streaming
platform
● Developing an Application Pipeline
● Developing an Operator, and the Operator Library
● Event Time Windowing
● Debugging, Scaling and Tuning
● Q & A

3. What is Apache Apex? Why?
● A platform for streaming applications
● Written in Java
● Uses YARN for resource management and HDFS for storage
● Has a comprehensive operator library.
● Scalable, fault tolerant, high throughput and low latency
● Provides low-level control for performance tuning
● Users include GE (Predix), Capital One, Royal Bank of Canada,
Pubmatic, SilverSpring Network, etc.
○ (more at https://apex.apache.org/powered-by-apex.html)

4. Technical
Overview

5. Application Development Model
● Directed Acyclic Graph (DAG)
● A stream is a sequence of data tuples
● An operator takes one or more input streams, performs computations, emits
one or more output streams
Filtered
Stream
Output
Stream
Tuple Tuple
FilteredStream
Enriched
Stream
Enriched
Stream
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator
er
Operator

6. From Logical DAG to Actual Work
● CLI issues a launch command with
App Package
● YARN starts up Application Master
● Logical DAG is translated to
Physical DAG
Apex CLI
YARN
RM
NM NM NM NM
AM
1 2
3
4
5 6Input Filter Transform Output

7. From Logical DAG to Actual Work
(cont'd)
● AM requests worker containers
from YARN to run physical
operators
● Worker Containers send data
using a pub-sub mechanism
Apex CLI
YARN
RM
AM Worker WorkerWorker
Worker
Worker
6
4
1
3
25
1 2
3
4
5 6
HDFS
PubSub
overflow &
Checkpoints

8. Streaming Windows & Checkpointing
● Stream is divided into fixed time slices
called streaming windows
● Checkpoint is performed by Worker
Containers at streaming window
boundaries
● Worker Containers send heartbeats to
AM
● Recovery is incremental without resetting
full DAG
● Checkpoints are purged after the
corresponding window is committed
● AM is also checkpointed
BeginWindow nEndWindow n
BeginWindow n+1
EndWindow n+1
Time
......
Bookkeeping &
Checkpointing done
here

9. Writing
an Apex Application
Pipeline

10. Writing an Apex Application Pipeline
● Creating a project: Maven archetype
● Two APIs to specify an Apex pipeline
○ Compositional
○ Declarative

11. Compositional API
Input Parser Counter Output
CountsWordsLines
Kafka Database
Filter
Filtered

12. Declarative API
StreamFactory.fromKafka09(brokers, topic)
.flatMap(input -> Arrays.asList(input.split("s+")))
.filter(input -> filterWords.contains(input))
.window(new WindowOption.GlobalWindow(),
new TriggerOption().accumulatingFiredPanes().withEarlyFiringsAtEvery(1))
.sumByKey(input -> new Tuple.PlainTuple<>(new KeyValPair<>(input, 1L)))
.map(input -> input.getValue())
.endWith(new JdbcOutput())
.populateDag(dag);
Input Parser Counter Output
CountsWordsLines
Kafka Database
Filter
Filtered

13. Operators

14. Operator API
Next
streaming
window
Next
streaming
window
Input Adapters - Starting of the pipeline. Interacts with external system to generate stream
Generic Operators - Processing part of pipeline
Output Adapters - Last operator in pipeline. Interacts with external system to finalize the processed stream
OutputPort::emit()

15. Operator Example (Stateless)

16. Operator Example (Stateful)

17. RDBMS
• JDBC
• MySQL
• Oracle
• MemSQL
NoSQL
• Cassandra, HBase
• Aerospike, Accumulo
• Couchbase/ CouchDB
• Redis, MongoDB
• Geode
Messaging
• Kafka
• JMS (ActiveMQ etc.)
• Kinesis, SQS
• Flume, NiFi
• MQTT
File Systems
• HDFS/ Hive
• Local File
• S3
Parsers
• XML
• JSON
• CSV
• Avro
• Parquet
Transformations
• Filters, Expression, Enrich
• Windowing, Aggregation
• Join
• Dedup
Analytics
• Dimensional Aggregations
(with state management for
historical data + query)
Protocols
• HTTP
• FTP
• WebSocket
• SMTP
Other
• Elastic Search
• Script (JavaScript, Python, R)
• Solr
• Twitter
Apex "Malhar" Operator Library

18. Queryable State
A set of operators in the library that supports real-time queries and updates of
operator state.
HashTag
Extractor
Top Counter
Twitter Feed
Input Operator
Unique
Counter
Query
Result
PubSub
Broker
Query
Input

19. Queryable State (cont'd)

20. Event Time
Windowing

21. What, Why, and How?
● Event Time Windows not to be confused with "streaming windows"
● Time of the event vs processing time and ingression time
● Data often arrives late and out of order
● Concepts outlined by Google's Millwheel and Apache Beam: Watermarks,
Allowed Lateness, Accumulation Modes, and Triggers

22. Example
StreamFactory.fromFolder("/tmp")
.flatMap(input -> Arrays.asList(input.split( "s+")))
.map(input -> new TimestampedTuple<>(System.currentTimeMillis(), input))
.window(new TimeWindows(Duration.standardMinutes(5)),
TriggerOption.AtWatermark()
.accumulatingFiredPanes()
.withEarlyFiringsAtEvery(Duration.standardSeconds(1)),
Duration.standardSeconds(15))
.sumByKey(input -> new TimestampedTuple<>(input.getTimestamp(), new KeyValPair<>(input.getValue(),
1L ))))
.map(new FormatAsTableRowFn()) // format for printing
.print()
.populateDag(dag);

23. Debugging, Scaling
& Tuning Your
Application

24. Debugging
● Logs
● Local Mode
● Remote JVM Attachment
● Tuple Recording

25. Scaling: Partitioning
Partitioning with Unifiers: NxM Partitioning:
0 1 2
0
1
a
1
b
1
c
U 2
Logical DAG
Physical DAG with operator 1 with 3 partitions
0 21 3
0
2
a
1
b 3
1
a
1
c
2
b
U U
0
2
a
1
b 3
1
a
1
c
2
b
U1
U
U2

26. Scaling: Partitioning (cont'd)
Parallel Partition: Cascading Unifiers:
0
1
a
1
b
U 2 3 4
0
1
a
1
b
U 4
2
a
3
a
2
b
3
b
1 2
1
1
1
1
U 2
1
1
1
1
U
1
2
U
2
U
3

27. Dynamic Partitioning
● Partitioning change while
application is running
○ Change number of partitions at
runtime based on stats
○ Supports re-distribution of state when
number of partitions change
○ API for custom scaler or partitioner
StatelessThroughputBasedPartitioner<MyOperator>
partitioner
= new StatelessThroughputBasedPartitioner<>();
partitioner.setCooldownMillis(45000);
partitioner.setMaximumEvents(30000);
partitioner.setMinimumEvents(10000);
dag.setAttribute(op, OperatorContext.STATS_LISTENERS,
Arrays.asList(new StatsListener[]{partitioner}));
dag.setAttribute(op, OperatorContext.PARTITIONER,
partitioner);

28. Locality
● Node-Local: The two operators should be placed in the same node
● Container-Local: Same worker container (saves serialization)
● Thread-Local: Same thread (simple function call)
// setting locality between op1 and op2 to be NODE_LOCAL
dag.addStream( "streamName", op1.output, op2.input)
.setLocality(DAG.Locality.NODE_LOCAL);

29. Recent Additions and Roadmap
● Apex runner in Apache Beam
● Iterative processing
● Integrated with Apache Samoa, opens up ML
● Integrated with Apache Calcite, allows SQL
---------------------------------------------
● Enhanced support for Batch Processing
● Support for Mesos and Kubernetes
● Encrypted Streams
● Python API

30. Q & A

31. Resources
● http://apex.apache.org/
● Learn more: http://apex.apache.org/docs.html
● GitHub Repositories
○ http://github.com/apache/apex-core
○ http://github.com/apache/apex-malhar
● Subscribe - http://apex.apache.org/community.html
● Download - http://apex.apache.org/downloads.html
● Follow @ApacheApex - https://twitter.com/apacheapex
● Meetups – http://www.meetup.com/topics/apache-apex/
● Slideshare: http://www.slideshare.net/ApacheApex/presentations
● https://www.youtube.com/results?search_query=apache+apex