Presenter: Priyanka Gugale, Committer for Apache Apex and Software Engineer at DataTorrent. In this session we will cover introduction to Yarn, understanding yarn architecture as well as look into Yarn application lifecycle. We will also learn how Apache Apex is one of the Yarn applications in Hadoop.

1. Introduction to YARN and
Apex as YARN Application
Priyanka Gugale (priyag@apache.org)
September 30th 2016

2. Apache Apex - Stream Processing
Easily Operable - Exposes an easy API for developing Operators (part of an
application) and Applications
Highly Scalable - Scales statically as well as dynamically
Highly Performant - Can reach single digit millisecond end-to-end latency
Fault Tolerant - Automatically recovers from failures - without manual
intervention
Stateful - Guarantees that no state will be lost
Apex Malhar library

3. Apex Platform Overview
3

4. An Apex Application is a DAG
(Directed Acyclic Graph)
A DAG is composed of vertices (Operators) and edges (Streams).
A Stream is a sequence of data tuples which connects operators at end-points called Ports
An Operator takes one or more input streams, performs computations & emits one or more output streams
● Each operator is USER’s business logic, or built-in operator from our open source library
● Operator may have multiple instances that run in parallel

5. DAG Components
• Tuple
● Atomic data that flows over a stream
• Operator
● Basic compute unit per tuple
• Stream
● Connector abstraction between operators
● Tuples flow over this
Operator
1
Operator
2
Stream
tuple
3
tuple
1
tuple
2

6. How Apex is Yarn
Native?

7. Introducing YARN
● YARN - Yet Another Resource Negotiator
● framework that facilitates writing arbitrary distributed processing frameworks and
applications.
● YARN Applications/frameworks:
e.g. MapReduce2, Apache Spark, Apache Giraph, Apache Apex etc.

8. Introducing YARN
Map Reduce 1
YARN
≈
≈
Proprietary and Confidential
Job Tracker
Resource
Manager
Application Master
Timeline Server
Task Tracker Node Manager
Map Slot
Reduce Slot

9. Hadoop beyond Batch
YARN for better
resource utilization
More applications
than MapReduce

10. • Resource Manager
Manages and allocates cluster resources
Application scheduling
Applications Manager
• Node Manager
Per-machine agent
Manages life-cycle of container
Monitors resources
• Application Master
Per-application
Manages application scheduling and task execution
Hadoop v2 (YARN) Architecture
App
Master
Cont
Node
Manager
Cont Cont
Node
Manager
App
Master
App
Master
Node
Manager
Resource
Manager
MapReduce Status
Job Submission
Node Status
Resource Request
Client
Client

11. Application Submission workflow
YarnClient
Node
RM
(ApplicationsManager +
Scheduler)
Node
NM
Node
NMApplication
Master
ContainerContainer
1) Submit
application
2) Launch application Master
RM = Resource Manager
NM = Node Manager
AM = Application Master
= Heartbeats
3) AM registers with RM
4) AM negotiates for containers
5) Launch Container
5) Launch
Container

12. Apex as YARN application
Node
ResourceManager
(AsM + Scheduler)
NM Node NM Node NM
YarnClient
AppMaster
YarnContainer
YarnContainer
YarnContainer
StrAM
(AppMaster)
YarnContainer
StrAMChild
O1 O2
YarnContainer
StrAMChild
O3
Apex cli
StrAMClient
YarnClient
Apache Apex Meetup
ClientRM
Protocol
AMRM
Protocol
ContainerManager
Protocol
ContainerManager
Protocol
ClientRM
Protocol
AMRM
Protocol
ContainerManager
Protocol

13. Application Components of Apex - StrAMClient
• Part of apex client interface
• Invoked by “launch” command of apex
• Tasks:
● Copy required the application package files into HDFS
● Validate Logical Plan
● Serialize Logical plan to HDFS
● Launch Application Master i.e. StrAM
Apache Apex Meetup

14. Application Components of Apex - StrAM
• Streaming Application Master
• Started by StrAMClient on a YarnContainer
• Tasks:
● Convert logical plan to physical plan
● Serialize operators to HDFS
● Request for resources to ResourceManager
● Start StrAMChild in YarnContainer(s)
● Monitor StrAMChild using ContainerManager protocol
● Generate Application statistics
● Host results on WebService (dtManage)
● Checkpointing/Committing Application States
● Fault Tolerance
● Support Security
● Shutdown Application
Apache Apex Meetup

15. Application Components of Apex - StrAMChild
• Deployed on YarnContainer
• Started by NodeManager as instructed by StrAM
• Instance of StreamingContainer
• Contains Operators (compute-related)
• Contains BufferServer (stream-related)
• Tasks:
● Regularly send heartbeat to StrAM
● Execute commands from StrAM
● Shutdown or Kill self if instructed
● Manage lifecycle of an Operator
● Network communication using BufferServer
Apache Apex Meetup

16. Apex as YARN application
Node
ResourceManager
(AsM + Scheduler)
NM Node NM
StrAM
(AppMaster)
YarnContainer
StrAMChild
O1 O2
YarnContainer
StrAMChild
O3
Apex cli
StrAMClient
YarnClient
Apache Apex Meetup
ClientRM
Protocol
AMRM
Protocol
ContainerManager
Protocol

17. Summary – Apex platform
• Enables YARN to be used for Streaming Applications
• Takes care of YARN specific work
• User can focus on business logic defined in Operators
Apache Apex Meetup

18. Q&A
18

19. Resources
19
• http://apex.apache.org/
• Learn more: http://apex.apache.org/docs.html
• Subscribe - http://apex.apache.org/community.html
• Download - http://apex.apache.org/downloads.html
• Follow @ApacheApex - https://twitter.com/apacheapex
• Meetups – http://www.meetup.com/pro/apacheapex/
• More examples: https://github.com/DataTorrent/examples
• Slideshare: http://www.slideshare.net/ApacheApex/presentations
• https://www.youtube.com/results?search_query=apache+apex
• Free Enterprise License for Startups -
https://www.datatorrent.com/product/startup-accelerator/