This talk introduces latest features in Apache Hadoop YARN 3.2 and how this is used in production environment on 10k+ nodes.

1. Apache Hadoop YARN:
State of the Union
Weiwei Yang/Chunde Ren
Hortonworks/Alibaba

2. Weiwei Yang
• Software Engineer at
Hortonworks, YARN dev team
• Apache Hadoop committer and
PMC member
Chunde Ren
• Staff Software Engineer at
Alibaba
• Leading the Hadoop team in
real-time computation platform

3. • State of the Union: Service, ML, Cloud and beyond
Scale and Performance
Unified platform
• Apache YARN 3.1 in Alibaba
Utilization+: balance & oversubscription
• Q&A
Hybrid clusters

4. State of the Union: Service,
ML, Cloud…
Weiwei Yang
Hortonworks, YARN

5. 1 Year Timeline: GA Releases
2.9.0 3.0.0 3.1.0 3.2.0
• Submarine
• Node attributes
• Service upgrade
• Containerize
improvements
• Global
Scheduling
• Multiple
Resource types
• New YARN UI
• Timeline v2
• GPU/FPGA
• YARN Native
Service
• Global
scheduling
• Placement
Constraints
• YARN Federation
• Opportunistic
Containers
• New YARN UI
• Timeline v2
Nov 17 Dec 17 Aug 18 Oct 182.9.1 3.0.3 3.1.1 3.2.0

6. Apache Hadoop YARN
Unified Data Operative System
ML
Streaming
Ad-hoc
Deep Learning
No-SQL
SQLService
Compute
Resource
SLA
Utilization

7. Focus area
• Continue to evolve at large scale
• Scale
• Global Scheduling
• Unified platform
• Container runtime and Services
• Placement constraints
• Beyond: Submarine/CSI

8. Scale at Today
• Tons of sites with clusters made
up by large amount of nodes
• Oath(Yahoo!), Twitter,
LinkedIn, Microsoft, Alibaba
etc.
• 50K nodes in a single cluster of
Microsoft[1]
• Roadmap: To 100K and beyond
https://azure.microsoft.com/en-us/blog/how-microsoft-drives-exabyte-analytics-on-the-world-s-largest-yarn-cluster/[1]

9. Global Scheduling
Thread 1
Proposal 1
Thread 2
Proposal 2
Thread 3
Proposal 3
Scheduler
state Placement
Committer

10. Global Scheduling: takeaways
• Addresses hotspot issues
• Allows to plug customized node scoring policies (customize slot-
selection themes)
• Scoring can be done at background or in-place
• Not fit for clusters merely run small batches

11. Docker Container
• Better packing model
• Light-weighted mechanism for packaging and resource
isolation
• Popularized and made accessible by Docker
• Native integrated in YARN
• Docker container runtime
• Many security/usability improvements added to 3.x

12. Container Runtime
Run both docker and non-docker
containers on same cluster

13. YARN Service
Simplified Service
Framework
• Service discovery: DNS
Registration
• Service timeout
• Upgrade
• Integrated REST API/CLI/UI

14. Service - UI

15. Placement Constraints
Anti-affinity
Don't place containers together
Affinity
Collocate containers
Cardinality
Control number of containers per node/rack
Expression, namespace, service spec and more

16. Node Attributes
rm.yarn.io/hostName=“host1”
rm.yarn.io/hostType=“new”
nm.yarn.io/javaVersion=“1.8”
rm.yarn.io/hostName=“host2”
rm.yarn.io/hostType=“old”
nm.yarn.io/javaVersion=“1.9”
RM STATE
hostName=host1
javaVersion>1.6
Host1 Host2
Allocation = host1
Allocation = host1 | host2
1. Centralized/Distributed node-attributes
2. Distributed node-attributes support config/script providers
3. Admin tools and restful APIs

17. Submarine: TF Hello world
yarn jar hadoop-yarn-applications-submarine-<version>.jar job run
--name tf-job-001 --docker_image <your docker image>
--input_path hdfs://default/dataset/cifar-10-data
--checkpoint_path hdfs://default/tmp/cifar-10-jobdir
--num_workers 2
--worker_resources memory=8G,vcores=2,gpu=2
--worker_launch_cmd "cmd for worker ..."
--num_ps 2
--ps_resources memory=4G,vcores=2,gpu=0
--ps_launch_cmd "cmd for ps"
Run distributed TF training with one commnad:

18. Submarine - Architecture

19. • Container Storage Interface (CSI): a vendor neural interface to bridge
Container Orchestrator and Storage Providers.
Amazon
EBS
Hadoop
Ozone
ContainerStorageInterface
CSI
The Adaption of CSI

20. Part of container resource request
Volume Resource

21. Deployment
Resource Manager
Node
Manager
Controller
Plugin
Node
Plugin
Identity
Plugin
CSI
Driver
Adaptor
Container
Container
Volume Mount
Storage System
Node
Manager
Controller
Plugin
Node
Plugin
Identity
Plugin
CSI
Driver
Adaptor
Container
Container
Volume Mount
Volume Manager
3rd Party CSI Driver
Master Slave Slave

22. Apache YARN 3.1 in Alibaba
Chunde Ren
Alibaba, Real-time Computation

23. Apache YARN
BI Recommendation SecuritySearch
Apache HDFS
Ads
Streaming + Batch
The Ecosystem

24. The challenges & solutions
• YARN Clusters
• Tens thousand nodes, version: 3.1.0+patches
• Long running streaming jobs >> Batch jobs, Machine Learning
• SLA: latency, priority, failover
• HA, performance, load-balancer, placement constraints, isolation
• Resource Utilization
• Elastic queue capacity & preemption, Oversubscription, Hybrid cluster

25. Enhanced Capacity Scheduler

26. Load Balance – Node Scores
Dynamically optimize
container distribution
across the cluster
• Policy: app or queue
• Shuffle candidates
• Auto resize allocate
thread
• Score Cache

27. Load Balance - Reschedule
27
Hotspot
Candidates
Selector
Rescheduler
App1 Container
App2 Container Scheduler
editScheduler
Scheduler States
Notify Scheduler
MARK_CONTAINER_FOR_PREEMPTION
Resource Manager
MainScheduler
Eliminate Hotspots and
Fragmentations
• NM/Container utilization metrics
• Identify hotspot & idle nodes
• Preemption:
- Priority
- candidate app cache
- preemption contract protocol

28. 28
Container execution type：Guaranteed/Opportunistic
Schedule Mode
• Distributed Scheduler
• extended AMS
• extended CapacityScheduler
Isolation
• CPU: O container share = 2
• Memory: OOM Controller + Killer vs QoS Monitor
• Network: net_cls + tc + switch
• Disk IO: HDD blk.weight vs SSD
Preemption
• High/Low watermark
• O(App Priority) < G(App Priority)
Result: auto operation, utilization >10%
Resource Oversubscription

29. 29
Hybrid Cluster
Improve Online Service Cluster
resource utilization
Architecture: YARN on K8s
Isolation: powered by aliKernel
Use Case: machine learning & deep
learning
Utilization:
K8s
NodeManager
G+O Containers
NM docker)
Services Allocated
Services
ResourceManager
Controller
API Server
Master