6. Application
Master
Example System
Request
Containers Assign work
Handle Failure
Handle
workload
Changes
Requirements
Big Data :-)
Partitioned, replicated
Fault tolerant, Scalable
Efficient resource utilization
Generate Data
Serve
M/R
Server 3
HDFS 3
7. Allocation + Assignment
HDFS
Server 1 Server 2Server 3
Partition Assignment - affinity, even distribution
Replica Placement - on different physical machines
Container Allocation - data affinity, rack aware placement
M/Rp1 p2 p3 p4 p5 p6
p1 p2
p5 p4
Server 3
p3 p4
p1 p6
Server 3
p5 p6
p3 p2
Multiple servers to serve
the partitioned data
M/R job generates partitioned data
8. Failure Handling
Server 1 Server 2Server 1
Acquire new container close to data if possible
Assign failed partitions to new container
On Failure - Even load distribution, while waiting for new container
Server 23 Server 3 Server 4
p5 p4 p1 p6 p3 p2
p1 p2 p3 p4 p5 p6
p3 p2
p5 p6
9. Workload Changes
Server 1 Server 2Server 3
Workload change - Acquire/Release containers
Container change - Re-distribute work
Monitor - CPU, Memory, Latency, Tps
p1 p2
p5 p4
Server 3
p3 p4
p1 p6
Server 3
p5 p6
p3 p2
Server 3
p4 p6
p2
10. Service Discovery
Server 1 Server 2Server 3
Dynamically updated on changes
Discover everything, what is running where
p1 p2
p1 p1
Server 3
p3 p4
p1 p1
Server 3
p5 p6
p1 p1
Client Client
Service Discovery
11. Building YARN Application
Writing AM is Hard and Error Prone
Handling Faults, Workload Changes is non-trivial and often overlooked
Request
container
How many
containers
Where
Assign work
Place
partitions &
replicas
Affinity
Workload
changes
acquire/
release
containers
Minimize
movement
Faults
Handling
Detect non
trivial failures
new v/s
reuse
containers
Other
Service
Discovery
Monitoring
Is there something that can make
this easy?
13. What is Helix?
Built at LinkedIn, 2+ years in production
Generic cluster management framework
Contributed to Apache, now a TLP: helix.apache.org
Decoupling cluster management from core functionality
15. Helix at LinkedIn
In Production
Over 1000 instances covering over 30000
partitions
Over 1000 instances for change
capture consumers
As many as 500 instances in a
single Helix cluster
(all numbers are per-datacenter)
17. Helix concepts
Resource
(Database, Index, Topic, Task)
Partitions
Replicas
p1 p2 p3 p4 p5 p6
r1
r2
r3
Container
Process
Container
Process
Container
Process
Assignment ?
18. Serve
bootstrap
State Model and Constraints
Helix Concepts
State
Constraints
Transition
Constraints
Partition
Resource
Node
Cluster
Serve: 3
bootstrap: 0
Max T1 transitions in
parallel
-
Max T2 transitions in
parallel
No more than
10 replicas
Max T3 transitions in
parallel
-
Max T4 transitions in
parallel
StateCount=
Replication factor:3
Stop
21. Helix Controller
Target Provider
Determine how many containers are required along with the spec
Fixed CPU Memory Bin Packing
monitoring system provides usage information
Default implementations, Bin Packing can be used to customize further
TargetProvider
Resources p1,p2 .. pn
Existing containers c1,c2 .. cn
Health of tasks,
containers
cpu, memory, health
Allocation
constraints
Affinity,
rack locality
SLA
Fixed: 10 containers
CPU headroom:30%
Memory Usage: 70%
time: 5h
Number of
container
release list
acquire list
Container spec
cpu: x
memory: y
location: L
22. Helix Controller
Provisioner
Given the container spec, interact with YARN RM to
acquire/release, NM to start/stop containers
YARN
Interacts with YARN RM and subscribes to notifications
23. Helix Controller
Rebalancer
Based on the current nodes in the cluster and constraints, find an
assignment of task to node
Auto Semi-Auto Static
Rebalancer
Tasks t1,t2 .. tn
Existing containers c1,c2 .. cn
Allocation
constraints &
objectives
Affinity,
rack locality,
Even distribution of
tasks,
Minimize movement
while expanding
Assignment
C1: t1,t2
C2: t3,t4
User defined
Based on the FSM, compute & fire the transitions to Participants
24. Example System: Helix-Based Solution
Solution
Configure App
Configure Target Provider
Configure Provisioner
Configure Rebalancer
Generate Data
Serve
M/R
Server 3
HDFS 3
25. Configure AppConfigure App
App Name Partitioned Data Server
App Master
Package
/path/to/
GenericHelixAppMaster.tar
App package
/path/to/
RedisServerLauncher.tar
App Config
DataDirectory: hdfs:/path/to/
data
Configure target providerConfigure target provider
TargetProvider RedisTargetProvider
Goal Target TPS: 1 million
Min container 1
Max containers 25
Configure ProvisionerConfigure Provisioner
YARN RM host:port
Configure RebalancerConfigure Rebalancer
Partitions 6
Replica 2
Max partitions per container 4
Rebalancer.Mode AUTO
Placement Data Affinity
FailureHandling Even distribution
Scaling Minimize Movement
app_config_spec.yaml
Example System: Helix-Based Solution