LinkedIn

2. The Past, Present, and Future of
Hadoop @ LinkedIn
Carl Steinbach
Senior Staff Software Engineer
Data Analytics Infrastructure Group
LinkedIn

3. The (Not So) Distant Past

4. PYMK (People You May Know)
First version implemented in 2006
6-8 Million members
Ran on Oracle (foreshadowing!)
Found various overlaps
School, Work… etc
Used common connections
Triangle closing (?)

5. Triangle Closing
?
Mary
Dave
Steve

6. PYMK Problems
By 2008, 40-50 Million members
Still running on Oracle
Failed often
Infrequent data refresh
6 weeks – 6 months!

7. Humble Beginnings Back in ‘08

8. Success! (circa 2009)
Apache Hadoop 0.20
20 node cluster (repurposed hardware)
PYMK in 3 days!

9. The Present

10. Hadoop @ LinkedIn Circa 2016
> 10 Clusters
> 10,000 Nodes
> 1000 Users
Thousands of workflows, datasets, and ad-hoc queries
MR, Pig, Hive, Gobblin, Cubert, Scalding, Tez, Spark,
Presto, …

11. Two Types of Scaling Challenges
Machines
People and Processes

12. Scaling Machines

13. Some Tough Talk About HDFS
Conventional wisdom holds that HDFS
Scales to > 4k nodes without federation*
Scales to > 8k nodes with federation*
What’s been our experience?
Many Apache releases won’t scale past a couple thousand nodes
Vendor distros usually aren’t much better
Why?
Scale testing happens after the release, not before
Most vendors have only a handful of customers with clusters larger than 1k nodes
* Heavily dependent on NN RPC workload, block size, average file size, average container size, etc, etc

14. March 2015 Was Not a Good Month

15. What Happened?
We rapidly added 500 nodes to a 2000 node cluster
(don’t do this!)
NameNode RPC queue length and wait time skyrocketed
Jobs crawled to a halt

16. What Was the Cause?
A subtle performance/scale regression was introduced
upstream
The bug was included in multiple releases
Increased time to allocate a new file
The more nodes you had, the worse it got

17. How We Used to do Scale Testing
1. Deploy the release to a small cluster (num_nodes = 100)
2. See if anything breaks
3. If no, then deploy to next largest cluster and goto step 2
4. If yes, figure out what went wrong and fix it
Problems with this approach
Expensive: developer time + hardware
Risky: Sometimes you can’t roll back!
Doesn’t always work: overlooks non-linear regressions

18. • Scale testing and performance
investigation tool for HDFS
• High fidelity in all the dimensions that
matter
• Focused on the NameNode
• Completely Black-box
• Accurately fakes thousands of DNs on a
small fraction of the hardware
• More details in forthcoming blog post
18
HDFS Dynamometer

19. Scaling People and Processes

20. 20

21. 21
v
Hadoop Performance Tuning

22. Too many dials!
Lots of frameworks: each one is
slightly different.
Performance can change over time.
Tuning requires constant monitoring
and maintenance!
22
Why Are Most User Jobs Poorly Tuned?
* Tuning decision tree from “Hadoop In Practice”

23. 23
Dr Elephant: Running Light Without Overbyte
Automated Performance
Troubleshooting for Hadoop
Workflows
● Detects Common MR and
Spark Pathologies:
○ Mapper Data Skew
○ Reducer Data Skew
○ Mapper Input Size
○ Mapper Speed
○ Reducer Time
○ Shuffle & Sort
○ More!
● Explains Cause of Disease
● Guided Treatment Process

24. Grab the source code
github.com/linkedin/dr-elephant
Read the blog post
engineering.linkedin.com/blog
24
Dr Elephant is Now Open Source

25. Upgrades are Hard
A totally fictional story:
The Hadoop team pushes a new Pig upgrade
The next day thirty flows fail with ClassNotFoundExceptions
Angry users riot
Property damage exceeds $30mm
What happened?
The flows depended on a third-party UDF that depended on a transitive
dependency provided by the old version of Pig, but not the new version of
Pig

26. Bringing Shading Out of the Shadows
What most people think it is
Package artifact and all dependencies in the same JAR + rename some or all of
the package names
What it really is
Static linking for Java
Unfairly maligned by many people
We built an improved Gradle plugin that makes shading
easier for inexperienced users

27. Audit Hadoop flows for
incompatible and unnecessary
dependencies.
Predict failures before they happen
by scanning for dependencies
that won’t be satisfied post-
upgrade.
Proved extremely useful during
Hadoop2 migration
27
Byte-Ray: “X-Ray Goggles for JAR Files”

28. Byte-Ray in Action

29. SoakCycle:
Real World Integration Testing

30. The Future?

31. Dali
2015 was the year of the table
We want to make 2016 the year of the view
Learn more at the Dali talk tomorrow

32. ©2014 LinkedIn Corporation. All Rights Reserved.©2014 LinkedIn Corporation. All Rights Reserved.