How did eBay move their ETL computation from conventional RDBMS environment over to Spark? What did it take to go from a strategic vision to a viable solution? This paper will take you through a journey which lead to an implementation of a 1000+ node Spark Cluster running 10,000+ ETL jobs daily, all done in a span of less than 6 months, by a team with limited Spark experience. We will share the vision, technical architecture, critical Management decisions, Challenges and Road ahead. This will be a unique opportunity to look into this awesome Spark success story at eBay!

1. Kim Curtis, Brian Knauss, eBay
Evolution of eBay’s Enterprise Data
Ecosystem with Apache Spark
#EntSAIS13

2. Evolution
2#EntSAIS13
Principles
Opportunities
Investment

3. Principles
• Expand Capabilities
• Increase Flexibility
• Optimize Cost/Performance
3#EntSAIS13

4. • Expand Capabilities
• Increase Flexibility
• Optimize Cost/Performance
Principles
4#EntSAIS13
DO MORE
WITH FEWER BARRIERS
CHEAPER/FASTER

5. Spark in the EDW
• Increase Flexibility
– Investment and Engineering
• Expand Capabilities
– Pre-Load Transformation
• Optimize Cost/Performance
– Let’s see…
5#EntSAIS13

6. TCO Comparison
6#EntSAIS13
Vendor Open Source
HW Depr. HW Depr.
HW/SW Maint.HW/SW Maint.
DC Costs
DC Costs

7. Opportunity Assessment
7#EntSAIS13
2014 2015 2016 2017 2018 2019
Vendor TCO Open Source
TCO

8. Scope Design Implement Optimize
Investment
8#EntSAIS13

9. WHAT HOW WHEN
THEN
WHAT?
DO IT
9#EntSAIS13

10. Scope
• Engage with Customers
– Isolated our impact
• Define Boundaries
– Relational Batch Processing
• Set Intermediate Targets
– Offset 2017 growth
10#EntSAIS13

11. Design
• Extensible Framework
– (ELnTn to ETLn)
• Optimize Hardware
– Processing and Storage Nodes
• Optimize Software
– Automated Spark SQL Tuning
11#EntSAIS13

12. Load/MergeLoad/MergeTransform/MergeTransform/MergeLoadLoadExtract Load Transform/Merge Extract Load/Merge
Before (ELnTn )

13. Load/MergeLoad/Merge
Extract Transform Load/Merge
After (ETLn)

14. Implement
• Production HDFS Data Environment
– Tight Alignment with Platform
• Prioritized Effort
– Minimize effort to hit goals
• Distributed Engineering (internal Open Source)
– Built and tested framework additions
14#EntSAIS13

15. Optimize
• Scaling/DR(DA)
– Multi-platform architecture
• Cost/Performance Optimization
– HW/SW tuning
• Feature Expansion
– ML on platform
15#EntSAIS13

16. Challenges
• Scale of Data and Workload
– Data Validation Between Targets
• Migration Automation
– Migration of Non-Standard Processes
• Enterprise Readiness of Open Source
– Job-Level Workload Tracking/Management
16#EntSAIS13

17. The End Beginning…
17#EntSAIS13
Opportunity
Investment
Optimization