2. Who We Are
Early Engineers On Hadoop team @ Uber
Kelvin Chu Reza ShiftehfarVinoth Chandar
3. Agenda
● Intro to Data @ Uber
● Trips Pipeline Into Warehouse
● Paricon
● INotify DStream
● Future
4. Edit or delete footer text in Master ipsandella doloreium dem isciame ndaestia nessed
quibus aut hiligenet ut ea debisci eturiate poresti vid min core, vercidigent.
Uber’s Mission
“Transportation as reliable as running water,
everywhere, for everyone”
300+ Cities 60+ Countries
And growing...
5. Data @ Uber
● Impact of Data is Huge!
○ 2000+ Unique Users Operating a massive transportation system
● Running critical business operations
○ Payments, Fraud, Marketing Spend, Background Checks …
● Unique & Interesting Problems
○ Supply vs Demand - Growth
○ Geo-Temporal Analytics
● Latency Is King
○ Enormous business value in making data available asap
7. Challenges
● Scaling to high volume Kafka streams
○ eg: Event data coming from phones
● Merged Views of DB Changelogs across DCs
○ Some of the most important data - trips (duh!)
● Fragile ingestion model
○ Projections/Transformation in pipelines
○ Data Lake philosophy - raw data on HDFS, transform later using Spark
● Free-form JSON data → Data Breakages
● First order of business - Reliable Data
8. New World Order: Hadoop & Spark
Kafka Logs
Schemaless
Databases
RDBMS Tables
Amazon S3
HDFS
OLAP
Warehouse
Applications
Adhoc SQL
Applications Adhoc SQL Machine Learning
Paricon
Spark
SQL
Spark
/Hive
Spark
Jobs/
Oozie
Spark?
Data
Delivery
Services
Raw
Data
Cooked
Spark/Spark
Streaming
9. Trips Pipeline : Problem
● Most Valuable Dataset in Uber (100% Accuracy)
● Trips stored in Uber’s ‘schemaless’ datastores (sharded
Mysql), across DCs, cross replicated
● Need a consolidated view across dcs, quickly (~1-2 hr
end-end)
Trip Store
(DC1)
Trip Store
(DC2)
Writes in DC1 Writes in DC2
Multi Master XDC
Replication
11. Trips Pipeline : ETL via SparkSQL
● Decouples raw ingestion from Relational Warehouse
table model
○ Ability to provision multiple tables off same data set
● Picks latest changelog entry in the files
○ Applies them in order
● Applies projections & row level transformations
○ Produce ingestible data into Warehouse
● Uses HiveContext to gain access to UDFs
○ explode() etc to flatten JSON arrays.
● Scheduled Spark Job via Oozie
○ Runs every hour (tunable)
12. Paricon : PARquet Inference and CONversion
● Running in production since Feburary 2015
○ first Spark application at Uber
13. Motivation 1: Data Breakage & Evolution
Upstream Data Producers
Downstream Data Consumers
JSON at S3 data evolving over time … and one day
14. Motivation 1: Why Schema
● Contract
○ multiple teams
○ producers
○ consumers
● Avoid data breakage
○ because we have schema evolution systems
● Data to persist in a typed manner
○ analytics
● Serve as documentation
○ understand data faster
● Unit testable
16. Motivation 2: Why Parquet
● Supports schema
● 2 to 4 times FASTER than json/gzip
○ column pruning
■ wide tables at Uber
○ filter predicate push-down
○ compression
● Strong Spark support
○ SparkSQL
○ schema evolution
■ schema merging in Spark v1.3
■ merge old and new compatible schema versions
■ no “Alter table ...”
17. Paricon : Transfer
● distcp on Spark
○ only subset of command-line options currently
● Approach
○ compute the files list and assign them to RDD partitions
○ avoid stragglers by randomly grouping different dates
● Extras
○ Uber specific logic
■ filename conventions
■ backup policies
○ internal Spark eco-system
○ faster homegrown delta computation
○ get around s3a problem in Hadoop 2.6
18. Paricon : Infer
● Infer by JsonRDD
○ but not directly
● Challenge: Data is dirty
○ garbage in garbage out
● Two passes approach
○ first: data cleaning
○ second: JsonRDD inference
19. Paricon : Infer
● Data cleaning
○ structured as rules-based engine
○ each rule is an expectation
○ all rules are heuristics
■ based on business domain knowledge
○ the rules are pluggable based on topics
● Struct@JsonRDD vs Avro:
○ illegal characters in field names
○ repeating group names
○ more
20. Paricon : Convert
● Incremental conversion
○ assign days to RDD partitions
○ computation and checkpoint unit: day
○ new job or after failure: work on those partial days only
● Most number of codes among the four tasks
○ multiple source formats (encoded vs non-encoded)
○ data cleaning based on inferred schema
○ home grown JSON decoder for Avro
○ file stitching
21. Stitching : Motivation
File size
Number of files
HDFS
block size
● Inefficient for HDFS
● Many large files
○ break them
● But a lot more small files
○ stitch them
23. Stitching : Algorithms
● Algo1: Estimate a constant before conversion
○ pros: easy to do
○ cons: not work well with temporal variation
● Algo2: Estimate during conversion per RDD partition
○ each day has its own estimate
○ may even self-tuned during the day
24. Stitching : Experiments
●
○ N: number of Parquet files
○ Si: size of the i-th Parquet file
○ B: HDFS block size
○ First part: local I/O - files slightly smaller HDFS block
○ Second part: network I/O - penalty of files going over a block
● Benchmark queries
25. Paricon : Validate
● Modeled as “Source and converted tables join”
○ equi-join on primary key
○ compare the counts
○ compare the columns content
● SparkSQL
○ easy for implementation
○ hard for performance tuning
● Debugging tools
26. Some Production Numbers
● Inferred: >120 topics
● Converted: >40 topics
● Largest single job so far
○ process 15TB compressed (140TB uncompressed) data
○ one single topic
○ recover from multiple failures by checkpoints
● Numbers are increasing ...
27. Lessons
● Implement custom finer checkpointing
○ S3 data network fee
○ jobs/tasks failure -> download all data repeatedly
○ to save money and time
● There is no perfect data cleaning
○ 100% clean is not needed often
● Schema parsing implementation
○ tricky and takes much time for testing
28. Komondor: Problem Statement
● Current Kafka->HDFS ingestion service does too much
work:
○ Consume from Kafka -> Write Sequence Files -> Convert to Parquet ->
Upload to HDFS, HIVE compatible way
○ Parquet generation needs a lot of memory
○ Local writing and uploading is slow
● Need to decouple raw ingestion from consumable data
○ Move heavy lifting into Spark -> Keep raw-data delivery service lean
● Streaming job to keep converting raw data into Parquet,
as they land!
29. Komondor: Kafka Ingestion Service
Komondor
Streaming Raw
Data Delivery
Kafka
HDFS
Streaming
Ingestion
Batch Verification
& File Stitching
Raw Data
Consumable
Data
30. Komondor: Goals
● Fast raw data into permanent storage
● Spark Streaming Ingestor to ‘cook’ raw data
○ For now, Parquet generation
○ But opens up polyglot world for ORC, RCFile,....
● De-duplicate of raw data before consumption
○ Shields downstream consumers from at-least-once delivery of pipelines
○ Simply replay events for an entire day, in the event of pipeline outages
● Improved wellness of HDFS
○ Avoiding too many small files in HDFS
○ File stitcher job to combine small files from past days
32. INotify DStream: Motivation
● Streaming Job to pick up raw data files
○ Keeps end-to-end latency low vs batch job
● Spark Streaming FileDStream not sufficient
○ Only works 1 directory deep,
■ At least have two levels for <topic>/<dc>/
○ Provides the file contents directly
■ Loses valuable information in file name. eg: partition num
○ Checkpoint contains an entire file list
■ Will not scale to millions of files
○ Too much overhead to run one Job Per Topic
33. INotify DStream: HDFS INotify
● Similar to Linux iNotify to watch file system changes
● Exposes the HDFS Edit Log as an event stream
○ CREATE, CLOSE, APPEND, RENAME, METADATA, UNLINK events
○ Introduced in Hadoop Summit 2015
● Provides transaction id
○ Client can use to resume from a given position
● Event Log Purged every time the FSImage is uploaded
34. INotify DStream: Implementation
● Provides the HDFS INotify events as a Spark DStream
○ Implementation very similar to KafkaDirectDStream
● Checkpointing is straightforward:
○ Transactions have unique IDs.
○ Just save Transaction ID to permanent storage
● Filed SPARK-10555, vote up if you think it is useful :)
35. INotify DStream: Early Results
● Pretty stable when running on YARN
● HDFS iNotify reads ALL events from NameNode
● Have to add filtering
○ to catch only events of interests (Paths/Ext.)
○ Performed at Spark level
● Memory usage increases on NN when iNotify is running
36. INotify DStream: Future Uses, XDC Replication
● Open possibility, provided INotify is a charm in production
● Uber’s thinking about all active-active data architecture
○ This means n HDFS clusters that need to be in-sync
● Typical batch-based distcp creates bursty network
utilization
○ Or go through scheduling trouble to smoothen it out
○ INotify DStream provides way to keep shipping files as they land
○ Power of Spark to do any heavy lifting such as filtering sensitive data
38. Future/Ongoing Work
● Spark SQL Based ETL-Platform
○ Powers all tables into warehouse
● Open up SQL-On-Hadoop via Spark SQL/Hive
○ Spark Shell is already so nifty!
● Machine Learning Platform using Spark
○ MLLib /GraphX Possibilities
● Standardized Support For Spark jobs
● Apollo: Next Gen Real-time analytics using Spark
Streaming
○ Watch for our next talk! ;)
43. Trips Pipeline : Consolidating Changelogs
● Data Model, very similar to BigTable/HBase
○ row_key : uuid for trip
○ col_key : One column in trip record
○ version & body : version of the column & json blob
○ cell : Unique tuple of {row_key, col_key, version}
● Provides REST endpoint to tail cell change log for
every shard
44. Trips Pipeline : Challenge
● Existing ingestion turned cell changes into Warehouse
upserts,
○ Losing the version information
○ Unable to reject older (& duplicate) cell changes in logs, coming
from XDC replication
{ trip-xxx :
{“FARE”=>{f1:{body:12.35,version:11},
f2:{body:val2,version:10}},
{“ETA”=>{f3:{body:val3,version:13},
f4:{body:val4,version:10}}
}
trip-uuid FARE_f1 ETA_f3 ETA_f4
trip-xxx 12.35 4 5
trip-xyz 14.50 2 1
45. Spark At Uber
● Today
○ Paricon: Turn Historical Json Into Parquet Gold Mine
○ Streamio/Spark SQL : Deliver Global View of Trip Database
into Warehouse in near real-time
● Tomorrow
○ INotify DStream :
■ Komondor - The ‘Uber’ data ingestor
■ XDC Data Replicator
○ Adhoc SQL Access to data: Hive On Spark/Spark SQL
○ Spark Apps: Directly accessing data on HDFS
46. Trips Pipeline : Raw Row Images in HDFS
● Streamio : Generic connector of partitioned streams
○ Pluggable in & out stream implementations
● Tails cell changes from both DCs into a Kafka topic
● Uses HBase to construct full row image (latest value
for each column for a trip)
○ Logs ‘row changelog’ to HDFS
● Preserves version of latest cell for each column/row
○ Can efficiently de-duplicate/reconcile.
● Extensible to all Schemaless datastores
