Pig TPC-H Benchmark and Performance Tuning

Running TPC-H On Pig Jie Li, Koichi Ishida, Muzhi Zhao, Ralf Diestelkaemper, Xuan Wang, Yin Lin CPS 216: Data Intensive Computing Systems Dec 9, 2011

Goals ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[1] https://issues.apache.org/jira/browse/HIVE-600

Benchmark Set Up ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Initial Result ,[object Object],[object Object]

Six Rules Of Writing Efficient Pig Scripts ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Rule 1: Reorder JOINs properly ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],* We focused on the default hash join. The replicated join does not apply to most of the TPC-H joins and its benefit is ignorable in most queries.

Apply Rule 1 to TPC-H ,[object Object],[object Object]

Rule 2: COGROUP ,[object Object],[object Object]

Rule 2 Example ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Apply Rule 2 to TPC-H Query 13 ,[object Object],[object Object]

Rule 3: FLATTEN ,[object Object],[object Object]

Rule 3 Example SQL Pig t1 = group A by x; t2 = foreach t1 generate FLATTEN (A), AVG(A.y) as avg_y; t3 = filter t2 by y < avg_y; ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Apply Rule 2 and 3 to TPC-H Query 17 ,[object Object],[object Object],[object Object]

Rule 4: Project before (CO)GROUP ,[object Object],[object Object],[object Object],[object Object],[object Object]

Rule 4 Example ,[object Object],[object Object],[object Object],[object Object]

Rule 5: Remove types in LOAD ,[object Object],[object Object],[object Object],[object Object],[object Object]

Apply Rule 5 to TPC-H Query 6 ,[object Object],[object Object]

Rule 6: Use hash-based aggregation ,[object Object],[object Object],[object Object],[object Object]

Query 1 (Rule 6 will be applicable soon) Q1 has a group-by and several aggregations.

Six Rules Summary ,[object Object],[object Object],[object Object]

All rewritten queries based on Rule 1~5

Acknowledgement ,[object Object],[object Object],[object Object]

Pig TPC-H Benchmark and Performance Tuning

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Pig TPC-H Benchmark and Performance Tuning

Similar to Pig TPC-H Benchmark and Performance Tuning (20)

Recently uploaded

Recently uploaded (20)

Pig TPC-H Benchmark and Performance Tuning