1. Hadoop Simulation and Performance Apache Hadoop India Summit 2011 Ranjit Mathew, Yahoo! R & D India Copyright © 2011 Yahoo! All rights reserved.

2. Overview 2 Introduction GridMix3 PigMix2 Tips Plans Q & A

3. 3 Introduction

4. Why? 4 Capacity Planning Benchmarking Comparative evaluation of releases Basis for improvements Debugging

5. Performance Evaluation Techniques 5 Analytical Modeling Use statistics, queuing theory, etc. to model system Use models to predict behavior Simulation Simulate work-load based on representation or traces Benchmarking used to compare variants Measurement Use metrics gathered from tools and logs Measure under peak, regular and light work-loads Ref.: “The Art of Computer Systems Performance Analysis”, Raj K. Jain (Wiley, 1991)

6. Hadoop Performance Evaluation Tools 6 GridMix3 PigMix2 TeraSort / GraySort DFSIO, NNBench, S-Live HiBench etc.

7. 7 GridMix3

13. Rumen 10 Comprises: TraceBuilder - Job Traces from Job History and Configuration Folder - Scales Job Traces to a given time-window Job Traces are in JSON format Insulation for release-to-release changes in format and contents Statistical information on Jobs in Trace Provides API to access Job Traces

14. GridMix3 Flow 11 GridMix3 Production Cluster Data Generator Job Submitter Job Histories & Configuration Job Trace Rumen Benchmark Cluster

15. GridMix3 Architecture 12 GridMix3 JobFactory JobSubmitter JobMonitor GridmixJob MapReduceJob Job JobStory Status JobTracker Rumen

16. GridMix3 Emulation-Accuracy 13

17. Submission Policies and Job Types 14 Submission policy determines when Jobs are submitted: STRESS - Keep cluster under stress (but not overwhelm it) REPLAY - Faithful emulation of inter-job submission times SERIAL - Submit a Job only after the previous one finishes Types of synthetic Jobs: LOADJOB - Emulates work-load from Job Trace SLEEPJOB - Do nothing for periods from Job Trace

18. 15 PigMix2

20. http://wiki.apache.org/pig/PigMix

23. PigMix2 Flow 18 Input Data Data Generator PigMix2 Benchmark Cluster

24. 19 Tips

25. Minimize Variance 20 Check hardware, especially for failing hard-drives Use large data-sets to minimize effects of overheads Beware of speculative execution Set ipc.ping.interval to 5000 (HADOOP-5380) Use appropriate PARALLEL clause in PigMix2 Pig scripts Several runs needed for proper analysis

26. Apples to Apples Comparison 21 Benchmarking versus Production Cluster: Same hardware Same software stack Same configuration Similar networking Same size (might not be feasible) Extrapolating results can be tricky

27. 22 Plans

28. Future Work 23 Greater emulation-accuracy in GridMix3: Distributed Cache Compression CPU usage Memory usage More comprehensive Job Traces from Rumen Integration of PigMix2 with Pig Statistics

29. 24 Q & A

30. ranjit mathew senior principal engineer ranjit@yahoo-inc.com