This document discusses Hadoop and MapReduce. It covers topics like API design, configuration, testing MapReduce jobs with MRUnit, running MapReduce jobs with JobRunner, monitoring jobs with the web UI, profiling performance with HPROF, and comparing MapReduce to other frameworks like Pig and Hive. The document contains code examples and explanations of MapReduce concepts.

1. Hadoop 5
MapReduce

2. ❖ blog: http://brass.to/blog/
❖ twitter: @akahigeg
❖ Rails
❖ Mac,Vim
❖ Linux(CentOS),Apache,MySQL
❖ Agile

3. ❖ X
❖ PHP CakePHP
❖ TDD
❖ 16 Puppet

4. 5.1 API P127
5.2 P130
5.3 P137
5.4 P140
5.5 P146
5.6 P161
5.7 MapReduce P164

6. 5.1 API P127
❖ =
XML 5-1 P128
❖
‣
❖
‣
‣ final

7. 5.1 API P129
❖
‣ DRY
❖
‣
❖

8. 5.2 P130
❖ Hadoop
❖ IDE
❖ P131
‣ -conf P132
‣ conf
‣ P133

9. 5.2 P133
❖ Tool
❖ GenericOptionParser ToolRunner
❖ P135
❖ P135
‣ JVM P136
‣ -conf

10. 5.3 P137
❖ map reduce
‣
‣ OutputCollector
‣
❖ =
❖ JUnit Mockito TDD
‣ MapReduce MRUnit

11. 5.3 P137
❖ MaxTemperatureMapper
‣ Test: 5-4 P137
‣ mapper ver.1 5-5 P138
‣ Test: P138
‣ mapper ver.2 P139
❖ MaxTemperatureReducer
‣ Test: P139
‣ reducer 5-6 P140

12. 5.4 P140
❖
‣ map reducer
‣ Tool
❖ JobRunner
‣ MapReduce
‣

13. 5.4 P142
❖
‣ mapper
-
❖ 5-8 P143
❖ mapper 5-9 P144
❖ TDD

14. 5.4 P144
❖ JobRunner
‣ Tool
‣ 5-10 P145
❖ P145
‣ HDFS MapReduce
‣ Hadoop
‣

15. 5.5 P146
❖
❖
‣ JobClient P147
‣ ID = job_YYYYMMDDHHMM_0000
‣ ID = ID + _m_000000
‣ ID = ID + _0000

16. 5.5 P148
❖ MapReduce Web UI http://jobtracker-host:50030/
‣ jobtracker P148
- Hadoop
-
-
-
-
❖ P150
-
-

17. 5.5 P152
❖
‣ reducer
- MapReduce
‣
‣
‣

18. 5.5 P153
❖
‣ Web UI
‣
‣ MapReduce
❖ Web UI
‣
‣

19. 5.5 P158
❖ P158
‣ P157
‣
❖ Hadoop P157
‣ Chukwa
‣
‣

20. 5.5 P159
❖
‣
‣
‣
‣
‣
‣ IDE

21. 5.6 P161
❖
✓ mapper
✓ reducer
✓ combiner
✓
✓
✓

22. 5.6 P161
❖
‣ JobRunner
P162
❖ HPROF P162
‣
‣
‣

23. 5.7 MapReduce P164
❖ MapReduce
❖
‣ UNIX
❖ MapReduce
Pig Hive

24. 5.7 MapReduce P166
❖
❖ 2
❖
❖
❖
‣ JobControl
‣ Hadoop Workflow Scheduler HWS ※

25. ❖ TDD
‣
‣
‣
❖
❖
❖