Алексей Демин (Senior R&D Engineer в Altoros) «Обзор Hadoop-дистрибутивов. Тюнинг «узких мест» Hadoop»

1. © ALTOROS Systems | CONFIDENTIAL
Diomin Aliaksey
R&D
2013, Minsk

2. © ALTOROS Systems | CONFIDENTIAL

3. © ALTOROS Systems | CONFIDENTIAL 3
OpenSource Monitoring Target Group
Apache Hadoop Yes X Developers
Cloudera Yes Good All
Hortonwork Yes Good All
MapR No Bad Enterprise
PivitalHD No Bad Enterprise

4. © ALTOROS Systems | CONFIDENTIAL 4
How to find bottleneck?

5. © ALTOROS Systems | CONFIDENTIAL 5

6. © ALTOROS Systems | CONFIDENTIAL 6

7. © ALTOROS Systems | CONFIDENTIAL

8. © ALTOROS Systems | CONFIDENTIAL 8

9. © ALTOROS Systems | CONFIDENTIAL 9

10. © ALTOROS Systems | CONFIDENTIAL 10

11. © ALTOROS Systems | CONFIDENTIAL 11

12. © ALTOROS Systems | CONFIDENTIAL 12

13. © ALTOROS Systems | CONFIDENTIAL 13
1. Increase size of cluster
2. Increase input block size
3. Increase buffer size

14. © ALTOROS Systems | CONFIDENTIAL 14
1. Increase size of cluster
2. Increase input block size
3. Increase buffer size

15. © ALTOROS Systems | CONFIDENTIAL 15

16. © ALTOROS Systems | CONFIDENTIAL 16

17. © ALTOROS Systems | CONFIDENTIAL 17

18. © ALTOROS Systems | CONFIDENTIAL 18
1. Increase size of cluster
2. Increase input block size
3. Increase buffer size

19. © ALTOROS Systems | CONFIDENTIAL 19

20. © ALTOROS Systems | CONFIDENTIAL 20
1. Increase size of cluster
2. Increase input block size
3. Increase buffer size

21. © ALTOROS Systems | CONFIDENTIAL 21
1. Compression

22. © ALTOROS Systems | CONFIDENTIAL 22
1. Compression
2. Combiner

23. © ALTOROS Systems | CONFIDENTIAL 23
Wordcount
Reduce function as Combine
combine 1: <a, 1> <b, 1> <a, 1> => <a, 2> <b, 1>
combine 2: <a, 1> <b, 1> => <a, 1> <b, 1>
Reduce: <a, {1, 2}> <b, {1, 1}> => <a, 3> <b, 2>

24. © ALTOROS Systems | CONFIDENTIAL 24
Mean
combine 1: <k,40> <k,30> <k,20> => <k, 30>
combine 2: <k,2> <k,8> => <k, 5>
Reduce: <k, {30, 5}> => <k, 17.5>

25. © ALTOROS Systems | CONFIDENTIAL 25
Mean
combine 1: <k,40> <k,30> <k,20> => <k, 30>
combine 2: <k,2> <k,8> => <k, 5>
Reduce: <k, {30, 5}> => <k, 17.5>
(40 + 30 + 20 + 2 + 8)/5 = 17.5

26. © ALTOROS Systems | CONFIDENTIAL 26
Mean
combine 1:
<k,<40,1>> <k,<30,1>>, <k,<20,1>> => <k, <90,3> >
combine 2:
<k,<2,1>> <k, <8,1>> => <k, <10, 2> >
Reduce: <k, {<90,3>, <10,2>} > => <k, 20>

27. © ALTOROS Systems | CONFIDENTIAL 27