다양한 Hadoop Eco 기술에 대한 이해

1. BVYdde ?Xd Midgn
gZZehl
.,-2* ,/
-

2. -* BVYdde
BVYdde :
) Cc -553( >dj =jii c ( FjXZcZ ~ ( c .,,,( deZc hdjgXZ
) Cc .,,-( eVX Z HjiX ~ ( QZW =gVlaZg
) - XdgZ( -A< L G( L C> aZkZa - dc 4 YY idiVa -N<
) CcYZm c gViZh 8 -,, eV Zh + hZX
) a b iZY i Z idiVa cjbWZg d eV Zh id -,, b aa dc
) QZW cYZm c w
) 0 w x
) y Y h ­y
CcYZm c *
N Z hidgn d BVYdde .

3. B> M :
eVX Z HjiX 0~
hX ZbVaZhh( YjgVWaZ( VcYa c V ajgZ( VjidbVi Vaan gZWVaVcXZY
Cc .,,/( Add aZ ejWa h ZY i Z Add aZ aZ MnhiZb eVeZg
Cc .,,0( YZkZadeZY HjiX > hig WjiZY aZ MnhiZb
) >jgVWaZ 6 20G WadX (
) BVcYa c V ajgZ 6 z HdYZ
) jidbVi XVaan gZWVaVcXZ 6 HVbZ HdYZ
/

4. B> M :
0iie6++lll*ha YZh VgZ*cZi+bVYk gjh+ Y h).443-012

5. B> M :
1

6. GVeLZYjXZ :
H> M :(
) x (
) xx (
) *
! Cc .,,0( uGVeLZYjXZ6 M bea ZY >ViV JgdXZhh c dc
FVg Z =ajhiZghv
2
GVe
HdYZ
LZYjXZ
GVe

7. GVeLZYjXZ :
3
-* ldgYXdjci GVeLZYjXZ *
.* ( Mea i( ( ( ( …
/* /~ Mea i *
0* bVe … 7 (-9 *
1* … Zn
*
2* gZYjXZ … kVajZ hjb
cVa kVajZ *
3*
<ZVg(. =Vg(/ >ZZg(. L kZg(.
cVa 7 Zn(kVajZ9 w

8. 4
RackN
NodeManager
NodeManager
NodeManager
Rack2
NodeManager
NodeManager
NodeManager
Rack1
NodeManager
NodeManager
NodeManager
C2.1
C1.4
AM2
C2.2 C2.3
AM1
C1.3
C1.2
C1.1
Hadoop Client 1
Hadoop Client 2
create app2
submit app1
submit app2
create app1
ASM Scheduler
queues
ASM Containers
NM ASM
Scheduler Resources
.......negotiates.......
.......reports to.......
.......partitions.......
ResourceManager
status report
GVeLZYjXZ :

9. BVYdde !
.,,2 BVYdde !
) B> M GVeLZYjXZ BVYdde =dgZ a W
) .,,2 S BII ­
.,,3 Nl iiZg( VXZWdd ( F c ZYCc
.,,4 MjW Jgd_ZXiw
) BWVhZ( Tdd ZZeZg( J SV dd ( B kZ VXZWdd q
) =adjYZgV Add aZ( VXZWdd ( SV dd ~
.,,5 bVodcZ GVeLZYjXZ
.,-- Bdgidcldg h SV dd ~
5

10. BVYdde** q
GVeLZYjXZw !!!!
) B> M
) =ajhiZg (
) DdW ( egdXZhh c
) =a Zci JC q!
<ViX :
) KjZgn( AgVe JgdXZhh c
** q:
-,

11. BVYdde** q w !
w :
) **
) ** **
B> M GVeLZYjXZ :
.,-. GVeLZYjXZ P. S LH
--

12. BVYdde .*, :
=ajhiZg …
GVeLZYjXZ w
BVYdde
S LH
-.

13. BVYdde MQw :
-/
N Z BVYdde ?XdhnhiZb NVWaZ
.,-2
-2, ~ MQ
iie6++ VYddeZXdhnhiZbiVWaZ* i jW* d+

14. BVYdde ?Xd r
-0
B> M
S LH
y
HdMKF
MKF dc BVYdde
x
BVYdde)gZaViZY egd_ZXih Vi eVX Z
HZlMKF
x

15. BVYdde ?Xd ( MKF dc BVYdde
-1
w +
!!
:
B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
J ( B kZ( Mi c Zg NZo ( NV_d(
CbeVaV( JgZhid( BVl
Add aZ >gZbZa
.,--
>ViV

16. BVYdde ?Xd ( MKF dc BVYdde
-2
• J .,,4 6 MXgei GVeLZYjXZ
• BkZ .,,4 6 MKF GVeLZYjXZ
• >g aa .,-- 6 B> M B<VhZ(Gdc d><(M/
GVeL
• CbeVaV .,-. 6 y ( ( =adjYZgV
• JgZhid .,-/ 6 _dcw VXZWdd
• B kZ .*, .,-2 6 B kZ NZo =C> ( ~
• NV_d .,-0 6 >Q dc BVYdde( MKF
• MeVg MKF .,-0 6 y
• B QK .,-/ 6 GJJ (
sMKF t MKF)Ic)BVYdde : MKF -,w

17. : y q
-3
B> M
GVeLZYjXZ
J MXg ei
B kZ MKF
B kZ .*, B kZ iZo
>g aa( JgZhid
CbeVaV y
NZo
MeVg kh NZo
NV_d
BVlf
~
MeVg MKF

18. q
-4
y …
y
NV_d( B kZ
w:
MKF
>g aa( CbeVaV( JgZhid( B QK(
B kZ dc NZo
!
MeVg L>>
MeVg MKF L>>( B kZ
y

19. q
-5
MnhiZb HVbZ
Vjai
NdaZgVcXZ
>ncVb X
MX ZYja c
Fd N bZ
KjZgn
Fdl
FViZcXn
eVX Z NV_d I I I I
CbeVaV R R R I
Mig c Zg
B kZ
I I I I
JgZhid R R R I
MKF)dc)BVYdde :

20. MKF dc BVYdde
.,
MKF)Ic)BVYdde y *
78 B> M & WVcYl Yi
-, ( 4~( M N 3.,, geb
-, & 4~ & -,,G<+hZX 8 4A<+hZX
-*1N< B kZ( NV_d( CbeVaV
*
hZaZXi Xdjci & gdb NUN?MN
VXidg " "( "
y"( " " * w w
w w VXidg * B kZ -
NV_d( CbeVaV -*1 p / * -,, ( .,, w
* Q Vi YdZh u-,, i bZh VhiZg i Vc B kZv VXijVaan bZVc:

21. MKF)Ic)BVYdde VXidg
.-
• BVYdde 6 jaa hXVc
• y 6 y gdje Wn( dgYZg Wn
• … 6 dgYZg Wn kZ
gZYjXZ
w *
• 6 L= aZ ( IL= aZ( eVgfjZi
• 6 IL=
*
• 6 (
• y 6

22. BVYdde :
..*** ”
IL= JVgfjZi
Bdgidcldg h
(
DVkV (
L= aZ
=adjYZgV
( (
**
(
McVeen
? X Zci >ViV MidgV Z dg cVani Xh l i eVX Z JVgfjZi .*,

23. K “ **
./
K JVgfjZi WadX h oZw Y h WadX h oZ w : ?LL 6 JVgfjZi aZ
h djaY cdi WZ hea i cid bjai eaZ Y h)WadX h
è B kZ eVgfjZi ( beVaV
è JVgfjZi Y h WadX h oZ
iie6++hiVX dkZg adl*Xdb+fjZhi dch+/,404331+hZi)eVgfjZi)hcVeen)djieji) aZ)h oZ) h) kZ

24. BVYdde ?Xd ( HdMKF :
.0
HdMKF y ~
HdMKF B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
BVYdde ><GM q
Add aZ < NVWaZ
.,-.
B> M >< :
BWVhZ( =VhVcYgV( Gdc d><

25. HdMKF w :
.1Gdc d><( =VhhVcYgV( B<VhZ y
• =VhcVYgV .,,4 6 VXZWdd
• B<VhZ .,,2 6 ><
• XXjbjad .,-- 6 B<VhZ ( ”
• LZYh 6 ><
• ?aVhiXMZVgX .,-, 6 FjXZcZ
• Gdc d>< 6 -, Zc( w ( Ddc w
• >ncVbd>< 6 QM ( ( MM>
y(
) =VX Z LZY h ><
) ZaVhi XhZVgX

26. HIMKF!! = J :
.2HdMKF y !
=dch hiZcXn 6 xx w 8 w y
*
kV aVW a in 6 w 8 w
JVgi i dc NdaZgVcXZ 6 8 HQ
y
HdMKF J w =
=
L><GM
=J
w (
B<VhZ
J
>ncVbd( =VhVcYgV( =djX ><
= J = J NZdgZb HdMKF

27. HIMKF :
B> M
w
) ­y w ) w
HdMKF y ~
.3

28. HdMKF BWVhZ
.4
iie6++lll*ijidg Vahed ci*Xdb+ WVhZ+ WVhZUijidg Va*eY
BWVhZ B> M HdMKF ><

29. HdMKF BWVhZ
.5

30. HdMKF
/,
HdMKF w è u v q :
>< w ( F hi
HdMKF y ~
KjZgn

31. HdMKF :
/-
…
-(
.* 6 8 - 6 -
/* 6 8 - 6 H
0* y
L><GM
… !
HdMKF

32. HdMKF :
/.
EZn)PVajZ *
PVajZ 0w * ><
PVajZw Mig c (
F hi( GVe
LZY h( >ncVbd><(
L V
EZn)PVajZ
PVajZ ~
=dajbc
BWVhZ( =VhVcYgV(
>gj Y( XXjbjad
=dajbc Vb an
PVajZ mba( Dhdc
Gdc d><( =djX ><(
?aVhi XhZVgX
>dXjbZci

33. HdMKF
//
iie6++c.b*Xd* g+c.b.,-,+YdlcadVY+=djX <VhZP-U0*eY
• Dd c Dd c w ( HdMKF
… >< _d c
• HdMKF Dd c w w

34. HdMKF =dajbc
/0

35. HdMKF >dXjbZci
/1
=dajbc
Vban
>dXjbZci

36. K =dajbc >< :
/2>< w: Xdajbc dg ZciZY YWbh
=dgZ
w *
)
)

37. HdMKF MKF dc BVYdde** w :
/3
HdMKF Hdi Ican MKF … ><GM
><
< M?
MKF
HdMKF
L><GM
MKF dc BVYdde
…
= J HdMKF y !
(

38. HdMKF MKF dc BVYdde** !
/4
B kZ MKF FVnZg :
B> M
S LH >ViV IM
B< M? B kZ
MKF =a Zci
CchZgi + OeYViZ >ZZe MKF cVanhh
B<VhZ VcY B kZ r <ZiiZg Nd Zi Zg
ü B< M?
>ViV
ü
ü BkZ
ü
KjZgn

39. HdMKF
w
:
HdMKF ee *!
è ‘ >< ee w *
ee ~ w * ~
P Zl
P Zl
à w
HdMKF
:
J
w =J
/5

40. HIMKF!! = J : < M? :
0,
<Vh XVaan kV aVWaZ 6 kV aVWaZ
Md i)hiViZ 6 w >ViVw Zme gZ
?kZcijVaan Xdch hiZcXn 6 y >ViV
= J
< M?
HdMKF y !
=dch hiZcXn 6 xx w 8 w
y *
kV aVW a in 6 w 8 w
*
JVgi i dc NdaZgVcXZ 6 8
*

41. HIMKF!! = J : < M? :
0-
=C> * (
. … w *
< M? ~ * =C>w ~
( < M? *
=C>( < M? = J
= J /~ .~ *
J ( *
‘( ‘ Jw w =
* = : =dch hiZcXn
?kZcijVaan =dch hiZci ~ *
HdMKF y ~ r ?kZcijVaan =dch hiZci

42. HZl MKF:
0.HZlMKF :
B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
HdMKF
HZlMKF
=C>
Hdc)adX c
h VgZY)cdi c
HdMKF VcY
L><GM …
:
Pdai><( Hjd><( bZbMKF( =ajhig m

43. HZl MKF :
0/
HdMKF L><GM
MKF
=C>
< o HZlMKF ><GM
(
HZlMKF y
HZlMKF LZXdgY FdX c
ad c
<j Zg edda

44. HZl MKF :
00
- MKF
. igVchVXi dch =C> WZ c( gdaaWVX ( Xdbb i
/ Hdc)adX c XdcXjggZcXn Xdcigda
NgVchVXi dc FdX c
Hdc)adX c w *
& ( HZlMKF w
0 B eZg)cdYZ eZg dgbVcXZ
x ><GM *
1 JVgVaaZa( h VgZY)cdi c VgX iZXijgZ
(
w x *

45. HZlMKF
01
GZbdgn
bnMKF
ü Pdai>< AHO ( GZbMKF
ü MXVaZ>< GnMKF hXVaVW a in (
ü Nd j>< AHO GVg V>< hXVaaVW a in
ü =ajhig m> Y hig WjiZY eZZg)id)eZZg MKF YViVWVhZ
ü Hjd>< Y hig WjiZY YViVWVhZ bVcV ZbZci
s-0 2 HZlMKF
=ajhig m( bZbMKF( Hjd><( Mea XZ GVX cZ( NgVch FVii XZ( Pdai><
-2 HdMKF( HZlMKF >ViVWVhZh Nd QViX

46. HZlMKF( HdMKF( L><GM :
02
ü L><GM dg cZlMKF
ü HdMKF dg HZlMKF
ü L><GM
ü HdMKF
ü < o( Cc gV

47. HZlMKF( HdMKF( L><GM :
03

48. HZlMKF :
04
FZVYZg’ IgVXaZ(
G Xgdhd i( M J( C<G H X Z JaVnZg w
…
uIcZ h oZ YdZh cdi i Vaav

49. 05
B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
HdMKF
(
BVYdde
(
q
BVYdde
:
ajbZ( H ( ajZciY( ad hiVh
HZlMKF

50. r w :
1,
• MXgWZ .,,4 6 VXZWdd (
• = j lV .,,4 6 SV dd(
• ajbZ .,-. 6 ( y ?NF
• ajZciY 6 ajbZ eaj c ( ? E w
• Fd hiVh 6 * * w
• H .,-0 6 ‘ HM
• MKIIJ 6 L><GM BVYdde
• EV V 6 Fc ZYCc( GZhhV Z KjZjZ >h
< o Fd ( (
) y EV Vw *
? E 6 ?aVhi XhZVgX ajZciY E WVcV Fd gZ Vidg
FIA
L><
GMA
KjZjZ

51. r :
1-
w
MdjgXZ )9 = VccZa )9 Mc
) MdjgXZ 6
) = VccZa 6
) Mc 6
Jaj c
Jaj c ~
w
) 6 ><( JdgdidXda( MHM( B> M
) 6 ><( KjZjZ(

52. r :
1.
:
JgdidXda
DMIH NZmi RGF
< cVgn DMIH ­y w(
hoZ w( NZmi JVghc
YViV dgbVi
N g i( kgd( JgdidWj ­ (

53. r :
1/
egdidWj w ( i g iw *
PLI
)
) (
i g i
) .,,3( VXZWdd
) ( w *
egdidWj
) .,,4( Add aZ
) w ( w
N g i kh JgdidXda <j Zgh kh kgd ) < VhZY =dbeVg hdc

54. y + MigZVb c
10
B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
HdMKF
higZVb c
:
GZbgdn
?kZci
GL …
) C+I HQ( > h (
MeVg ( Midgb( hoboV
HZlMKF
y

55. y r w :
11
• M0 .,-- 6 SV dd( higZVbc .,-/
• Midgb .,-/ 6 NliiZg( higZVbc (
• MVboV .,-0 6 Fc ZYCc( higZVbc ( >h
• JjahVg .,-1 6 ZWVn( higZVbc =?J( >gjY
• MeVg .,-0 6 >ViV<gX h( L>> + ( GF+MKF
• a c .,-0 6 YViV gihVch( MeVg ( egdXZhh
MeVg a W
y MigZVb c < >ViV cZmi hiZe dg eVX Z a c
a c

56. y w:
12
GL
NZo
MeVg
a c
>ViV
>ViV
MVboVMidgb
JjahVg
B> M( aZ HdMKF
…
MHM( =Vaa(
> A
BVYdde
> A
L>>

57. 13NZo
B> M *
GVe LZXYjXZ *
GVeLZYjXZ kh NZo 6 ~ :

58. 14
6 GZbdgn ( > A
<ViX MigZVb c w G Xgd <ViX
GVeLZYjXZ kh MeVg 6 ~ :

59. 15
MigZVb c < >ViV6 Midgb( MeVg VcY MVboV
6 GZbdgn ( > A
<ViX MigZVb c w G Xgd <ViX
eVX Z Midgb

60. 2,

61. y y
2-

62. y
2.

63. y
2/

64. Qdg adl( Gdc idg c
20
BVYdde
w OC …+
MQ !!!!
:
bWVg ( =adjYZgV GVcV Zg
Ido Z( o VWVc( Fj
BVYdde
>ViV
B> M
S LH
GVeLZYjXZ
MKF dc BVYdde
HdMKF
HZlMKF
y

65. Qdg adl
21
OC
o VWVc
F c ZYCc
BVYdde
o VWVc
Ido Z
RGF
OC
Fj
Jni dc
=dYZ
Fj
Jni dc ~
Qdg adl ?c cZ =dbeVg hdc

66. 22
BVYdde =ajhiZg
=adjYZgV GVcV Zg
=adjYZgV
bWVg
BdgidcQIg h
aVb c d
‘ =adjY cZ
< >ViV

67. 23
BVYdde =ajhiZg bWVg

68. 24
BVYdde =ajhiZg bWVg