Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

AWS Black Belt Online Seminar 2017 Amazon EMR

16.376 visualizaciones

Publicado el

20171101
AWS Black Belt Online Seminar 2017 Amazon EMR

Publicado en: Tecnología
  • Sé el primero en comentar

AWS Black Belt Online Seminar 2017 Amazon EMR

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. w , +1(++( + r4JF 5YNP 5RYa B YV R FRZV N_s 4ZNg 8@E
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • z • w × w w w × • 7MI w 7 Rk _ CH& 7 Rk _ 9I
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. v • – ()/ )) ) n x v v k –4JF Uaa 2 Nd &NZNg &P Z# vk • – x zj 4JF zu j4JF v k • – v k x z n j v k • 7MI U Vd _ e WWVc SZ_UZ_X acZTV bf eVd( 7MI acZTZ_X Zd afS ZT j RgRZ RS V R_U Zd dfS VTe e TYR_XV Z_ RTT cUR_TV hZeY eYV 7MI 9fde Vc 7XcVV V_e RgRZ RS V Re Yeea4))Rhd(R Rk _(T )RXcVV V_e)( 7_j acZTZ_X Z_W c ReZ _ Z_T fUVU Z_ eYZd U Tf V_e Zd ac gZUVU _ j Rd R_ VdeZ ReV W fdRXV TYRcXVd W c 7MI dVcgZTVd SRdVU _ TVceRZ_ Z_W c ReZ _ eYRe j f YRgV ac gZUVU( C _eY j TYRcXVd hZ SV SRdVU _ j fc RTefR fdV W 7MI dVcgZTVd& R_U Rj gRcj Wc eYV VdeZ ReVd ac gZUVU(
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4TR QN • • 7 Rk _ CH • 7 Rk _ CH w , +0)+ $ • 7 Rk _ CH .(i • w 7 Rk _ CH •
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. –~
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E ❏ , 3 w • CRaHVUfTV p + ❏ >RU a × • FZX& >ZgV& >8RdV& FcVde & >fV& >RU a ,(i& IaRc ,(i& Z_ ❏ 7MI w • Ia e& 7C& :j_R :8& LF9& 9 fUMReTY& 9B & AZ_VdZd& ACI → – @N ERQbPR – vm Storage S3 (EMRFS), HDFS YARN Cluster Resource Management Batch MapReduce Interactive Tez In Memory Spark, Flink Applications Hive, Pig, Spark SQL/Streaming/ML, Mahout, Sqoop HBase/Phoenix Presto Hue (SQL Interface/Metastore Management) Zeppelin (Interactive Notebook) Ganglia (Monitoring) HiveServer2/Spark Thriftserver (JDBC/ODBC) Amazon EMR service
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. n S3 EMR EMR Redshift Machine Learning Collect Process Analyze Store Data Collection and Storage Data Processing Event Processing Data Analysis Data Answers Athena Quicksight Kinesis DynamoDB AWS IoT AWS Lambda KCL Apps Glue
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ❏ w ) • CRdeVc)9 cV)JRd _ UV • CH I • 8 edecRa 7TeZ _& IeVad n ❏ w • >RU a& O7HD • >ZgV& FZX& >8RdV& IaRc & Z_ & FcVde & VeT( 4ZNg 8@E
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E n
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Task Node Task Instance Group 4ZNg 8@E n security group security group Master Node Master Instance Group Amazon S3 Amazon DynamoDB Amazon Kinesis Core Node Core Instance Group HDFS HDFS HDFS HDFS Task Node Task Instance Group w >: I 7MI w AWS Glue Data Catalog
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E @N aR_ aN PR :_ b ❏ CRdeVc D UV + • RZ gVc ❏ w • DR VD UV HVd fcTVCR_RXVc • 9 cV D UV JRd D UV Master Node Master Instance Group Hadoop1: JobTracker Hadoop2: ResourceManager HDFS: NameNode Hive: HiveServer2, MetaStore Presto: Coordinator
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E 6 _R aN PR :_ b ❏ + 9 cV D UV ❏ w • D UVCR_RXVc • :ReRD UV w >: I • 8Ip EA HDFS Core Node Core Instance Group Hadoop1: TaskTracker Hadoop2: NodeManager HDFS: DataNode Presto: Worker
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E 6 _R aN PR :_ b ❏ 9 cV D UV • >: I • 9FK)H7C ❏ >: I p • .(+( HDFS Core Node Core Instance Group Hadoop1: TaskTracker Hadoop2: NodeManager HDFS: DataNode Presto: Worker
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E GN aN PR :_ b ❏ >: I 9 cV ❏ >: I w 9 cV D UV ❏ >: I • >: I w :VT ZddZ _$ Task Node Task Instance Group Hadoop1: TaskTracker Hadoop2: NodeManager Presto: Worker
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E GN aN PR :_ b ❏ =c fa • Ia e SZU • _deR_TV JjaV ❏ H p Ia e Task Instance Group 2 Task Instance Group 1 c3.xlarge * 2 bid: $0.1 r3.xlarge * 2 bid: $0.5
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. F a aN PR Task Instance GroupCore Instance Group ❏ ❏ ❖ B QRZN Q 9 cV D UV ❖ ” × E_ UV R_U ❖ F a aN PR JRd D UV ❖ E_ UV R_U 3 w 4 .(i RcXV 0 + +, p/ $ 6 _R A QR B QRZN Q# 4 (-+2o+ Yo0 GN A QR F a# 4 (-+2o/Yo0 % ( .o/Yo0 , +1 ++ + $ w qIa e q 5 +3( 2 + 5 + (1. /
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E l
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 8@E9F2 4ZNg F 79F w ❏ w ❏ • w ❏ w • w ❏ w EMR EMR Amazon S3
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 8@E9F ❏ ld-4))n ” ❏ 7 Rk _ I- • 4 w 7 Rk _ = RTZVc ❏ 7 Rk _ I- w s • 7 Rk _ I- w ❏ w
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 8@E9F 6 V aR a IVRd ❏ 7 Rk _ I- • FKJ HVRU RWeVc hcZeV T _dZdeV_Tj$ • FKJ): B J gV_efR T _dZdeV_Tj$ ❏ CH I 9 _dZdeV_e LZVh • 7 Rk _ :j_R :8 w • BZde 9 _dZdeV_Tj • :V VeV 9 _dZdeV_Tj • HVRU RWeVc hcZeV 9 _dZdeV_Tj • Amazon S3 Amazon DynamoDB CH I w Yeead4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)CR_RXV V_e=fZUV)V c a R_ T _dZdeV_e gZVh(Ye
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E – v ❏ ” • • ❏ w 7 Rk _ I- • w • w t
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E 2 5 a a_N 4PaV ❏ D UV • EA • 8RdY& HfSj& FjeY _& VeT( • 7 Rk _ I- p • 7aRTYV 8ZXe a w w p8 edecRa Yeea4))Rhd(ejaVaRU(T )dR a), +0) 3)SfZ UZ_X R_U UVa jZ_X Tfde Raa ZTReZ _d hZeY RaRTYV 8ZXe a R_U R Rk _ V c(Ye
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E 2 FaR ❏ ) • p • 4 JB >ZgVGB ❏ 7 Rk _ I- KH p p w • I- Rc • IecVR Z_X& >ZgV& FZX& IaRc CH w T R_U cf__Vc( Rc $ • dTcZae cf__Vc( Rc SRdY ❏ IeVa 7fe eVc Z_ReV$
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E y ❏ IeVa • 7 Rk _ CH • w w 9B • 7MI I:A • 7C • ) • ❏ 7MI w • BR SUR • :ReR FZaV Z_V ❏ w Amazon EMR Amazon EMR Step API AWS Data Pipeline Airflow, Luigi, or other schedulers on EC2 AWS Lambda
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E y ❏ w w • CRdeVc D UV ddY p p IIC 7 Rk _ 9, IjdeV d CR_RXVc$ Hf_ T R_U • >ZgVIVcgVc,& daRc dfS Ze • PVaaV Z_& >fV& daRc SdVcgVc =K • BZgj HVde IaRc • E kZV w w Web UIs: Hue SQL editor, Zeppelin notebooks, R Studio, and more! Connect with ODBC / JDBC using HiveServer2/Spark Thriftserver Use Spark Actions in your Apache Oozie workflow to create DAGs of jobs. (start using start-thriftserver.sh) Or, use the native APIs and CLIs for each application
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. C_VcNaR FbO Ra 8@E z ❏ IfS_Ve FcZgReV • CH .(,( ❏ FfS ZT • I-):j_R :8 LF9 V_Ua Z_e • I-):j_R :8 w D7J =M • CH IVcgZTV IVTfcZej =c fa Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +/)+,) Rf_TY R Rk _ V c T fdeVcd Z_ R Rk _ gaT acZgReV dfS_Ved)
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 NPUR F N_ GRg NQ @N ERQbPR C_R a n z Yeea4))U Td(Rhd(R Rk _(T )V c) ReVde)HV VRdV=fZUV)V c UReR V_TcjaeZ _ aeZ _d(Ye ❏ Re cVde pZ_ ecR_dZe p w ❏ Re cVde • CH I& >: Ip p w × w ❏ Z_ ecR_dZe • JBI ❏ CH × • 9 fUW c ReZ _
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E n (). )( #
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E n (). )( # ❏ 7aRTYV Z_ w ❏ w ❏ w w w ❏ 7aRTYV >8RdV w I- ❏ 7 Rk _ 9 fUMReTY gV_ed ❏ w w ❏ 7C w w ❏ Z- dVcZVd 7 Rk _ 9, Z_deR_TVd w ❏ 7 Rk _ CH w
  30. 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 NPUR 9YV n Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)_ h cf_ cVR eZ V decVR ac TVddZ_X Re dTR V hZeY RaRTYV W Z_ _ R Rk _ V c) ❏ HV VRdV /(+( w ❏ HV VRdV /(3( p7aRTYV Z_ +(-(, • w p • IaRc p pIGB • 7 Rk _ AZ_VdZd IecVR d w • w • ViRTe j _TV × • wp × w
  31. 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. n y z Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)R Rk _ V c _ h dfaa ced WZ_V XcRZ_VU RTTVdd T _ec fdZ_X T fdeVc eRXd) ❏ CH w p 7C w – + +( :Vae5: L w p × – + ,( :Vae5: Lp pJVde w pz – , +( :Vae5: L p w – , ,( :VaezLR fV pz ❏ w w 9 fdeVc : – ( :VdTcZSV9 fdeVc pBZde9 fdeVcd
  32. 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. n n n ❏ HV VRdV .(i /(i w ❏ w – HV VRdV /(+( • p p w p w • – HV VRdV /(+( • p w Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)R Rk _ V c _ h dfaa ced Rfe dTR Z_X R_U T _WZXfcRS V dTR V U h_ SVYRgZ c)
  33. 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 NPUR 5N R n F z ❏ HV VRdV /(,( w – w p>8RdV w I- >: I – w I- w ❏ HV VRdV /(1( – >8RdV I- w ❏ KF – CH w w I- – CH I 9 _dZdeV_e LZVh – M7B Z_ V cj W c McZeV – /=8 c Ie cV – Z_ V cj) _ UZd W c HVRU Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)R Rk _ V c _ h dfaa ced fdZ_X R Rk _ d- Rd R UReR de cV W c RaRTYV YSRdV)
  34. 34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 6Y bQJNaPU 8cR a Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)+,)R Rk _ V c _ h dfaa ced VgV_ed R_U _ eZWZTReZ _d hZeY R Rk _ T fUhReTY VgV_ed) ❏ CH – w • IJ7HJ D=& HKDD D=& M7 J D=& J HC D7J :& J HC D7J :QM J>Q HHEHI – w • Wc H I P D= e HKDD D=z & Wc HKDD D= e H I P D=z – w w w • F D: D=& 7JJ79> :& : J79> :& 7 B : – × • F D: D=& 97D9 BQF D: D=& HKDD D=& 9ECFB J :& 97D9 BB :& 7 B :
  35. 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. n n Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +1) -)R Rk _ V c R__ f_TVd Z_deR_TV W VVed W c R Rk _ VT, da e Z_deR_TVd R_U da e S T d) ❏ p CH w ❏ – w p / r × r – – p+ 0 – C9 × g9FK • g9FK p 9, g9FK p CH O7HD D UVCR_RXVc gT cV – w w • w × p w w
  36. 36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +1) -)R Rk _ V c R__ f_TVd Z_deR_TV W VVed W c R Rk _ VT, da e Z_deR_TVd R_U da e S T d) n n +( × ,( -( .( c × /( 8I w 0( 1( g9FK
  37. 37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg V be 4@ n n Yeead4))Rhd(R Rk _(T ) a)RS fe Rhd)hYRed _Vh), +1) 1)R Rk _ V c _ h dfaa ced Rf_TYZ_X T fdeVcd hZeY Tfde R Rk _ Z_fi R Zd) ❏ 7 Rk _ BZ_fi 7 Rk _ CRTYZ_V RXV 7C $ 7 Rk _ CH w ❏ 7C w p w 7C ❏ w /(1( 7 Rk _ CH w
  38. 38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. V n 4ZNg 86 V aN PR n BPa OR_ ) ()/# Yeead4))Rhd(R Rk _(T ) a)RS fe Rhd)hYRed _Vh), +1)+ )R Rk _ V c _ h dfaa ced Z- Z_deR_TVd) ❏ w 7 Rk _ 9, w Z- ❏ Z- pD _ L ReZ V CV cj iacVdd DLCV$ II: w p E E ❏ w /(3( 7 Rk _ CH w
  39. 39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E n Yeead4))Rhd(R Rk _(T ) a)RS fe Rhd)hYRed _Vh), +1)+ )R Rk _ V c _ h dfaa ced aVc dVT _U SZ Z_X) ❏ , +1 + , p 7MI w ❏ w w ❏ w + ❏ p+ + $
  40. 40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E ,&e
  41. 41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E ERYRN R ()- / # • HV VRdV .( ( m , +/) 1 • HV VRdV .(+( m , +/) 3 • HV VRdV .(,( m , +/)++ • HV VRdV .(-( m , +0) + • HV VRdV .(.( m , +0) - • HV VRdV .(/( m , +0) . • HV VRdV .(0( m , +0) . • HV VRdV .(1(i m , +0) 0 1 • HV VRdV .(2(i m , +0) 2 , +1) , • HV VRdV .(3(+ m , +1) - • HV VRdV .(3(, m , +1) 1 7aRTYV 8ZXe a w w × +, +, w >RU a × Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV)V c cV VRdV T a _V_ed(Ye • HV VRdV /( ( m , +0) 2 • HV VRdV /( (- m , +0)+ • HV VRdV /(+( m , +0)++ • HV VRdV /(,(i m , +0)+, • HV VRdV /(-(i m , +1) + , • HV VRdV /(.( m , +1) , • HV VRdV /(/( m , +1) . • HV VRdV /(0( m , +1) 0 • HV VRdV /(1( m , +1) 1 • HV VRdV /(2( m , +1) 2 • HV VRdV /(3( m , +1)+
  42. 42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E ERYRN R ❏ w • 7 Rk _ CH p w w ❏ w • p w ❏ w w p • GfZT 9cVReV • pCR_RXV V_e =fZUV HV VRdV =fZUV 7C w w p http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-release-differences.html
  43. 43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R2 n ❏ w w • w • 4 • jRc_(cVd fcTV R_RXVc(hVSRaa(RUUcVdd a ce 3 ,0 56 2 22 • )Y V)YRU a)YZgV 56 )fdc) ZS)YZgV& )VeT)YZgV)T _W& )gRc) X)YZgV ❏ w w w • YRU a w ” • jRc_ w pYZgV w
  44. 44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R2 n ❏ 9 RddZWZTReZ _ % Fc aVceZVd • 8 edecRa 7TeZ _ • w ” Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV)V c T _WZXfcV Raad(Ye e.g. /etc/hadoop/conf/mapred-site.xml
  45. 45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R2 n j ❏ w w GfZT 9cVReV • p w p × - ” • w = e RUgR_TVU aeZ _d ❏ w • CR_RXV V_e =fZUV • 7 Rk _ CH w w • Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)CR_RXV V_e=fZUV • HV VRdV =fZUV • w w • Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV
  46. 46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. n o-&1&(# Yeea4))U Td(Rhd(R Rk _(T ))V c) ReVde)HV VRdV=fZUV)V c cV VRdV T a _V_ed(Ye
  47. 47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&ey -&e ❏ w • IaRc ,(i • FPNYN &)) • >ZgV ,(i • Z_ • BZgj ❏ • NcN 0 • JVk >ZgV& FZX$ ❏ • IeVa ❏ IR_US i • w • IR_US i
  48. 48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E 4 YVPNaV S _ -&1&( • >RU a& >ZgV& JVk& IaRc & FcVde & FZX& CRY fe& Ib a& Z_ w w • >8RdV& FY V_Zi =K • >fV& PVaaV Z_ w w • E kZV • =R_X ZR& >9ReR X& P VVaVc& YVcf
  49. 49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 YVPNaV 2 ❏ >RU a • qO7HD w $%>: I w $ p qCRaHVUfTV + q CH O7HD p q ❏ >ZgV • O7HD p IGB q CRaHVUfTV qI- w q ❏ JVk • O7HD :7= $ q >ZgV ” p >ZgV q
  50. 50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 YVPNaV 2 ❏ IaRc • w w qO7HD pCRaHVUfTV qIaRc IaRc IGB >ZgV $ IaRc IecVR Z_X w $& CB ZS $ p q ❏ Z_ • w w q O7HD p w p qIaRc p pIGB q w ViRTe j _TV × q ❏ FcVde • IGB qO7HD ” pIGB q w E D qIGB 7DI IGB q
  51. 51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 YVPNaV 2 ❏ FZX • CRaHVUfTV :IB q q ❏ CRY fe • CRaHVUfTV IaRc q q ❏ Ib a • H:8CI p>: I I- w CRaHVUfTV q
  52. 52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 YVPNaV 2 n n ❏ >8RdV • >: I ALIq w w q ❏ FY V_Zi • >8RdV w IGB w q
  53. 53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 YVPNaV 2 :H n n ❏ >fV • >ZgV ×w p =K q ❏ PVaaV Z_ • =K IaRc w IaRc IGB p p s w q ❏ E kZV • w w w qCRaHVUfTV >ZgV q
  54. 54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 YVPNaV 2 ❏ =R_X ZR • w 9FK&CV cj&DVeh c $ p q ❏ >9ReR X • >ZgV ×w FZX w q ❏ P VVaVc • w w pALI p w w q >8RdV $ q
  55. 55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 YVPNaV 2 ❏ BZgj • IaRc CH w H IJ w p pHF9 w w
  56. 56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. n 4ZNg 8@E
  57. 57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E x n ❏ w w • ” w • HV VRdV BRSV )7C LVcdZ _ w p w ❏ w w • 7aRTYV 8ZXe a w w p8 edecRa • w 7C Yeea4))Rhd(ejaVaRU(T )dR a), +0) 3)SfZ UZ_X R_U UVa jZ_X Tfde Raa ZTReZ _d hZeY RaRTYV 8ZXe a R_U R Rk _ V c(Ye
  58. 58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. n 2 NQ ❏ O7HD 5 OVe 7_ eYVc HVd fcTV DVX eZRe c ❏ >RU a, w • SJcRT Vc ❏ HVd fcTV HVbfVde w w • 7aa ZTReZ _CRdeVc p w HVd fcTVCR_RXVc MapReduce Pig Hive Other Other HDFS Other Map ReducePig Hive YARN HDFS Hadoop V1 Hadoop V2
  59. 59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. K4EA2 ER b_PR@N NTR_ ❏ w w ❏ w w • 9FK& CV cj& VeT( ❏ w ❏ • 7aa ZTReZ _CRdeVc Yeea4))YRU a(RaRTYV( cX)U Td)TfccV_e)YRU a jRc_)YRU a jRc_ dZeV)O7HD(Ye
  60. 60. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. K4EA2 A QR@N NTR_ ❏ w w ❏ w w HC ❏ w 9 _eRZ_Vc • 9 _eRZ_Vc Yeea4))YRU a(RaRTYV( cX)U Td)TfccV_e)YRU a jRc_)YRU a jRc_ dZeV)O7HD(Ye
  61. 61. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. K4EA2 6 aNV R_ ❏ w w ❏ DC ❏ • :VWRf e9 _eRZ_Vc iVTfe c • • BZ_fi9 _eRZ_Vc iVTfe c • TXc fad • : T Vc9 _eRZ_Vc iVTfe c • : T Vc Yeea4))YRU a(RaRTYV( cX)U Td)TfccV_e)YRU a jRc_)YRU a jRc_ dZeV)O7HD(Ye
  62. 62. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. K4EA2 4 YVPNaV @N aR_ ❏ 9 _eRZ_Vc • SJcRT Vc • + 7C ❏ 9 _eRZ_Vc Yeea4))YRU a(RaRTYV( cX)U Td)TfccV_e)YRU a jRc_)YRU a jRc_ dZeV)O7HD(Ye
  63. 63. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg 8@E K4EA ❏ CRaHVUfTV& >ZgV& FZX& IaRc _ O7HD • 7 Rk _ CH pO7HD w ❏ × • CH >ZgV IaRc p w
  64. 64. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. n 2 VcR ❏ IGB Z V p w w ❏ w w w p ×w • >: I w w • 7 Rk _ CH CH I 7 Rk _ I-$ • p7 Rk _ AZ_VdZd& 7 Rk _ :j_R :8 ❏ • CRaHVUfTV& JVk& IaRc Yeead4))YZgV(RaRTYV( cX)
  65. 65. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VcR Metastore HDFS Cluster Amazon S3 HiveServer2 CREATE TABLE SELECT FROM +4 ×w >: I 7 Rk _ I- w ,4 ×w w -4 ×w .4 ×w w /4 04 w w Amazon DynamoDB Amazon Kinesis
  66. 66. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VcR @RaN a _R ❏ IGB CVeRde cV ❏ CVeRde cV CjIGB • 7 Rk _ CH CRdeVc D UV w • p7 Rk _ H:I ❏ w 7MI = fV :ReR 9ReR X CVeRde cV CVeRde cV Amazon S3 HDFS
  67. 67. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. v n BE6 9VYR CN_ bRa ❏ w • w • • EaeZ ZkVU H h 9 f _Rc EH9$ Z V ) FRcbfVe ❏ w • + • + p w • J NJ B & I GK D9 B EH9 w Yeead4)) cT(RaRTYV( cX)U Td)daVT Z_ec (Ye
  68. 68. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. n w ❏ • w • w w ❏ w ❏ w p+ w +
  69. 69. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VcR BE6 9VYR CN_ bRa v ❏ ×w ” ❏ CREATE TABLE t ( col1 STRING, … ) STORED AS [ORC/PARQUET]; INSERT INTO t (…); SELECT col1 FROM t;
  70. 70. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VcR 4ZNg V R V n ❏ 7 Rk _ AZ_VdZd IecVR w p>ZgV ×w • 7 Rk _ AZ_VdZd p w w • 4 >ZgV p w ❏ IecVR IYRcU CRaaVc w • 7 Rk _ AZ_VdZd w • 7 Rk _ :j_R :8 $ p ” w IYRcU IYRcU + CRaaVc CRaaVc SELECT FROM 123 id data 1 … 2 … 3 … Amazon Kinesis
  71. 71. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VcR 4ZNg V R V n VcR n 7 Rk _ AZ_VdZd IecVR D # >GB ” q 7 Rk _ AZ_VdZd w
  72. 72. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. n 2 F N_ ❏ w w • p • $ ❏ w w p w • IGB& w & & http://spark.apache.org/
  73. 73. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. F N_ FD 7NaN Ra 4C F N_ & # ❏ IaRc IGB • >ZgVGB p>ZgV CVeRde cV • :89)E:89 • IGB p :ReRdVe):ReR cR V ❏ :ReRdVe 7F • w • ITR R) RgR • :ReR cR V :ReRdVe Yeead4))UReRScZT d(T )S X), +0) 1)+.)R eR V W eYcVV RaRTYV daRc RaZd cUUd UReRWcR Vd R_U UReRdVed(Ye Yeea4))daRc (RaRTYV( cX)U Td) ReVde)db ac XcR Z_X XfZUV(Ye
  74. 74. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ❏ :ZdTcVeZkVU IecVR $ p ❏ HVdZ ZV_e :ZdecZSfeVU :ReRdVe $ w • p ❏ pIaRc ,(, CH HV VRdV /(2( $ pIecfTefcVU IecVR Z_X 7F =7 DStream RDD@T1 RDD@T2 Messages Receiver F N_ Fa_RNZV T i Yeea4))daRc (RaRTYV( cX)U Td) ReVde)decVR Z_X ac XcR Z_X XfZUV(Ye Yeead4))daRc (RaRTYV( cX)U Td) ReVde)decfTefcVU decVR Z_X ac XcR Z_X XfZUV(Ye
  75. 75. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4ZNg V R V F N_ Fa_RNZV T 7 Rk _ AZ_VdZd IecVR p7 Rk _ CH IaRc :IecVR • IaRc 7 Rk _ AZ_VdZd • pAZ_VdZd 9 ZV_e BZScRcjzA9B x p • w p http://spark.apache.org/docs/latest/streaming-kinesis-integration.html IYRcU IYRcU + A9B eYcVRU A9B eYcVRUAmazon Kinesis DStream RDD@T1 RDD@T2
  76. 76. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. F N_ 7f NZVP ER b_PR 4YY PNaV ❏ iVTfe c H:: • >VRa p ❏ :j_R ZT HVd fcTV 7 TReZ _ 5 iVTfe c • IecVR Z_X 7aa ZTReZ _ w • daRc dYV w • .(.( pdaRc (Uj_R ZT7 TReZ _(V_RS VU a_bR • daRc (dYfWW V(dVcgZTV(V_RS VU a_bR https://spark.apache.org/docs/latest/job-scheduling.html
  77. 77. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  78. 78. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ❏ w ) n ❏ w 4ZNg 8@E w p + “ u
  79. 79. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. p q • 7 Rk _ CH CR_RXV V_e =fZUV Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)CR_RXV V_e=fZUV • 7 Rk _ CH HV VRdV =fZUV Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV • 7 Rk _ CH 7F HVWVcV_TV Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)7F ) • 7 Rk _ CH :VgV aVc =fZUV ,(i R_U -(i $ Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde):VgV aVc=fZUV) • 7MI 9B V c Yeea4))U Td(Rhd(R Rk _(T )T Z) ReVde)cVWVcV_TV)V c)Z_UVi(Ye
  80. 80. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. D 4
  81. 81. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. n 7MI w • Yeea4))Rhd(R Rk _(T ) a)Rhd a Z_ec UfTeZ _) 7MI I feZ _d 7cTYZeVTe • p w G 7 • Yeea4))Rhd(ejaVaRU(T )dR a)
  82. 82. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GdVaaR_ 9NPRO 4JF x 3Nd PY bQMW p p p w v u – Uaa 2 &SO&ZR )cE0fJZ
  83. 83. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4JF jx v 7MI p p ” Yeead4))Rhd(R Rk _(T ) a)T _eRTe fd)Rhd dR Vd) 7MI ”
  84. 84. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  85. 85. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 R QVe
  86. 86. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4@ E YR z , +/ 0 - p 7C H V 7C H V , p • CH c V • 7 Rk _ CH p7 Rk _ 9, • 9, Z_deR_TV ac WZ V • 7 Rk _ 9, p7 Rk _ I- Yeea4))Rhd(ejaVaRU(T )dR a), +/) 0)V c R_URe cj ZR faUReV(Ye
  87. 87. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&( w w • >RU a ,(0( • >ZgV +( ( • FZX (+.( • CRY fe (+ ( • IaRc +(.(+ 7aRTYV 8ZXe a w w × • p
  88. 88. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&) w • IaRc +(/( • >fV -(1(+ $ p w • FcVde (++3 • PVaaV Z_ (0 I_RadY e$ • E kZV .( (+
  89. 89. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,& w • IaRc +(/(, • =R_X ZR -(0( $ • FcVde (+,/ • PVaaV Z_ (/(/ • E kZV .(,(
  90. 90. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,& w • >RU a ,(1(+ • IaRc +(0( • =R_X ZR -(1(, • FcVde (+-
  91. 91. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&, w • >9ReR X +( ( $ • CRY fe (++(+ • Ib a +(.(0 $ • FcVde (+-0 • PVaaV Z_ (/(0 IaRc • :j_R ZT R TReZ _ RgR 2 • 7L7Q>EC ”
  92. 92. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&- w • IaRc +(0(+ • >RU a ,(1(, • FcVde (+. CH I • I- II ACI • 7MI ACI 9CA
  93. 93. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&. w • >8RdV +(,( $ m D IGB _ >RU a • P VVaVc -(.(2 $ • FcVde (+.-
  94. 94. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&/&) t zu ,&/&(– w • FY V_Zi .(1( $ m IGB _ >8RdV • JVk (2(- $ m :7= V_XZ_V _ O7HD • >8RdV +(,(+ • CRYRfe (+,( • FcVde (+.1
  95. 95. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&/& w • CRY fe (+,(, • IaRc +(0(, • FcVde (+.2
  96. 96. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&0&( w • >8RdV +(,(, • JVk (2(. • PVaaV Z_ (0(+ • FcVde (+/+
  97. 97. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&0& w • >RU a ,(1(- • FcVde (+/,(-
  98. 98. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&0& w • IaRc +(0(- • FcVde (+/1(+ • P VVaVc -(.(3 • CH :j_R :8 9 __VTe c – .(,( – >ZgV& IaRc
  99. 99. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R ,&1&) w • FjeY _ ,(0 w • w >ZgV • > L 3310 • > L + + 0
  100. 100. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -&(&( w • >ZgV ,(+( • FcVde (+/ • IaRc ,( ( • FPNYN &)) • >fV -(+ ( • FZX (+0( • JVk (2(. • >8RdV +(,(, • PVaaV Z_ (0(+ • NcN 0 • JVk >ZgV& FZX$ • IeVa IR_US i • w • IR_US i
  101. 101. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 85F z j @, 6, n CH 7F 8I • >: I • p • 9 fU c ReZ _ 8I I9+)IJ+ w • >: I C. 9. Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0) ,)R Rk _ V c _ h dfaa ced R Rk _ VSd g f Vd R_U . R_U T. R Rk _ VT, WR Z ZVd)
  102. 102. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 4 NPUR VcR 4 NPUR F N_ 8@E 7f NZ 75 n BFF :j_R :8 :j_R :8 ×w p CH w E D =Ze>fS • Yeead4))XZeYfS(T )Rhd RSd)V c Uj_R US T __VTe c Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0) 3)eYV R Rk _ V c Uj_R US T __VTe c W c RaRTYV YZgV R_U RaRTYV daRc Zd _ h aV_ d fcTV)
  103. 103. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -&(& w • >RU a ,(1(- • FcVde (+/,(- • IaRc ,( (+
  104. 104. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -&)&( w • Z_ +(+(- $ • >8RdV +(,(- • PVaaV Z_ (0(,
  105. 105. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -& &( w • IaRc ,( (, >8RdV _ I- • >8RdV w I-
  106. 106. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -& &) w • FcVde (+/1(+ • P VVaVc -(.(3 • CH :j_R :8 9 __VTe c – .(,( – >ZgV& IaRc
  107. 107. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -& &( w • >ZgV ,(+(+ • >fV -(++( • IaRc ,(+( • E kZV .(-( • Z_ +(+(.
  108. 108. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -& &) w • PVaaV Z_ • P FF B D +2/0 • P FF B D +311
  109. 109. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -&,&( w • FcVde (+00 • FY V_Zi .(3( • >8RdV +(-( • Z_ +(,( • PVaaV Z_ (1(
  110. 110. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. E, n Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +1) -)R Rk _ V c cV VRdV / . R_U dfaa ce W c c. Z_deR_TVd _ h RgRZ RS V) H. • × w w
  111. 111. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -&-&( w • >fV -(+, • FcVde (+1 • B:7F w • PVaaV Z_ (1(+ • P AVVaVc -(.(+
  112. 112. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -&.&( w • >8RdV +(-(+ • IaRc ,(+(+ • Z_ +(,(+ • CRY fe (+-( • IaRc ,(i w FcVde Z_ ecR_dZe
  113. 113. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -&/&( w • Z_ +(-( • FY V_Zi .(++( • PVaaV Z_ (1(, w p 7 Rk _ BZ_fi 7C 5N R • w w C_R a • _ UV(ac aVceZVd K4EA • T _eRZ_Vc X. (ac aVceZVd F • IGEEF ,22 w
  114. 114. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -&0&( w • 7MI I:A +(++(+0 • Z_ +(-(+ • >ZgV ,(-( • IaRc ,(,( • IecfTefcVU IecVR Z_X =7 7MI = fV :ReR 9ReR X • >ZgV IaRc IGB 7MI = fV :ReR 9ReR X >ZgV w B gVR • EEP ,1.2 w bR • >K /2/3 w 5N R • CN $ >8RdV w w w • w RgR CR_RXV V_e ieV_dZ _d
  115. 115. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -&1&( w • 7MI I:A W c RgR +(++(+2- • Z_ +(-(, • >fV .( (+ • >fV D eVS W c IaRc w • FZX (+1( • FcVde (+2. • BZgj (.( Z_TfSReZ_X $ Z- dVcZVd 7 Rk _ 9, Z_deR_TVd w ETe SVc +,& , +1$
  116. 116. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ERYRN R -&1&( F N_ • w p p7fe ITR Z_X XcRTVWf • IF7HA ,+.3. w LR RYV • P FF B D ,-11 w ON R • >87I +2/-- bR • >ZgV bfVcj VUZe c 7MI = fV :ReR 9ReR X w •

×