2010/9/30の頓智・さんとの勉強会で使ったスライドです

1. Elastic MapReduce
Hadoop EMR

2. • (@sasata299)
• NoSQL
•
• http://blog.livedoor.jp/sasata299/

3. Hadoop

4. etc…

5. •
• EC2 Hadoop & S3
• Cloudera (CDH1)
•
• Hadoop Streaming (Ruby )
•

6. •
• ( )
•
• master ssh
• Hadoop (HADOOP-6254)
• S3 cpu
• S3 → …

7. SocketTimeoutException

9. HADOOP-6254
Elastic MapReduce !!
https://issues.apache.org/jira/browse/HADOOP-6254

10. HADOOP-6254
Cloudera (CDH2) !!
http://archive.cloudera.com/cdh/2/hadoop-0.20.1+169.88.releasenotes.html

12. Elastic Mapreduce
(EMR)

13. • EC2, S3
•
•
• GUI( )

14. • EC2, S3 →
• →
• →
• GUI( ) →CUI
•

15. • EC2, S3 →
• →
• →
• GUI( ) →CUI
•

16. EMR CDH2
AMI
(Amazon Machine
UP Image)
EMR
CDH2

17. EMR CDH2
AMI
(Amazon Machine
UP Image)
EMR
CDH2

18. EMR !!
(eHarmony)

19. …

22. EMR
BootStrap Action
Step (Hadoop Job)
Job Flow ( )

23. EMR
BootStrap Action
Step (Hadoop Job)
Job Flow ( )

24. EMR
BootStrap Action
Step (Hadoop Job)
Job Flow ( )

25. EMR
BootStrap Action
Step (Hadoop Job)
Job Flow ( )

26. ( )
elastic-mapreduce
--create #
--num-instances 10 # master:1 , slave:9
--bootstrap-action s3n://xxx/hoge.sh #
--alive #

27. ( )
elastic-mapreduce
--create #
--num-instances 10 # master:1 , slave:9
--bootstrap-action s3n://xxx/hoge.sh #
--alive #
Created job flow j-8IXS98OW1WEE
ID

28. ( )
elastic-mapreduce
--stream # Hadoop streaming
--input, --output, --mapper, --reducer #
--cache s3n://xxx/fuga.rb #
--jobconf xxx=yyy #
--jobflow j-xxxxx # ID

29. ( )
elastic-mapreduce
--stream # Hadoop streaming
--input, --output, --mapper, --reducer #
--cache s3n://xxx/fuga.rb #
--jobconf xxx=yyy #
--jobflow j-xxxxx # ID

30. •
•
•
• --alive
• AMI
• Cloudera AMI
• BootStrap Action

31. •
• mapred.child.java.opts
• Java
• Streaming
•
•
• ElasticMapReduce-master 5100

32. • EMR
Hadoop
• EMR
•
• --alive