This document discusses using Hadoop and HBase on Amazon Web Services (AWS) for distributed data storage and analytics. It introduces Hadoop and describes how AWS Elastic MapReduce simplifies implementing and managing Hadoop clusters. It also discusses using HBase, an open-source NoSQL database, on AWS for scalable access to large, unstructured datasets. Finally, it covers strategies for optimizing costs when running big data workloads on AWS infrastructure.

1. Hadoop & HBase
with Amazon Web Services
Dr. Matt Wood
matthew@amazon.com

2. Thank you.

3. 3
Introducing Hadoop

4. 3
Introducing Hadoop
g
HBase on AWS

5. 3
Introducing Hadoop
g
HBase on AWS
v
Cost optimization

6. Data for competitive advantage.

7. Using data
Customer segmentation,
financial modeling,
system analysis,
line-of-sight,
business intelligence...

8. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

9. Cost of data generation
is falling.

10. lower cost,
increased throughput
Generation
Collection & storage
Analytics & computation
Collaboration & sharing

11. Generation
HIGHLY CONSTRAINED
Collection & storage
Analytics & computation
Collaboration & sharing

12. Very high barrier to turning
data into information.

13. Move from a data generation challenge
to analytics challenge.

14. Enter the AWS Cloud.

15. Remove the constraints.

16. Enable data-driven innovation.

17. Move to a distributed data approach.

18. Maturation of two things.

19. Software for distributed
storage and analysis
Maturation of two things.

20. Software for distributed
storage and analysis
Maturation of two things.
Infrastructure for distributed
storage and analysis

21. Software
Frameworks for data-intensive
workloads.
Distributed by design.

22. Infrastructure
Platform for data-intensive
workloads.
Distributed by design.

23. Support the data life cycle.

24. Generation
HIGHLY CONSTRAINED
Collection & storage
Analytics & computation
Collaboration & sharing

25. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

26. Lower the
barrier to entry.

27. Accelerate time to market and
increase agility.

28. Enable new business
opportunities.

29. Washington Post
Pinterest
NASA

30. “AWS enables Pfizer to explore
difficult or deep scientific questions in
a timely, scalable manner and helps us
make better decisions more quickly”
Michael Miller, Pfizer

31. 3
Introducing Hadoop

32. Software for distributed
storage and analysis
Maturation of two things.
Infrastructure for distributed
storage and analysis

33. Software for distributed
storage and analysis
Maturation of two things.
Infrastructure for distributed
storage and analysis

34. Apache Hadoop
Software for distributed storage and analysis
Implements the map/reduce pattern
Focus on your data

35. Built for uncertainty
Hadoop provides tools to navigate data
Allows discovery
Query flexibility at scale

36. Built for flexibility
Java native
Executes code in any language
Just a distribution mechanism

37. Rich ecosystem
Diverse tools
Machine learning, recommendations, predictive
analytics, segmentation, real time analysis
Lots of innovation

38. But...
A very big project
500k+ lines of code
Challenging to configure and optimize

39. G
Undifferentiated heavy lifting

40. Amazon Elastic MapReduce

41. Amazon Elastic MapReduce
Web service for data processing
Hosted Hadoop
Configured and optimized

42. Amazon Elastic MapReduce
Job flows
Elastic platform
Maintain clusters or run once and terminate
Debugging tools

43. S3
Input data

44. S3
Input data
Code Elastic
MapReduce

45. S3
Input data
Code Elastic Name
MapReduce node

46. S3
Input data
Code Elastic Name
MapReduce node
Elastic
cluster

47. S3
Input data
Code Elastic Name
MapReduce node
HDFS
Elastic
cluster

48. S3
Input data
Code Elastic Name
MapReduce node
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

49. S3
Input data
Code Elastic Name Output
MapReduce node S3 + SimpleDB
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

50. S3
Input data
Output
S3 + SimpleDB

62. Hadoop all the way down
Amazon Hadoop distribution
HDFS
Streaming interface
Hive, Pig, Mahout, Spark, Shark

63. Data integration
Optimized and integrated into AWS environment
Reads and writes to S3
Analytics on DynamoDB data
Can process data from any source:
Cassandra, Mongo, Couch, Amazon RDS

64. Data movement
Multi-part upload
Import/Export
AWS Direct Connect
Aspera

65. Cluster scalability
Resize running job flows
Add capacity for shorter runs
Remove capacity during off peak hours
Balance scale and cost

66. Cluster scalability
14 hours remaining

67. Cluster scalability
7 hours remaining

68. Cluster scalability
3 hours remaining

69. Cluster scalability
Steady state Steady state
Large batch task

70. Cluster availability
Canonical source of data
Any one in the engineering team
IAM integration
Monitoring

71. Click stream analysis for retail
3.5 billion records
71 million unique cookies
1.7 million targeted ads
13 Tb of clickstream logs
Each day

72. Click stream analysis for retail
Workflow time from 2 days to 8 hours
Procurement time from 2 months to 5 minutes
$13k per month
500% increase return on advertising spend

74. Log data stored in Amazon S3
Amazon S3 Months of user click-through data
Search terms
Ads displayed
Premium listing inventory

75. Elastic Map Reduce spins up
200 instance cluster
Hadoop Cluster
Amazon S3 Amazon EMR

76. Find patterns across logs. Write results to S3.
Hadoop Cluster
Amazon S3 Amazon EMR

77. Hadoop in the AWS Cloud
Elastic MapReduce for hosted Hadoop
Optimized, configured, ready to roll
Focus on the business benefit of data
Hadoop all the way down

78. Software for distributed
storage and analysis
Maturation of two things.
Infrastructure for distributed
storage and analysis

79. g
HBase on AWS

80. Vibrant ecosystem
Mahout for machine learning
Mesos for cluster management
Spark for fast analytics
HBase for unstructured data

81. HBase
NoSQL data store
Runs on top of HDFS
Scalable
Rapid retrieval across large datasets

82. Architecture
Huge, distributed map/hash
Distributed
Implements Bloom filters
Sortable

83. Column based
Columns are similar to fields
Rows are records

84. Built for data
Built to scale across billions of rows
The more data, the better the relative
performance

85. But...
Large, complex project
Running in production can be challenging
Distributed system

86. G
Undifferentiated heavy lifting

87. HBase for Elastic MapReduce

96. Using HBase
Social media firehose
Customer information
Usage and application logs
Hadoop analytics

97. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

98. Amazon DynamoDB
NoSQL database service
Provisioned throughput
Unlimited storage
Very easy to use

99. DynamoDB & Amazon EMR
SQL like queries
Query flexibility at scale
Integrate queries across datasets
Hive

100. NoSQL on the AWS Marketplace
CouchDB
Cassandra
MongoDB
aws.amazon.com/marketplace

101. v
Cost optimization

102. Lowered prices 19 times in the past six years.

103. On-demand

104. Reserved capacity

105. 100%
Reserved capacity

106. 100%
On-demand
Reserved capacity

107. 100%
On-demand
Reserved capacity

108. Spot market

112. $0.08 vs $0.007
(yesterday evening)

115. Reserved Instance Marketplace

117. 3
Introducing Hadoop
g
HBase on AWS
v
Cost optimization

118. B
aws.amazon.com/elasticmapreduce

119. Thank you
aws.amazon.com/rds
matthew@amazon.com
@mza