Big data technologies let you work with any velocity, volume, or variety of data in a highly productive environment. Join the General Manager of Amazon EMR, Peter Sirota, to learn how to scale your analytics, use Hadoop with Amazon EMR, write queries with Hive, develop real world data flows with Pig, and understand the operational needs of a production data platform.

1. Scaling Your Analytics
with Amazon Elastic MapReduce
Peter Sirota, General Manager - Amazon Elastic MapReduce
November 14, 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. Agenda
• Amazon EMR: Hadoop in the cloud
• Hadoop Ecosystem on Amazon EMR
• Customer Use Cases

3. Hadoop is the right system for Big Data
• Scalable and fault tolerant
• Flexibility for multiple languages
and data formats
• Open source
• Ecosystem of tools
• Batch and real-time analytics

4. Challenges with Hadoop
On Premise
On Amazon EC2
• Manage HDFS, upgrades,
and system administration
• Pay for expensive support
contracts
• Select hardware in
advance and stick with
predictions
• Difficult to integrate with
AWS storage services
• Independently manage
and monitor clusters

5. Amazon EMR is the
easiest way to run Hadoop in the cloud

6. Why Amazon EMR?
•
•
•
•
Managed services
Easy to tune clusters and trim costs
Support for multiple data stores
Unique features and ecosystem support

7. S3, DynamoDB, Redshift
Input data

8. S3, DynamoDB, Redshift
Input data
Code
Elastic
MapReduce

9. S3, DynamoDB, Redshift
Input data
Code
Elastic
MapReduce
Name
node

10. S3, DynamoDB, Redshift
Input data
Code
Elastic
MapReduce
Name
node
S3/HDFS
Elastic
cluster

11. S3, DynamoDB, Redshift
Input data
Code
Elastic
MapReduce
Name
node
Queries
+ BI
S3/HDFS
Via JDBC, Pig, Hive
Elastic
cluster

12. S3, DynamoDB, Redshift
Input data
Code
Elastic
MapReduce
Output
Name
node
Queries
+ BI
S3/HDFS
Via JDBC, Pig, Hive
Elastic
cluster

13. S3, DynamoDB, Redshift
Input data
Output

14. Elastic clusters
Customize size and type to reduce costs

15. Choose your instance types
Try out different configurations to find your
optimal architecture
CPU
c1.xlarge
cc1.4xlarge
cc2.8xlarge
Memory
m1.large
m2.2xlarge
m2.4xlarge
Disk
hs1.8xlarge

16. Long running or transient clusters
Easy to run Hadoop clusters short-term or 24/7, and
only pay for what you need
=

17. Resizable clusters
Easy to add and remove compute
capacity on your cluster
10 hours

18. Resizable clusters
Easy to add and remove compute
capacity on your cluster
6 hours

19. Resizable clusters
Easy to add and remove compute
capacity on your cluster
Peak capacity

20. Resizable clusters
Easy to add and remove compute
capacity on your cluster
Matched compute
demands with cluster sizing
10 hours

21. Use Spot and Reserved Instances
Minimize costs by supplementing on-demand pricing

22. Easy to use Spot Instances
Name-your-price supercomputing to minimize costs
Spot for
task nodes
On-demand for
core nodes
Up to 90%
off Amazon
EC2
on-demand
pricing
Standard
Amazon EC2
pricing for
on-demand
capacity

23. 24/7 clusters on Reserved Instances
Minimize cost for consistent capacity
Reserved
Instances for
long running
clusters
Up to 65% off
on-demand
pricing

24. Your data, your choice
Easy to integrate Amazon EMR with your data stores

26. Using Amazon S3 and HDFS
Data aggregated
and stored in
Amazon S3
Ad-hoc Query
Long running EMR cluster
holding data in HDFS for
Hive interactive queries
Weekly Report
Data Sources
Transient EMR cluster
for batch map/reduce jobs
for daily reports

27. Use Amazon EMR with Amazon Redshift
and Amazon S3
Processed data
loaded into
Amazon Redshift
data warehouse
Daily data
aggregated in
Amazon S3
Data Sources
Amazon EMR
cluster used to
process data

28. Use the Hadoop Ecosystem
on Amazon EMR
Leverage a diverse set of tools to get the most out of your data

29. Hadoop 2.x
•
•
•
•
•
and much more...
Databases
Machine learning
Metadata stores
Exchange formats
Diverse query languages

30. Use Hive on Amazon EMR to interact with
your data in HDFS and Amazon S3
• Data warehouse for Hadoop
• Integration with Amazon S3 for
better performance reading and
writing to Amazon S3
• SQL-like query language to make
iterative queries easier
• Easy to scale in HDFS on a
persistent Amazon EMR cluster

31. Use HBase on a persistent Amazon EMR cluster
as a column-oriented scalable data store
• Billions of rows and millions
of columns
• Backup to and restore from
Amazon S3
• Flexible datatypes
• Modulate your HBase tables
when adding new data to
your system

32. Use ad-hoc queries on your cluster to
drive insights in real-time
Spark / Shark
• In-memory MapReduce
for faster queries
• Use HiveQL to interact
with your data

33. Use ad-hoc queries on your cluster to
drive insights in real-time
Spark / Shark
• In-memory MapReduce
for faster queries
• Use HiveQL to interact
with your data
Impala (coming soon!)
• Parallel database
engine for Hadoop
• Use SQL to query data
in HDFS on your cluster
in real-time

34. “Hadoop-as-a-Service [Amazon EMR] offers a
better price-performance ratio [than bare-metal Hadoop].”
1. Elastic clusters and cost optimization
2. Rapid, tuned provisioning
3. Agility for experimentation
4. Easy integration with diverse datastores

35. Diverse set of partners to build on Amazon EMR
BI / Visualization
Hadoop Distribution
Monitoring
Business Intelligence
Data Transformation
Data Transfer
Performance Tuning
Available on AWS Marketplace
BI / Visualization
ETL Tool
Graphical IDE
Available as a distribution in Amazon Elastic MapReduce
BI / Visualization
Encryption
Graphical IDE

36. Thousands of customers

37. How Netflix scales Big Data Platform on
Amazon EMR
Eva Tse, Director of Big Data Platform, Netflix
November 14, 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

38. Hadoop ecosystem as
our Data Analytics platform
in the cloud

39. How we got here?

42. How do we scale?

44. Separate compute and
storage layers

45. Amazon S3 as our DW

46. S3
Source
of
truth

47. S3
S3mper-enabled
Source
of
truth

48. Multiple clusters

49. Ad hoc
SLA
zone y
zone x
S3
Source
of
truth

50. Ad hoc
SLA
Bonus
zone x
Bonus
zone y
S3
Source
of
truth
Bonus
zone z

51. Unified and global big data
collection pipeline

52. Events Pipeline
SLA
cloud
apps
Suro
Ursula
S3
Bonus
Source
of
truth
Aegisthus
Dimension Pipeline
Adhoc

53. Innovate – services and tools

54. Sting
CLIs
Gateways

55. Putting into perspective …
•
•
•
•
•
Billions of viewing hours of data
~3000 nodes clusters
Hundred billion events / day
Few petabytes DW on Amazon S3
Thousands of jobs / day

56. Adhoc querying

57. Simple Reporting

58. E
E
T
T
L
T
L

59. Analytics and statistical modeling

61. Open Connect

62. What works for us?
Scalability

63. What works for us?
Hadoop integration on Amazon EC2 / AWS

64. What works for us?
Let us focus on innovation and build a solution

65. What works for us?
Tight engagement with Amazon EMR & Amazon
EC2 teams for tactical issues and strategic
roadmap

66. Next Steps …
• Heterogeneous node cluster
• Auto expand shrink
• Richer monitoring infrastructure

67. We strive to build the best of class
big data platform in the cloud

68. Big Data at Channel 4
Amazon Elastic MapReduce for Competitive Advantage
Bob Harris – Channel 4 Television
14th November 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

69. Channel 4 – Background
•
Channel 4 is a public service, commercially funded, not-for-profit, broadcaster.
•
We have a remit to deliver innovative, experimental, distinctive, and diverse
content across television, film, and digital media.
•
We are funded predominantly by television advertising, competing with the other
established UK commercial broadcasters, and increasingly with emerging,
Internet based, providers.
•
Our content, is available across our portfolio of around 10 core and time-shift
channels, and our on demand service 4oD is accessible across multiple devices
and platforms.

70. Why Big Data at C4

71. Business Intelligence at C4
•
Well established Business Intelligence capability
•
Based on industry standard proprietary products
•
Real-time data warehousing
•
Comprehensive business reporting
•
Excellent internal skills
•
Good external skills availability

72. Big Data Technology at C4
•
2011 - Embarked on Big Data initiative
–
–
•
2012 - Ran Amazon EMR in parallel with
conventional BI
–
–
•
Ran in-house and cloud-based PoCs
Selected Amazon EMR
Hive deployed to Data Analysts
Amazon EMR workflows deployed to production
2013 – Amazon EMR confirmed as primary Big Data
platform
–
–
Amazon EMR usage growing, focus on automation
Experimenting with Mahout for Machine Learning

73. What problems are we solving?
Single view of the viewer
recognising them across
devices and serving
relevant content
Personalising the viewer experience

74. How are we doing this?
• Principal tasks…
– Audience segmentation
– Personalisation
– Recommendations
• What data do we process…
–
–
–
–
Website clickstream logs
4oD activity and viewing history
Over 9m registered users
Majority of activity now from “logged-in” users

75. High-Level Architecture

76. High-Level Architecture
•
Amazon EMR and existing BI technology are
complementary
•
Process billions of data rows in Amazon EMR,
store millions of result rows in RDBMS
•
No need to “rip and replace”, existing technology
investment is protected
•
Amazon EMR will continue to underpin major
growth in data volumes and processing
complexity

77. Where Next?
• Continued growth in usage of Amazon EMR
• Migrate to Hadoop 2.x
• Adopt Amazon Redshift
• Improved integration between C4 and AWS
• Shift toward “near real-time” processing

78. Please give us your feedback on this
presentation
BDT301
As a thank you, we will select prize
winners daily for completed surveys!