1. Glenn Gore
Big Data Analytics
Sr. Manager, Solutions Architects, AWS

2. Overview
• The Big Data Challenge
• Big Data tools and what can we do with them ?
• Packetloop – Big Data Security Analytics
• Intel technology on big data.

3. An engineer’s definition
When your data sets become so large that you have to start
innovating how to collect, store, organize, analyze and
share it

4. Generation
Collection & storage
Analytics & computation
Collaboration & sharing

5. Generation
Collection & storage
Analytics & computation
Collaboration & sharing
Lower cost,
higher throughput

6. Generation
Collection & storage
Analytics & computation
Collaboration & sharing
Lower cost,
higher throughput
Highly
constrained

7. Generated data
Available for analysis
Data volume
Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011
IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares

8. Amazon Web Services helps remove
constraints

9. Remove constraints = More experimentation
More experimentation = More innovation
More Innovation = Competitive edge

10. Elastic MapReduce and Redshift
Big Data tools

11. EMR is Hadoop in the Cloud

12. What is Amazon Redshift ?
Amazon Redshift is a fast and powerful, fully managed,
petabyte-scale data warehouse service in the AWS
cloud
Easy to provision and scale
No upfront costs, pay as you go
High performance at a low price
Open and flexible with support for popular BI tools

13. Elastic MapReduce and Redshift
Big Data tools

14. How does EMR work ?
EMR
EMR Cluster
S3
Put the data
into S3
Choose: Hadoop distribution, # of
nodes, types of nodes, custom
configs, Hive/Pig/etc.
Get the output from
S3
Launch the cluster using the
EMR console, CLI, SDK, or
APIs
You can also store
everything in HDFS

15. What can you run on EMR…
S3
EMR
EMR Cluster

16. EMR
EMR Cluster
Resize Nodes
S3
You can easily add and
remove nodes

17. Resize Nodes with Spot Instances
Cost without Spot
10 node cluster running for 14 hours
Cost = 1.2 * 10 * 14 = $168

18. Resize Nodes with Spot Instances
Cost without Spot Add 10 nodes on spot
10 node cluster running for 14 hours
Cost = 1.2 * 10 * 14 = $168
20 node cluster running for 7 hours
Cost = 1.2 * 10 * 7 = $84
= 0.6 * 10 * 7 = $42

19. Resize Nodes with Spot Instances
Cost without Spot Add 10 nodes on spot
10 node cluster running for 14 hours
Cost = 1.2 * 10 * 14 = $168
20 node cluster running for 7 hours
Cost = 1.2 * 10 * 7 = $84
= 0.6 * 10 * 7 = $42
= Total $126
25% reduction in price
50% reduction in time

20. Ad-Hoc Clusters – What are they ?
EMR Cluster
S3
When processing is complete, you
can terminate the cluster (and stop
paying)
1

21. Ad-Hoc Clusters – When to use
EMR Cluster
S3
Not using HDFS
Not using the cluster 24/7
Transient jobs
1

22. EMR
EMR Cluster
“Alive” Clusters – What are they ?
S3
If you run your jobs 24 x 7 , you
can also run a persistent cluster
and use RI models to save costs
2

23. EMR
EMR Cluster
“Alive” Clusters – When ?
S3
Frequently running jobs
Dependencies on map-reduce-map
outputs
2

24. S3 instead of HDFS
S3
EMR
EMR Cluster
• S3 provides 99.99999999999% of
durability
• Elastic
• Version control against failure
• Run multiple clusters with a single
source of truth
• Quick recovery from failure
• Continuously resize clusters
3

25. S3 and HDFS
S3
EMR
EMR Cluster
Load data from S3 using S3DistCP
Benefits of HDFS
Master copy of the data in S3
Get all the benefits of S3
HDFS
S3distCP
4

26. Elastic MapReduce and Redshift
Big Data tools

27. Reporting Data-warehouse
RDBMS
Redshift
OLTP
ERP
Reporting
and BI
1

28. Live Archive for (Structured) Big Data
DynamoDB
Redshift
OLTP
Web Apps Reporting
and BI
2

29. Cloud ETL for Big Data
Redshift
Reporting
and BI
Elastic MapReduce
S3
3

30. Streaming Hive Pig DynamoDB Redshift
Unstructured Data ✓ ✓
Structured Data ✓ ✓ ✓ ✓
Language Support Any* HQL Pig Latin Client SQL
SQL ✓SQL-Like ✓
Volume Unlimited Unlimited Unlimited Relatively
Low
1.6 PB
Latency Medium Medium Medium Ultra Low Low

31. Collection & storage
Analytics & computation
Collaboration & sharing
Remove
Constraints
Generation