In this talk from the Cloud World Forum Big Data event in June this year, I discuss the benefits of using the AWS Cloud for large scale computation and data processing workloads.

1. ianmas@amazon.com
@IanMmmm
LARGE SCALE DATA
ANALYSIS WITH AWS
Ian Massingham – Technical Evangelist

2. THE MORE DATA YOU COLLECT
THE MORE VALUE YOU CAN
DERIVE FROM IT!

5. THE COST OF DATA
GENERATION IS FALLING!

6. We are constantly producing more data

7. From all types of industries

9. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

10. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Lower cost,
higher throughput

11. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Lower cost,
higher throughput
Highly
constrained

12. + ELASTIC AND HIGHLY SCALABLE
+ NO UPFRONT CAPITAL EXPENSE
+ ONLY PAY FOR WHAT YOU USE
+ AVAILABLE ON-DEMAND
= REMOVE CONSTRAINTS

13. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

14. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
AWS Import / Export
AWS Direct Connect

15. Inbound data transfer is free
Multipart upload to S3
Physical media
AWS Direct Connect

16. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Amazon S3,
Amazon Glacier,
Amazon DynamoDB,
Amazon RDS,
Amazon Redshift,
AWS Storage Gateway,
Data on Amazon EC2

17. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Amazon EC2
Amazon Elastic
MapReduce

18. AMAZON EC2
ELASTIC COMPUTE CLOUD!

24. 3 HOURS
FOR $4828.85/hr!

25. Instead of
$20+ MILLIONS
in infrastructure!

27. SIMULATING
205,000 COMPOUNDS!

29. 18 HOURS
FOR $1833.33/hr!

30. Instead of
$68+ MILLIONS
in infrastructure!

31. GPU INSTANCES"
"
G2"
CG1
1x NVIDIA Kepler GK104
8 vCPU (Intel Xeon E5-2670)
2x NVIDIA Fermi M2050
16 vCPU (Intel Xeon X5570)
$0.65/h
$2.10/h

32. ON A SINGLE INSTANCE
COMPUTE TIME: 4h
COST: 4h x $2.1 = $8.4

33. ON MULTIPLE INSTANCES
COMPUTE TIME: 1h
COST: 1h x 4 x $2.1 = $8.4

38. AMAZON ELASTIC
MAPREDUCE
HADOOP AS A SERVICE!

39. • SPLITS DATA INTO PIECES
• LETS PROCESSING OCCUR
• GATHERS THE RESULTS!

40. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Amazon S3,
Amazon DynamoDB,
Amazon RDS,
Amazon Redshift,
Data on Amazon EC2

41. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

42. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
BATCH
PROCESSING

43. GENERATE ➔ ➔ SHARE!
STREAM
PROCESSING

44. AMAZON KINESIS
REAL-TIME DATA STREAM PROCESSING!

45. Real-time response to content
in semi-structured data streams
Relatively simple computations
on data (aggregates, filters,
sliding window, etc.)

46. Hourly server logs: how your
systems went wrong an hour ago
Weekly / Monthly Bill: What you
spent this past billing cycle
Daily customer report from your
website: tells you what deal or ad
to try next time
Daily fraud reports: tells you if there
was fraud yesterday
Daily business reports: tells me
how customers used AWS services
yesterday
Real-time metrics: what just went
wrong now
Real-time spending alerts/caps:
guaranteeing you can’t overspend
Real-time analysis: what to offer
the current customer now
Real-time detection: blocks
fraudulent use now
Fast ETL into Amazon Redshift:
how are customers using services
now

47. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

48. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Amazon S3,
Amazon DynamoDB,
Amazon RDS,
Amazon Redshift,
Data on Amazon EC2
Amazon EC2
Amazon Elastic
MapReduce
Amazon S3,
Amazon Glacier,
Amazon DynamoDB,
Amazon RDS,
Amazon Redshift,
AWS Storage Gateway,
Data on Amazon EC2
AWS Import / Export
AWS Direct Connect

49. GENERATE ➔ ➔ SHARE!
STREAM
PROCESSING

50. GENERATE ➔ ➔ SHARE!
STREAM
PROCESSING
Amazon S3,
Amazon DynamoDB,
Amazon RDS,
Amazon Redshift,
Data on Amazon EC2
Amazon Kinesis
Stream Processing on
Amazon EC2

51. WANT TO KNOW MORE?
aws.amazon.com/solutions/case-studies/big-data/!

54. ianmas@amazon.com
@IanMmmm
THANK YOU
Ian Massingham – Technical Evangelist

56. Further References
Atomic Fiction Case-Study Video!
https://www.youtube.com/watch?v=ljHo1_5sWxo!
Slideshare with full details on the Schrodinger Materials Science case-study!
http://www.slideshare.net/insideHPC/cycle-computing-recordbreaking-peta-scale-hpc-run!
Real-time Streaming and Querying with Amazon Kinesis and Amazon EMR Video!
https://www.youtube.com/watch?v=NIa33ZwFa8E!
!
!