Big Data Analysis in the Cloud - AWS Korea (정윤진, Solutions Architect)

1. Cloud

2. Thank you

3. In the next 30 minutes
1
3
What is big data
Big data on AWS
How customers using AWS
2

5. Where is this data coming from ?

6. Human generated
Machine generated
Tweet
Surf the internet
Buy and sell products
Upload images and videos
Play games
Check in at restaurants
Search for cafes
Find deals
Watch content online
Look for directions
Use social media

7. Human generated
Machine generated
Networks and security
devices
Mobile phones
Cell phone towers
Smart grids
Smart meters
Telematics from cars
Sensors on machines
Videos from traffic and
security cameras

8. What is it used for ?

9. Data for competitive
advantage

10. Data for competitive
advantage
Customer Segmentation
Financial modeling,
System analysis,
Line-of-sight,
Replacing Human decisions
Business intelligence..

11. Data for competitive
advantage
Customer Segmentation
Financial modeling,
System analysis,
Line-of-sight,
Replacing Human decisions
Business intelligence..
Innovating new business and
revenue models

12. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation

13. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation
lower cost,
increased
throughput

14. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation
lower cost,
increased
throughput
constraint

15. Very high barrier to
turning data into
information…

16. Very high barrier to
turning data into
information.
Infrastructure capacity
Technical Skills
Questions to ask
Cheap experimentation

17. Amazon Web Services Cloud

18. Elastic and highly scalable
No upfront capital expense
Only pay for what you use
+
+
Available on-demand
+
=
Remove
constraints

19. Remove constraints = More experimentation
More experimentation = More innovation
More Innovation = Competitive edge

20. Amazon Web Services
Removes constraints
Focus on your data
Leave undifferentiated heavy lifting to us

21. HOW

22. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation

23. 26

24. AWS CloudCorporate Data center
Virtual Private Cloud
VPN
Internet
Direct Connect
Storage Gateway
AWS Import/Export
S3 EMR RedShift
How to move your data into AWS

25. AWS
Import/Export
Corporate
data center
Amazon
Elastic
MapReduce
Amazon
Simple
Storage
Service (S3)
BI Users
Clickstream data
from 500+
websites and VoD
platform

26. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation

27. More than 25 Million Streaming Members
50 Billion Events Per Day
30 Million plays every day
2 billion hours of video in 3
months
4 million ratings per day
3 million searches
Device location , time ,
day, week etc.
Social data

28. 10 TB of streaming data per day

29. What is S3?
Highly scalable data storage
Access via APIs
Fast
(850K requests
per sec)
Highly available & durable
(99.999999999% Durability
Economical
($0.095 per GB)*
Web store

31. Velocity of data
Amazon Dynamodb

32. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation

33. “Who buys video games?”

34. 3.5 billion records
13 TB of click stream logs
71 million unique cookies
Per day:

35. 500% return on ad spend
17,000% reduction in
procurement time
Results:

38. What is EMR?
Map-Reduce engine Integrated with tools
Hadoop-as-a-service
Massively parallel
Cost effective AWS wrapper
Integrated to AWS services

39. +
Source: http://nerds.airbnb.com/redshift-performance-cost
Table Size Query type Hive Redshift
3 billion
rows
Simple range
query
1680
seconds (28
min)
360 seconds
(6 min)
1 million
rows
2 complex
joins
182 seconds 8 seconds
$13.60/hour on Redshift versus $57/hour on
HIVE

40. Every day is crucial and costly

41. Challenge: To run a virtual screen with a higher
accuracy algorithm & 21 million compounds

43. Metric Count
Compute Hours of
Work
109,927 hours
Compute Days of
Work
4,580 days
Compute Years of
Work
12.55 years
Ligand Count ~21 million ligands
Using Cycle Computing and Amazon
Web Services

44. 3 Hours
for $4828.85/hr

45. Instead of $20+
Million in
Infrastructure

46. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation

47. Open web index.
3.4 billion records.
Available to all.
1000 Genomes
project

49. Generation
Collect
Store
Collaboration & sharing
Analysis and Computation

50. Game instances
DB instances Proxy farms
Amazon EMR
Amazon
Glacier
Amazon
RedShift
Amazon
DynamoDB
Game traffic Analysis
Users
Sample architecture

51. Thank you! aws.amazon.com/big-data
younjin@amazon.com
May 21st, COEX Intercontinental, Seoul
One day Free training
Walk through of services
http://aws.amazon.com/apac/awsday/seoul/