大數據與企業應用專題 From Batch to Streaming: How Amazon Flex Uses Real-time Analytics (Level 200) Speaker: Yubo Wang, BDM, AWS

1. Big Data Analytics Evolvement and Innovation
From Batch to Streaming
Yubo Wang
Product Mgr, GCR
Time 16:00-16:40
Level 200

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Real-time streaming data overview
• Streaming data services
• Benefits of streaming analytics
• Batch to streaming best practices
• How Amazon Flex moved from batch to streaming

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What is batch processing?
Execution of a series of jobs in a program on a
computer without manual intervention - Wikipedia
• Data is collected over a period of time
• Process and analyze on a schedule
• Combine several processes to obtain final result

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What Is Real-time Data?
Mobile Apps Web Clickstream Application Logs
Metering Records IoT Sensors Smart Buildings
[Wed Oct 11 14:32:52
2000] [error] [client
127.0.0.1] client denied
by server configuration:
/export/home/live/ap/htdo
cs/test

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Hourly server logs
• Weekly or monthly bills
• Daily web-site clickstream
• Daily fraud reports
• Real time metrics
• Real time spending alerts/caps
• Real time clickstream analysis
• Real time detection
It’s All About the Pace
Batch Processing Stream Processing

6. The diminishing value of data
Recent data is highly valuable
• If you act on it in time
• Perishable insights (M. Gualtieri,
Forrester)
Old + recent data is more
valuable
• If you have the means to combine
them

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Simple Pattern for Streaming
Data
Continuously creates
data
Continuously writes
data to a stream
Can be almost anything
Data Producer
Durably stores data
Provides temporary
buffer that preps data
Supports very high-
throughput
Streaming Service
Continuously processes
data
Cleans, prepares, &
aggregates
Transforms data to
information
Data Consumer
Mobile Client Amazon Kinesis Amazon Kinesis app

8. Processing real-time, streaming data
• Durable
• Continuous
• Fast
• Correct
• Reactive
• Reliable
What are the key requirements?
Collect Transform Analyze React Persist

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Analytics
Amazon Kinesis
Data Firehose
Build custom
applications that process
and analyze streaming
data
Easily process and
analyze streaming data
with standard SQL
Easily load streaming
data into AWS

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Streams
• Easy administration and low cost
• Build real time applications with framework of choice
• Secure, durable storage

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Analytics
• Powerful real time applications
• Easy to use, fully managed
• Automatic elasticity

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Firehose
• Zero administration and seamless elasticity
• Direct-to-data store integration
• Serverless, continuous data transformations
Amazon S3
Amazon Redshift

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Analytics Applications
Easily write SQL code to process streaming data
Connect to streaming source
Continuously deliver SQL results

14. Video is critical to many applications
Smart Home Security MonitoringSmart City
Industrial Automation Computer Vision

15. Unlocking Real-Time Video Analytics

16. Find the Red Car

17. Amazon Kinesis—Real-Time Analytics
Easily collect, process, and analyze video and data streams in real time
Capture, process,
and store video
streams for analytics
Load data streams
into AWS data stores
Analyze data streams
with SQL
Build custom
applications that
analyze data streams
Kinesis Video Streams Kinesis Data Streams Kinesis Data Firehose Kinesis Data Analytics
New at re:Invent 2017

18. Stream video from millions of devices
Easily build vision-enabled apps
Secure
Durable, searchable storage
Fully managed
Amazon Kinesis Video Streams
S t r e a m v i d e o a n d t i m e - e n c o d e d d a t a f o r a n a l y t i c s
Amazon AI
Services
Apache MxNet
TensorFlow
Custom Video
Processing
3rd Party
Partners
Kinesis Video
Streams

19. Use case: Smart Home
Example: Pet Monitor

20. Use case: Smart City
Example: Amber Alert System

21. Use case: Industrial Automation
Example: Equipment Preventive Maintenance

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of streaming analysis
Immediate results
• Real-time
aggregations
• Filtering
• Anomaly detection
Reduced
complexity
• Fewer scheduled
jobs to manage
• Kinesis is a fully-
managed solution
Scalable
• Enables parallel
processing
• Horizontally scales,
based on your
ingest rate

23. Real-Time Analytics Core Use
CasesScenarios/
Verticals
Accelerated Ingest-
Transform-Load
Continuous Metrics
Generation
Machine Learning and
Actionable Insights
Digital Ad
Tech/Marketing
Publisher, bidder data
aggregation
Advertising metrics like
coverage, yield, and
conversion
User engagement with ads,
optimized bid/buy engines
IoT Sensor, device telemetry
data ingestion
Operational metrics and
dashboards
Device operational
intelligence and alerts
Gaming Online data aggregation,
e.g., top 10 players
Massively multiplayer online
game (MMOG) live
dashboard
Leader board generation,
player-skill match
Consumer
Online
Clickstream analytics Metrics like impressions and
page views
Recommendation engines,
proactive care
Operation
Security
DevOps tools, ingesting
VPC Flow Logs
Subscribe to Amazon
CloudWatch Logs and
analyze logs in real-time
Anomaly detection

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Streaming
Ingest-
Transform-
Load
• Continuous
Metric
Generation
• Actionable
Insights
Three Common Scenarios
Compute analytics as the data is generated
React to analytics based off of insights
Deliver data to analytics tools faster and
cheaper

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Web Analytics and
Leaderboards
Amazon
Kinesis Data
Analytics
AWS
Lambda
function
Amazon
Cognito
Lightweight JS
client code
Web Server on
Amazon EC2
Instance
OR
Amazon
DynamoDB
Table
Amazon
Kinesis Data
Streams
Compute top 10 usersIngest web app data Persist to feed live apps

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring IoT Devices
IoT sensors AWS IoT
Amazon
RDS
MySQL DB
instance
Amazon
Kinesis
Data
Streams
Amazon
Kinesis
Data
Analytics
AWS
Lambda function
Compute avg temp
every 10 sec
Ingest sensor data
Persist time series
analytic to database

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Analyzing CloudTrail Event Logs
AWS
CloudTrail
Amazon
CloudWatch
events trigger
AmazonKinesis
DataAnalytics
AWS Lambda
function
Amazon S3
bucket for raw
data
Amazon S3 bucket
for processed data
Amazon
DynamoDB
Table(s)
Chart.JS
Dashboard
Compute
operational metrics
Ingest and deliver raw
log data
Deliver to a real time
dashboards and archival
AmazonKinesis
DataFirehose
AmazonKinesis
DataFirehose

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Batch to streaming best practices
Migrate incrementally
• Don’t boil the ocean
• Begin by streaming data
in parallel to existing
batch processes
• Persist streaming data
into durable storage,
like Amazon S3
• Add in streaming
analysis results to
replace batch analysis
Application databases Data warehouseData producer
Amazon Kinesis
ETL
ETL
Amazon S3
Streaming
data

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Batch to streaming best practices
Perform ITL rather than ETL
• ITL: Ingest-Transform-Load
• ETL: Extract-Transform-Load
• Transform data in near-real time
rather than a scheduled job
• Enrich data in near-real time
• Persist transformed and/or
enriched data
Data producer
Amazon Kinesis
Firehose
Raw streaming
data
AWS Lambda
function
Amazon S3
Transformed
data
Transform
data
Enrichment
source data
Raw data Transformed and/or
enriched data

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Batch to streaming best practices
Aggregate upon arrival
• Continuously write raw data
to persistent data store for
archival and other analysis
• Aggregate in real time when
window size < 1 hour
• Write aggregated data to
persistent data store for
immediate value
Amazon Kinesis
Firehose
Raw streaming
data
Amazon S3
Raw
data
Aggregated
data
Amazon Kinesis
Analytics
Aggregate
Results
Data producer

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Batch to streaming example

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Amazon delivery app (Android/iOS)
• Crowd-sourced model launched in
30+ U.S. cities
• Used by Amazon Logistics worldwide

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Deliveries for Amazon.com, Prime
Now, Amazon Fresh, restaurants,
grocery stores
• Millions of packages per year

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The problem
• Collecting, processing, and storing telemetry data
• Telemetry data = remote measurements
• Includes metrics, crashes, logs, sensor data, clickstream data, etc.

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The goal
• Understand what’s happening in the field
• Analyze all the data and make performance optimizations
• Focus our time on improving the app and the delivery flow

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Getting from batch to streaming
• To solve our use cases, we had to incrementally improve our system
• We evolved from a batch-based system to a stream-based system
• Let’s walk through the iterations

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Collect metrics and send to an existing metrics service
• ETL jobs to load data into a big Oracle Data Warehouse
Iteration 1: Use existing systems
Existing metrics serviceApp DW
ETL
Data
collection

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• 1. Batch process with 24-hour delay
• 2. Fixed, inflexible DB schema
• 3. Analysis difficult and slow via SQL
Iteration 1: Use existing systems
Existing metrics serviceApp DW
ETL
Data
collection

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Collect metrics in the app using AWS Amazon Mobile Analytics SDK, which
automatically loads data into Redshift
Iteration 2: Use AWS
App
CloudFormation
ETL system
Data
collection

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• 1. Batch process with 24-hour delay 2-hour delay
• 2. Fixed, inflexible DB schema
• 3. Analysis difficult and slow via SQL
Iteration 2: Use AWS
App
CloudFormation
ETL system
Data
Collection

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Add shared configuration that is used in the app and automatically updates
the Redshift schema
Iteration 3: Automated DB schema
App
CloudFormation
ETL system
Data
collection
Schema config

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• 1. Batch process with 24-hour delay 2-hour delay
• 2. Fixed, inflexible Auto-updating DB schema
• 3. Analysis difficult and slow via SQL
Iteration 3: Automated DB
schema
App
Schema config
CloudFormation
ETL system
Data
collection

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Introduce a Kinesis stream and Kinesis Firehose to publish to Redshift
• Partition data by date to simplify data retention policies
Iteration 4: Use Streams
App
Data
collection Via Pinpoint
Schema
config

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• 1. Batch Streaming process with 24-hour 2 hour a delay of a couple
minutes
• 2. Fixed, inflexible Auto-updating DB schema
• 3. Analysis difficult and slow via SQL
Iteration 4: Use Streams
App
Data
collection Via Pinpoint
Schema
config

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Use generic message types
• Publish the data to:
• S3
• Redshift
• ElasticSearch
Iteration 5: Generic message
types
App
ElasticSearch

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Iteration 5
App
Data
collection
ElasticSearch
Consumer Lambdas
SQL reports
Dashboards
ProtoBuf
Consumer Redshifts

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• 1. Batch Streaming process with 24-hour 2 hour a few seconds delay
• 2. Fixed, inflexible Auto-updating DB schema and generic message types
• 3. Analysis difficult and slow via SQL flexible by processing message
payload
Iteration 5: Generic message
types

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of Streaming
1. Agility: real-time data means your business can react quicker
2. Flexibility: generic message types give you flexible schemas so your
system can handle multiple data types and future use cases
3. Shareability: streams allow you to multiplex and share your data easily
with your consumers
4. Extensibility: Processing streams of data allows us to write it to multiple
data storage systems, which enables a variety of analytics tools

49. Thank you