Industry 4.0 demands greater insight into data to bring people, processes, and equipment together. In this workshop, we illustrate how to gain business insights from video and voice data sources, highlighting the data pipeline. We ingest source feeds, efficiently store the data, and perform advanced analytics using AWS ML services and analysis tools. Typical applications include anomaly detection (detecting spills or hazardous objects and predictive maintenance) and voice sentiment analysis (customer service insights). By the end of the session, you will be able to quickly analyze data for uncommon characteristics using those detections to initiate a wide variety of actions.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Lake Patterns for Voice, Vision, Advanced
Analytics, & ML Using Serverless
Paul Armstrong
Principal Solutions
Architect
Greg Share
Enterprise Solutions
Architect
Jagadeesh Pusapadi
Enterprise Solutions
Architect
A R C 3 2 0

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Session abstract
Industry 4.0 demands greater insight into data to bring people, processes, and
equipment together. This workshop will illustrate how to gain business insights from
video and voice data sources, highlighting the data pipeline. We
will ingest source feeds, efficiently store the data, and perform advanced analytics
using Amazon Machine Learning (Amazon ML) services and analysis tools. Typical
applications include anomaly detection (detecting spills or hazardous objects
and predictive maintenance), voice sentiment analysis (customer service insights). By
the end of the session you will be able to quickly analyze data for uncommon
characteristics, using those detections to initiate a wide variety of actions.

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Collect Store
Process/
analyze
Consume
Data Answers
Time to answer (latency)
Throughput
Cost
The data pipeline

5. Streaming
Collect Store ConsumeProcess/analyze
Apache Kafka
HotHotWarm
Fast
Stream
SQLNoSQLCacheFileStream
Mobile apps
Web apps
Devices
Sensors
IoT platforms
Data centers
Migration
Logging
RECORDS
FILES
STREAMS
Analysis&visualizationDataScience
DataTransport&LoggingIoTApplications
Presto
FastSlow
BatchInteractivePredictive
AmazonAI
Apps
Model
Train/
Eval
Models
Deploy
ETL

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workshop objectives
• Work through the data pipeline
• Focus on voice and video
• Demonstrate how to use the services to solve typical problems and use cases
• Leave with a working end to end solution

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Target audience
• Architects
• Developers/engineers
• Level 300 – good working knowledge of AWS services, CLI
• All code will be provided
• Note ML services will be used and the models shared, not an in-depth ML
session

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What do I need?
• AWS account with admin access
• AWS Command Line Interface (AWS CLI) command tools installed on laptop
• Understanding of AWS CloudFormation
• We will cover a broad spectrum of topics
• Voice/video
• Archive/storage/retrieval
• Analytics and ML
• Notification services

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workshop setup steps
• Login to AWS account
• Create an Amazon Simple Storage Service (Amazon S3) bucket
• Download the zip file from the link provided
• Un-compress and sync with the Amazon S3 bucket
• Open the workbook for detailed instructions

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Video footage and voice ingestion
Industrial/Smart City/Smart Home video feeds
Customer services/operations voice inputs

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Objectives
Collect video feed from camera
• CCTV footage
• Production line camera
Collect voice feed using hotline
• Automated hotline
• Analyze contact center recordings
Collect Store
Process/
analyze
Consume

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data collection process
• Call center hotline interaction
to data lake
• Data combined with footage
data
• CCTV footage streamed into
the data lake
• Optional customer producer
interaction with incoming
footage
Collect Store
Process/
analyze
Consume
ML inference
at the edge
(anomaly/face detection)
Proxied video
stream
(optional)
Footage
Customer interaction data
Voice
CCTV
Image
Contact
center
Streaming
ingest
engine
Data
persistence
(data lake)
Footage direct into
streaming ingest engine
Streaming
Ingest Engine
Customer
Producer
Streaming
Ingest Engine

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Collect target architecture
• Phone hotline writing
contact records and voice
recordings to S3
• Shared bucket for images
• Cameras set up in workshop
and shared streams for cross
account access to Amazon
Kinesis Video Streams for
CCTV
• Optional direct access RTSP
stream and use own
producer to Kinesis Video
Streams for CCTV
Collect Store
Process/
analyze
Consume
Producer SDK writes directly to
the video stream
Model inference at the
edge to detect faces
Producer SDK
RTSP stream
read by proxy and
send to stream
Cropped images
from video stream
Contact records and
call recordings
Voice
CCTV
Image

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Module activities
• Stream from local web camera or test setup of S3 sample
• Set up connect or use a sample audio file for analysis
Key activities
• Connect to Kinesis Video Streams streams to view content on the stream or
simulate sample frames for analysis
• Optionally configure connect to create voice recordings
• Ensure can access S3 of call recordings
Collect Store
Process/
analyze
Consume

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Module outcomes
• Video stream enabled
• [Optional] Camera producer sending content to the stream
Collect Store
Process/
analyze
Consume

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Video footage and voice persistence
Video and voice persistence options for analysis use cases

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data storage process
• Voice/video data combined in
the data lake
• Authenticated access
mechanism for streaming
video footage access –
persisted to the data lake
Collect Store
Process/
analyze
Consume
Voice
CCTV
Video
Get content
from stream
Data
persistence
(data lake)
Serverless
processing
Data
persistence
(data lake)
Serverless ingested
footage
access mechanism

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Store—Target architecture
• Voice/image stored in S3
• CCTV API to request video
from steam and store in S3
with signed URL
Collect Store
Process/
analyze
Consume
Voice
CCTV
Image
Get content
from stream

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Module activities
• Test uploading files to S3 for
• Video analysis
• Voice analysis
• Sample video frame for video object detection
Collect Store
Process/
analyze
Consume

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Module outcomes
• Store locations understood and accessible with sample data
Collect Store
Process/
analyze
Consume

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Video and voice analysis
Combined analysis for sentiment/anomaly detection
(Industrial H&S across manufacturing, travel and transport safety, and customer service)

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Video and voice data processing
• Voice/image processing >
analysis > indexing
• CCTV run model for object
detection
Collect Store
Process/
analyze
Consume
Voice
CCTV
Image
Serverless
processing
Serverless
processing
workflow
Video
analysis
Voice to text
conversion
Sentiment
analysis
Analysis
indexing
Machine learning
image
analysis
Serverless
processing

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Process—Target architecture
• Voice/image use AWS managed
services for indexing
• CCTV run model for object
detection
Collect Store
Process/
analyze
Consume
Voice
CCTV
Image
Deployed
model and
endpoint

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Module activities
• Test the ingestion of video or sample frame
• Validate that the image is processed and view the output image with detected
object and bounding box. Explanation of how the lambda parameters are
managed
• Explanation of model and further activities for building and enhancing the
deployed model
Collect Store
Process/
analyze
Consume

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Module outcomes
• Solution using Amazon SageMaker to detect objects
• Solution that can analyze voice recordings
Collect Store
Process/
analyze
Consume

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data visualizations
Share CCTV footage with third parties
Real-time dashboards and alerting for both internal and external parties

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Output consumption
• Authenticated access to
Web based visualizations
• Indexed views
• Detected anomaly alerts
and alarm notifications
Collect Store
Process/
analyze
Consume
Voice
CCTV
Video
Get content from stream
Get Content
Notifications
Serverless
processing
Serverless
processing
Serverless
access
endpoint
Data
lake
Alerts and
alarms
Consumption
auth.
mechanism

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Consume—Target architecture
• HTML authenticated viewer
with Amazon Cognito
• View indexes
• Kibana dashboard viewer
• Amazon Simple Notification
Service (Amazon SNS)
notifications of detected
spills
Collect Store
Process/
analyze
Consume
Voice
CCTV
Image
Get content
from stream
Get content
Stream
processor

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Module activities
• View video from web page
• Configure alerts receive alert when an object is detected
• View analytics of voice streams
• View sentiment analysis (voice/video)
[Log in to local example or use AWS deployed solution]
Collect Store
Process/
analyze
Consume

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Module outcomes
• Visualization of the solutions in real time/historic
• Alerting to key events
• Reference architecture to use cases for real time voice/video detection and
alerting
Collect Store
Process/
analyze
Consume

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Voice and video analysis framework
Collect Store
Process/
analyze
Consume
Get content
from stream
Deployed model
and endpoint
Object/
image
detection
Video and customer
interaction processing
Contact
center
Streaming
ingest engine
Customer interaction data
Data
lake
Streaming
ingest engine
Footage direct into
streaming ingest engine
ML inference
at the edge
(anomaly/face detection)
Customer
producer
Proxied video
stream
(optional)
Streaming
ingest engine
Serverless processing
workflow
Video
analysis
Voice to text
conversion
Sentiment
analysis
Analysis
indexing
Serverless
processing
Serverless
access
endpoint
Data
lake
Consumption
auth. mechanism
Get
content
Serverless
processing
Serverless
processing
Machine learning
image
analysis
Notifications
Alerts and
alarms

33. Overall architecture
Collect Store
Process/
analyze
Consume
Get content
from stream
Get content
Deployed
model and
endpoint
Object/image detection
Producer SDK writes directly to
the video stream
Model inference at the
edge to detect faces
RTSP stream
read by proxy and
send to stream
Cropped images
from video stream
Contact records and
call recordings
Process
media

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Summary
• Ingest data securely and at scale
• Analyze new data sources to augment transactional data sources
• Use data sources for ML training and inference
• Integrate with existing applications and workflows
• Provide visualization on demand
• Real-time alerting

35. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.