AI based technology provides unique opportunities to enhance existing media delivery services. This session will provide an introduction to the Amazon AI stack and best practices on how to integrate AI services such as Amazon Rekognition into your own media applications.

1. Enhanced Media Workflows Using Amazon AI
Tobias Börjeson, Solutions Architect, AWS
December 7, 2017
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Media & Entertainment Cloud
Symposium | Stockholm

2. Artificial Intelligence at Amazon

6. Thousands Of Employees Across The Company Focused on AI
Discovery &
Search
Fulfilment &
Logistics
Enhance
Existing Products
Define New
Products
Bring Machine
Learning To All
Artificial Intelligence At Amazon
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deep Learning for Media – Glass to Glass
Playout &
Distribution
Filtering & Quality
Control
Visual Effects &
Editing
Application & Filesystem
Texture & Asset Search
Analytics
Sentiment Analysis
Other Amazon AI
Services
(Lex, Polly)
DAM & Archive
Auto-categorization
Metadata Augmentation
Digital Supply
Chain
Tag on Ingest
Live and VOD Feature
Extraction
Celebrity Detection
Publishing
Value Add
API-based services
OTT
Filtering &
Quality Control
Acquisition
Pre-
processing &
optimization

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
So where should we start?

10. © NVIDIA
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The 10,000ft Intro to Deep Learning
Raw Data Low Level Features Mid Level Features High Level Features
Result
Application
Components
Task
Identify a Face
Training
10-100M images
Network
~ 10 layers
1B parameters
Learning
~ 30 Exaflops
~ 30 GPU days
© 2016 NVIDIA
Input

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Confidence Labels
99.2%
Animal
Dog
Chihuahua
98.6%
Food
Dessert
Muffin
97.9% Collage
Dog or Muffin?

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Amazon AI Stack
Services
Platforms
Frameworks
Infrastructure
MXNet TorchCTKKerasGluonCaffe
TensorFlo
w
AWS Deep Learning AMI
Amazon
Sagemaker
Mechanical
Turk
AWS
DeepLens
Amazon ML Spark & EMR
Vision
Rekognition
Speech
Polly, Transcribe
Language
Lex, Translate, Comprehend
GPU / FPGA ServerlessCPU IoT Mobile

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Frameworks & Infrastructure
Services
Platforms
Frameworks
Infrastructure
MXNet TorchCTKKerasGluonCaffe
TensorFlo
w
AWS Deep Learning AMI
Amazon
Sagemaker
Mechanical
Turk
AWS
DeepLens
Amazon ML Spark & EMR
Vision Speech Language
GPU / FPGA ServerlessCPU IoT Mobile

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deep Learning Compute
AWS Deep Learning AMI
MXNet
Torch
CTKKeras
TheanoCaffe
TensorFlow
Amazon EC2
AnacondaIntel MKL
CUDA+cuDNN Python 2+3
Caffe2
• One Click launch
• Machine Image or Stack-based
• Single node or distributed
• GPU, CPU (& FPGA)
• NVIDIA & Intel acceleration
• Anaconda Data Science Platform
• Python w/ AI/ML/DL libraries

15. Media-Centric Models & Datasets
CelebAPlaces CIFAR-10/100
Object, Network & Gateway
Storage Services
Frameworks, 3rd Party Enablement, & Industry Initiatives
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Choosing the right Instance Type for AI
Instance
Name
GPU
Count
vCPU
Count
Memory Network EBS
p3.xlarge 1 8 61 GiB ~10Gbps 1.5 Gbps
p3.8xlarge 4 32 244 GiB 10Gbps 7Gbps
p3.16xlarge 8 64 488 GiB 25Gbps 14Gbps
P2 & P3: Distributed Training &
Inference
Hyper-scale performance on NVIDIA V100s
G3: Multi-User Modeling
NVIDIA M60 GPUs, 16,384 cores
F1: High Speed Inference
Xilinx Ulstrascale Plus, 6,800 engines
X1: Specialized AI/ML/DL
128 vCPUs, 3,904 GiB RAM
P3 Instances Provide up to 1 Petaflop of mixed precision performance,
and 125 Teraflops of single precision floating point

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why is this Important?
Amazon EC2 Compute & EBS block storage supports second-level billing.
Combined with EC2 SPOT Fleet, this provides a up to ~90% cost savings over on-demand.

18. Artificial Artificial Intelligence
Ground Truth Generation & Niche
Image Categorization at scale is time
consuming & untenable
• Crowdworking to the rescue
• Deep Learning for Unlabeled Data
• Amazon Mechanical Turk – build
machine learning datasets using HITs
(human intelligence tasks), Requesters &
Workers
• Human Inference can be as high as 100s
of HITs/min (2/s @ ~200ms)
Price Task Images Labels Time
Worker
s
$0.01
Label at least
one object in
the image
237 678 13.5 37
- ~50,000 AMT Workers
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building A Deep Learning Pipeline in the Cloud
Training
Version Controlled
Datasets
Amazon
S3
Amazon
API Gateway
Amazon
EC2
Amazon
S3
AWS Lambda
Amazon
EFS
AWS
Batch
Amazon
ECS
Amazon
Glacier
AWS
Snowball
Data
Scientist
Amazon
Mechanical Turk

20. Amazon SageMaker
B u i l d , Tr a i n , a n d D e p l o y M a c h i n e L e a r n i n g M o d e l s a t S c a l e
• Zero setup with Managed Notebooks
• Built-in, High Performance Algorithms
• One-click Training
• Automatic Model Tuning
• Reduced model training time
• One-Click Deployment
• Pay by the second
One-click
training
Automatic, built-
in model tuning
Highly-
optimized ML
algorithms
Deployment
without
engineering
effort
Fully-managed
hosting at scale
Build
Managed
notebook
instancesDeploy
Train

21. Can Amazon Rekognition add value to my content?
‘What about using the same process to enhance my Media Workflows?’
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

22. Ingest Store Analyze Deliver
PETABYTES OF IMAGES
AND VIDEO ASSETS
CENTRALIZED STORAGE
& GLOBAL REGISTRY
METADATA ENRICHMENT
THROUGH DEEP LEARNING
ENHANCED VALUE
AND SEARCH EXPERIENCE
Media Workflow Enrichment with Rekognition
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FEEDBACK
LOOP

23. Amazon Rekognition Image
Deep learning-based image recognition service
Search, verify, and organize millions of images
Object & Scene
Detection
Facial
Analysis
Face
Comparison
Facial
Recognition
Celebrity
Recognition
Image
Moderation
Text
Detection
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

24. Amazon Rekognition Video
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

25. Workflow Considerations
Content
VOD, LIVE &
IMAGE-BASED
Processing
COMPLEXITY &
TENANCY
Quality
MEZZANINE
vs. UGC
Integration
AWS, INTERNAL
& PARTNER
FILE
FORMATS
STREAMING
PROTOCOLS
STORAGE
& NETWORK
IO
PROCESSING
VELOCITY
SIDECAR
DATA
CMS,
MAM & DAM
CONFORMANCE
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

26. Maple
Villa
Bushes
Grass
Tree
House
Window
Sky
Mountain Range
Forest
Clouds
Object and scene detection makes it easy for you to add features
that search, filter, and curate large libraries.
Identify objects and scenes and provide confidence scores
DetectLabelsObject & Scene Detection
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

27. Demographic Data
Facial Landmarks
Sentiment Expressed
Image Quality
Brightness: 23.6
Sharpness: 99.9
General Attributes
Facial Analysis DetectFaces
Analyze facial characteristics in multiple dimensions
Smiling
99.1%
Female
100%
Mouth Closed
99.5%
Age Range
26 – 43 years old
Crowd Mode
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

28. Celebrity Recognition & Image Moderation
Detect explicit and suggestive contentRecognize thousands of famous individuals
RekognizeCelebrities
DetectModerationLabel
s
Safe
Appropriate
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

29. {
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
Rekognition APIs – Overview
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
Non-storage API Operations
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
Storage-based API Operations
ListFaces
Detect-Text
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deterministic Response Time
~500ms Object & Scene Detection
<500ms Search for 1mil Face Collection

31. moderation level = safe
Building Rich Metadata Indexes using Rekognition
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Name: You?

32. Complementary Services Decoupling
Amazon
SQS
Amazon
SNS
Amazon
KinesisProcessing
Amazon API
Gateway
AWS Batch
Amazon
EC2 Amazon
ECS
Compute
Applications
AWS
Lambda
Storage
Amazon
EFSAmazon
S3
Persistence
Amazon
DynamoDB
Amazon
ElasticSearch
AWS
Elemental
MediaConvert AWS
Elemental
MediaLive
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

33. ASSET INGEST AMAZON S3 AWS LAMBDA
CLIENT APP AMAZON ELASTICSEARCH
AMAZON REKOGNITION
Mobile app uploads
the image to S3
A Lambda function is triggered
and calls Rekognition
Rekognition retrieves the image from S3 and
returns labels for the property and amenities
Other users can search properties by
landmarks, category, etc.
MEDIA ASSET
MANAGEMENT
S3TA or 3rd Party Asset
Ingest
Event-Based Processing
Via S3 Notifications
Lambda Triggers Rekognition
asynchronously
Return Object, Scene & Face
Data for Video
Inject Asset Tags via REST API
Into specific fields
Searchable Tags stored as bag of
words collections
Authenticated Users can
Search Assets via Custom/Web App
Metadata Enrichment using Amazon Rekognition
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

34. Amazon Rekognition media use cases
• Automated metadata. Classification of videos and still images – objects,
scenes, activity, sentiment, demographics
• Real time demographic & sentiment tracking. Touchless capture of
audience demos and reactions to displayed content, synchronized to
timeline
• Dynamic display enrichment. OTT users seamlessly access details of
actors appearing on screen in real-time
• Content moderation. Identify suggestive and explicit content to improve
alignment with rating standards
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

35. • Built in 3 weeks
• Indexed against 99,000 people
• Index created in one day
• Saved ~9,000 hours a year in manual
curation costs
• Live video with frame sampling
Automating footage tagging with Amazon
Rekognition
Previously, only about half of all footage was indexed due to the
immense time requirements required by manual processes
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

36. Solution Architecture
EncodersStills Extraction &
Feeds
Results
Cache
Bucket
R3
Amazon
Rekognition
Users
Stills
Frames
SQS
Trigger
1
2
3
4

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key Takeaways
• AI-based technology provides unique opportunities to enhance
existing media delivery services
• Running Deep Learning Infrastructure is hard
• Managed services can be used to eliminate ‘undifferentiated
heavy lifting’, allowing for niche AI focus
• AI for media is a cross-functional tech undertaking
• Don’t overcomplicate the pipeline & infrastructure
• Many traditional ‘in the cloud’ paradigms map to deep learning
• When you have to, utilize compute diversification across GPU,
CPU & FPGA, combined with Object Storage & Fractional
Billing

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank You!