Autonomous cars need to identify road signs in real time, drones need to recognize objects with or without network connectivity. In this breakout session, you will learn what is machine learning (ML) inference at the edge and why it matters. We will show you how to use AWS Greengrass to locate cloud trained machine learning models, deploy them to your Greengrass devices, enable access to on-device GPU or FPGA, and apply the models to locally generated data without a need for connection to the cloud.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Push Intelligence to the Edge
M a c h in e L e arn in g o n A W S G r e en gras s D e v ic es
S a t y e n Y a d a v , G M , I o T E d g e & D e v i c e S e r v i c e s
J a s o n C h e n , P r i n c i p a l P r o d u c t M a n a g e r , I o T E d g e & D e v i c e S e r v i c e s
N o v e m b e r 3 0 , 2 0 1 7
AWS re:INVENT

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Architecture: 2016
AWS AWS IoT Core
Gateway
Endpoints
Greengrass
Things
Sense & Act
Cloud
Storage & Compute
Intelligence
Insights & Logic → Action

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Architecture: 2017
Secure device
connectivity
and messaging
Endpoints
AWS IoT Core
Fleet onboarding,
management and
SW updates
Fleet
audit and
protection
IoT data
analytics and
intelligence
Gateway
AWS Greengrass
Things
Sense & Act
Cloud
Storage & Compute
Amazon
Intelligence
Insights & Logic → ActionAWS IoT 1-Click

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How can I extend
AWS intelligence
to the edge?

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data and
state sync
Local
actions
Local
triggers
Security
AWS Greengrass
Extend intelligence to the edge
Local ML
inference
Preview today
Over the air
updates
Protocol
adapter for
OPC-UA
Local
resource
access

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine Learning at the Edge
Greengrass ML Inference

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Use Cases
Voice/sound
recognition
Collision
avoidance
Image
recognition
Anomaly
detection
More
!
Smart
Agriculture
Predictive
maintenance
Self-driving
cars
Video
surveillance
Robotics

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Greengrass ML Inference
Build and train ML
models in the cloud
Accelerate ML inference
applications on the edge
Devices take
action quickly –
even when
disconnected
Use Greengrass to
deploy optimized
models on your
target device

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inference Training
Machine Learning at the Edge
Local
actions
Edge Cloud

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ryuji Takehara
Cloud System Architect

11. 12
Machine Learning in SONY Factories
Already running Greengrass on our factory floors, now developing ML for:
 Factory operator positioning
• Operator location & time resource management
• Using beacon and ML predict position of operator
 Predictive maintenance
• Detect aging degradation of bearing with acceleration sensor
 Size limitation (ML library is big)
 Tight coupling ML model and Lambda
 HW resource access
Needed more AWS Lambda ability

12. 13
Using Greengrass ML Inference
Deploy and run customized ML model on devices using Greengrass for “Anomaly Detection”
 Customize model without size limitation
 Continuous adjustment , delivery model
 HW resource access
(high resolution sensor)
SONY: Spritzer
Host board (GG Core)
Accelerometer
Manufacturing
Machine
Special device + host
(Accelerometer)
SONY ML dashboard
Customize model
for various factory machine
Easy to customize model
Management cloud
Detection
acceleration
x,y,z
Class label
Cloud
E d g e
Greengrass ML Inference
Sony now integrates “Cloud, Edge, Firmware and Sensor Devices”

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Latency Bandwidth Availability Privacy
Value of ML Inference at the Edge

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
We create the technology to connect the world
Khamis Abulgubein
Product Line Manager, Emerging IoT Applications

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What
is an
anomaly?
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Processing video analytics
LAN Scenario
LAN
10 GB/day
10 GB/day
10 GB/day/camera 1% relevant data
10 GB/day

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Processing video analytics
WAN Scenario
WAN
10 GB/day/camera 1% relevant data
10 GB/day
10 GB/day
10 GB/day

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Processing video analytics
WAN Scenario
WAN
100 MB/day/camera 1% relevant data
10 GB/day
10 GB/day
10 GB/day
IMPACT Scene
Analytics Gateway
Scene anomalies
Scene Metadata Greengrass
core
Trigger local actions based
on object recognition
AI-based object classification
MQTT

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Greengrass ML Inference Overview

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What do you want to achieve?
Sense
Generate and receive rich data
about the environment
Infer
Extract relevance from huge
amounts of data in real time
Action
Take smart actions

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why is it hard?
Collect and
moving data
to the cloud
Process
data, build
and train
your model
Deploy
model to
the target
device
Build ML
framework
(e.g., MXNet)
for different
device
Write
Inference
app and
deploy it to
the target
device
Utilize
accelerator
such as GPU

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ML Inference using AWS Greengrass
Train in the cloud
• Massive computing power
• Large repository of data
Trained models
and Lambdas
Extracted
IntelligenceInferences and
take actions
locally on device
AWS Cloud
for training
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inference at the edge
• Low latency
• bandwidth saving
• regulation/privacy

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deploy cloud trained
models to target devices
for you
• Add your trained model as a
“Machine Learning”resource
to Greengrass group
• Deploy to Greengrass devices
• Locate Amazon SageMaker
trained models in
Greengrass console
• Bring your own models
Deploy cloud trained models

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Access on-device ML
accelerators such as GPU
and FPGA from Lambda
functions to speed up
inference
• No code required
• Simply declare the
accelerator as a “Local
Resource” that Lambda
functions need to access
Access hardware accelerators

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pre-built MXNet package
so you don’t need build it
from scratch for your
devices
• Intel Atom E3900
(Apollo Lake)
• NVIDIA Jetson TX2
• Raspberry Pi
You can always bring
your own framework (e.g.
TensorFlow, Caffe2, and CNTK)
Pre-built MXNet for devices

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lambda examples
to help you create
inference apps,
showing you how to
• Load trained models
• Applying them to locally
generated data for local
inferencing
• Take actions
Lambda inference examples

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of Greengrass ML Inference
Deploy cloud
trained models
Enable
GPU access
Use pre-build
MXNet, or bring
your own ML
framework
Lambda
actions

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS DeepLens
Deep Learning video camera
using AWS AI and AWS Greengrass on Intel Atom

30. Corporate Vice President, Intel Corporation
GM, Artificial Intelligence Product Group
31

31. 33
AWS Greengrass & Intel – Powerful,
Optimized edge solutions
Delivers a secure, intelligent edge
Developers can easily create new applications,
from edge to cloud

32. AWS Greengrass & Intel- Machine learning
capabilities at the edge
34
Intel Atom Processor with integrated graphics
Deep Learning Optimized Software
Compute performance, agility, and speed
to run real time machine learning on the device.

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Summary

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customers and partners

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Apply for preview today!
aws.amazon.com/
greengrass/ml

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Learn more
AWS DeepLens Through Thursday Venetian - Casanova
604 & Marco Polo 702
Flight simulator using
FreeRTOS and Greengrass
Through Thursday Aria, Builder Fair
Echo Greeter Booth: face
detection on device
Through Friday Aria, level 3
Dynamic sorter: object
recognition on device
Through Thursday Aria, Builder Fair
Robotic arms Through Friday Aria, Pinyon1
Live Nokia Traffic Camera using
Greengrass ML Inference
Through Thursday Venetian, Maker’s place

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Backup

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
IoT with AWS
Enterprise
Applications
Enterprise
Users
Corp Apps
Amazon
QuickSight
Amazon
EMR
Amazon
Redshift
Amazon
S3
Machine
Learning
AWS
Lambda
All
AWS
IoT Partners
Edge
ARM, Broadcom, Digi,
Expressif, Intel, MediaTek,
Microchip, NXP, ST, TI,
Qualcomm, …
Gateway
Adlink Technology,
Advantech, MachineShop,
Samsung, Technicolor, …
Platform
Ayala, Bright Wolf,
BSquare, C3IoT, Mnubo,
PTC, Salesforce, Splunk,
Thinglogix, …
Connectivity
Amdocs, Asavie, AT&T,
Eseye, Soracom, TATA
Communications, Telus,
Verizon, …
Consulting / ISVs
Accenture, Aricent,
Clearscale, CTP, Luxoft,
Mobiquity, Solstice,
Storm Reply, Sturdy
Networks, TCS, Trek10, …
MQTT
MQTT
Endpoints Gateway/PLC
Device
Shadow
Snowball
Edge
AWS
Greengrass
Lambda
Functions
Message
Router
Local Comms Long-range Comms
Amazon
FreeRTOS
Certificate
Authority
Local
Resources
OPC-UA
Adapter
IoT SDK
OPC-UA
MQTT
Edge
Users
Cert
WiFi
MQTT
Edge
OTA
OTA
Amazon
FreeRTOS
Integrated
Client
Cloud
Device
Shadow
Rules
Engine
AWS IoT
Core
Certificate
Authority
AWS IoT
Device
Management
AWS
IoT
Users
Over-The-Air
(OTA)
Updates
Analytics
Data Store
Data
Pipelines
Templated
Reports
Batch Fleet
Provisioning
Real-Time
Fleet Index &
Search
AWS IoT
Device
Defender
Ad-hoc & In-
depth Analysis
Risk
Mitigation
Monitor
Device
Behavior
Alerts
Message
Broker
Audit Device
Configurations
Amazon
Kinesis
AWS IoT
Analytics