Machine Learning (ML) works by using powerful algorithms to discover patterns in data, and constructing complex mathematical models using these patterns. Once a model is built, you perform inference by applying data to the trained model to make predictions for your application. Building and training ML models requires massive computing resources so it is a natural fit for the cloud. But, inference takes a lot less computing power and is typically done in real-time when new data is available, so getting inference results with very low latency is important to making sure your applications can respond quickly to local events. AWS Greengrass ML inference gives you the best of both worlds. You use ML models that are built and trained in the cloud, and you deploy and run ML inference locally on connected devices. For example, autonomous cars need to identify road signs in real time; and drones need to recognize objects with or without network connectivity.

1. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Massimiliano Angelino
Solution Architect, Amazon Web Services
Alessandro Pirrotta
Ph.D., Data Scientist, DFDS
ML inference at the Edge

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Architecture: 2016
AWS AWS IoT Core
Gateway
Endpoints
Greengrass
Things
Sense & Act
Cloud
Storage & Compute
Intelligence
Insights & Logic → Action

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS IoT Architecture: 2018
Secure device
connectivity
and messaging
Endpoints
AWS IoT Core
Fleet onboarding,
management and
SW updates
Fleet
audit and
protection
IoT data
analytics and
intelligence
Gateway
AWS Greengrass
Things
Sense & Act
Cloud
Storage & Compute
Amazon
Intelligence
Insights & Logic → ActionAWS IoT 1-Click
CO
M
ING
IN
2018

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How can I extend
AWS intelligence
to the edge?

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Greengrass extends AWS intelligence to your devices, so they can act
locally on the data they generate, while still taking advantage of the cloud.
Extend AWS intelligence to the Edge
AWS Greengrass
Edge Cloud
Law of economics
Law of physics
Law of the land

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Extend intelligence to the Edge
AWS Greengrass
Machine
learning
inference
Local execution
of ML models
Over-the-air
updates
Easily update AWS
Greengrass core
Protocol
adapters
Local
resource
access
Lambda interacts
with peripherals
Easy integrations
with local
protocols
ʥ
A
Data and
state sync
Security
Local
device shadows
Local
actions
Lambda
functions
Local
messages
and triggers
Local
message broker
High-quality
AWS
security

7. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Machine Learning at the Edge

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Use Cases
Voice/sound
recognition
Collision
avoidance
Image
recognition
Anomaly
detection
More
!
Smart
Agriculture
Predictive
maintenance
Self-driving
cars
Video
surveillance
Robotics

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Greengrass ML Inference
Build and train ML
models in the cloud
Accelerate ML inference
applications on the edge
Devices take
action quickly –
even when
disconnected
Use Greengrass to
deploy optimized
models on your
target device

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Machine Learning at the Edge
Inference Training
Local
actions
Edge Cloud
Triggers

11. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Greengrass ML Overview

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What do you want to achieve?
Sense
Generate and receive rich data
about the environment
Infer
Extract relevance from huge
amounts of data in real time
Action
Take smart actions

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why is it hard?
Collect and
moving data
to the cloud
Process data,
build and
train your
model
Deploy
model to the
target device
Build ML
framework
(e.g., MXNet)
for different
device
Write
Inference
app and
deploy it to
the target
device
Utilize
accelerator
such as GPU

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ML Inference using AWS Greengrass
Train in the cloud
• Massive computing power
• Large repository of data
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inference at the edge
• Low latency
• bandwidth saving
• regulation/privacy
Trained models
and Lambdas
Extracted
Intelligence
Inferences and
take actions
locally on device
AWS Cloud
for training
SageMaker

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deploy cloud trained models
to target devices for you
• Add your trained model as a
Machine Learning resource
to Greengrass group
• Deploy to Greengrass devices
• Locate Amazon SageMaker
trained models in
Greengrass console
• Bring your own models
Deploy cloud trained models

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Access GPU and FPGA
from Lambda functions
to speed up inference
• No code required
• Simply declare the
accelerator as a Local
Resource that Lambda
functions need to access
Access hardware accelerators

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pre-built MXNet and
Tensorflow packages
• Intel Atom E3900
(Apollo Lake)
• NVIDIA Jetson TX2
• Raspberry Pi
You can always bring
your own framework (e.g.
Caffe2, and CNTK)
Pre-built ML frameworks for devices

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lambda examples
to help you create
inference apps,
showing you how to
• Load trained models
• Applying them to locally
generated data for local
inferencing
• Take actions
Lambda inference examples

19. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Demo

20. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Guest speaker - DFDS

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of Greengrass ML Inference
Deploy cloud
trained models
Enable
GPU access
Use pre-build
MXNet and TensorFlow,
or bring your own ML
framework
Lambda
actions

22. AWS DeepLens
HD video camera
Custom-designed
deep learning
inference engine
Micro-SD
Mini-HDMI
USB
USB
Reset
Audio out
Power
HD video camera
with on-board
compute optimized
for deep learning
Tutorials, examples,
demos, and pre-
built models
From unboxing
to first inference
in <10 minutes
Integrates with
Amazon SageMaker
and AWS Lambda
10
MIN
The world’s first deep learning-enabled video camera for developers

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS DeepLens Architecture
Video out
Data out
I N F E R E N C E
D E P L O Y P R O J E C T S
Manage device
Security
Console Project
Management
AWS Cloud
Intel: Model Optimizer
cIDNN and Driver

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customers and partners

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Consultancy Partners – ML Competency

26. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Go build!

27. Problem
Nokia saw a need in industrial IoT to analyze
video streams at the edge and send the data
to remote centers only when anomalies are
detected.
Solution
Deploying AWS Greengrass on Nokia Multi-
access Edge Computing platform and
combining it with Nokia private mobile
network solutions. This joint solution makes
it possible for the oil industry to pair real-
time drilling data with production data
of nearby wells.
Impact
Due to the high cost of bandwidth being, this
solution enables Nokia to optimize the data
that is sent to other wells and to the cloud
based on rules and alerts set up on
the locally processed data.

28. Problem
Wärtsilä needed to accurately predict when the
marine engines they manufactured needed to
get serviced. Understanding and predicting the
service schedule is vital for Wärtsilä to increase
their service and parts revenue.
Solution
Accenture worked with AWS account SAs, AoD
SAs, and Salesforce SAs to architect an IoT
solution using Salesforce and AWS IoT Core to
collect data and build predictive models. The
solution they developed is scalable and
extensible beyond just this use case, as Wärtsilä
has 14,000 ships with 35,000 engines installed.
There are great possibilities for sensor-driven IoT
use cases.
Impact
The entire solution should result in an increase
in parts and service sales for Wärtsilä and higher
customer retention.