In this talk, we consider the unique challenges of the biosphere that recently opened in downtown Seattle. We address two of those challenges using modern deep learning techniques: computer-vision-based plant health monitoring, and microclimate anomaly detection using autoencoders on time-series data extracted from multiple sensors. Our focus is on architecting the inference pipelines for solving these problems at scale. Specifically, we highlight the inference steps and TensorRT optimizations for AWS Greengrass ML inference.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inference at the Edge:
A Case Study at the Amazon Spheres
WenMing Ye
Specialist Solution Architect
Amazon
A R C 3 1 8
Miro Enev, PhD
Sr. Solution Architect
NVIDIA

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Introduction: AI @ Amazon Spheres
Video: Welcome to the Amazon Spheres [ living wall video ]
Approach:
Anomaly detection using DL on time-series sensor streams
Architecture:
Training (SageMaker)
Inference (NVIDIA Jetson TX2)
Results:
Improved alerting
Future Work:
Computer vision based plant stress

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our Goal = Help the Caretakers
“We take care of 40,000 plants
from over 700 species!”
Claire
Ben

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sensor Types

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Temperature

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Co2

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
InstLightLevels

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenge 1: Lots of Systems to Manage

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenge 2: Too Many Suspicious Values

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
When Issues Occur…They Go Unnoticed
Example 1: During a product launch (Alexa microwave
integration), event organizers requested that the temperature be
lowered for media and the air velocity reduced for better acoustics
Problem: Incorrect temp. and air velocity for fourth floor plants
for a week
Example 2: Building automation staff suspended the irrigation for
the living wall to update/repairs several sensors
Problem: 24 hours without water for living wall
[ low irrigation pressure warning was ignored ]

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AI to Assist the Caretakers
• Accurate Alerts [ low false alarm
rate ]
• Real-time & Low Cost
• Enable Current/Future Science
• Scalability & Availability of
Technology

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AI
ML
DL
AI

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ML
TRIBES

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why DL

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep Learning @ Spheres
[ AutoEncoder Network ]

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Preparing Data for Model Training
Split into sliding windows
[ heavily overlapped ]
Z Normalization

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Detecting Anomalies
Reconstruction error (RE) as a proxy to outliers
Whenever RE is high, get an alert

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-Sensor Models
[ AutoEncoder Network ]

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Living
Wall
Inside the Spheres
[ First floor ]
Cafe
North Conservatory

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Aircuity
Sensors
AC-46-1-2
AC-46-1-1
AC-46-1-4
AC-46-1-3
AC-46-1-5

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sensor Zones
Living Wall [ 4 floors ]
t,rh,d,co2 [ X, AC-46-2-2, AC-46-3-2, AC-46-4-3 ]
light level [ DLI-46-1-DG1, DLI-46-2-DG2, DLI-46-3-DG3, DLI-46-
4-DG5 ]
North Conservatory [ 1st floor ]
t,rh,d,co2 [ AC-46-1-1, AC-46-1-2, AC-46-1-3, AC-46-1-4 ]
light level [ DLI-46-1-DS1, DLI-46-1-DM2, DLI-46-1-DM3 ]
South Conservatory [ 2nd floor ]
t,rh,d,co2 [ AC-46-2-3, AC-46-2-4, AC-46-2-5, AC-46-2-6 ]
light level [ DLI-46-2-DM1, DLI-46-2-DM2, DLI-46-2-DM3 ]
Canopy [ 3 floors above N. Conservatory ]
t,rh,d,co2 [ AC-46-2-1, AC-46-3-1, AC-46-4-2 ]
light level [ DLI-46-4-DL1, DLI-46-4-DL2, DLI-46-4-DL3,
DLI-46-4-DL4, DLI-46-4-DL5, DLI-46-4-DL6,
DLI-46-4-DL7, DLI-46-4-DL8,
DLI-46-4-DL13, DLI-46-4-DL14 ]

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sensors
SpheresAIArchitecture
v01
Camera
CO2
Light Level
Thermostat
AWS Cloud
Model_1
Model_2
Model_N

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Cloud
AWS Cloud
SpheresAIArchitecture
v01 On-Premise
Email alert
Sensors
Camera
CO2
Light Level
Thermostat

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Inference @ Edge
NVIDIA Jetson
TX2
Scaling Inference [ edge processing ]

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jupyter Notebook

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

37. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
WenMing Ye - wye@amazon.com
Miro Enev - menev@nvidia.com

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Types of ML/DL Used

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Model Training

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Anomaly Detection

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Anomaly Detection

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DL Anomaly Detection
[ Autoencoder Network ]

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Spectral Matching