Jerome Mies presented on using edge computing and AWS IoT Greengrass to monitor street conditions in Amsterdam via bicycle inspections. The project aims to cover the entire city with LiDAR and video AI to provide an accurate overview of deficiencies. Edge devices like NVIDIA Jetson Nano mounted on bicycles can perform real-time image analysis and inference to detect issues like light posts, garbage, and road damage. AWS Greengrass enables managing remote edge devices and orchestrating event-driven workflows. Models are trained with Amazon SageMaker and optimized for battery-powered edge applications. The cloud allows scalability to multiple edge devices for real-time data processing and updates to improve public safety.

1. Jerome Mies (Lecturer/ Researcher at Hogeschool van Amsterdam)
23 May, 2023
Biking on the edge
Monitoring the conditions
of streets in Amsterdam
with AWS IoT Greengrass

2. Introduction video –
http://vimeo.com/769075918/f2159baf0b

3. Overview
• What are we working on?
• What is edge computing?
• How do we use edge computing?
• How do we use AWS?
• How can we integrate AWS?
• Example models

4. Together improving the wellbeing in public spaces
Hogeschool van Amsterdam
Centre of expertise for applied
artificial intelligence
Smart Asset Management Lab
SURF
SURF is the collaborative
organisation for IT in Dutch
education and research
Municipalities
Velotech Solutions
Sustainable object detections
in the public space

5. Conditions of the assets in the city
impact our safety and wellbeing
Light posts
Traffic lights
Garbage
Traffic signs
Road
markings
Road surface
Trees
1

6. Our ambition is to cover the complete city maintenance
with LiDAR & video AI
Video image
recognition
LiDAR mapping
Light posts
Traffic lights
Garbage
Traffic signs
Road
markings
Road surface
Trees
1

7. Provide an accurate & complete overview of deficiencies
in the city through eco-friendly inspections
3
Detailed routes per area Image analysis using algorithms Registration in eform-app
Bicycles can go where cars can’t Validation of the inspections
Integration with maintenance
subcontractor systems
Key features of the inspections
Inspections (periodically) of designated areas by bicycle/vehicle
Number of identified objects, defects & exact GPS locations
Reporting & dashboard that is linked with the existing workflows
Purpose
Improve social & traffic safety
Minimize liability risks
Preventing peak workload for
maintenance parties in winter

8. Route
planning
Schedule
riders
Data gathering
& mobile
mapping
Before & after
inspections
Data
analysis
Reporting
municipality
Reporting
maintenance
contractor
Green and fast methods to get insights in the public space
by using bikes & technology
AI
2
Our approach:

9. Overview
• What are we working on?
• What is edge computing?
• How do we use edge computing?
• How do we use AWS?
• How can we integrate AWS?
• Example models

10. Edge computing is real-time computing at the end devices

11. What are edge devices?

12. A cloud computing provider can help integrate
applications with edge devices
• Cloud providers
• Amazon AWS
• Microsoft Azure
• Edge Impulse
• IoT (Edge) devices
• Jetson Nano
• Raspberry Pi
• Arduino Nano

13. Edge computing versus Cloud computing
Advantages of edge computing
• Real-time computing
• Limited sending of
information
• Privacy

14. Overview
• What are we working on?
• What is edge computing?
• How do we use edge computing?
• How do we use AWS?
• How can we integrate AWS?
• Example models

15. Light post detection at the edge
4
Inspections on a bicycle
Real-time inference on
images to detect defects
Bicycle-mounted edge
devices operate
autonomously.

16. From light post detection to tilted street lights
The angle of a light post
detected in an image is
determined in multiple
steps.
The angle of the light post is compared
to the horizon.
5
Light posts in a training set are labelled
using the Roboflow application and
models are trained.

17. Modular edge
device
• NVIDIA Jetson Nano or
NVIDIA Xavier NX
• 32 GB SD card
• 4G and WiFi connectivity
• Battery pack
• Camera
• GPS sensor and
gyroscope

18. Modular edge
device
Multi-vehicle use

19. Overview
• What are we working on?
• What is edge computing?
• How do we use edge computing?
• How do we use AWS?
• How can we integrate AWS?
• Example models

20. • Deploy software to all remote devices at
once from a central cloud location
• Software versioning
• CI/CD, Infrastructure as Code
• Modular Greengrass components make it
possible to design flexible architectures
adapting to multiple use cases
• Docker support
• Dealing with unstable network connections
reliably
Managing remote devices
with AWS Greengrass

21. • Every Greengrass component acts as a
microservice in a data pipeline, e.g.
taking pictures with a camera, doing the AI
inference, removing sensitive data,
uploading to the cloud, etc.
• Orchestrate event-driven workflows at the
device level, with components talking over
inter-process communication
• Trigger or schedule workflows remotely over
MQTT from AWS IoT Core
• Address individual edge devices using a
dedicated MQTT topic structure
Greengrass components
and event-driven workflow

22. • Using publicly available models
• Dedicated models to detect objects of interest, e.g. models from
municipalities for garbage detection
• Pre-trained models, such as YOLOv5 trained on a COCO dataset, to
detect faces for further removal
• Training custom models with transfer learning
• Training data is collected from the bicycles and labelled manually.
• Train models with Amazon SageMaker Studio
• Track training experiments and model performance with MLflow
Models and training

23. • Model optimization is crucial for battery-powered edge
applications.
• Limited to GPU capacity of the device (NVIDIA Jetson Nano vs NVIDIA
Xavier NX)
• Faster models (and more powerful devices) will allow us to move
from image processing to video processing to LIDAR processing at
the edge.
Model optimization and limitations of the edge

24. • With remote device management, scaling towards multiple edge
devices is easy.
• Remote connectivity enables real-time updates and event-driven
data processing.
• Modular design allows us to accommodate multiple use cases.
• Let’s make cities a safer place through cloud and edge technology
and AI.
• We have developed a blueprint for smart edge-computing
applications for use in public sector and research.
Cloud is the key to scalability at the edge

25. We can use AWS to integrate with our dashboard for simpler use
Integration AWS with a dashboard

26. Overview
• What are we working on?
• What is edge computing?
• How do we use edge computing?
• How do we use AWS?
• How can we integrate AWS?
• Example models

27. Examples of use cases for VeloTech
• Road deterioration
• Broken street lights
• Tilted lanterns
• Street signs pollution
• Lidar scans

28. Overview
• What are we working on?
• What is edge computing?
• How do we use edge computing?
• How do we use AWS?
• How can we integrate AWS?
• Example models

29. Road deterioration example
- text in Dutch

30. Broken street lights
example

31. Tilted street lights
example

32. Street signs pollution
example

33. Lidar scans – some ideas
Tilted street lights
Falling trees
Collapsing quays (‘kades’) and
bridges
Mapping the city

34. Questions?