Whether you are new to machine learning or ready to build on your existing skills, we can help you get ready to race. Developers with no prior machine learning experience can get started by taking the AWS DeepRacer digital training course or this hands-on workshop with Q&A session to get familiar with the basics of reinforcement learning (a branch of machine learning that's ideal for training autonomous vehicles) and AWS DeepRacer.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Developer, Start Your Engine!
Donnie Prakoso
Technical Evangelist, ASEAN
Amazon Web Services
@donnieprakoso
donnieprakoso

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
> hello, world
Donnie Prakoso, MSc
Technical Evangelist, AWS
@donnieprakoso
donnieprakoso
• Speak in Go and Python
• Microservices and Machine Learning
• I AWS User Groups
• Self-proclaimed Barista and Café Racer enthusiasts

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Agenda
• AWS DeepRacer origin
• RL for the Sunday driver
• Virtual simulator
• Under the hood
• Rubber meets the road

4. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

6. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How can we put machine learning in the hands of all
developers?

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ML
FRAMEWORKS &
INFRASTRUCTU
RE
AI SERVICES
REKOGNITION
IMAGE POLLY TRANSCRIBE
TRANSLATE COMPREHEND LEX
REKOGNITION
VIDEO
Vision Speech Language Chatbots
AMAZON
SAGEMAKER
BUILD
TRAIN
FORECAST
Forecasting
TEXTRACT PERSONALIZE
Recommendations
DEPLOY
Pre-built algorithms & notebooks
Data labeling (GROUND TRUTH)
One-click model training & tuning
Optimization (NEO)
One-click deployment & hosting
ML SERVICES
Frameworks Interfaces Infrastructure
EC2 P3
& P3N
EC2 C5 FPGAs GREENGRASS ELASTIC
INFERENCE
Reinforcement learning
Algorithms & models (AWS MARKETPLACE
FOR MACHINE LEARNING)
The Amazon ML stack: Broadest & deepest set of
capabilities

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How can we put
reinforcement learning
in the hands of all
developers? literally

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer: An exciting way for developers to get hands-on experience
with reinforcement learning
Global Racing League
DeepRacer Console
Virtual simulator, to
train and evaluate
1/18 scale autonomous
race car

10. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reinforcement learning in the broader AI context
Reinforcement
Learning
Supervised
Learning
Unsupervised
Learning

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Machine learning overview
SUPERVISED UNSUPERVISED REINFORCEMENT

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reinforcement learning in the real world
Reward positive
behavior
Don’t reward
negative
behavior
The result!

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Reinforcement learning terms
AGENT ENVIRONMENT STATE
ACTION
EPISODEREWARD

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The reward function
The reward function incentivizes particular
behaviors and is at the core of
reinforcement learning

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The reward function in a race grid
S G = 2
GOALAGENT

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Incentivizing centerline behavior
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
S 2 2 2 2 2 2 G = 2
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
8.6 9.5 8.5 7.5 6.3 5.0 3.5 1.9
S 10.4 9.4 8.2 6.9 5.4 3.8 G = 2
8.6 9.5 8.5 7.5 6.3 5.0 3.5 1.9
Discount per step
0.9

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer problem formulation
STATE

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How does learning happen? VALUE FUNCTION
POLICY FUNCTION

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
RL algorithms: Vanilla policy gradient
* Image Source: Landscape image is CC0 1.0 public domain
Data is only used once
• High variance of rewards
• Magnitude of update could be too large
J(q)New
weights
New
weights
0.4 ± ! 0.3 ± !

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
RL algorithms: Proximal policy optimization (PPO)
(State, action, reward,
next state)
(st,at, rt, st+1)
Advantage
Improved model

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
METHOD Supervised learning
HOW IT WORKS Expert driver controls a real
world car, that has a camera. Save the images
from the camera as inputs and corresponding
driving actions (speed and steering angle) as
outputs. Train a model.
RESULT Provide state(image) into model and
receive driving action
RL vs. other approaches for robotic racing
METHOD Reinforcement learning
HOW IT WORKS Virtual agent repeatedly
interacts with a simulated environment and
logs experience (image, action, new state,
reward). Experience is used to train a model,
and new model is used to get more
experience.
RESULT Provide state(image) into model and
receive driving action

23. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Cloud
AWS
DeepRacer
NAT gateway
VPC
AWS DeepRacer
Models
Simulation
video
Metrics
AWS DeepRacer simulator architecture

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer console diagram

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Programming your own reward function

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Track components
TRACK CENTER
TRACK WALL
TRACK SURFACE aka ON-TRACK
FIELD aka OFF-TRACK
TRACK BOUNDARIES

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Coordinate system and track waypoints
WAYPOINTS
X
Y
TRACK WIDTH
HEADING

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Hyper parameters control the training algorithm

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Action space

31. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DeepRacer car specifications
CAR 18th scale 4WD with monster truck chassis
CPU Intel Atom Processor
MEMORY 4 GB RAM
STORAGE 32 GB (expandable)
WI-FI 802.11ac
CAMERA 4 MP camera with MJPEG
DRIVE BATTERY 1000 mAh lithium polymer
COMPUTE BATTERY 13600 mAh USB-C
SENSORS Integrated accelerometer and gyroscope
PORTS 4x USB-A, 1x USB-C, 1x Micro-USB, 1x HDMI
SOFTWARE Ubuntu OS 16.04.3 LTS, Intel OpenVINO
toolkit, ROS Kinetic

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
ROS msg node
Stored file
ROS nodes
Web
Server
Publisher
Model
Optimizer
Video
M-JPEG
Web Server
Video
Inference
Results
Autonomous
Drive
Control
Node
Optimized
Model
Media engine
Camera
Model
Inference
engine
Manual
Drive
Navigation
Node
Servo & Motor
AWS DeepRacer software architecture

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Simulation-to-real domain transfer
SIM-to-REAL CHALLENGE
Train model using simulated images, but
the race car using the images the car
experiences in the real world
STRATEGIES
Environment control
Domain randomization
Modularity and abstraction

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Lab 1 – AWS DeepRacer service
OBJECTIVE Build your first AWS DeepRacer RL model
TIME 50 min.
1. Find the lab content here:
https://github.com/aws-samples/aws-deepracer-workshops/
2. Navigate to: Workshops/2019-AWSSummits-AWSDeepRacerService/Lab1

36. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Race for prizes and glory in the AWS DeepRacer League
Train your AWS DeepRacer model and compete:
• Online in the Virtual Circuit
• In person in the Summit Circuit (visit the Expo Hall)
www.deepracerleague.com

38. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.