The Path to Product Excellence: Avoiding Common Pitfalls and Enhancing Commun...
EDGE CASE TESTING IN A SENSOR-REALISTIC SIMULATION ENVIRONMENT
1. Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
Edge case testing in a
sensor realistic simulation
environment
Mike Dempsey
Managing Director
Claytex
2. Copyright Claytex Services Limited 2022
• Multi-domain modelling and simulation
with Dymola, Modelica and FMI
• Virtual testing for ADAS and
Autonomous Vehicles with AVSandbox
• Requirements Traceability with Reqtify
• Software, Consulting, Training
• Part of TECHNIA, PLM & Engineering
Knowledge Leaders
3. Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
Edge case testing in a sensor
realistic simulation environment
Why do we want to do this?
4. Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
The journey to AV safety
• Proving an AV is safe to deploy requires
testing on diverse scenarios
• Edge cases are those scenarios that occur
infrequently but cover all the risk
• Due to the risk involved and number of
edge cases we need to use simulation for
these tests
• The simulation platform must provide a
realistic, immersive, environment for the AV
controller
• New EU regulations place stringent
requirements on the simulation platform
used to generate evidence for the safety
case
5. Copyright Claytex Services Limited 2022
• UK government funded project, led by ∂RISK, other
partners are Imperial College London and DG Cities
• Funded through Innovate UK and CCAV
• https://www.drisk-project.org/
• 3 year project, concluded September 2020
• Project goal: to develop the ultimate driving test for
autonomous vehicles
• Key output is a Virtual Verification and Validation platform
• Edge case database
• Sensor realistic simulation platform
• Research on safety topics including “What is safety” and
“How do we measure safety”
7. Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
Simulation Platform
• Scenario basedAutonomous
Vehicle simulation platform
• Sensor realistic simulation with
physics-basedmodels
• Multibody vehicle dynamics
• Real-time, high-fidelity and fully
synchronous modes
• Supports open standards including
OpenSCENARIO, OpenDRIVE and
FMI
• Built around rFpro
8. Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
Powered by rFpro
• Engineering simulation tool, not a
gaming engine
• rFpro focuses on physically modelling
the real world
• Physics based rendering
• High fidelity road surface
• Millimetre accurate digital twins of real
world locations
• Fully synchronous, deterministic
simulation when distributed across
multiple processes and multiple
machine
• Real-time simulation mode to support
HiL and DiL (Passenger-in-the-Loop)
9. Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
System under test
• Autoware based controller
• Setup with assistance from
WMG, University of Warwick
• No changes required to use
with the simulator
• Sensor setup
• Velodyne Ultra Puck
• Single forward-facing camera
with RCCC filter
• GPS
• ODD limited to handling basic
junctions, no traffic lights, with
traffic and pedestrians in
good weather conditions
12. Copyright Claytex Services Limited 2022
Camera Simulation
• Lens distortion models including
360-degree fisheye camera
• Colour filter
• Dynamic exposure control
• Detailed shutter effects
• Apply masks to simulate dirt on
the lens
• Publish direct to ROS camera
topic
• HiL solutions available for real-
time camera simulation
13. Copyright Claytex Services Limited 2022
LiDAR model details
• Include rotational dynamics of spinning sensors
• Range measurement
includes noise that
effects precision
• Also has a deterministic
mode
• Weather model effects
measurement precision,
intensity and includes
random drop-outs
• Output the same
message stream as the
real device
14. Copyright Claytex Services Limited 2022
ROS Interface
• ROS bridge developed for
Autoware and Project Aslan
• Subscribes and publishes
ROS topics directly
• Sends state information
• Reads driving demands
• Publishes /clock to ensure
synchronization between
simulator and ROS
• ROS-camera sensor
published camera topics
direct to ROS
15. Copyright Claytex Services Limited 2022
Copyright Claytex Services Limited 2022
Testing our AV
Using a sensor realistic simulation
platform allows us to integrate the AV
controller without modification
16. Copyright Claytex Services Limited 2022
Test Automation
• Parameter sweeps can be
configured to cover nominal and
critical scenarios
• Interface to scenario database
which uses adaptive testing to
cover ODD
• OpenSCENARIO format used to
exchange scenario and results
• Integrate with CI/CD through
command line driven testing and
test-series input files
17. Copyright Claytex Services Limited 2022
KPI calculation
• Open API to allow you to
define your own metrics
and dashboards
• For example:
• RSS
• Safe space
• Accurate parking
• TTC, TTLC, etc.
• Standard reports linked
to the scenario
21. Copyright Claytex Services Limited 2022
How does this get us to a safe AV?
• Sensor realistic simulation
platform means no changes
needed to the controller to run in
simulation
• High fidelity models include noise
to explore robustness
• Open and extendible with support
for open standards
• Interfaces to scenario databases
provide the test gauntlet
• We continue to develop and
increase the level of detail
available and the validation of
sensors against real-world data
