AirSim is a simulator for autonomous vehicles built on Unreal Engine. AirSim provides a rich platform to develop autonomy by enabling use of AI technologies, such as deep learning, computer vision, reinforcement learning etc. It is open-source, cross platform and supports hardware-in-loop simulation, thus allowing rapid development and testing of the system. The simulation is developed as a plugin and can be simply be dropped into any Unreal environment. AirSim supports AI development capabilities by exposing APIs to enable data logging and controlling vehicles in a platform independent manner. We will give an overview of how to use AirSim for building realistic simulation environments and doing development for quadrotors that use popular flight controllers such as Pixhawk. It is developed as a plugin that can simply be dropped in to any Unreal environment you want. We will also showcase how the system can be used to incorporate machine learning components useful for building such autonomous systems.
製品/テクノロジ: AI (人工知能)/Deep Learning (深層学習)/Linux/Machine Learning (機械学習)/OSS/Windows
Ashish Kapoor
Microsoft Corporation
Microsoft Research
Principal Researcher
Shital Shah
Microsoft Corporation
Microsoft Research
Principal Research Software Development Engineer
6. Many Recent Successes of Machine Learning
Peng et al. 2016
Mnih et al. 2013Silver et al. 2016
1. Close-world systems (also only in software)
2. Luxury to obtain near-infinite data via
simulation etc.
7.
8. Can we use such ML methods to build
systems that operate in real-world?
9. Successes of Machine Intelligence Real World Flying Systems
Peng et al. 2016
Mnih et al. 2013Silver et al. 2016
14. Three Fundamental Challenges
Lack of large
amount of data
• High Sample
Complexity of
ML Methods
Computational
Constraints
• Real-time
Performance,
Limited
Memory and
Compute
Power
Operating in the
Open World
• Safety,
Uncertainty,
Mixed-
Initiative
Autonomy
19. What Does Simulator Enables?
• Wide variety of environment, day of time, weather patterns
• No legal hassle, much cheaper and safe
Generate lots of
training data
• Run exact same code that would be run onboard
• Slow down or accelerate simulated time
Develop autonomy
algorithms
• If it doesn’t work in simulator then it won’t work in real world
• Use sensors with varies parameters
Test perception
algorithms
• Can fail thousands of time to learn patterns
• Run in cloud for distributed learning
Reinforcement
Learning techniques
20. Why Use Unreal Engine?
Actual footage captured from simulated drone in AirSim with Open World Environment
21. The Heart of the Vehicle
Sensor Input
Motor Output
Sensors generate
information about the
world around
Motor voltages drives the
propeller
Humans provide
desired state
Sensor Input
Desired
State
22. How Simulator Works?
Sensor Input Motor Output
Desired
State
Humans provide
desired state
Generate all the
sensor data using
ground truth
Run physics engine
to get Lin & Ang
• Position
• Velocity
• Acceleration
23. What Does Physics Engine Do?
Physics
Engine
Force
Torque
Position
Velocity
Acceleration
Linear & Angular Flavors
Total of 3 + 3 = 6 vectors
25. May the Force & Torque be with You
𝐹𝐹 = 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 ∗ 𝐾𝐾𝐹𝐹
𝑇𝑇 = 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 𝑀𝑀𝑀𝑀𝑀𝑀𝑀𝑀 ∗ 𝐾𝐾𝑇𝑇
𝐾𝐾𝐹𝐹 ∝ 𝜌𝜌𝐷𝐷4
𝐾𝐾𝑇𝑇 ∝ 𝜌𝜌𝐷𝐷5
26. Physics - Linear Dynamics
𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 =
𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓
𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚
𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑦𝑦𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 += 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 ∗ 𝑑𝑑𝑑𝑑
𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 += 𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑦𝑦𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 ∗ 𝑑𝑑𝑑𝑑
*In real physics engine you would use better integrator than Euler
27. Physics - Angular Dynamics
𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 =
𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 − 𝜔𝜔 × (𝐼𝐼 𝜔𝜔)
𝐼𝐼
𝜔𝜔𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 += 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 ∗ 𝑑𝑑𝑑𝑑
𝑞𝑞𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 = 𝑞𝑞𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 ∗ 𝑄𝑄(𝜔𝜔 ∗ 𝑑𝑑𝑑𝑑)
*In real physics engine you would use better integrator than Euler
*Calculations are in body frame
28. Simulating IMU
Physics engine tells us,
• Angular velocity
• Linear acceleration
So just add noise, bias and drift!
𝐼𝐼 𝐼𝐼 𝐼𝐼 = 𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺 + 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴
31. AirSim is Open Source
Unreal Engine 4
• Designed as UE4
plugin
• Just drop in to
100s of realistic
environments
APIs for Dozens
of Languages
• Get camera
images
• Send commands
to the vehicle
C++ Header-
Only Library
• Eigen library as
only dependency
• Cross-platform
Highly
Extensible
• Add new vehicles
models
• Add new modes
33. AirSim Has APIs
from PythonClient import *
import cv2
import sys
client = AirSimClient('127.0.0.1')
# get depth image
result = client.setImageTypeForCamera(0, AirSimImageType.Scene)
# show image in opencv
rawImage = np.fromstring(result, np.int8)
png = cv2.imdecode(rawImage, cv2.IMREAD_UNCHANGED)
cv2.imshow("Camera Image", png)
Few lines of Python code can get you FPV image from drone!
Scene from Unreal Boy with a Kite environment
34. Make Drone Move in AirSim Using APIs
from PythonClient import *
import sys
client = AirSimClient('127.0.0.1')
# Stay 5 meters above ground
z = -5
# Fly!
client.moveOnPath([(0,-253,z),(125,-253,z),(125,0,z),(0,0,z)],
15, 0, DrivetrainType.ForwardOnly,
YawMode(False,0), 20, 1)
Same code can be ran from offboard computer on real drone!
Other languages available: C++, C#, Java and many more!
36. AirSim Extensibility
New Vehicle
Types
Car
Apollo Moon
Lander
New Sensor
Types
Ultrasonics
LIDAR
New Physics
Engines
PhysX
Bullet
New Sim
Modes
Computer
Vision
Just Physics
You can contribute on GitHub: https://github.com/microsoft/airsim
38. AgIoT: Precision Agriculture
Scan farm using drone to capture low level details on daily basis, analyze differences each day
and fuse with sensor information to identify areas that needs specific work
41. Autonomous 3D Scanning of Large Structures
3D reconstruction
in simulator using a
simulated drone
flight
3D reconstruction
in real world using
actual drone
49. Optimal Scanning: What Trajectory to Fly?
• Safety: How risky are the flight maneuvers?
• Efficiency: How long does the process take?
• Performance: How good is the 3D model?
