Mitchell Reifel (pmdtechnologies ag): pmd Time-of-Flight – the Swiss Army Knife of 3D depth sensing

c o n f i d e n t i a l© 2017 pmdtechnologies ag Infineon Technologies AG
pmd ToF
The Swiss army knife of 3D sensing
AWE 2018

Why 3D? Two Driving Forces
1. More and more image sensors are never viewed by a human
(robots, autonomous cars, security, etc.). As seen by this
image, computers need 3D to view the world accurately.
2. Human-viewed 3D imaging is driven by using depth to improve
2D images, overlaying data on top of real-world images
(augmenting reality) and for man-machine interface (MMI)
This is NOT two guys with a 1 meter Eiffel Tower!
© 2018 pmdtechnologies ag www.pmdtec.com | can you imagine

2015 2016 2017 2018 2019 2020 2021
Markets and 3D Rollout
Mobile
3D Selfie
Mobile
3D Rear Facing
Consumer Mobile
Augmented Reality
Interior
Automotive
Exterior 3D for
Autonomous
Driving
Personal
Robotics
Drones Retail
Analytics
VR with
3D
OtherAutoMobile
Enterprise AR

3D Technology Overview

1. Stereoscopic – Uses parallax, or the displacement of two, 2D-images,
to reconstruct a 3D image.
2. Structured Light – Projects a infrared pattern (or multiple patterns)
into the real world, and uses a 2D imager to measure
distortion/displacement of the pattern. It ultimately uses triangulation to
get to a 3D reconstruction.
3. Time-of-Flight (TOF) – Like “RADAR”, it bounces light off an object.
By calculating the time it takes light to leave the camera and return it
can reconstruct the a 3D map of the real world.
3D Imaging Technologies Can Broadly be Categorized
into Three areas
Baseline
Baseline

Classification of 3D Imaging Techniques
3D
Techniques
Radio Wave Light Wave
Stereo
Passive
Stereo
Active Stereo
Structured
Light
Spatial Temporal
Time of
Flight
(TOF)
Direct
(LIDAR,
SPAD)
Indirect
(Phase TOF)
Sound Wave
Intel
Intuitive
Mantis
Intel
PMD
Sony
Apple Internal
QCOM/Himax
ST Micro (SPAD)
Velodyne (LIDAR)
(Rarely used in
consumer
electronics)

confidential
pmd Time-of-Flight (“indirect” ToF)
• Allows integrated one-chip solutions on standard CMOS processes
• No baseline requirement, allows smaller modules
• Allows multiple “modes” to do both near and far with the same module
• Provides both depth as well as reflected IR intensity signal
• The camera has a carrier light source
(usually 850nm or 940nm) and a
modulated signal
• The phase shift of the reflected signal
compared to the outgoing signal
provides an indication of the distance

Structured Light vs TOF for Size

Upcoming Time of Flight 3D Selfie
TOF features are resolved as in iPhone, eye-sockets even clearer
TOF also provides IR intensity image, which can improve face ID
algorithms
iPhone X Structured Light
Provides Depth Map Only
PMD Time of Flight
Provides Depth Map plus IR
Intensity Image

PMD Modes for the CamBoard pico flexx
Use case
Range
[m]
Frame
rate
Exposure
time (us)
Modulation
Frequencies
Indoor room reconstruction 1.0 – 4.0 5 fps 2000 80 & 60 MHz
Indoor navigation 1.0 – 3.0 10 fps 1000 80 & 60 MHz
3D object reconstruction 0.5 - 1.5 15 fps 700 80 & 60 MHz
Face reconstruction 0.3 – 1.0 25 fps 450 80 & 60 MHz
Small object recognition 0.3 – 2.0 35 fps 600 60 MHz
Hand tracking 0.1 – 1.0 45 fps 500 60 MHz
4 m 3 m 2 m 1 m 0 m
TANGOIndoor room reconstruction (5 frames/s, 2 ms int. time)
Room scanning, indoor navigation (10 frames/s, 1 ms int. time)
3D object reconstruction (15 frames/s, 0.7 ms int.)
Medium size object recognition, face reconstruction (25 frames/s, 0.45 ms int. t)
Remote collaboration, table-top gaming (35 frames/s, 0.6 ms int. time)
Hand tracking (45 frames/s, 0.5 ms int.)

3D Technology Comparison
Feature Active Stereo Structured
Light
Phased Time-of-
Flight
Multiple Distances No No Yes
3D Data Generation/Processing
Requirements
Triangulation;
Heavy SW processing
Triangulation or dot
dispersion placement;
Medium SW Processing
Direct Measurement;
Light Software Processing
Time from 3D capture to output
(Latency)
Slow Medium Fast
Active Illumination Yes
(Laser or LED)
Yes
(Laser Only)
Yes
(Laser or LED)
Low-Light Performance Excellent Excellent Excellent
Sun-Light Performance Excellent Poor Good
Power Consumption Low on Camera
High on Processor
Medium on Camera
High on Processor
Medium on Camera
Low on Processor
Size limited by baseline
between cameras
limited by baseline
camera& projector
Small; No Baseline
Resolution Determined by pixel
count and filtering
software
Determined by emitter
pattern
Determined by pixel count
Calibration and Alignment Difficult due to
Baseline
Difficult due to Baseline Simple since no Baseline

Questions to Ask for Your 3D Application
 Framerate?
 Distance?
 Multiple Distances?
 Resolution?
 Accuracy?
 Size?
 Outdoor use?
 Memory Available?
 Processing Available?
 Power Budget?
 Cost Budget?
 Calibrate before or after product assembly?
 Is Drop Test Important?
 Supply Chain?
 Second Source Required?
 …

Resolution and Framerate Vs. Functions
3D Depth Points
High
(60fps)
Medium
(30fps)
Low
FrameRate(framespersecond)
Photo Composition
Flash Level Control
Auto Exposure
Object Detection AF
Object Tracking AF
Face
Unlock
Advanced
Gestures
Simple
Gestures
Background
Subtraction
SLAM
VR/AR Natural hand
interaction
10 100 1K 10K 100K
Boke
Effect
3D Ruler
VR/AR Object/Room
Interaction
Ranging
Range Finding Full Point Cloud
AR/VR

Console VR Mobile VR
Virtual Reality Augmented Reality
Enterprise Consumer Mobile
Will be a major segment
of the gaming market and
Will be a few major players:
Sony Playstation VR, Oculus,
HTC Vive, Eventually Xbox
On the market today with
Samsung VR and Google Cardboard
It is not seeing broad customer
acceptance and seems will be a
niche market
Expensive high-end
Enterprise systems for
R&D, etc. Not forecast to
ramp into a consumer product
Forecasted to be an
extension to and possibly
a replacement of the
mobile phone
VR/AR Segmentation
3D In Design 3D in R&D 3D in Production ?

3D in Enterprise AR Headsets
 Most enterprise and developer AR
headsets today include 3D
sensors, including Microsoft, Meta,
Daqri, etc, and 3D will increase
penetration in this segment
 The depth camera is used to find
and track hands for gesture and
hand interaction with the virtual
world
Microsoft Hololens Sensor List
• 1 IMU
• 4 environment understanding
cameras
• 1 depth camera for Gesture Sensing
• 1 2MP photo / HD video camera
• Mixed reality capture
• 4 microphones
• 1 ambient light sensor
Virtual Displays
3D Detection of
hand

3D in Virtual Reality
 Today’s VR systems use controllers to manipulate within the
virtual world, which is somewhat unnatural. 3D will allow a
more natural interaction.
 In “Long Range Mode” surroundings of the user (desk, couch,
table) to provide warnings when a user is too close to an
obstacle while being immersed
Depth Sensor
Users real hands brought into
Virtual world

3D Imaging, the Mobile Handset
Market and the Evolution to Augmented
Reality

Mobile 3D So Far
Mobile 3D Selfie Mobile 3D World Facing
Models Launched
Today
Apple iPhone X ASUS ZenfoneAR
Models Launched September 2017 January 2017
3D Tech/Vendor Used Internal Structured Light Infineon/PMD Time-of-Flight
Use-Cases Face unlock, Animojis AR, Room Scanning
Competitive Entries In Late 2018/2019 2019/20
Competitive Entries will
use…
Time-of-Flight, Structured
Light
Time-of-Flight, Maybe
Structured Light

Rear Facing 3D Market Evolution
 3D hardware development has been pushed by Google on the
Android side, ending up with the Asus ZenFone launch, which is
the first commercial mobile phone with a 3D sensor (front or rear)
 However in late 2017 Google pulled the Tango effort into the 2D
ARCore Augmented Reality effort.
 This is seen as an effort to establish ecosystem first using existing 2D
sensors and accelerometers, then bring in full 3D sensors later
Qualcomm announces
Tango cellphone reference
Design with world facing
3D imaging
Lenovo launches the first
Tango mobile device with
world facing 3D imaging, the
Phab 2 Pro
Asus launches the first Tango
smartphone with world facing
3D imaging, the ZenFone AR
Google launches Tango
project and tablet reference
design with world facing 3D
imaging
Google merges Tango into
their ARCore development kit
December 2017

Why ARCore and ARKit Will Move to 3D
Using 3D data with AR adds realism, immersion and
details not capable with just a 2D camera
Feature 2D+IMU
(ARCore and ARKit)
3D Depth
6 Degree of Freedom
Tracking
Yes Yes
3D Awareness Basic Plane Full Environmental
Light and Shadows Faked Photorealistic
3D Object Placement Yes Yes
Occlusion No Yes
3D Scanning No Yes
Visualize Objects in Scale No Yes
Depth of Field No Yes
Comparison of using 2D Sensor vs. 3D Sensor for AR

3D Sensors “World Facing” Sensors
Are Mobile Phone AR Displays Next?
Handset
AR with 2D Overlay
Handset
AR with 3D Overlay
HEADset
AR with 3D Overlay
Establish ecosystem
With ARCore, ARKit
Add rear 3D imagers
to improve experience
Migrate to head’s up
displays as accessory
(with 3D sensors)

Why AR Glasses Will Succeed Next Time
 Google Glass, introduced in 2013, never took off because the technology was early, and smart phone
penetration and “always on” aps were still relatively low penetration.
 Now that people live through their cellphones even while doing other things acceptance of mobile AR
will come more naturally. In addition, the technology will be better, smaller and more stylish
 3D sensors, which allow for gesture interaction in front of the device on a virtual interactive display, will
be required

Major Tech Signaling for Mobile AR
Apple: “We’re already seeing things that will transform the way you
work, play, connect and learn. Put simply, we believe AR is going to
change the way we use technology forever.” – Apple CEO Tim Cook,
Apple earnings call, November 2, 2017*
All the major tech players are already signaling a big jump into AR
And they will need 3D sensors!
Facebook:

Summary
 Of the 3D technologies, TOF provides the most
flexibility and size advantage
 3D imaging is a quickly growing market, and will be a
key part of VR/AR hardware
 TOF is the 3D imager of choice in most enterprise
headsets today
 As AR moves to mass consumer, 3D will be a key
enabler
7/12/2018 24

More Info?
 Come see PMD in Booth 824 (In Enterprise Pavilion) for demos and info
 Buy/Play with one of our USB development kits, in stock at Automation
www.automation24.com

Mitchell Reifel (pmdtechnologies ag): pmd Time-of-Flight – the Swiss Army Knife of 3D depth sensing

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Mitchell Reifel (pmdtechnologies ag): pmd Time-of-Flight – the Swiss Army Knife of 3D depth sensing

Similar a Mitchell Reifel (pmdtechnologies ag): pmd Time-of-Flight – the Swiss Army Knife of 3D depth sensing (20)

Más de AugmentedWorldExpo

Más de AugmentedWorldExpo (20)

Último

Último (20)

Mitchell Reifel (pmdtechnologies ag): pmd Time-of-Flight – the Swiss Army Knife of 3D depth sensing

Notas del editor