For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/09/when-2d-is-not-enough-an-overview-of-optical-depth-sensing-technologies-a-presentation-from-ambarella/ Dinesh Balasubramaniam, Senior Product Marketing Manager at Ambarella, presents the “When 2D Is Not Enough: An Overview of Optical Depth Sensing Technologies” tutorial at the May 2021 Embedded Vision Summit. Camera systems used for computer vision at the edge are smarter than ever, but when they perceive the world in 2D, they remain limited for many applications because they lack information about the third dimension: depth. Sensing technologies that capture and integrate depth allow us to build smarter and safer applications across a wide variety of applications including robotics, surveillance, AR/VR and gesture detection. In this presentation, Balasubramaniam examines three common technologies used for optical depth sensing: stereo camera systems, time-of-flight (ToF) sensors and structured light systems. He reviews the core ideas behind each technology, compares and contrasts them, and identifies the tradeoffs to consider when selecting a depth sensing technology for your application, focusing on accuracy, sensing range, performance under difficult lighting conditions, optical hardware requirements and more.

1. © 2021 Ambarella Inc.
When 2D Is Not Enough: An
Overview of Optical Depth
Sensing Technologies
Dinesh Balasubramaniam
Ambarella

2. © 2021 Ambarella Inc. 2
Ambarella
Founded on the premise that video is a unique type of data requiring
an optimized SoC architecture
• DNN AI processors integrated with state-of-the-art video processors
yield highly optimized computer vision SoCs
Viewing and sensing SoCs with leading performance and efficiency
• Low-power and high-performance SoCs for IoT and Automotive
camera markets
2004
Ambarella founded
2012
IPO NASDAQ: AMBA
2021
280+ million
HD/4K cameras enabled
2015
Acquisition of
Santa Clara, CA
Company HQ
Tokyo
Hsinchu
Total Headcount
Detroit Parma
VisLab
Shanghai Seoul
Munich
785
Shenzhen
World-class engineering team
• >600 software, algorithm, VLSI and
image processing engineers globally
AI Vision Processors For Edge Applications

3. © 2021 Ambarella Inc.
Outline
• Why Depth?
• Depth Sensing Modalities
• Stereo Camera, Time of Flight, and Structured Light
• Operating Principles
• Design considerations: Resolution, Range, Precision, Bright light and Low-light
performance
• Review and comparison
• Applications
3

4. © 2021 Ambarella Inc.
Why Depth?
4

5. © 2021 Ambarella Inc.
Why Depth?
5
Which face is real? What is in the way?
Are all objects flat?

6. Modalities

7. © 2021 Ambarella Inc.
Depth Sensing Modalities
7
Sonar
Time of Flight module
Stereo camera
Radar Lidar
Structured Light module

8. Stereo Camera System

9. © 2021 Ambarella Inc.
Operating Principle – Stereo Camera System
= disparity
Right Image
Left Image

10. © 2021 Ambarella Inc.
Operating Principle – Stereo Camera System
B
f
dl
dr
𝐵
𝐷
=
𝐵 − 𝑑𝑙 − 𝑑𝑟
𝐷 − 𝑓
𝐷 = 𝑓
𝐵
𝑑𝑙 − 𝑑𝑟
B – baseline (distance between optical center)
D – distance to be measured
f – focal length of lens
d – distance of projected point from optical axis
• Distance inversely proportional to disparity
• Epipolar geometry helps to reduce search
space for matching point
• Finding matching point is called
correspondence problem
D

11. © 2021 Ambarella Inc.
Design Considerations – Stereo Camera Systems
• Systems are flexible and scalable – can accommodate many configurations with
appropriate change in sensor resolution, field of view (FoV), and baseline.
• Depth precision decreases with distance – non-linear error model.
• Good performance in bright light conditions. Low-light performance depends on
output of image pipeline.
• Challenged on scenes with low texture – improved with active illumination.
• Manufacturing process can be complicated – alignment of optical units is critical
during and after manufacture. Improved with periodic calibration or autocalibration
software.
11

12. © 2021 Ambarella Inc.
Design Consideration – Stereo Camera Systems
12
Stereo Left Stereo Right

13. © 2021 Ambarella Inc.
Performance – Stereo Camera Systems
13
Stereo Left Stereo Right

14. © 2021 Ambarella Inc.
Design Considerations – Stereo Camera Systems
14

15. Time of Flight

16. © 2021 Ambarella Inc.
Operating Principle – Time of Flight
• Two types of ToF sensors: direct ToF and indirect ToF.
• Direct Tof sensors emit a pulse of light and measure the time it takes for the reflection
to arrive at the sensor. Pulse strength and duration can be varied to allow long range
measurements (up to kms).
• Indirect ToFs also called continuous-wave ToFs emit a pulsed, modulated light signal
and measure the phase difference of the reflected signal to derive depth.

17. © 2021 Ambarella Inc.
Operating Principle – Time of Flight
Phase
Depth
Integration time Read out time
∅ = 90
∅ = 0 ∅ = 180 ∅ = 270

18. © 2021 Ambarella Inc.
Operating Principle – Time of Flight
Depth Amplitude

19. © 2021 Ambarella Inc.
• Depth range depends on modulating frequency of the IR source - maximum
measurable distance for given frequency called unambiguous range.
• LEDs can be used up to 30MHz. VCSELs (vertical-cavity surface-emitting laser) can be
used if higher modulating frequencies are required (up to 100MHz).
Design Consideration– Time of Flight
Modulating Frequency
(MHz)
Unambiguous Range
(m)
20 7.5
50 3
75 2
100 1.5

20. © 2021 Ambarella Inc.
Design Considerations – Time of Flight
• Comparatively low resolution (VGA current max) but unaffected by scene texture
• Dense depth information where field of view and field of illumination overlap
• Range limited by modulating frequency of illuminator. Depth precision decreases with
distance for a given modulating frequency.
• Good performance in low-light conditions. Sunlight affects bright light performance –
improved when using 940 nm IR illuminator.
• Scattered reflections and cross talk can impact depth measurement.
• Integration time impacts number of fps but can handle motion.
• Combining multiple systems can be challenging – can be separated by frequency.
20

21. © 2021 Ambarella Inc.
Design Considerations – Time of Flight
21
* Data from vendor module

22. Structured Light

23. © 2021 Ambarella Inc.
Operating Principle – Structured Light
• Light source plus a pattern generator
project a random pattern.
• When image plane moves pattern
shifts depending on object distance.
• Reflected pattern correlated against a
reference stored on the system.
• Depth depends on disparity between
the reference and reflected pattern.

24. © 2021 Ambarella Inc.
Operating Principle – Structured Light
B
D
f
R
R1 R2
p1 p2
𝑓
𝑝2 − 𝑝1
=
𝑅
𝑅2 − 𝑅1
𝐷
𝐵
=
𝑅 − 𝐷
𝑅2 − 𝑅1
𝐷 =
𝑅𝐵𝑓
𝑅 𝑝2 − 𝑝1 + 𝐵𝑓
B – baseline (distance between optical center)
f – focal length of lens
R – distance to reference plane
D – distance to be measured
𝑅𝑛 – Point on reference plane
𝑝𝑛 – projected points on image plane

25. © 2021 Ambarella Inc.
System Considerations – Structured Light
• Moderately flexible – sensor resolution, baseline, and pattern density can be varied
for different applications.
• Depth precision decreases with distance.
• Good performance in low-light conditions. Lower performance when sunlight is
present – improved when using 940 nm IR illuminator.
• Spatial resolution higher than time of flight sensors but depth density depends on
projector pattern.
• Frame rate depends on sensor and depth processing speed.
25

26. © 2021 Ambarella Inc.
System Considerations – Structured Light
26

27. Comparison

28. © 2021 Ambarella Inc.
Comparison
Stereo Time of Flight Structured Light
Depth Range Short to long range Low to mid range Low to mid range
Depth Precision mm to cm mm to cm mm to cm
Depth Resolution
High – can detect finer features or
objects further away
Up to VGA Pattern dependent
Active emitter Optional Required Required
Bright light performance Good Fair – improved with 940 nm IR Fair – improved with 940 nm IR
Low-light performance Depends on ISP Good Good
Cost High Low Mid

29. Applications

30. © 2021 Ambarella Inc.
Applications
30
Long-range stereo for
autonomous driving
ToF for occupancy sensing Structured light for access control

31. Thank You