Imaging objects obscured by occluders is a significant challenge for many applications. A camera that could “see around corners” could help improve navigation and mapping capabilities of autonomous vehicles or make search and rescue missions more effective. Time-resolved single-photon imaging systems have recently been demonstrated to record optical information of a scene that can lead to an estimation of the shape and reflectance of objects hidden from the line of sight of a camera. However, existing non-line-of-sight (NLOS) reconstruction algorithms have been constrained in the types of light transport effects they model for the hidden scene parts. We introduce a factored NLOS light transport representation that accounts for partial occlusions and surface normals. Based on this model, we develop a factorization approach for inverse time-resolved light transport and demonstrate high-fidelity NLOS reconstructions for challenging scenes both in simulation and with an experimental NLOS imaging system.
Non-line-of-sight Imaging with Partial Occluders and Surface Normals | TOG 2019
1. Non-line-of-sight Imaging with Partial
Occluders and Surface Normals
Felix Heide1,2, Matthew O’Toole1,3, Kai Zang1, David B. Lindell1, Steven Diamond1, Gordon Wetzstein1
1Stanford University 2Princeton University 3CMU
ACM SIGGRAPH 2019
10. Non-line-of-sight Imaging
[1] Velten et al. (2012)
[2] Gupta et al. (2012)
[3] Wu et al. (2012)
[4] Buttafava et al. (2015)
[5] O’Toole et al. (2018)
[6] Lindell et al.(2019)
Direct Pulsed
Measurement
[1] Katz et al. (2012)
[2] Katz et al. (2014)
[3] Smith et al. (2018)
[4] Kadambi et al. (2016)
[5] Heide et al. (2015)
Coherent and Modulated
Measurment
Accurate Image Formation Models
and Priors
[1] Xin et al. (2019)
[2] Thrampoulidis et al. (2019)
[3] Tancik et al. (2018)
[4] Chen et al. (2019)
Tracking using
Intensity Imaging
[1] Klein et al. (2012)
[2] Smith et al. (2018)
[3] Boger-Lombard and Katz (2018)
[4] Bouman et al.
11. Non-line-of-sight Imaging
[1] Velten et al. (2012)
[2] Gupta et al. (2012)
[3] Wu et al. (2012)
[4] Buttafava et al. (2015)
[5] O’Toole et al. (2018)
[6] Lindell et al.(2019)
Direct Pulsed
Measurement
[1] Katz et al. (2012)
[2] Katz et al. (2014)
[3] Smith et al. (2018)
[4] Kadambi et al. (2016)
[5] Heide et al. (2015)
Coherent and Modulated
Measurment
Accurate Image Formation Models
and Priors
[1] Xin et al. (2019)
[2] Thrampoulidis et al. (2019)
[3] Tancik et al. (2018)
[4] Chen et al. (2019)
Tracking using
Intensity Imaging
[1] Klein et al. (2012)
[2] Smith et al. (2018)
[3] Boger-Lombard and Katz (2018)
[4] Bouman et al.
Image Formation Without Occlusions *
* ( or explicitly requires them )
12. Continuous Image Formation without
Occlusions
Steered Laser
& SPAD
Detector
Scanned Wall
Area
x′i
y′
Confocal Measurements:
𝑡 [ps]
j 𝜌j
x
y
z
Path Length
Gating
Voxel
Volume
13. Continuous Image Formation with
Occlusions
Steered Laser
& SPAD
Detector
Scanned Wall
Area
x′i
y′
j
Confocal Measurements:
𝑡 [ps]
Visibility
Factors
Voxel
Volume
Normal
Factors
x
y
z
Path
Gating
j
14. Steered Laser
& SPAD
Detector
Scanned Wall
Area
x′i
y′
j Confocal Measurements:
𝑡 [ps]
Visibility
Factors
Voxel
Volume
Normal
Factors
x
y
z
Path
Gating
Continuous Image Formation with
Occlusions
15. Steered Laser
& SPAD
Detector
Scanned Wall
Area
x′i
y′
Confocal Measurements:
𝑡 [ps]
Retro
Reflection
n
Diffuse
Reflection
Visibility
Factors
Voxel
Volume
Normal
Factors
𝜔
x
y
z
j
Continuous Image Formation with Normals
16. Full Continuous Image Formation
Model
Steered Laser
& SPAD
Detector
Scanned Wall
Area
x′i
y′
Confocal Measurements:
𝑡 [ps]
Retro
Reflection
n
Diffuse
Reflection
𝜔
x
y
z
j
21. Copy across all time-stamps.
Copy Matrix:
Time-dependent Transport.
Sampling Matrix:
Discretized Operator Size
Discrete
Model:
Visibility Matrix:
Steered Laser
& SPAD
Detector
Scanned Wall
Area
𝑁
𝑁
𝑁
𝑁
𝑁
i
j
Normal Matrix:𝜔
nj
Matrix-Free
Implementation
22. Recover Factorized Transient Light Transport
Discrete
Model:
Optimization
Problem:
Spherical Normal
Parametrization
MAP Estimate
Regularizer (Prior)
23. Memory Limitations
Visibility Vars:
Steered Laser
& SPAD
Detector
Scanned Wall
Area
𝑁
𝑁
𝑁
𝑁
𝑁
i
j
Normal Vars:
Proposed Method:
Backprojection:
Matrix-Free Inverse:
For a volume of 643
Matlab (CPU) ~ 500GB and 2h
For a volume of 403
Cuda (GPU) ~ 14GB and 180s
24. Occluded Hidden Scene in 2D
2D Setup
Geometry:
No
Occlusion
Proposed
Measuremen
t
Detector
25. Occluded Hidden Scene in 2D
Backprojection
Method
Filtered
Backprojection
Linear Inverse
Method
Proposed
Approach
Ground
Truth
2D Setup
Geometry:
Detector
26. Occluded Hidden Scene in 3D
Filtered
Backprojection
Linear Inverse
Method
Proposed
Approach
Ground
Truth
27. Partially Occluded Hidden Scene in 3D
Filtered
Backprojection
Linear Inverse
Method
Proposed
Approach
Ground
Truth
Transient Measurement
𝑡 [ps]
44. Future Directions
Scattering Media and
Active Sources
Learned Visibility Matrices
And Scene Representations
1
6
4
1
2
8
2
5
6
1
2
8
3
2
6
4
5
1
2
2
5
6
3
2
Higher-Order Bounces
Every Surface Becomes a “Sensor”
45. Non-line-of-sight Imaging with Partial
Occluders and Surface Normals
Felix Heide1,2, Matthew O’Toole1,3, Kai Zang1, David B. Lindell1, Steven Diamond1, Gordon Wetzstein1
1Stanford University 2Princeton University 3CMU
ACM SIGGRAPH 2019
