Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Non-line-of-sight Imaging with Partial
Occluders and Surface Normals
Felix Heide1,2, Matthew O’Toole1,3, Kai Zang1, David ...
Non-line-of-sight Imaging
Non-line-of-sight Imaging
Hidden Objects Behind Hidden Objects
Experimental Non-line-of-sight Imaging
Experimental Non-line-of-sight Imaging
Reconstruction (TV-Prior):
Simulated Transients :
𝑡 [ps]
Partially-Occluded
Non-line-of-sight Imaging
Reconstruction (TV-Prior):
Time-tagged Photon Counts – Transient Images
Experimental measurements Photograph of hidden scene
𝑡 [ps]
Measurement Slice Reconstruction
Transient Image Slice
Non-line-of-sight Imaging
[1] Velten et al. (2012)
[2] Gupta et al. (2012)
[3] Wu et al. (2012)
[4] Buttafava et al. (2015...
Non-line-of-sight Imaging
[1] Velten et al. (2012)
[2] Gupta et al. (2012)
[3] Wu et al. (2012)
[4] Buttafava et al. (2015...
Continuous Image Formation without
Occlusions
Steered Laser
& SPAD
Detector
Scanned Wall
Area
x′i
y′
Confocal Measurements...
Continuous Image Formation with
Occlusions
Steered Laser
& SPAD
Detector
Scanned Wall
Area
x′i
y′
j
Confocal Measurements:...
Steered Laser
& SPAD
Detector
Scanned Wall
Area
x′i
y′
j Confocal Measurements:
𝑡 [ps]
Visibility
Factors
Voxel
Volume
Nor...
Steered Laser
& SPAD
Detector
Scanned Wall
Area
x′i
y′
Confocal Measurements:
𝑡 [ps]
Retro
Reflection
n
Diffuse
Reflection...
Full Continuous Image Formation
Model
Steered Laser
& SPAD
Detector
Scanned Wall
Area
x′i
y′
Confocal Measurements:
𝑡 [ps]...
Factorized Transient Light Transport
Continuous Model:
Discretize
Measurement
𝑥′
, 𝑦′
, 𝑡:
Discrete
Model:
Discretize Spac...
Discretized Operator Size
Discrete
Model:
Steered Laser
& SPAD
Detector
Scanned Wall
Area
𝑁
𝑁
𝑁
𝑁
𝑁
i
j
How much light can...
Visualization of the Visibility Matrix
Ground Truth Scene Visibility Term for
Pixel Position indicated on the leftViewing ...
Discretized Operator Size
Discrete
Model:
Steered Laser
& SPAD
Detector
Scanned Wall
Area
𝑁
𝑁
𝑁
𝑁
𝑁
i
j
How does forshorte...
Copy across all time-stamps.
Copy Matrix:
Time-dependent Transport.
Sampling Matrix:
Discretized Operator Size
Discrete
Mo...
Recover Factorized Transient Light Transport
Discrete
Model:
Optimization
Problem:
Spherical Normal
Parametrization
MAP Es...
Memory Limitations
Visibility Vars:
Steered Laser
& SPAD
Detector
Scanned Wall
Area
𝑁
𝑁
𝑁
𝑁
𝑁
i
j
Normal Vars:
Proposed Me...
Occluded Hidden Scene in 2D
2D Setup
Geometry:
No
Occlusion
Proposed
Measuremen
t
Detector
Occluded Hidden Scene in 2D
Backprojection
Method
Filtered
Backprojection
Linear Inverse
Method
Proposed
Approach
Ground
T...
Occluded Hidden Scene in 3D
Filtered
Backprojection
Linear Inverse
Method
Proposed
Approach
Ground
Truth
Partially Occluded Hidden Scene in 3D
Filtered
Backprojection
Linear Inverse
Method
Proposed
Approach
Ground
Truth
Transie...
Backprojection
Method
Filtered Backprojection
Method
Linear Inverse
Method
Proposed
Approach
Albedo Reconstructions
Backprojection
Method
Filtered Backprojection
Method
Linear Inverse
Method
Proposed
Approach
More Difficult Albedo Reconst...
Backprojection
Method
Linear Inverse
Method
Proposed
Approach
Self-Occlusions
Estimated Normals
Physical Measurement Setup
Single Photon Avalanche Detector
MPD PD-100
100 𝜇𝑚 active area
Relay Optics
75mm achromatic
doublet lens
Beam-Splitter in Coaxial Alignment
Short-Pulsed Laser Illumination (50ps FWHM)
AlphaLas
PICOPOWER-LD-670-50
Scanning Galvo Mirror System
Illumination Path
Detection Path
Backprojection
Method
Linear Inverse
Method
Proposed
Approach
Experimental Reconstruction Results
LCT
Method
Scene Photogr...
Backprojection
Method
Linear Inverse
Method
Proposed
Approach
Experimental Reconstruction Results
LCT
Method
Scene Photogr...
Backprojection
Method
Linear Inverse
Method
Proposed
Approach
Experimental Reconstruction Results
LCT
Method
Scene Photogr...
Backprojection
Method
Linear Inverse
Method
Proposed
Approach
Experimental Reconstruction Results
LCT
Method
Scene Photogr...
Backprojection
Method
Linear Inverse
Method
Proposed
Approach
Experimental Reconstruction Results
LCT
Method
Estimated Nor...
Future Directions
Scattering Media and
Active Sources
Learned Visibility Matrices
And Scene Representations
1
6
4
1
2
8
2
...
Non-line-of-sight Imaging with Partial
Occluders and Surface Normals
Felix Heide1,2, Matthew O’Toole1,3, Kai Zang1, David ...
Próxima SlideShare
Cargando en…5
×

Non-line-of-sight Imaging with Partial Occluders and Surface Normals | TOG 2019

155 visualizaciones

Publicado el

Imaging objects obscured by occluders is a significant challenge for many applications. A camera that could “see around corners” could help improve navigation and mapping capabilities of autonomous vehicles or make search and rescue missions more effective. Time-resolved single-photon imaging systems have recently been demonstrated to record optical information of a scene that can lead to an estimation of the shape and reflectance of objects hidden from the line of sight of a camera. However, existing non-line-of-sight (NLOS) reconstruction algorithms have been constrained in the types of light transport effects they model for the hidden scene parts. We introduce a factored NLOS light transport representation that accounts for partial occlusions and surface normals. Based on this model, we develop a factorization approach for inverse time-resolved light transport and demonstrate high-fidelity NLOS reconstructions for challenging scenes both in simulation and with an experimental NLOS imaging system.

Publicado en: Ingeniería
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Non-line-of-sight Imaging with Partial Occluders and Surface Normals | TOG 2019

  1. 1. Non-line-of-sight Imaging with Partial Occluders and Surface Normals Felix Heide1,2, Matthew O’Toole1,3, Kai Zang1, David B. Lindell1, Steven Diamond1, Gordon Wetzstein1 1Stanford University 2Princeton University 3CMU ACM SIGGRAPH 2019
  2. 2. Non-line-of-sight Imaging
  3. 3. Non-line-of-sight Imaging
  4. 4. Hidden Objects Behind Hidden Objects
  5. 5. Experimental Non-line-of-sight Imaging
  6. 6. Experimental Non-line-of-sight Imaging Reconstruction (TV-Prior): Simulated Transients : 𝑡 [ps]
  7. 7. Partially-Occluded Non-line-of-sight Imaging Reconstruction (TV-Prior):
  8. 8. Time-tagged Photon Counts – Transient Images Experimental measurements Photograph of hidden scene 𝑡 [ps]
  9. 9. Measurement Slice Reconstruction Transient Image Slice
  10. 10. Non-line-of-sight Imaging [1] Velten et al. (2012) [2] Gupta et al. (2012) [3] Wu et al. (2012) [4] Buttafava et al. (2015) [5] O’Toole et al. (2018) [6] Lindell et al.(2019) Direct Pulsed Measurement [1] Katz et al. (2012) [2] Katz et al. (2014) [3] Smith et al. (2018) [4] Kadambi et al. (2016) [5] Heide et al. (2015) Coherent and Modulated Measurment Accurate Image Formation Models and Priors [1] Xin et al. (2019) [2] Thrampoulidis et al. (2019) [3] Tancik et al. (2018) [4] Chen et al. (2019) Tracking using Intensity Imaging [1] Klein et al. (2012) [2] Smith et al. (2018) [3] Boger-Lombard and Katz (2018) [4] Bouman et al.
  11. 11. Non-line-of-sight Imaging [1] Velten et al. (2012) [2] Gupta et al. (2012) [3] Wu et al. (2012) [4] Buttafava et al. (2015) [5] O’Toole et al. (2018) [6] Lindell et al.(2019) Direct Pulsed Measurement [1] Katz et al. (2012) [2] Katz et al. (2014) [3] Smith et al. (2018) [4] Kadambi et al. (2016) [5] Heide et al. (2015) Coherent and Modulated Measurment Accurate Image Formation Models and Priors [1] Xin et al. (2019) [2] Thrampoulidis et al. (2019) [3] Tancik et al. (2018) [4] Chen et al. (2019) Tracking using Intensity Imaging [1] Klein et al. (2012) [2] Smith et al. (2018) [3] Boger-Lombard and Katz (2018) [4] Bouman et al. Image Formation Without Occlusions * * ( or explicitly requires them )
  12. 12. Continuous Image Formation without Occlusions Steered Laser & SPAD Detector Scanned Wall Area x′i y′ Confocal Measurements: 𝑡 [ps] j 𝜌j x y z Path Length Gating Voxel Volume
  13. 13. Continuous Image Formation with Occlusions Steered Laser & SPAD Detector Scanned Wall Area x′i y′ j Confocal Measurements: 𝑡 [ps] Visibility Factors Voxel Volume Normal Factors x y z Path Gating j
  14. 14. Steered Laser & SPAD Detector Scanned Wall Area x′i y′ j Confocal Measurements: 𝑡 [ps] Visibility Factors Voxel Volume Normal Factors x y z Path Gating Continuous Image Formation with Occlusions
  15. 15. Steered Laser & SPAD Detector Scanned Wall Area x′i y′ Confocal Measurements: 𝑡 [ps] Retro Reflection n Diffuse Reflection Visibility Factors Voxel Volume Normal Factors 𝜔 x y z j Continuous Image Formation with Normals
  16. 16. Full Continuous Image Formation Model Steered Laser & SPAD Detector Scanned Wall Area x′i y′ Confocal Measurements: 𝑡 [ps] Retro Reflection n Diffuse Reflection 𝜔 x y z j
  17. 17. Factorized Transient Light Transport Continuous Model: Discretize Measurement 𝑥′ , 𝑦′ , 𝑡: Discrete Model: Discretize Space 𝑥′ , 𝑦′ , 𝑥, 𝑦, 𝑧:
  18. 18. Discretized Operator Size Discrete Model: Steered Laser & SPAD Detector Scanned Wall Area 𝑁 𝑁 𝑁 𝑁 𝑁 i j How much light can get reflected from j to i ? vij Wall pos. Vol pos. j i
  19. 19. Visualization of the Visibility Matrix Ground Truth Scene Visibility Term for Pixel Position indicated on the leftViewing direction towards wall
  20. 20. Discretized Operator Size Discrete Model: Steered Laser & SPAD Detector Scanned Wall Area 𝑁 𝑁 𝑁 𝑁 𝑁 i j How does forshortening at j reduce the light back to i ? 𝜔 nj Nij Vol pos. j i Wall pos.
  21. 21. Copy across all time-stamps. Copy Matrix: Time-dependent Transport. Sampling Matrix: Discretized Operator Size Discrete Model: Visibility Matrix: Steered Laser & SPAD Detector Scanned Wall Area 𝑁 𝑁 𝑁 𝑁 𝑁 i j Normal Matrix:𝜔 nj Matrix-Free Implementation
  22. 22. Recover Factorized Transient Light Transport Discrete Model: Optimization Problem: Spherical Normal Parametrization MAP Estimate Regularizer (Prior)
  23. 23. Memory Limitations Visibility Vars: Steered Laser & SPAD Detector Scanned Wall Area 𝑁 𝑁 𝑁 𝑁 𝑁 i j Normal Vars: Proposed Method: Backprojection: Matrix-Free Inverse: For a volume of 643 Matlab (CPU) ~ 500GB and 2h For a volume of 403 Cuda (GPU) ~ 14GB and 180s
  24. 24. Occluded Hidden Scene in 2D 2D Setup Geometry: No Occlusion Proposed Measuremen t Detector
  25. 25. Occluded Hidden Scene in 2D Backprojection Method Filtered Backprojection Linear Inverse Method Proposed Approach Ground Truth 2D Setup Geometry: Detector
  26. 26. Occluded Hidden Scene in 3D Filtered Backprojection Linear Inverse Method Proposed Approach Ground Truth
  27. 27. Partially Occluded Hidden Scene in 3D Filtered Backprojection Linear Inverse Method Proposed Approach Ground Truth Transient Measurement 𝑡 [ps]
  28. 28. Backprojection Method Filtered Backprojection Method Linear Inverse Method Proposed Approach Albedo Reconstructions
  29. 29. Backprojection Method Filtered Backprojection Method Linear Inverse Method Proposed Approach More Difficult Albedo Reconstructions
  30. 30. Backprojection Method Linear Inverse Method Proposed Approach Self-Occlusions Estimated Normals
  31. 31. Physical Measurement Setup
  32. 32. Single Photon Avalanche Detector MPD PD-100 100 𝜇𝑚 active area
  33. 33. Relay Optics 75mm achromatic doublet lens
  34. 34. Beam-Splitter in Coaxial Alignment
  35. 35. Short-Pulsed Laser Illumination (50ps FWHM) AlphaLas PICOPOWER-LD-670-50
  36. 36. Scanning Galvo Mirror System
  37. 37. Illumination Path
  38. 38. Detection Path
  39. 39. Backprojection Method Linear Inverse Method Proposed Approach Experimental Reconstruction Results LCT Method Scene PhotographMeasurement Slice
  40. 40. Backprojection Method Linear Inverse Method Proposed Approach Experimental Reconstruction Results LCT Method Scene Photograph
  41. 41. Backprojection Method Linear Inverse Method Proposed Approach Experimental Reconstruction Results LCT Method Scene Photograph
  42. 42. Backprojection Method Linear Inverse Method Proposed Approach Experimental Reconstruction Results LCT Method Scene Photograph
  43. 43. Backprojection Method Linear Inverse Method Proposed Approach Experimental Reconstruction Results LCT Method Estimated NormalsScene Photograph
  44. 44. Future Directions Scattering Media and Active Sources Learned Visibility Matrices And Scene Representations 1 6 4 1 2 8 2 5 6 1 2 8 3 2 6 4 5 1 2 2 5 6 3 2 Higher-Order Bounces Every Surface Becomes a “Sensor”
  45. 45. Non-line-of-sight Imaging with Partial Occluders and Surface Normals Felix Heide1,2, Matthew O’Toole1,3, Kai Zang1, David B. Lindell1, Steven Diamond1, Gordon Wetzstein1 1Stanford University 2Princeton University 3CMU ACM SIGGRAPH 2019

×