This document proposes a new approach called temporal frequency probing to analyze light transport using time-of-flight (ToF) cameras and projectors. By projecting patterns at different modulation frequencies and capturing the returning light with a ToF camera, it can separate direct from indirect light transport and further separate caustic from diffuse indirect light. This allows it to reconstruct high temporal resolution "light-in-flight" videos depicting the propagation of light over time within a scene. The approach leverages decades of research on light transport analysis and provides a new paradigm for studying transient 5D light transport using commercially available ToF sensors and projectors.

1. Temporal Frequency Probing for
5D Transient Analysis of
Global Light Transport
Matt O’Toole1 Felix Heide2 Lei Xiao2 Matthias Hullin3 Wolfgang Heidrich2,4 Kyros Kutulakos1
1University of Toronto 2University of British Columbia 3University of Bonn 4KAUST
http://www.dgp.toronto.edu/~motoole/temporalprobing.html

2. the time-of-flight (ToF) sensing revolution
estimating poses
[Schwarz et al. 11]
Kinect for Xbox One
femto-photography
[Velten et al. 2013]
looking-around-corners
[Velten et al. 2012]
gesture recognition
[Droeschel et al. 11]
Google’s self-driving car

3. temporal probing for scene analysis
capture depth
robust to indirect light
reconstruct “light-in-flight” video
with high temporal resolution
key insight: complex-valued image formation model

4. contributions
the transient-frequency transport matrix
unified theory for light transport analysis
analyze ToF images using classical techniques
ToF imaging using projectors & masks
capture depth robust to indirect light
reconstruct “light-in-flight” video with high temporal resolution

5. what is a ToF photo?
ToF camera

6. what is a ToF photo?
ToF camera point light

7. what is a ToF photo?
ToF camera ToF projector

8. what is a ToF photo?
ToF camera ToF projector

9. what is a ToF photo?
ToF camera ToF projector

10. what is a ToF photo?
ToF camera ToF projector

11. what is a ToF photo?
ToF camera ToF projector

12. what is a ToF photo?
ToF camera ToF projector

13. what is a ToF photo?
ToF camera ToF projector

14. what is a ToF photo?
ToF camera ToF projector

15. what is a ToF photo?
intensity
ToF camera ToF projector

16. what is a ToF photo?
phase delay
ToF camera ToF projector

17. what is a ToF photo?
transport coefficient
ToF camera ToF projector

18. what is a ToF photo?
transport coefficient
ToF camera ToF projector

19. what is a ToF photo?
transport coefficient
ToF camera ToF projector

20. mirror
what is a ToF photo?
transport coefficient
ToF camera ToF projector

21. it is a vector of complex-valued pixels…
photo
mirror
camera
projector

22. it is a vector of complex-valued pixels…
photo pattern
camera

23. … produced by a complex-valued matrix
photo transport matrix for pattern
camera

24. conventional image under ambient lighting

25. visualizing a complex ToF image
brightness
hue
phase

26. transient-frequency light transport equation
photo transport matrix for pattern
camera

27. transient-frequency light transport equation
photo transport matrix for pattern
 conventional transport matrix [Ng et al. 03; …]

28. Techniques Reference(s)
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global
separation
[Nayar et al. SIGGRAPH 2006]

29. Techniques Reference(s) Conventional Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global
separation
[Nayar et al. SIGGRAPH 2006]

30. Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global
separation
[Nayar et al. SIGGRAPH 2006]

31. Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global
separation
[Nayar et al. SIGGRAPH 2006]

32. Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global
separation
[Nayar et al. SIGGRAPH 2006]

33. Techniques Reference(s) Time-of-Flight Analysis
transport equation [Ng et al. 03]
dual photography [Sen et al. 05; Sen and Darabi 09]
radiometric compensation [Wetzstein and Bimber 07]
radiosity equation [Goral et al. 84]
radiosity solution [Goral et al. 84]
inverse light transport [Seitz et al. 05; Bai et al. 10]
transport eigenvectors [O’Toole and Kutulakos 10]
structured light transport [O’Toole et al. CVPR 2014]
fast direct/global
separation
[Nayar et al. SIGGRAPH 2006]

34. structured light transport [O’Toole et al. CVPR 2014]
acquire direct & indirect ToF images using epipolar constraints
fast direct/global separation [Nayar et al. SIGGRAPH 2006]
acquire caustic indirect & diffuse indirect ToF images using radiometric constraints

35. structured light transport [O’Toole et al. CVPR 2014]
acquire direct & indirect ToF images using epipolar constraints
fast direct/global separation [Nayar et al. SIGGRAPH 2006]
acquire caustic indirect & diffuse indirect ToF images using radiometric constraints

36. structured light transport for freq.
mirror
ToF camera ToF projector

37. structured light transport for freq.
mirror
ToF camera ToF projector

38. structured light transport for freq.
mirror
direct paths obey
epipolar geometry
ToF camera ToF projector

39. structured light transport for freq.
mirror
indirect paths DONT
obey epipolar geometry
ToF camera ToF projector

40. structured light transport for freq.
mirror
projection
pattern
mask
pattern
ToF camera ToF projector

41. structured light transport for freq.
mirror
mask
pattern
projection
pattern
ToF camera ToF projector

42. structured light transport for freq.
mirror
projection
pattern
mask
pattern
1. open electronic shutter
2. for i = 1 to N
use random epipolar mask &
project complementary pattern
3. close electronic shutter
ToF camera ToF projector

43. conventional ToF imaging
brightness
hue

44. conventional ToF imaging structured light transport
brightness
hue
indirect ToF

45. conventional ToF imaging structured light transport
direct ToF
brightness
hue
indirect ToF

46. application: 3D from ToF

47. ToF depth maps & indirect illumination

48. conventional vs direct-only ToF
direct-only
conventional

49. 3D from conventional vs direct-only ToF
conventional direct-only

50. conventional vs direct-only ToF
conventional direct-only

51. conventional vs direct-only ToF
conventional direct-only

52. 3D from conventional vs direct-only ToF
conventional direct-only

53. structured light transport [O’Toole et al. CVPR 2014]
acquire direct & indirect ToF images using epipolar constraints
fast direct/global separation [Nayar et al. SIGGRAPH 2006]
acquire caustic indirect & diffuse indirect ToF images using radiometric constraints

54. effect of indirect transport on spatial frequencies
mirror
ToF camera ToF projector

55. effect of indirect transport on spatial frequencies
mirror
caustic  high frequency
non-caustic  ToF camera low frequency ToF projector

56. effect of indirect transport on spatial frequencies
mirror
caustic  high frequency
non-caustic  low frequency
ToF camera ToF projector

57. effect of indirect transport on spatial frequencies
mirror
1. for i = 1 to N
project high spatial-freq. pattern
capture image
2. non-caustic = min of images
caustic  high frequency
non-caustic  low frequency
ToF camera ToF projector

58. conventional ToF imaging structured light transport
direct ToF
indirect ToF
brightness
hue

59. structured light transport fast caustic/non-caustic
direct ToF non-caustic ToF
indirect ToF
separation
conventional ToF imaging
brightness
hue

60. structured light transport fast caustic/non-caustic
separation
direct ToF non-caustic ToF
indirect ToF caustic ToF
conventional ToF imaging
brightness
hue

61. direct ToF non-caustic ToF
caustic ToF

62. application: light-in-flight imaging

63. “light-in-flight” imaging for time instant t
mirror
goal: reconstruct
image for time t
ToF camera ToF projector

64. “light-in-flight” imaging: prior work
femto-photography
[Velten et al. 2013]
very high cost
well-resolved wavefronts
PMD-based system
[Heide et al. 2013]
many modulation frequencies
strong scene priors
optical wavefronts not resolvable
our work
many modulation frequencies
weaker, transport-specific priors
resolvable optical wavefronts

65. piecewise “light-in-flight” video construction
conventional ToF for = 100 MHz conventional image

66. piecewise “light-in-flight” video construction
direct ToF for = 100 MHz light-in-flight video
direct (no regularization)

67. piecewise “light-in-flight” video construction
caustic ToF for = 100 MHz light-in-flight video
direct (no regularization)
caustic indirect (no regularization)

68. piecewise “light-in-flight” video construction
non-caustic ToF for = 12 to 140 MHz light-in-flight video
direct (no regularization)
caustic indirect (no regularization)
non-caustic indirect (regularized
using [Heide et al. 2013])

69. comparison to [Heide et al. 2013]
light-in-flight video [Heide et al. light-in-flight video (ours)
13]

70. comparison to [Heide et al. 2013]
light-in-flight video [Heide et al. light-in-flight video (ours)
13]

71. conclusion
• a new paradigm for ToF sensing
• combines ToF sensors with projectors & masks
• can now leverage from decade of light transport research
• widespread implications for spatio-temporal transport analysis

72. come visit us at our E-Tech booth
visualizing light transport phenomena

73. Temporal Frequency Probing for
5D Transient Analysis of
Global Light Transport
Matt O’Toole1 Felix Heide2 Lei Xiao2 Matthias Hullin3 Wolfgang Heidrich2,4 Kyros Kutulakos1
1University of Toronto 2University of British Columbia 3University of Bonn 4KAUST
http://www.dgp.toronto.edu/~motoole/temporalprobing.html

74. temporal probing setup
max. exposure time of 8 ms, max. modulation frequency of 150 MHz