Optical Computing for Fast Light Transport Analysis
Matthew O'Toole and Kiriakos N. Kutulakos. ACM SIGGRAPH Asia, 2010.
We present a general framework for analyzing the transport matrix of a real-world scene at full resolution, without capturing many photos. The key idea is to use projectors and cameras to directly acquire eigenvectors and the Krylov subspace of the unknown transport matrix. To do this, we implement Krylov subspace methods partially in optics, by treating the scene as a black box subroutine that enables optical computation of arbitrary matrix-vector products. We describe two methods—optical Arnoldi to acquire a low-rank approximation of the transport matrix for relighting; and optical GMRES to invert light transport. Our experiments suggest that good-quality relighting and transport inversion are possible from a few dozen low-dynamic range photos, even for scenes with complex shadows, caustics, and other challenging lighting effects.
3. The Light Transport Matrix [Debevec et al. 2000]
photo
pixels
pattern
pixels
elements
4. optical domain
Computing with Light
Key idea: analyze the transport matrix by implementing
iterative numerical algorithms directly in optics
transport
matrix
illumination
pattern
photo
numerical domain
6. Computing with Light
1. project
2. capture
numerical domain optical domain
Key idea: analyze the transport matrix by implementing
iterative numerical algorithms directly in optics
7. Computing with Light
numerical domain optical domain
Key idea: analyze the transport matrix by implementing
iterative numerical algorithms directly in optics
8. Optical Computing for
Fast Light Transport Analysis
optical
power iteration
find principal
eigenvector of
optical
Arnoldi
find best rank-k
approximation of
optical
GMRES
given photo p,
solve
• calibrated projectors
• calibrated cameras
• unknown, static scene
• scene physically accessible
9. project capture
Eigenvector of a square matrix T
when projected onto scene,
we get the same photo back
(multiplied by a scalar)
Computing Transport Eigenvectors
Numerical goal
find such that
and is maximalprojector
camerabeam
splitter
10. Optical Power Iteration
Goal: find principal eigenvector of
Observation: it is a fixed point of the sequence
numerical domain
Properties
• linear convergence [Trefethen and Bau 1997]
• eigenvalues must be distinct
• cannot be orthogonal to
principal eigenvector
11. Optical Power Iteration
numerical domain optical domain
Goal: find principal eigenvector of
Observation: it is a fixed point of the sequence
12. Optical Power Iteration
optical domainnumerical domain
projectcapture
initialize
normalize
Goal: find principal eigenvector of
Observation: it is a fixed point of the sequence
13. Optical Power Iteration
optical domain
projectcapture
initialize
normalize
projector
camera
beam
splitter
Goal: find principal eigenvector of
Observation: it is a fixed point of the sequence
14. Optical Power Iteration
optical domain
projectcapture
initialize
normalize
projector
camera
beam
splitter
Goal: find principal eigenvector of
Observation: it is a fixed point of the sequence
15. Optical Power Iteration
optical domain
projectcapture
initialize
normalize
Goal: find principal eigenvector of
Observation: it is a fixed point of the sequence
16. Optical Power Iteration
optical domain
projectcapture
initialize
normalize
Goal: find principal eigenvector of
Observation: it is a fixed point of the sequence
17. Optical Power Iteration
optical domain
projectcapture
initialize
normalize
Goal: find principal eigenvector of
Observation: it is a fixed point of the sequence
18. Optical Power Iteration
optical domain
projectcapture
initialize
normalize
Goal: find principal eigenvector of
Observation: it is a fixed point of the sequence
19. Optical Power Iteration
optical domain
(approximate)
principal eigenvector
Goal: find principal eigenvector of
Observation: it is a fixed point of the sequence
20. Optical Power Iteration
optical domain
(approximate)
principal eigenvector
...
Goal: find principal eigenvector of
Observation: it is a fixed point of the sequence
21. Computational microscopy [Bimber et al. 2010]
connection to numerical methods not exploited
Computational illumination [Wang et al. 2010]
optical feedback avoided or ignored
Krylov subspace methods [Saad 2003]
use recurrence relations
designed for extremely large matrices
Computing with Feedback Loops
Analog optical computing [Rajbenbach et al. 1987]
encode matrix with optical masks
optical feedback for iterative methods
22. Optical Computing for
Fast Light Transport Analysis
optical
power iteration
find principal
eigenvector of
optical
Arnoldi
find best rank-k
approximation of
optical
GMRES
given photo p,
solve
• calibrated projectors
• calibrated cameras
• unknown, static scene
• scene physically accessible
23. Space of Camera-Projector Arrangements
one viewpoint? yes
coaxial optical
paths?
[Garg et al, 2006]
“point-source”
illumination?
[Wang et al, 2009]
24. Space of Camera-Projector Arrangements
one viewpoint? yes
symmetric
coaxial optical
paths?
[Garg et al, 2006]
“point-source”
illumination?
[Wang et al, 2009]
25. Space of Camera-Projector Arrangements
one viewpoint? no
coaxial optical
paths?
[Garg et al, 2006]
“point-source”
illumination?
[Wang et al, 2009]
yes
symmetric
26. Space of Camera-Projector Arrangements
one viewpoint? yes no
nonsymmetricsymmetric
coaxial optical
paths?
[Garg et al, 2006]
“point-source”
illumination?
[Wang et al, 2009]
27. Space of Camera-Projector Arrangements
one viewpoint? yes no
nonsymmetricsymmetric
coaxial optical
paths?
[Garg et al, 2006]
no
“point-source”
illumination?
[Wang et al, 2009]
yes
28. Space of Camera-Projector Arrangements
one viewpoint? yes no
nonsymmetricsymmetric
coaxial optical
paths?
[Garg et al, 2006]
yes no
computable computable
“point-source”
illumination?
[Wang et al, 2009]
29. Space of Camera-Projector Arrangements
one viewpoint? yes no
nonsymmetricsymmetric
coaxial optical
paths?
[Garg et al, 2006]
yes no
computable computable
“point-source”
illumination?
[Wang et al, 2009]
yes no
30. one viewpoint?
coaxial optical
paths?
[Garg et al, 2006]
“point-source”
illumination?
[Wang et al, 2009]
dense, low-rank sparse, high-rank
no
nonsymmetric
no
computable
yes no
yes
symmetric
yes
computable
Space of Camera-Projector Arrangements
31. Domain of Optical Arnoldi
one viewpoint? yes no
nonsymmetricsymmetric
coaxial optical
paths?
[Garg et al, 2006]
yes no
computable computable
“point-source”
illumination?
[Wang et al, 2009]
yes no
sparse, high-rankdense, low-rank
32. Domain of Optical GMRES
one viewpoint? no
nonsymmetric
coaxial optical
paths?
[Garg et al, 2006]
no
computable
“point-source”
illumination?
[Wang et al, 2009]
symmetric
computable
dense, low-rank sparse, high-rank
33. Optical Computing for
Fast Light Transport Analysis
optical
power iteration
find principal
eigenvector of
optical
Arnoldi
find best rank-k
approximation of
optical
GMRES
given photo p,
solve
• calibrated projectors
• calibrated cameras
• unknown, static scene
• scene physically accessible
34. Rank-k Transport Approximation
Numerical goal [Simon and Zha 2000]
find matrices
such that
Symmetric
• 1 camera, 1 projector
• 2 photos for rank- approx.
Nonsymmetric
• 2 cameras, 2 projectors
• 4 photos for rank- approx.
projector
camerabeam
splitter
35. Numerical goal [Simon and Zha 2000]
find matrices
that minimize
Rank-k Transport Approximation
Symmetric
• 1 camera, 1 projector
• 2 photos for rank- approx.
Nonsymmetric
• 2 cameras, 2 projectors
• 4 photos for rank- approx.
projector
camerabeam
splitter
36. Prior Work on Transport Acquisition
Brute force [Schechner et al. 2007] [Sen et al. 2005]
must capture to completion
Methods draw from set of scene independent illumination
Kernel Nyström [Wang et al. 2009]
designed for dense, low-rank matrices
requires intense illumination and HDR photography
Compressive sensing [Peers et al. 2009] [Sen and Darabi 2009]
designed for sparse, high-rank matrices
computationally very intensive
37. Optical Arnoldi for Symmetric T
numerical domain
Goal: compute matrices such that
38. Optical Arnoldi for Symmetric T
numerical domain
Goal: compute matrices such that
39. Optical Arnoldi for Symmetric T
numerical domain
Goal: compute matrices such that
40. Optical Arnoldi for Symmetric T
numerical domain optical domain
projectcapture
initialize
orthogonalize
projected patterns
captured photos
Goal: compute matrices such that
41. Optical Arnoldi for Symmetric T
numerical domain optical domain
projectcapture
initialize
projected patterns
captured photos
orthogonalize
Goal: compute matrices such that
43. Optical Arnoldi for Symmetric T
optical domain
projectcapture
initialize
projected patterns
captured photos
orthogonalize
Goal: compute matrices such that
rows of
...
projected patterns
44. Optical Arnoldi for Symmetric T
...columns of
Goal: compute matrices such that
captured photos
...
rows of
projected patterns
47. left camera
& projector
Optical Arnoldi for Nonsymmetric T
optical domain
projectcapture
initialize
left-projected patterns
right-captured photos
normalize
Goal: compute matrices such that
right camera
& projector
orthogonalize
leftright
captureproject
leftright
48. left camera
& projector
Optical Arnoldi: Implementation
right camera
& projector
Just 1 line of MATLAB code:
• cameras & projectors calibrated
• RAW images, LDR capture
• Timings: 12s per Arnoldi iteration
left-project &
right-capture
right-project &
left-capture
55. Optical Computing for
Fast Light Transport Analysis
optical
power iteration
find principal
eigenvector of
optical
Arnoldi
find best rank-k
approximation of
optical
GMRES
given photo p,
solve
• calibrated projectors
• calibrated cameras
• unknown, static scene
• scene physically accessible
56. Numerical goal
given photo , find illumination
that minimizes
Light Transport Inversion
Remarks
• low-rank or high-rank
• unknown & not acquired
• illumination sequence will be
specific to input photo
projector
camerabeam
splitter
57. Defocus Compensation [Zhang and Nayar 2006]
inverts defocus kernel
Prior Work on Transport Inversion
Radiometric Compensation [Wetzstein and Bimber 2007]
inversion after capturing matrix
Transport matrix is known for all methods
Interreflectance Cancellation [Seitz et al. 2005]
decomposition into n-bounce images
Stratified Inverse [Ng et al. 2009] [Bai et al. 2010]
approximates inverse using geometric series
59. Optical GMRES for Symmetric T
optical domain
projectcapture
initialize
orthogonalize
Goal: find illumination such that
input
photo
projector
camera
beam
splitter
projected patterns
captured photos
60. Optical GMRES for Symmetric T
optical domain
projectcapture
initialize
orthogonalize
Goal: find illumination such that
input
photo
projector
camera
beam
splitter
77. Concluding Remarks
Implement numerical algorithms directly in optics:
• methods that operate on large, unknown matrices
are well established
• easy to build optical algorithms from
many existing numerical implementations
• theoretical bounds for convergence rate
Limitations:
• optical Arnoldi inefficient for high rank matrices
• optical GMRES only inverts a single image
78. Optical Computing for
Fast Light Transport Analysis
Matthew O’Toole and Kyros Kutulakos
University of Toronto
http://www.dgp.toronto.edu/~motoole
