Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
End-to-end Optimization of Optics and Image Processing for
Achromatic Extended Depth of Field and Super-Resolution
Imaging...
Courtesy of Michael Bok Courtesy of CSIR Notes
Courtesy of MicrodacCourtesy of Jeffrey Beach
Loss
Optimize optics end-to-end with higher-level processing!
Regular bi-convex lens (focus close) Optimized camera
Results: Achromatic extended DOF
How do computer vision pipelines
work in real life?
What is this?
Step 1: Build camera
Optimize optics to minimize aberrations:
Blur/spot size, chromatic aberrations, distortions, …
Point ...
Step 2: Image Signal Processing
Maximize PSNR:
Demosaicking, Denoising, Deblurring, …
PSF -1
Step 3: Image Post-Processing
Minimize final loss:
L2, perceptual loss, classification error, …
PSF
...PSF
-1
Bunny
PSF
-1
Teapot
?
PSF
-1
+ 𝜖 + 𝜖
+ 𝜖
+ 𝜖
+ 𝜖
𝜕
𝜕𝑥
𝜕
𝜕𝑥
Bunny
𝜕
𝜕𝑥
Vision: The Deep Computational Camera
Optimize end-to-end
PSF
-1
Bunny
𝜕
𝜕𝑥
Vision: The Deep Computational Camera
• Performance & robustness gains
• Domain-specific hardware may reduce fo...
Bunny
𝜕
𝜕𝑥
This project: Enable optimization of optics!
𝜕
𝜕𝑥
PSF
𝜕
𝜕𝑥
-1
Prior Work
Computational Cameras
Optics Optimization
Differentiable ISPs
Deep Computational Photography
• Deep Joint Demos...
A differentiable optics model
𝜕
𝜕𝑥
PSF
Image formation model
* =+
How does the optical element map to the PSF?
?
* =+
𝜕
𝜕𝑥
Wave Optics PSF simulator
PSF𝜕
𝜕𝑥
Spherical wave from point source
Phaseshift by optical element
𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
Phaseshift by optical element
𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
Phaseshift by optical element
Φ
𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
Height Map parameterization
(diffractive)
Zernike basis parameterization
(refractive)
Φ 𝑥 = [ 𝑎11, 𝑎12, … , … ] Φ[𝑥] = 𝑍𝑖
...
Phaseshift by optical element
Φ
𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
Fresnel propagation to sensor
𝑈′
𝑥 = 𝑈 𝑥 ∗ exp(
𝑗𝑘
2𝑧
𝑥2
)
Intensity measurement at sensor
𝑈′
𝑥 2
PSF
𝜌 𝑧′,𝜆 = exp 𝑗𝑘 𝑥2 + 𝑦2 + 𝑧′2 + 𝑛 − 1 𝜙 𝑥, 𝑦 ∗ exp 𝑗
𝑘
2𝑧
𝑥2
+ 𝑦2
2
Calculating the PSF
PSF
Calculating the PSF
𝜌 𝑧′,𝜆 = exp 𝑗𝑘 𝑥2 + 𝑦2 + 𝑧′2 + 𝑛 − 1 𝜙 𝑥, 𝑦 ∗ exp 𝑗
𝑘
2𝑧
𝑥2
+ 𝑦2
2
• Differentiable with respect to 𝜙...
Sanity check:
Optimizing a focusing lens
𝜕
𝜕𝑥
Convolve natural images with PSF of optics simulator
Minimize L2 loss with
Stochastic Gradient Descent
∙ 2
Sanity check: Optimizing a collimator lens
𝜕
𝜕𝑥
Iteration
Height Map
PSF
Fabrication
Refractive: Single-point Diamond TurningDiffractive: Photolithography
Application: Achromatic EDOF
Problem with single lens:
Limited Depth of Field, chromatic aberrations
Scene depth
Focal plane
Classic EDOF:
Better depth of field, but not easily invertible
Scene depth
Wiener
deconvolution
-1
Extended depth of field...
End-to-end optimization for EDOF
-1
Wiener
deconvolution
-1
Wiener
deconvolution
Add Gaussian Noise
End-to-end optimization for EDOF
-1
Wiener
deconvolution
Wiener deconvolution for reconstruction
End-to-end optimization for EDOF
Scene depth
During training:
Place input image at random depth
-1
Wiener
deconvolution
End-to-end optimization for EDOF
End-to-end optimized with deconvolution:
Depth-independent and easily invertible
Scene depth
𝜕
𝜕𝑥
Wiener
deconvolution
𝜕
𝜕...
Refractive Achromatic EDOF element
• Polymethyl methacrylate (PMMA)
• 5 mm aperture size
• Sensor distance 35.5𝑚𝑚
• F-numb...
Regular bi-convex lens Optimized lens
Elephant (0.5m) Book (2.0m)
Sensor image
Test scene
Processed image
Regular bi-convex lens Optimized lens
Elephant (0.5m) Book (2.0m)
Test scene
Regular bi-convex lens (focus close) Optimized camera
Real-world capture
Diffractive Achromatic EDOF element
• Fused silica processed via 16-level
photolithography
• 5 mm aperture size
• Sensor d...
Regular Fresnel Lens Optimized Camera w. diffractive element
Diffractive EDOF: Test scene
Experimental application: Super-Resolution
Experimental application: Super-Resolution
Please refer to paper for more detail!
Summary
-1
𝜕
𝜕𝑥
Jointly optimizing optics and post-processing
PSF
𝜕
𝜕𝑥
𝜕
𝜕𝑥
Bunny
𝜕
𝜕𝑥
𝜕
𝜕𝑥
PSF
-1
• Most optimization algorithms for image reconstruction can be
made differentiable by unrollin...
Bunny
PSF
Optical convolutional neural networks with optimized diffractive optics for image classification, Chang et al., ...
Felix Heide
Evan Peng Wolfgang Heidrich
Stephen Boyd Gordon Wetzstein
Xiong Dun
End-to-end optimization of optical sensing pipelines
https://vsitzmann.github.io/deepoptics/
𝜕
𝜕𝑥
Bunny
𝜕
𝜕𝑥
𝜕
𝜕𝑥
PSF
-1
End-to-end Optimization of Cameras and Image Processing - SIGGRAPH 2018
End-to-end Optimization of Cameras and Image Processing - SIGGRAPH 2018
End-to-end Optimization of Cameras and Image Processing - SIGGRAPH 2018
Próxima SlideShare
Cargando en…5
×

End-to-end Optimization of Cameras and Image Processing - SIGGRAPH 2018

2.045 visualizaciones

Publicado el

Jointly optimizing high-level image processing and camera optics to design novel domain-specific cameras.

Publicado en: Ingeniería
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

End-to-end Optimization of Cameras and Image Processing - SIGGRAPH 2018

  1. 1. End-to-end Optimization of Optics and Image Processing for Achromatic Extended Depth of Field and Super-Resolution Imaging Vincent Sitzmann* Stanford University Steven Diamond* Stanford University Yifan Peng* University of British Columbia Xiong Dun KAUST Wolfgang Heidrich KAUST Stephen Boyd Stanford University Felix Heide Stanford University Gordon Wetzstein Stanford University
  2. 2. Courtesy of Michael Bok Courtesy of CSIR Notes Courtesy of MicrodacCourtesy of Jeffrey Beach
  3. 3. Loss Optimize optics end-to-end with higher-level processing!
  4. 4. Regular bi-convex lens (focus close) Optimized camera Results: Achromatic extended DOF
  5. 5. How do computer vision pipelines work in real life?
  6. 6. What is this?
  7. 7. Step 1: Build camera Optimize optics to minimize aberrations: Blur/spot size, chromatic aberrations, distortions, … Point Spread Function (PSF)
  8. 8. Step 2: Image Signal Processing Maximize PSNR: Demosaicking, Denoising, Deblurring, … PSF -1
  9. 9. Step 3: Image Post-Processing Minimize final loss: L2, perceptual loss, classification error, … PSF
  10. 10. ...PSF -1
  11. 11. Bunny PSF -1
  12. 12. Teapot ? PSF -1 + 𝜖 + 𝜖 + 𝜖 + 𝜖 + 𝜖
  13. 13. 𝜕 𝜕𝑥 𝜕 𝜕𝑥 Bunny 𝜕 𝜕𝑥 Vision: The Deep Computational Camera Optimize end-to-end PSF -1
  14. 14. Bunny 𝜕 𝜕𝑥 Vision: The Deep Computational Camera • Performance & robustness gains • Domain-specific hardware may reduce footprint, cost, power… • New design space: The “BunnyCam” 𝜕 𝜕𝑥 PSF 𝜕 𝜕𝑥 -1
  15. 15. Bunny 𝜕 𝜕𝑥 This project: Enable optimization of optics! 𝜕 𝜕𝑥 PSF 𝜕 𝜕𝑥 -1
  16. 16. Prior Work Computational Cameras Optics Optimization Differentiable ISPs Deep Computational Photography • Deep Joint Demosaicking and Denoising (Gharbi et al. 2016) • Unrolled Optimiziation with Deep Priors (Diamond et al.) • … • EDOF through wave-front coding (Dowski & Cathey, 1995) • Recovering HDR radiance maps from photographs (Debevec & Malik 1997) • … • Diffractive Achromat (Peng et al. 2016) • Lens Factory (Sun et al. 2015) • Zemax • … • HDR image reconstruction from a single exposure using deep CNNs (Eiltertsen et al. 2016) • Learning to synthesize a 4d rgbd light field from a single image (Srinivasan et al. 2017) • … -1 Co-Design, but no true joint optimization Wiener deconvolution For efficient joint optimization: need to make differentiable!
  17. 17. A differentiable optics model 𝜕 𝜕𝑥 PSF
  18. 18. Image formation model * =+
  19. 19. How does the optical element map to the PSF? ? * =+ 𝜕 𝜕𝑥
  20. 20. Wave Optics PSF simulator PSF𝜕 𝜕𝑥
  21. 21. Spherical wave from point source
  22. 22. Phaseshift by optical element 𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
  23. 23. Phaseshift by optical element 𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
  24. 24. Phaseshift by optical element Φ 𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
  25. 25. Height Map parameterization (diffractive) Zernike basis parameterization (refractive) Φ 𝑥 = [ 𝑎11, 𝑎12, … , … ] Φ[𝑥] = 𝑍𝑖 𝑗 𝑥 ∙ 𝑎𝑖𝑗
  26. 26. Phaseshift by optical element Φ 𝑈 𝑥 = exp 𝑗𝑘 𝑥2 + 𝑧′2 + 𝑛 − 1 Φ x
  27. 27. Fresnel propagation to sensor 𝑈′ 𝑥 = 𝑈 𝑥 ∗ exp( 𝑗𝑘 2𝑧 𝑥2 )
  28. 28. Intensity measurement at sensor 𝑈′ 𝑥 2 PSF
  29. 29. 𝜌 𝑧′,𝜆 = exp 𝑗𝑘 𝑥2 + 𝑦2 + 𝑧′2 + 𝑛 − 1 𝜙 𝑥, 𝑦 ∗ exp 𝑗 𝑘 2𝑧 𝑥2 + 𝑦2 2 Calculating the PSF PSF
  30. 30. Calculating the PSF 𝜌 𝑧′,𝜆 = exp 𝑗𝑘 𝑥2 + 𝑦2 + 𝑧′2 + 𝑛 − 1 𝜙 𝑥, 𝑦 ∗ exp 𝑗 𝑘 2𝑧 𝑥2 + 𝑦2 2 • Differentiable with respect to 𝜙 • Implemented as TensorFlow module • Can easily combine with other models!
  31. 31. Sanity check: Optimizing a focusing lens 𝜕 𝜕𝑥
  32. 32. Convolve natural images with PSF of optics simulator
  33. 33. Minimize L2 loss with Stochastic Gradient Descent ∙ 2
  34. 34. Sanity check: Optimizing a collimator lens 𝜕 𝜕𝑥 Iteration Height Map PSF
  35. 35. Fabrication Refractive: Single-point Diamond TurningDiffractive: Photolithography
  36. 36. Application: Achromatic EDOF
  37. 37. Problem with single lens: Limited Depth of Field, chromatic aberrations Scene depth Focal plane
  38. 38. Classic EDOF: Better depth of field, but not easily invertible Scene depth Wiener deconvolution -1 Extended depth of field through wave-front coding (Dowski & Cathey, 1995) Metasurface optics for full-color computational imaging (Colburn et al 2018)
  39. 39. End-to-end optimization for EDOF -1 Wiener deconvolution
  40. 40. -1 Wiener deconvolution Add Gaussian Noise End-to-end optimization for EDOF
  41. 41. -1 Wiener deconvolution Wiener deconvolution for reconstruction End-to-end optimization for EDOF
  42. 42. Scene depth During training: Place input image at random depth -1 Wiener deconvolution End-to-end optimization for EDOF
  43. 43. End-to-end optimized with deconvolution: Depth-independent and easily invertible Scene depth 𝜕 𝜕𝑥 Wiener deconvolution 𝜕 𝜕𝑥 L2 loss Fresnel Lens Multifocal Lens Cubic Phase Plate Diffractive Achromat Refr. / Diffr. Hybrid Lens Ours 17.95 dB 18.32 18.33 20.20 18.92 24.30 -1
  44. 44. Refractive Achromatic EDOF element • Polymethyl methacrylate (PMMA) • 5 mm aperture size • Sensor distance 35.5𝑚𝑚 • F-number 7.1 • One active optical surface • Feature size 3.69𝜇𝑚 Optical surface Caustics
  45. 45. Regular bi-convex lens Optimized lens Elephant (0.5m) Book (2.0m) Sensor image Test scene
  46. 46. Processed image Regular bi-convex lens Optimized lens Elephant (0.5m) Book (2.0m) Test scene
  47. 47. Regular bi-convex lens (focus close) Optimized camera Real-world capture
  48. 48. Diffractive Achromatic EDOF element • Fused silica processed via 16-level photolithography • 5 mm aperture size • Sensor distance 35.5𝑚𝑚 • F-number 7.1 • One active optical surface • Feature size 2𝜇𝑚
  49. 49. Regular Fresnel Lens Optimized Camera w. diffractive element Diffractive EDOF: Test scene
  50. 50. Experimental application: Super-Resolution
  51. 51. Experimental application: Super-Resolution Please refer to paper for more detail!
  52. 52. Summary
  53. 53. -1 𝜕 𝜕𝑥 Jointly optimizing optics and post-processing PSF 𝜕 𝜕𝑥
  54. 54. 𝜕 𝜕𝑥 Bunny 𝜕 𝜕𝑥 𝜕 𝜕𝑥 PSF -1 • Most optimization algorithms for image reconstruction can be made differentiable by unrolling • Jointly optimize CNNs with optics, image signal processing Future work: Better image reconstruction, higher-level tasks Unrolled Optimization with Deep Priors (Diamond et al.) Deep Joint Demosaicking and Denoising (Gharbi et al.) DeepISP (Schwartz et al.) FlexISP (Heide et al.) …
  55. 55. Bunny PSF Optical convolutional neural networks with optimized diffractive optics for image classification, Chang et al., 2018 -1 𝜕 𝜕𝑥 𝜕 𝜕𝑥 𝜕 𝜕𝑥
  56. 56. Felix Heide Evan Peng Wolfgang Heidrich Stephen Boyd Gordon Wetzstein Xiong Dun
  57. 57. End-to-end optimization of optical sensing pipelines https://vsitzmann.github.io/deepoptics/ 𝜕 𝜕𝑥 Bunny 𝜕 𝜕𝑥 𝜕 𝜕𝑥 PSF -1

×