1. Compressive Displays:
Managing Lightfield Bandwidth with
Math + Layered LCDs
Ramesh Raskar
Asso. Prof.
MIT Media Lab
http://cameraculture.info

2. Slow Glass: Time Shifted Display
Light of Other Days by Bob Shaw
http://www.fantasticfiction.co.uk/s/bob-shaw/other-days-other-eyes.htm http://baens-universe.com/articles/otherdays

3. Shift Glass
Space Shifting
Angle Shifting
Time Shifting
Illumination Shifting
4D 4D t 4D 4D

4. Motivation: Glasses Free 3D Displays
Eliminating Eyewear Expanding FoV and DoF for Glasses-free Displays
Eliminating Moving Parts Thin Holographic Displays
(Preserving Depth Cues) (Developing or Avoiding High-Res SLM)
Favarola et al Michael Bove et al

5. Glasses-free 3D Displays
Bandwidth: 600 Gbytes/sec
1M * 60 Hz * 100x100 Views

6. Compressive Displays:
Lossy Compression in Optics
Bandwidth: 600 Gbytes/sec
2 Gbytes/sec

7. Opportunities: New Hardware + New Math
Emerging Displays
Multilayer High Frame Rate Directional Backlighting
Compression and Embedded Processing
Non-negative Matrix Factorization (NMF) Non-negative Tensor Factorization (NTF)

8. Camera Culture
Creating new ways to capture and share visual information
MIT Media Lab
Ramesh Raskar
http://cameraculture.info
Facebook.com/cameraculture
Computational Photography Femtosecond Imaging 3D Displays
1. Looking around corners 1. Tensor Display
1.Light-Field Camera A family of compressive light field
A new camera design exploiting the Using short laser pulses and fast detector, we aim to build
a device that can look around corners with no imaging displays comprising all architectures
fundamental dictionary of light-fields
device in the line of sight using time resolved transient employing a stack of time-multiplexed,
for a single-capture capture of light-
imaging. light-attenuating layers illuminated by
fields with full-resolution refocusing
uniform or directional backlighting
effects.
2. Layered 3D
2. Color Primaries Tomographic techniques for image
A new camera design with
synthesis on displays composed of
switchable color filter arrays for compact volumes of light-attenuating
optimal color fidelity and picture material. Such volumetric attenuators
quality on scene geometry, color recreate a 4D light field or high-contrast
and illumination. 2D image when illuminated by a uniform
2. Reflectance Recovery backlight.
We demonstrate a new technique that allows a camera to
3. Flutter-Shutter rapidly acquire reflectance properties of objects 'in the wild' 3. Glasses-free 3D HDTV
A camera that codes the exposure from a single viewpoint, over relatively long distances and Light field displays with increased
time with a binary pseudo-sequence without encircling equipment. brightness and refresh rate by stacking a
to de-convolve and remove motion
pair of modified LCD panels, exploiting
blur in textured backgrounds and rank and constraint of 3D displays
partial occluders. 3. Trillion Frames per Second Imaging
A camera fast enough to capture light pulses moving
through objects. We can use such a camera to understand 4. BIDI Screen
reflectance, absorption and scattering properties of A thin, depth-sensing LCD for 3D
4. Compressive Capture materials. interaction using light fields which
We analyze the gamut of visual
supports both 2D multi-touch and
signals from low-dimensional
unencumbered 3D gestures.
images to light-fields and propose
non-adaptive projections for
efficient sparsity exploiting 5. Living Windows 6D Display
reconstruction. A completely passive display that
responds to changes in viewpoint and
changes in incident light conditions.
May 2012

9. Health & Wellness Human Computer Interaction Visual Social Computing
1. Retinal Imaging 1. Bokode 1. Photocloud
With simplified optics and cleaver Low-cost, passive optical design so A near real-time system for
illumination we visualize images of the that bar codes can be shrunk to fewer interactively exploring a collectively
retina in a standalone device easily than 3mm and read by ordinary captured moment without explicit 3D
operated by the end user. cameras several meters away. reconstruction.
2. NETRA/CATRA
Low-cost cell-phone attachments that 2. Specklesense 2. Vision Blocks
measures eye-glass prescription and Set of motion-sensing configurations On-demand, in-browser,
cataract information from the eye. based on laser speckle sensing . The customizable, computer-vision
3. Cellphone Microscopy underlying principles allow application-building platform for the
A platform for computational microscopy interactions to be fast, precise, masses. Without any prior
and remote healthcare extremely compact, and low cost. programming experience, users can
create and share computer vision
4. High-speed Tomography 3. Sound Around applications.
A compact, fast CAT scan machine using
Soundaround is a multi-viewer
no mechanical moving parts or
interactive audio system, designed to
synchronization.
be integrated into multi-view displays
3. Lenschat
presenting localized audio/video LensChat allows users to share
5. Shield Fields channels with no need for glasses or mutual photos with friends or borrow
3D reconstruction of objects from a single headphones. the perspective and abilities of many
shot photo using spatial heterodyning. cameras.
6. Second Skin
Using 3D motion tracking with real-time
Visit us online at
vibrotactile feedback aids the correct of
movement and position errors to improve
motor learning. Cameraculture.info
fb.com/cameraculture
Light Propagation Theory and Fourier Optics
1. Augmented Light Fields 2. Hologram v Parallax Barrier
Expands light field representations to Defines connections between parallax 3. Ray–Based Diffraction Model
describe phase and diffraction effects by barrier displays and holographic displays by
Simplified capture of diffraction model for
using the Wigner Distribution Function analyzing their operations and limitations in
computer graphics applications.
phase space
Post-Doctorial Researchers: Doug Lanman, Gordon Wetzstein, Alex Olwal, Christopher Barsi
Research Assistants: Matthew Hirsch, Otkrist Gupta, Nikhil Naik, Jason Boggess, Everett Lawson, Aydın Arpa, Kshitij Marwah
Visiting Researchers & Students: Di Wu, Daryl Lim

10. Office of the Future,
UNC, 1997

11. Pocket Projectors
1999-2004
UNC Chapel Hill and
Mitsubishi Electric Research Labs

12. Software Compression
Scalable Display Inc.

13. Time Illumination Space/Angle
Slow Display 6D Display Layered 3D
tinyurl.com/slow-display tinyurl.com/6d-display www.layered3d.info
High-Rank 3D (HR3D) BiDi Screen Polarization Fields
www.hr3d.info www.bidiscreen.com tinyurl.com/polarization-fields

14. Optical Sensing: Touch + 3D Gestures

15. BiDi Screen

16. Light Field Transfer Application

17. MIT media lab camera culture EyeNetra.com
NETRA: Refractive Error on Mobile Phone
Siggraph 2010
Vitor Pamplona Ankit Mohan Manuel Oliveira Ramesh Raskar
18

18. 6D Display: Respond to View + Ambient Illumination

19. Camera Culture: Compressive Displays Team
Gordon Wetzstein Matthew Hirsch Douglas Lanman
Postdoctoral Associate Graduate Student Postdoctoral Associate
Wolfgang Heidrich, Professor, University of British Columbia
Yunhee Kim, Postdoctoral Fellow, MIT Media Lab

20. 3D Display: Light and Rank Deficient
Parallax
barrier
Front
Back
LCD display

21. Input 4D Light Field: Horizontal and Vertical Parallax

22. Parallax Barrier: Front Layer

23. Parallax Barrier: Rear Layer

24. Light Field of Parallax Barriers: Rank 1
k
L[i,k]
i
k
g[k]
i `
f[i]
light box
L[i, k ]  f [i]  g[k ] L f g

25. Dual LCD: Content-Adaptive Parallax Barriers
G
L[i,k]
k
g[k]
~
i F L`
f[i]
light box
~
L  FG

26. Dual LCD: Content-Adaptive Parallax Barriers
G
L[i,k]
k
g[k]
i F ~
f[i] L`
light box
~
L  FG
Lanman, Hirsch, Kim, Raskar Siggraph Asia 2010

27. High-Rank 3D (HR3D): Front Layer

28. High-Rank 3D (HR3D): Rear Layer

29. Algebraic Rank Constraint
Rank-1 Rank-1
s1*
m2
s1 s1

30. Light Field vs. Holographic Displays

31. Is a hologram just another ray-based light field?
Can a hologram create any intensity distribution in 3D?
Why does a hologram create a “wavefront”, but parallax barrier does not?
Why does a hologram create accommodation cues?
What are the effective resolution and depth of field for holograms vs. barriers?

32. Parallax Barrier: Np=103 pix. Hologram: NH=105 pix.
ϕP∝w/d ϕH∝λ/tH
θp=10 pix θH =1000 pix
Fourier Patch
w
Horstmeyer, Oh, Cuypers, Barbastathis, Raskar, 2009

33. Augmented Light Field
wave optics based
rigorous but cumbersome
Wigner
WDF
Distribution
Function Augmented LF
Traditional Traditional
Light Field Light Field
ray optics based
Interference & Diffraction
simple and powerful Interaction w/ optical elements
Oh, Raskar, Barbastathis 2009: Augmented Light Field
34

34. Glasses-free 3D Displays
Raw: 600 Gbytes/sec
Compressive: 2 Gbyte/sec

36. View Dependent Appearance and Iridescent color
Cross section through a single M. Rhetenor scale

37. Generalizing Parallax Barriers: Rank 1
mask K
…
mask 3
mask 2 mask 2 mask 2
mask 1 mask 1 mask 1
light box light box light box
Conventional Parallax Barrier High-Rank 3D (HR3D) Layered 3D and Polarization Fields
 Parallax barriers use heuristic design: front mask with slits/pinholes, rear mask with interlaced views
 High-Rank 3D (HR3D) considers dual-layer design with arbitrary opacity and temporal multiplexing
 Layered 3D and Polarization Fields considers multi-layer design without temporal multiplexing

38. Layered 3D: Multi-Layer Automultiscopic Displays
mask K
…
mask 3
mask 2
mask 1
light box
Layered 3D

39. Tomographic Light Field Synthesis
virtual plane
Image formation model:
attenuator
ò
- m (r )dr
L(x, q ) = I 0 e C
æ L(x, q ) ö
L(x, q ) = ln ç ÷ = - ò m (r)dr
è I0 ø C
backlight l = -Pa
Tomographic synthesis:
2
arg min l + Pa , for a ³ 0
a
2D Light Field

40. Tomographic Light Field Synthesis
virtual plane
Image formation model:
attenuator
ò
- m (r )dr
L(x, q ) = I 0 e C
æ L(x, q ) ö
L(x, q ) = ln ç ÷ = - ò m (r)dr
è I0 ø C
backlight l = -Pa
Tomographic synthesis:
2
arg min l + Pa , for a ³ 0
a
2D Light Field

41. Multi-Layer Light Field Decomposition
Reconstructed Views
Target 4D Light Field
Multi-Layer Decomposition

42. Prototype Layered 3D Display
Transparency stack with acrylic spacers Prototype in front of LCD (backlight source)

44. Polarization Fields
Four Stacked Liquid Crystal Panels
Two Crossed Polarizers

46. Simulation Results

47. Codesign of Optics + Computation
Materials Shift Glass
Photons Optics Displays
Lighting Capture
Sensors
HCI
Signal Processing
CV / Machine Learning
Bits

49. Speed beats Resolution

50. Tensor Displays
Siggraph 2012
Highlight Paper

51. Tensor Displays

52. Tensor Displays

53. Tensor Displays

59. Next Generation of Stacked LCDs
Asks
High Frame Rate (x 10)
Moiré: Non-uniform pixel grid
Transmission Limitations
OLED and emerging tech Needs ‘Compressible’ views
Slightly lower brightness
Collaboration Within intrinsic display properties
Open Architecture
CompDisp Consortium
Visit ML,
Join at cameraculture.info

60. • Higher Dimensional Displays
Compressive Display
– 3D, video, wavelength ..
• 600 GB  2GB/sec
– Lossy compression in OPTICS
• Emerging Hardware
– Multi-layer, high frame rate, directional backlt
– Frame rate beats resolution
G
• Mathematical Optimization ~
`
L
= F
– Tomography, sparsity, tensor factorization

61. Camera Culture: Compressive Displays Team
Gordon Wetzstein Matthew Hirsch Douglas Lanman
Postdoctoral Associate Graduate Student Postdoctoral Associate
Wolfgang Heidrich, Professor, University of British Columbia
Yunhee Kim, Postdoctoral Fellow, MIT Media Lab

62. Raskar, Lanman, Wetzstein, Hirsch MIT Media Lab http://cameraculture.info
Shift Glass
Capture Display
5D: Looking Compressive Displays
around corners 6D: View and Lighting Aware
4D: Rank Deficient, multilayer
4D: Netra for Optometry
WDF
Analyze G
Augmented
Light LF
~
`
L
= F
Field
4D, 6D, 8D: Augmented Light Field

63. Raskar, Lanman, Wetzstein, Hirsch MIT Media Lab http://cameraculture.info
Layered 3D Polarization Fields High-Rank 3D (HR3D)
www.layered3d.info tinyurl.com/polarization-fields www.hr3d.info
Slow Display 6D Display BiDi Screen
tinyurl.com/slow-display tinyurl.com/6d-display www.bidiscreen.com

64. Compressive Display Research in Camera Culture
Ramesh Raskar, Douglas Lanman, Gordon Wetzstein, Matthew Hirsch
http://cameraculture.media.mit.edu/compressivedisplays

65. Faster Horse  Car