Keynote sludes

1. Compressive Displays
Combining Optical Fabrication, Computation, and Perceptual
Tricks to Build the Displays of the Future
Gordon Wetzstein
MIT Media Lab
media.mit.edu/~gordonw
displayblocks.org

2. Evolution?
This slide has a 16:9 media window
1928
2013

3. displayblocks.org

4. Images: Wikipedia, Shinya Yoshioka

5. Nature
This slide has a 16:9 media window
Image: Desafio Monteverde and Arenal Volcano Tours Costa Rica

6. Nature
Image: National Geographic

7. Three-layer Tensor Display

8. This slide has a 16:9 media window

9. This slide has a 16:9 media window
Compressive Light
Field Displays

10. This slide has a 16:9 media window

12. viewer moves right
viewer moves down
This slide has a 16:9 media window
4D Light Field

13. Compressive Displays
Display Optics
Computational
Processing

14. Some Depth Cues of the HVS
binocular disparity
convergence
current glasses-based (stereoscopic) displays
near-term: compressive light field displays
longer-term: holographic displays
motion parallax
accommodation/blur

16. Parallax Barriers – Ives 1903
barrier
2D display
• low resolution & very dim
• switchable 2D/3D with LCDs

17. lenslets
Integral Imaging – Lippmann 1908
2D display
• brighter than parallax barriers
• always low resolution, even for 2D

18. Next-generation Displays
computation
optics & electronics
interaction
human visual
system

19. Structural Formula for Compressive Display Innovation
M
N
C
N
U
C
O
C
multiplexing
nonlinear
compression
optimization
user experience
content
community
next

20. I. Multiplexing

21. Multiplexing
Most displays are 2D!
Spatial Multiplexing

22. Space vs Time
visual acuity / retina display: 20/20 = 1 arc minute
@ 10” need >300 dpi
multiplexing requires 300++ dpi
flicker fusion threshold: 20-60 Hz (depends on light)
LCDs: 120-240 Hz
DMDs, ferroelectric LCDs: KHz
much easier with currently available hardware!

23. Combine Space and Time
mask 2
mask 1
Conventional Parallax Barriers

24. Combine Space and Time
mask 2
mask 1
Time-shifted Parallax Barriers [Kim et al. 2007]
High Resolution through High Speed

25. I. Multiplexing
Use temporal multiplexing if you can!

26. II. Nonlinear Pixel Interaction
& Coupling

27. mask 2
mask 1
Conventional Parallax Barriers

28. nonlinear
l = p1*p2
p2
mask 2
mask 1
blocked!
p1
Conventional Parallax Barriers

29. linear
l = p1+p2
p2
mask 2
mask 1
not blocked!
p1
Additive Layers / Most Volumetric Displays
e.g., LEDs or transparent OLEDs

30. Holography – Nonlinear Interaction
plane wave
emitted wavefront: U(x)
received intensity:
I(x) = T {U(x)}
2
= Re {T {U(x)}} + Im {T {U(x)}}
2
hologram
2
screen or retina

31. Holography – Coupling
plane wave
æ
x'ö æ
x'ö
W (x, u = l sin(q )) = ò t ç x + ÷ t ç x - ÷ e2 p ix'u dx '
è
2ø è
2ø
Fourier transform of all points interacting
with each other!
hologram
nonlinear interaction & pixel coupling!

32. Layered 3D – SIGGRAPH 2011
attenuation layers with
spacers
backlight

33. Layered 3D – SIGGRAPH 2011
mask K
…
mask 2
mask 1

34. Layered 3D – SIGGRAPH 2011
l = p1*p2*…*pK
mask K
pK
…
mask 2
mask 1
p2
p1

35. Layered 3D – SIGGRAPH 2011
mask K
…
mask 2
mask 1

37. Tensor Displays – SIGGRAPH 2012
Reconfigurable
directional backlight
three layer

38. Three Layer Prototype

40. Directional Backlight Prototype

41. Tensor Displays – Directional Backlight
Perceptual Integration
mask 2
microlens array
mask 1
time

42. view from above
LCD with directional backlight, rank 6
LCD + directional BL
conventional lenslets

43. LCD with directional backlight, rank 6 (as seen by obserer)

44. Filmed with High-speed Camera
directional backlight
front LCD

45. Polarization Fields – SIGGRAPH ASIA 2011

46. four stacked liquid crystal panels
two crossed polarizers

47. Polarization Fields – SIGGRAPH ASIA 2011
design optimization
LCD 3
LCD 2
LCD 1
backlight
• eliminate redundant polarizers
 sequentially-crossed design

48. Polarization Fields – SIGGRAPH ASIA 2011
design optimization
LCD 3
• eliminate redundant polarizers
 use sequentially-crossed
LCD 2
• exploit field-sequential color
 0.33 = 2.7% brightness
LCD 1
backlight

49. Polarization Fields – SIGGRAPH ASIA 2011
design optimization
LCD 3
• eliminate redundant polarizers
 use sequentially-crossed
LCD 2
• exploit field-sequential color
 0.33 = 2.7% brightness
LCD 1
• further optimize polarizers
 minimum is a crossed pair
backlight

50. Polarization Fields – SIGGRAPH ASIA 2011
f3
LCD 3
f2
LCD 2
f1
LCD 1
backlight
image formation
K
Q(x, q ) = åfk (x, q )
k=1
L(x, q ) = sin 2 (Q(x,q ))

52. II. Nonlinear Pixel Interaction
& Coupling
Design optical systems that provide
nonlinear pixel coupling!

53. III. Compression

54. Which Light Field is More Compressible?
a
b

55. Which Light Field is More Compressible?
a
b

56. Compressive
Optics
16:9 media window
This slide has a
Display-adaptive
Compression
Computed Tomography
4D Light Field
Nonnegative Tensor
Factorization
Uniform or
Directional Backlight
Stacked Layers
(LCDs or Transparencies)
Observer = Decoder

57. Give those Pixels a Break!
Applied Mathematics
Benefits for Optics & Electronics
• sparse optimization
• fewer pixels
• low-rank factorization
• relaxation on refresh rate
• …
• thinner form factors
• …

58. From Conventional to Compressive Displays
mask 2
mask 1
Conventional Parallax Barriers
t
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
HR3D
SIG Asia 2010
t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
t

59. From Conventional to Compressive Displays
mask 2
mask 1
Time-shifted Parallax Barriers [Kim et al. 2007]
High Resolution through High Speed
t
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
HR3D
SIG Asia 2010
t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
t

60. From Conventional to Compressive Displays
Perceptual Integration
time
Time-shifted Parallax Barriers [Kim et al. 2007]
High Resolution through High Speed
t
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
HR3D
SIG Asia 2010
t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
t

61. From Conventional to Compressive Displays
mask 2
mask 1
High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010]
Compression in Time – Nonnegative Matrix Factorization
t
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
HR3D
SIG Asia 2010
t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
t

62. From Conventional to Compressive Displays
Perceptual Integration
time
High-Rank 3D [Lanman et al., SIGGRAPH Asia 2010]
Compression in Time – Nonnegative Matrix Factorization
t
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
HR3D
SIG Asia 2010
t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
t

63. From Conventional to Compressive Displays
mask K
…
mask 2
mask 1
Layered 3D [Wetzstein et al., SIGGRAPH 2011]
Compression in Pixels & Depth – Computed Tomography
t
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
HR3D
SIG Asia 2010
t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
t

64. From Conventional to Compressive Displays
mask K
…
mask 2
mask 1
Layered 3D [Wetzstein et al., SIGGRAPH 2011]
Compression in Pixels & Depth – Computed Tomography
t
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
HR3D
SIG Asia 2010
t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
t

65. From Conventional to Compressive Displays
mask K
…
mask 2
mask 1
Layered 3D [Wetzstein et al., SIGGRAPH 2011]
Compression in Pixels – Computed Tomography
t
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
HR3D
SIG Asia 2010
t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
t

66. From Conventional to Compressive Displays
…
Perceptual Integrat
…
…
time
Tensor Displays
Compression in Time & Pixels –Tensor Factorization
t
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
HR3D
SIG Asia 2010
t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
t

67. From Conventional to Compressive Displays
…
Perceptual Integrat
…
…
time
Tensor Displays – Multilayer & Directional Backlighting
t
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
HR3D
SIG Asia 2010
t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
t

68. From Conventional to Compressive Displays
thin!
Perceptual Integrat
time
Tensor Displays – Directional Backlighting
t
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
HR3D
SIG Asia 2010
t
Layered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
t

69. III. Compression
Design new optical architectures that
directly exploit data compression!

70. IV. Optimization
via High-performance Computing

71. Layered 3D – SIGGRAPH 2011
attenuation layers with
spacers
backlight

73. Computed Tomography (CT)
This slide has a 16:9 media window
Image: Wikipedia
x-ray sensor
3D Reconstruction
x-ray source
Reconstructed 2D Slices

74. Computed Tomography (CT)
This slide has a 16:9 media window

75. Tomographic Light Field Synthesis
image formation
virtual planes
x
L(x, q ) = e
-
ò c m (r )dr
log ( L ( x, q )) = - ò m (r)dr
attenuation volume
c
backlight
tomographic synthesis
2D light field
argmin log( L) P
0
x
2
2

76. Tomographic Polarization Field Synthesis
virtual planes
f3
LCD 3
f2
LCD 2
image formation
K
x
f1
Q(x, q ) = åfk (x, q )
k=1
L(x, q ) = sin 2 (Q(x,q ))
LCD 1
tomographic synthesis
backlight
Q(x,q ) = ±sin-1
2D light field
(
argmin Q - Pf
fmin £f £fmax
x
)
L(x, q ) mod p
2
2

77. Low-rank Light Field Factorization
light field
two-layer light field display
L(
front layer
fm(2)(
rear layer
backlight
fm(1)(

78. Low-rank Light Field Factorization
two-layer light field display
front layer
front layer
fm(2)(
rear layer
backlight
Lanman et al. – SIGGRAPH Asia 2010
fm(1)(
rear layer
L(
`
rank-1

79. Low-rank Light Field Factorization
high-speed LCDs = perceptual average
two-layer light field display
L(
front layer
fm(2)(
rear layer
backlight
Lanman et al. – SIGGRAPH Asia 2010
fm(1)(
`
rank-4

80. Low-rank Light Field Factorization
high-speed LCDs = perceptual integration
G
~
L
FG
Lanman et al. – SIGGRAPH Asia 2010
F
~
`
L
rank-4

81. Low-rank Light Field Factorization
arg min L - FG Wfield F,G ³ 0
, for
emitted light
2
F,G
`
L
=
G
F

82. Low-rank Light Field Factorization
light field
light field tensor
multi-layer light field display
L(
fm(3)(
middle layer
fm(2)(
rear layer
backlight
fm(1)(
rear layer
front layer
L(

83. Low-rank Light Field Factorization
Target Light Field Tensor
Rank-M Approximation
nonnegative tensor
factorization (NTF)
+ ... +
+
frame 1
perceptual
integration
frame 2
frame M

84. Low-rank Light Field Factorization
nonlinear (multilinear)
optimization problem
iterative update rules
(see paper for details)
Efficient GPU Implementation
forward projection (multiview rendering)
back projection (projective texture mapping)

86. IV. Optimization
via High-performance Computing
Find optimal pixel states via optimization!

87. V. User Experience

88. medical?
Nvidia
University of Louisiana
entertainment?
Gregg Favalora’s Perspecta
James Cameron’s Avatar
What’s the Killer App of 3D Displays?
scientific visualization?
games?

89. (Interactive) Experiences!
leap motion
ms kinect

90. Vision-correcting Display

91. unfortunately, this content is not public at the moment

92. V. User Experience
Design user experiences, not devices!

93. VI. Community

95. displayblocks.org
Code, data & instructions online!

96. VI. Community
Let’s share our knowledge!

97. VII. Content

98. Content Generation
Live Action
consumer light field
cameras
camera rigs
Rendered Footage

99. Lytro Light Field Camera

100. sensor
Lytro Light Field Camera
lenslets

101. Compressive Light Field Photography – SIG 2013
Exploit
redundancy
Sparsify
Preserve
information
Compressive
Reconstruction
Light Field Atoms
Mask-based
Light Field Coding

102. Prototype Setup with a Variable Mask
LCoS
Virtual sensor
Polarizing
Imaging Beamsplitter
Lens
Camera
Image sensor

104. Display-Adaptive Image Synthesis – SIG 2013
“conventional pipeline”
1. render trillions of rays
2. large-scale optimization
3. display pixel states

105. Display-Adaptive Image Synthesis – SIG 2013

106. Display-Adaptive Image Synthesis – SIG 2013
samples
simulation
prototype
1-3% of rendered & optimized samples = same quality

107. VII. Content
Content is crucial!
How you capture / render it too!

108. VIII. What’s Next?

109. Accommodative Depth Cues

110. Transactions on Graphics 2013

111. Compressive
Superresolution Display

112. unfortunately, this content is not public at the moment

113. Compressive
Light Field Projection

114. unfortunately, this content is not public at the moment

115. Head-mounted Displays
Google Glass
Oculus Rift
Avegant Glyph
Nvidia Near Eye Display

116. Structural Formula for Compressive Display Innovation
M
N
C
N
U
C
O
C
multiplexing
nonlinear
compression
optimization
user experience
content
community
next

117. Gordon Wetzstein
MIT Media Lab
media.mit.edu/~gordonw
displayblocks.org
collaborators:
Matt Hirsch (MIT)
Doug Lanman (NVIDIA)
Andrew Maimone (UNC)
Felix Heide (UBC)
Fu-Chung Huang (UC Berkeley)
Vincent Lee (MIT)
James Gregson (UBC)
Brian Barsky (UC Berkeley)
Henry Fuchs (UNC)
Ramesh Raskar (MIT)
Wolfgang Heidrich (UBC)
code, data & instructions online
use Layered 3D in your class!