1. Media Lab, MIT Cambridge, MA From 4D Capture to 6D Display: A mask-based approach Ramesh Raskar

3. R Raskar, H Nii, B de Decker, Y Hashimoto, J Summet, D Moore, Y Zhao, J Westhues, P Dietz, M Inami, S Nayar, J Barnwell, M Noland, P Bekaert, V Branzoi, E Bruns Siggraph 2007 Prakash: Lighting-Aware Motion Capture Using Photosensing Markers and Multiplexed Illuminators

4. Vicon Motion Capture High-speed IR Camera Medical Rehabilitation Athlete Analysis Performance Capture Biomechanical Analysis

5. Imperceptible Tags under clothing, tracked under ambient light Hidden Marker Tags Outdoors Unique Id

6. Labeling Space (Indoor GPS) Each location receives a unique temporal code But 60Hz video projector is too slow Projector Tags Pos=0 Pos=25 5 Time

8. Inside of Multi-LED Emitter

9. Tag

10. Analog Space Labeling Multi-LED Beacon 1 Beacon 2 Beacon 3 Tag N ?

11. Imperceptible Tags Location

12. Location Orientation

13. 3D Overlay Orientation

14. Imperceptible Tags Incident Illumination

16. On-set MoCap: Location + Orientation + Incident Illumination

19. Towards a 6D Display Passive Reflectance Field Display Martin Fuchs, Ramesh Raskar, Hans-Peter Seidel, Hendrik P. A. Lensch Siggraph 2008 1 2 1 1 1 MPI Informatik, Germany 2 MIT

20. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

22. electronic: [Nayar et al. 2004] slit based / different patterns: [Scharstein 1996] Martin Fuchs <mfuchs@mpi-inf.mpg.de>

23. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

24. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

25. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

26. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

27. Improved Design Martin Fuchs <mfuchs@mpi-inf.mpg.de>

29. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

30. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

31. Martin Fuchs <mfuchs@mpi-inf.mpg.de>

32. Observer-Variance Martin Fuchs <mfuchs@mpi-inf.mpg.de>

33. 6D Construction Martin Fuchs <mfuchs@mpi-inf.mpg.de>

34. Illumination + Spatial Variation Martin Fuchs <mfuchs@mpi-inf.mpg.de>

35. Variance with Observation Angle Martin Fuchs <mfuchs@mpi-inf.mpg.de>

36. Towards 6D Martin Fuchs <mfuchs@mpi-inf.mpg.de>

37. 6D Results Martin Fuchs <mfuchs@mpi-inf.mpg.de>

39. Coded Aperture Camera The aperture of a 100 mm lens is modified Rest of the camera is unmodified Insert a coded mask with chosen binary pattern

40. In Focus Photo LED

41. Out of Focus Photo: Open Aperture

42. Out of Focus Photo: Coded Aperture

43. Captured Blurred Photo

44. Refocused on Person

45. Mask? Sensor 4D Light Field from 2D Photo: Heterodyne Light Field Camera Full Resolution Digital Refocusing: Coded Aperture Camera Mask? Sensor Mask Sensor Mask? Sensor Mask Sensor

46. Light Field Inside a Camera

47. Lenslet-based Light Field camera [Adelson and Wang, 1992, Ng et al. 2005 ] Light Field Inside a Camera

49. Digital Refocusing [Ng et al 2005] Can we achieve this with a Mask alone?

50. Mask based Light Field Camera [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor

51. Heterodyne Light Field Camera Scanner sensor Mask [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ] Mask Sensor

52. How to Capture 4D Light Field with 2D Sensor ? What should be the pattern of the mask ?

53. Radio Frequency Heterodyning Receiver: Demodulation High Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz Baseband Audio Signal

54. Optical Heterodyning Photographic Signal (Light Field) Carrier Incident Modulated Signal Reference Carrier Main Lens Object Mask Sensor Recovered Light Field Software Demodulation Baseband Audio Signal Receiver: Demodulation High Freq Carrier 100 MHz Reference Carrier Incoming Signal 99 MHz

55. Captured 2D Photo Encoding due to Mask

56. 2D FFT Traditional Camera Photo Heterodyne Camera Photo Magnitude of 2D FFT 2D FFT Magnitude of 2D FFT

57. Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*9 4D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9

61. How to Capture 2D Light Field with 1D Sensor ? f θ f x f θ 0 f x0 Band-limited Light Field Sensor Slice Fourier Light Field Space

62. Extra sensor bandwidth cannot capture extra dimension of the light field f θ f x f θ 0 f x0 Sensor Slice Extra sensor bandwidth

63. f θ f x ??? ??? ??? ???

64. Solution: Modulation Theorem Make spectral copies of 2D light field f θ f x f θ 0 f x0 Modulation Function

65. f θ Modulated Light Field f x f θ 0 f x0 Modulation Function Sensor Slice captures entire Light Field

66. Demodulation to recover Light Field f θ f x Reshape 1D Fourier Transform into 2D 1D Fourier Transform of Sensor Signal

67. f θ f x f θ 0 f x0 Modulation Function == Sum of Impulses Physical Mask = Sum of Cosines

68. 1/f 0 Mask Tile Cosine Mask Used

69. Where to place the Mask? Mask Sensor Mask Modulation Function Mask Modulation Function f x f θ Mask Sensor

70. Mask Sensor Where to place the Mask? Mask Modulation Function f x f θ

71. Mask Sensor d v α α = (d/v) ( π /2) Where to place the Mask? Mask Modulation Function

72. Captured 2D Photo Encoding due to Cosine Mask

73. Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform 2D FFT Rearrange 2D tiles into 4D planes 200*200*9*9 4D IFFT 4D Light Field 9*9=81 spectral copies 200*200*9*9

74. Digital Refocusing Only cone in focus Captured Photo

75. Full resolution 2D image of Focused Scene Parts Captured 2D Photo Image of White Lambertian Plane divide

76. Coding and Modulation in Camera Using Masks Coded Aperture for Full Resolution Digital Refocusing Heterodyne Light Field Camera Mask? Sensor Mask Sensor Mask Sensor