Slides from a talk I gave at: Wissens-Austausch-Workshop: The 3rd Workshop on Visualisierung großer Datenmengen in der Wissenschaft (VisDa3? WAW), DLR@Oberpfaffenhofen, Germany, Jun. 9 - 10, 2015. For more detail of my work, see: http://campar.in.tum.de/Main/YutaItoh

1. FAR: Fachgebiet Augmented Reality Technische Universität München
Head Mounted Displays
How to realize ultimate AR experiences?
Yuta Itoh (TUM)
Workshop Visualisierung großer Datenmengen, 9th June 2015
1

2. FAR: Fachgebiet Augmented Reality
Prof. Gudrun Klinker
2/52

3. Industrial Augmented Reality
3/52

4. Head-Mounted Displays (HMDs)
Optical See-Through (OST-HMD)
Non Optical See-Through
4/52

5. OST-HMD in science fiction
From Ironman, 2008 ©Marvel Entertainment
5/52

6. Everything is perfect in Science Fiction
 World sensing
 Image recognition
 Eye tracking
6/52

7. 7/52

8. Arianespace, 2009Le Voyage dans la Lune, 1902
1902 2009
8/52

9. What is the current status
of the OST-HMD technology?
What’s still missing?
9/52

10. Hugo Gernsback, TV Glasses
1963
10/52

11. Ivan Sutherland, The Sword of Damocles
1968
11/52

12. 2013-
12/52

13. 13/52

14. 14/52

15. OST-HMDs Overwrite the reality
NVIS nVisor ST60
15/52

16. Related Work | AR X-ray vision
Avery, Ben, Christian Sandor, and Bruce H. Thomas. "Improving spatial perception for augmented reality
x-ray vision." Virtual Reality Conference, 2009. VR 2009. IEEE. IEEE, 2009.
16/52

17. Related Work | Visual Assistance
Hwang, Alex D., and Eli Peli. "An augmented-reality edge enhancement application for Google Glass."
Optometry & Vision Science 91.8 (2014): 1021-1030.
17/52

18. Issues in OST-HMDs
18/52

19. Design Factor (non-OST)
Toshiba, 2006 Oculus Rift DK2, 2014
Photo from IFIXIT
19/52

20. Design Factor
Google Glass, 2013
Photo by Mikepanhu
20/52

21. Wide Field of View
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused Point Light Sources” Maimone et al., TOG’14
21/52

22. Wide Field of View
Issues on OST-HMD
“Pinlight Displays: Wide Field of View Augmented Reality Eyeglasses
using Defocused Point Light Sources” Maimone et al., TOG’14
22/52

23. Consistency issues in OST-HMDs
Temporal
[Zheng ’14] [Didier ’05]
Visual
[Kiyokawa ’01] [Liu ’08] [Lee ’09]
Spatial
[Azuma ’95] [Tuceryan ’00] [Genc ’02]
Social
Image from Google
Glass: Don't Be A
Glasshole | Mashable
23/52

24. Temporal Inconsist. in OST-HMD
“How fast is fast enough? : a study of the effects of latency in
direct-touch pointing tasks” Jota et al. CH’13
“latencies down to 2.38 ms are required to alleviate
user perception when dragging”
24/52

25. Temporal Inconsist. in OST-HMD
Digital Light Processing Projector
“Minimizing Latency for Augmented Reality Displays: Frames
Considered Harmful” Zheng et al. ISMAR’14
25/52

26. Visual Consistency
Occlusion, Depth, Color, Shadow, etc..
Lee and woo, 2009
Kiyokawa et al. 2003
Liu et al. 2008
26/52

27. Occlusion (physical “Z-buffer”)
27/52

28. Depth (eye accommodation)
Liu et al. 2008
28/52

29. Grubert, J., Tümler, J., & Mecke, R. (2008). Untersuchungen zur optimierung der see-through-
kalibrierung fuer mobile augmented reality assistenzsysteme. Michael Schenk, Hrsg, 6(7).
Spatial Calibration of OST-HMDs
29/52

30. Spatial Calibration of OST-HMDs
30/52

31. Eye-HMD Calibration
3D
2D
31/52

32. P is a Black Box
Find 3D-2D Projection:
3D2D
32/52

33. 2D
Say, is a Perspective Projection
33/52

34. Find 3D-2D Projection:
In the Eye of the Beholder…
= K*[R t]
34/52

35. Manual alignment
– Intensive user interaction
– User-dependent noise
Predictive tracking for augmented reality
RT Azuma - 1995 - ronaldazuma.com
35/52

36. Manual alignment
SPAAM: Single Point Active Alignment Method
– Medium user interaction
– User-dependent noise
Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR
M Tuceryan, N Navab - Augmented Reality, 2000.(ISAR 2000). …, 2000 - ieeexplore.ieee.org
Practical solutions for calibration of optical see-through devices
Y Genc, M Tuceryan, N Navab - … of the 1st International Symposium on …, 2002 - dl.acm.org
N times
36/52

37. SPAAM (Manual Calibration) video
37/52

38. Idea: user-perspective rendering
Spindler, Martin, Wolfgang Büschel, and Raimund Dachselt. "Use your head: tangible windows for 3D information
spaces in a tabletop environment."Proceedings of the 2012 ACM international conference on Interactive tabletops
and surfaces. ACM, 2012.
38/52

39. Utilizes 3D Eye Localization
–Interaction-free, thus do not bother users
–More accurate than a realistic SPAAM setup
Automated alignement
Yuta Itoh, and Gudrun Klinker. "Interaction-free calibration for optical
see-through head-mounted displays based on 3d eye localization." 3D
User Interfaces (3DUI), 2014 IEEE Symposium on. IEEE, 2014.
39/52

40. 1. Estimate a 2D iris ellipse
– Iris detector + Fitting by RANSAC
2. Back project it to 3D circle
Robust real-time pupil tracking in highly off-axis images
L Świrski, A Bulling, N Dodgson - … of the Symposium on
Eye Tracking …, 2012 - dl.acm.org
3D Eye Position Estimation
Image-based Eye Pose and Reflection Analysis for Advanced Interaction Techniques andScene Understanding
C Nitschke, A. Nakazawa, H. Takemura - CVIM,[ …, 2011 - ci.nii.ac.jp
40/52

41. Estimated 3D Eye Position
[m]
>10 cm
41/52

42. Wearable eye-trackers
Pupil Labs Tobii
42/52

43. OST-HMD + eye trackers?
43/52

44. OST-HMD + eye trackers?
A compact, eye-tracked optical see-through head-mounted display
H Hua, C Gao - Proc. SPIE, 2012 - ebooks.spiedigitallibrary.org
44/52

45. Pupil Center Cornea Reflection
Villanueva ‘08
Cornea reflection
45

46. HMD Screen resembles a plane
HMD
Image
Screen
46

47. Corneal Reflection
47

48. Detected Checkerboard in 2D
Detected grid corners
48

49. Scene Reflection in the Eye
49

50. Detected and Projected
Corners
Reprojected Points (3D  2D)
Original Image
50

51. Estimated Cornea Position
Limbus tracking Corneal Imaging
Plopski, Alexander, et al. "Corneal-imaging calibration for
optical see-through head-mounted
displays." Visualization and Computer Graphics, IEEE
Transactions on 21.4 (2015): 481-490.
51

52. Optical See-Through HMDs are hard fun
• Consistency issues
(Spatial, visual, temporal, etc.)
• Eye sensing will be a hot topic
• Big players => big markets
Takeaways:
52/52

53. Appendix
53/52

54. An ideal eye-HMD system
Eye
Image source
Environment
Combiner
(a half mirror)
54/52

55. HMD-dependent parameters
User-dependent parameters
Image by Soerfm
Photo by Laitr Keiows
55

56. • Shipments of smart watches&glasses
–130 million by 2018
•10 times than 2013
Dawn of a Commercial Wearable Devices
56/52

57. Innovega iOptik
57/52

58. Simple
No user interaction
Accurate
 better than Degraded manual
calibrations
Summary of INDICA
Calibration of OST-HMDs using
3D eye position
58/52

59. Eye Position Estimation
E
<
C1
Eyeball
sphere
C3
Eyeball
orbit
g2
g3
g1
C2
59