Guest lecture on Grand Challenges for Mixed Reality. Taught by Mark Billinghurst on October 19th 2021 at the University of Canterbury

1. GRAND CHALLENGES FOR
MIXED REALITY
Mark Billinghurst
mark.billinghurst@auckland.ac.nz
October 19th 2021

2. 1967 – IBM 1401 – half of the computers in the world, $10,000/month to run

3. Sketchpad (1963)
• Ivan Sutherland
• First interactive graphics
• Pen based input

4. The ultimate display would, of course, be a room within
which the computer can control the existence of matter. A
chair displayed in such a room would be good enough to sit
in. Handcuffs displayed in such a room would be confining,
and a bullet displayed in such a room would be fatal..
Sutherland, Ivan. "The ultimate display." (1965).

5. Sutherland Display (1968)
https://www.youtube.com/watch?v=NtwZXGprxag

6. Star Trek – HoloDeck (1974)

7. Star Wars – Hologram (1977)

8. My First VR Experience - 1990
• Silicon Graphics Reality Engine
• 500,000 polygons/second
• VPL Eyephone HMD
• 320 x 240 resolution
• Magnetic tracking
• Glove input
• Expensive - $250,000+

9. Star Wars – Collaborative AR

10. 1999 – Shared Space Demo
• Face to face collaborative AR like Star Wars concept

11. Shared Space Demo

12. Marker Based Tracking: ARToolKit (1999)
https://github.com/artoolkit

13. CPU: 300 Mhz
HDD; 9GB
RAM: 512 mb
Camera: VGA 30fps
Graphics: 500K poly/sec
1998: SGI O2 2008: Nokia N95
CPU: 332 Mhz
HDD; 8GB
RAM: 128 mb
Camera: VGA 30 fps
Graphics: 2m poly/sec
By 2008 phones had the same hardware as used in Shared Space demo

14. 2005: Mobile AR version of Shared Space
• AR Tennis
• Shared AR content
• Two user game
• Audio + haptic feedback
• Bluetooth networking

15. ARTennis Demo

16. VR and AR Today
• Large growing market
• > $25 Billion USD in 2020
• Hundreds of millions of users
• Many available devices
• HMD, phones, tablets, HUDs
• Robust developer tools
• Vuforia, MRTK, Unity, etc
• Large number of applications
• > 150K developers, > 100K apps
• Strong research/business communities
• ISMAR, IEEE VR, AWE conferences, AugmentedReality.org, etc

17. https://www.youtube.com/watch?v=aUPMDwypBkA

18. https://www.youtube.com/watch?v=xRSF31dbLBU

20. Future Visions of VR: Ready Player One
• https://www.youtube.com/watch?v=LiK2fhOY0nE

21. Today vs. Tomorrow
VR in 2021 VR in 2045
Graphics High quality Photo-realistic
Display 110-150 degrees Total immersion
Interaction Handheld controller/some gesture Full gesture/body/gaze
Navigation Limited movement Natural
Multiuser Few users Millions of users

22. https://www.youtube.com/watch?v=gg-ZakMEwDU

24. “.. the technologies that will significantly
affect our lives over the next 10 years
have been around for a decade. The
future is with us ... The trick is learning
how to spot it”
October 2004
Bill Buxton

25. Key Technologies for MR Systems
• Display
• Stimulate visual, hearing/touch sense
• Tracking
• Changing viewpoint, registered content
• Interaction
• Supporting user input

26. DISPLAY

27. • Past
• Bulky Head mounted displays
• Current
• Handheld, lightweight head mounted
• Future
• Projected AR
• Wide FOV see through
• Retinal displays
• Contact lens
Evolution in Displays

28. North Focals (2020)
• https://www.bynorth.com
• Socially acceptable smart glasses
• $599 USD, small field of view
• Ring input device

29. Wide FOV See-Through (3+ years)
• Waveguide techniques
• Wider FOV
• Thin see through
• Socially acceptable
• Pinlight Displays
• LCD panel + point light sources
• 110 degree FOV
• UNC/Nvidia
Lumus DK40
Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., & Fuchs, H. (2014). Pinlight displays:
wide field of view augmented reality eyeglasses using defocused point light sources. In ACM SIGGRAPH
2014 Emerging Technologies (p. 20). ACM.

30. https://www.youtube.com/watch?v=P407DFm0PFQ
Pinlight Display Demo

31. Kura Gallium Glasses (2020)
• https://www.kura.tech/
• Pinpoint wide field of view effect - 150° FOV;
• Resolution in the range 4K-8K;
• Very transparent screen

32. Contact Lens (15 + years)
• Contact Lens only
• Unobtrusive
• Significant technical challenges
• Power, data, resolution
• Babak Parviz (2008)
• MojoVision
• Mojo Lens
• Prototype smart contact lens
• https://www.mojo.vision/
http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-a-contact-lens/

34. https://www.youtube.com/watch?v=TzVAMRe3kmA

35. INTERACTION

36. Evolution of Interaction
• Past
• Limited interaction
• Viewpoint manipulation
• Present
• Screen based, simple gesture
• tangible interaction
• Future
• Natural gesture, Multimodal
• Intelligent Interfaces
• Physiological/Sensor based

37. Natural Gesture
• Freehand gesture input
• Depth sensors for gesture capture
• Move beyond simple pointing
• Rich two handed gestures
• Eg Microsoft Research Hand Tracker
• 3D hand tracking, 30 fps, single sensor
• Commercial Systems
• Hololens2, Oculus, Intel, MagicLeap, etc
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D. K. C. R. I., ... & Izadi, S.
(2015, April). Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. CHI (Vol. 8).

38. https://www.youtube.com/watch?v=LblxKvbfEoo

39. Multi-Scale Gesture
• Combine different gesture types
• In-air gestures – natural but imprecise
• Micro-gesture – fine scale gestures
• Gross motion + fine tuning interaction
Ens, B., Quigley, A., Yeo, H. S., Irani, P., Piumsomboon, T., & Billinghurst, M. (2018). Counterpoint:
Exploring Mixed-Scale Gesture Interaction for AR Applications. In Extended Abstracts of the 2018 CHI
Conference on Human Factors in Computing Systems (p. LBW120). ACM.

40. https://www.youtube.com/watch?v=TRfqNtt1VxY&t=23s

41. Eye Tracking Input
• HMDs with integrated eye-tracking
• Hololens2, MagicLeap One
• Research questions
• How can eye gaze be used for interaction?
• What interaction metaphors are natural?

42. Eye Gaze Interaction Methods
• Gaze for interaction
• Implicit vs. explicit input
• Exploring different gaze interaction
• Duo reticles – use eye saccade input
• Hardware
• HTC Vive + Pupil Labs integrated eye-tracking
Piumsomboon, T., Lee, G., Lindeman, R. W., & Billinghurst, M. (2017, March). Exploring natural eye-gaze-based
interaction for immersive virtual reality. In 3D User Interfaces (3DUI), 2017 IEEE Symposium on (pp. 36-39). IEEE.

43. Duo-Reticles (DR)
Inertial Reticle (IR)
Real-time Reticle (RR) or Eye-gaze Reticle (original name)
A-1
As RR and IR are aligned,
alignment time counts down
A-2 A-3
Selection completed

44. Duo-Reticles (DR) – Video 1

45. Physiological Sensor Input
• Using physiological sensors for implicit input
• Systems that recognize user intent/activity
• EEG
• Measuring brain activity
• EMG
• Measuring muscle activity

46. https://www.youtube.com/watch?v=K34p7RwjWt0

47. HP Reverb G2 Omnicept
• Wide FOV, high resolution, best in class VR display
• Eye tracking, heart rate, pupillometry, and face camera

48. NextMind
• EEG attachment for AR/VR HMD
• 9 dry EEG electrodes
• https://www.next-mind.com/

49. https://www.youtube.com/watch?v=yfzDcfQpdp0

50. Multimodal Input
• Combine gesture and speech input
• Gesture good for qualitative input
• Speech good for quantitative input
• Support combined commands
• “Put that there” + pointing
• E.g. HIT Lab NZ multimodal input
• 3D hand tracking, speech
• Multimodal fusion module
• Complete tasks faster with MMI, less errors
Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in Space: Gesture Interaction with
Augmented-Reality Interfaces. IEEE computer graphics and applications, (1), 77-80.

51. Intelligent Interfaces
• Move to Implicit Input vs. Explicit
• Recognize user behaviour
• Provide adaptive feedback
• Move beyond check-lists of actions
• E.g. AR + Intelligent Tutoring
• Constraint based ITS + AR
• PC Assembly (Westerfield, 2015)
• 30% faster, 25% better retention
Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent Augmented Reality Training for
Motherboard Assembly. International Journal of Artificial Intelligence in Education, 25(1), 157-172.

52. https://www.youtube.com/watch?v=vifHh4WjEFE

53. TRACKING

54. Evolution of Tracking
• Past
• Location based, marker based,
• magnetic/mechanical
• Present
• Image based, hybrid tracking
• Future
• Ubiquitous
• Model based
• Environmental

55. Model Based Tracking
• Track from known 3D model
• Use depth + colour information
• Match input to model template
• Use CAD model of targets
• Recent innovations
• Learn models online
• Tracking from cluttered scene
• Track from deformable objects
Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., & Navab, N. (2013). Model based training, detection
and pose estimation of texture-less 3D objects in heavily cluttered scenes. In Computer Vision–ACCV 2012 (pp. 548-562).

56. Deformable Object Tracking
https://www.youtube.com/watch?v=t2vqsitWLKs

57. Environmental Tracking (3+ yrs)
• Environment capture
• Use depth sensors to capture scene & track from model
• InifinitAM (www.robots.ox.ac.uk/~victor/infinitam/)
• Real time scene capture, dense or sparse capture, open source
• iPad Pro LiDAR
• Scene scanning up to 5m

58. https://www.youtube.com/watch?v=qZY6y1IVIfw

59. Fusion4D (2016)
• Real capture and dynamic reconstruction
• RGBD sensors + incremental reconstruction
Dou, M., Khamis, S., Degtyarev, Y., Davidson, P., Fanello, S. R., Kowdle, A., ... & Izadi, S. (2016). Fusion4d:
Real-time performance capture of challenging scenes. ACM Transactions on Graphics (TOG), 35(4), 1-13.

60. Fusion4D Demo
https://www.youtube.com/watch?v=rnz0Kt36mOQ

61. Wide Area Outdoor Tracking
• Process
• Combine panoramas into point cloud model (offline)
• Initialize camera tracking from point cloud
• Update pose by aligning camera image to point cloud
• Accurate to 25 cm, 0.5 degree over very wide area
Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking. In Mixed
and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.

62. Wide Area Outdoor Tracking
https://www.youtube.com/watch?v=8ZNN0NeXV6s

63. AR Cloud Based Tracking
• AR Cloud
• a machine-readable 1:1 scale model of the real world
• processing recognition/tracking data in the cloud
• Can create cloud from input from multiple devices
• Store key visual features in cloud, Stitch features from multiple devices
• Retrieve for tracking/interaction
• AR Cloud Companies
• 6D.ai, Vertical.ai, Ubiquity6, etc

64. 6D.ai Demo
• https://www.youtube.com/watch?v=AwwU14gllS0

65. PERCEPTION AND
NEUROSCIENCE

66. AR/VR as Perceptual Phenomena
• Virtual Reality
• Do I perceive myself as being in the Virtual Environment?
• Sense of Presence
• Augmented Reality
• Is that virtual object part of my real world?
• Sense of Object Presence
• What perceptual cues create a sense of Presence/Object Presence?
• How can Presence/Object Presence be measured?

67. Measuring Presence
• Presence is very subjective so how to measure it ?
• Subjective Measures
• Self report questionnaire
• University College London Questionnaire (Slater 1999)
• Witmer and Singer Presence Questionnaire (Witmer 1998)
• ITC Sense Of Presence Inventory (Lessiter 2000)
• Continuous measure
• Person moves slider bar in VE depending on Presence felt
• Objective Measures
• Behavioural
• reflex/flinch measure, startle response
• Physiological measures
• change in heart rate, skin conductance, skin temperature
Presence Slider

68. Using Neuro-Physiological Presence Measures
• Put people in High Presence/Low Presence VR
• Measure physiological cues
• EEG, ECG, GSR
• Measure subjective cues
• Presence surveys
• SUS, Witmer-Singer
• Correlate subjective and physiological results
Dey, A., Phoon, J., Saha, S., Dobbins, C., & Billinghurst, M. (2020, November). A Neurophysiological Approach for
Measuring Presence in Immersive Virtual Environments. In 2020 IEEE International Symposium on Mixed and Augmented
Reality (ISMAR) (pp. 474-485). IEEE.

69. Results
• Significant difference in subjective presence scores between HP/LP VE
• Significant difference in EEG power and heart rate between HP/LP VE
EEG Power Heart Rate

70. Perception Based Graphics
• Eye Physiology
• Rods in eye centre = colour vision, cones in periphery = motion, B+W
• Foveated Rendering
• Use eye tracking to draw highest resolution where user looking
• Reduces graphics throughput

71. Foveated Rendering
• https://www.youtube.com/watch?v=lNX0wCdD2LA

72. Making AR Content Appear Real
• Appearance
• Lighting
• Audio cues
• Occlusion
• Touch/Haptic feedback
• Etc..

73. Key Perceptual Issues in AR
• Classification of Perceptual Issues
• Environment, Capturing, Augmentation
• Display device, User
Kruijff, E., Swan, J. E., & Feiner, S. (2010, October). Perceptual issues in augmented reality revisited.
In 2010 IEEE International Symposium on Mixed and Augmented Reality (pp. 3-12). IEEE.

74. Example Perceptual Issues
• zxcZ

75. SOCIAL AND ETHICAL ISSUES

76. Social Acceptance
• People don’t want to look silly
• Only 12% of 4,600 adults would be willing to wear AR glasses
• 20% of mobile AR browser users experience social issues
• Acceptance more due to Social than Technical issues
• Needs further study (ethnographic, field tests, longitudinal)

77. TAT AugmentedID
https://www.youtube.com/watch?v=tb0pMeg1UN0

80. Ethical Issues
• Persuasive Technology
• Affecting emotions
• Behaviour modification
• Privacy Concerns
• Facial recognition
• Space capture
• Personal data
• Safety Concerns
• Sim sickness, Distraction
• Long term effects
Pase, S. (2012). Ethical considerations in augmented reality applications. In Proceedings of the International Conference on
e-Learning, e-Business, Enterprise Information Systems, and e-Government (EEE) (p. 1). The Steering Committee of The
World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp).

81. COLLABORATION

82. Using AR/VR for Enhanced Collaboration
• Changing perspective, Sharing views
• Copying spaces, Changing scale
• Copy bodies, Sharing non-verbal cues

83. AR for Remote Collaboration
• Camera + Processing + AR Display + Connectivity
• First person Ego-Vision Collaboration

84. AR View Remote Expert View

85. Shared Sphere – 360 Video Sharing
Shared
Live 360 Video
Host User Guest User
Lee, G. A., Teo, T., Kim, S., & Billinghurst, M. (2017). Mixed reality collaboration through sharing a
live panorama. In SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications (pp. 1-4).

86. https://www.youtube.com/watch?v=FFF6qP5Ap44

87. 3D Live Scene Capture
• Use cluster of RGBD sensors
• Fuse together 3D point cloud

89. View Sharing Evolution
• Increased immersion
• Improved scene understanding
• Better collaboration
2D 360 3D

90. Sharing Virtual Communication Cues
• Collaboration between AR and VR
• Gaze Visualization Conditions
• Baseline, FoV, Head-gaze, Eye-gaze

91. https://www.youtube.com/watch?v=K_afCWZtExk

92. Multi-Scale Collaboration
• Changing the user’s virtual body scale

94. Sharing a View

95. Sharing: Separating Cues from Body
• What happens when you can’t see your colleague/agent?
Piumsomboon, T., Lee, G. A., Hart, J. D., Ens, B., Lindeman, R. W., Thomas, B. H., & Billinghurst, M. (2018, April). Mini-me: An adaptive
avatar for mixed reality remote collaboration. In Proceedings of the 2018 CHI conference on human factors in computing systems (pp. 1-13).
Collaborating Collaborator out of View

96. Mini-Me Communication Cues in MR
• When lose sight of collaborator a Mini-Me avatar appears
• Miniature avatar in real world
• Mini-Me points to shared objects, show communication cues
• Redirected gaze, gestures

97. https://www.youtube.com/watch?v=YrdCg8zz57E

98. Scaling Up
• Supporting Large Groups of People
• Social VR spaces
• Large scale events
• Hybrid Interfaces
• AR/VR users with desktop/mobile
• Persistent virtual worlds

99. Facebook Horizon/Workrooms
• Key features
• Avatar creation, multi-user social VR
• Large scale 3D virtual spaces
• Meeting support, using real devices
• In world content creation/authoring

100. https://www.youtube.com/watch?v=Is8eXZco46Q

101. Hybrid Interfaces - Spatial
https://www.youtube.com/watch?v=PG3tQYlZ6JQ

102. Research Issues
• Avatars
• How to easily create?
• How realistic should they be?
• How can you communicate social cues?
• Hybrid Interfaces
• How can you provide equity across difference devices?
• Social Presence
• How can you objectively measure Social Presence?
• How use AR/VR cues to increase Social Presence?

103. Collaboration Technology Trends
1. Improved Content Capture
• Move from sharing faces to sharing places
2. Increased Network Bandwidth
• Sharing natural communication cues
3. Implicit Understanding
• Recognizing behaviour and emotion

104. Natural
Collaboration
Implicit
Understanding
Experienc
e Capture

105. Natural
Collaboration
Implicit
Understanding
Experienc
e Capture
Empathic
Computing

106. Empathic Computing
Can we develop systems
that allow us to share what
we are seeing, hearing and
feeling with others?
Piumsomboon, T., Lee, Y., Lee, G. A., Dey, A., & Billinghurst, M. (2017). Empathic Mixed
Reality: Sharing What You Feel and Interacting with What You See. In Ubiquitous Virtual
Reality (ISUVR), 2017 International Symposium on (pp. 38-41). IEEE.

107. Empathy Glasses (CHI 2016)
• Combine together eye-tracking, display, face expression
• Implicit cues – eye gaze, face expression
+
+
Pupil Labs Epson BT-200 AffectiveWear
Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of
the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.

108. Remote Collaboration
• Eye gaze pointer and remote pointing
• Face expression display
• Implicit cues for remote collaboration

109. https://www.youtube.com/watch?v=CdgWVDbMwp4

110. Brain Synchronization
• Measure EEG of people collaborating
• Brain activity synchronizes
• More synchronization = better collaboration

111. Brain Synchronization

112. Pre-training (Finger Pointing) Session Start

113. Post-Training (Finger Pointing) Session End

114. Brain Synchronization in VR

117. asfd

119. Technology Trends
• Advanced displays
• Wide FOV, high resolution
• Real time space capture
• 3D scanning, stitching, segmentation
• Natural gesture interaction
• Hand tracking, pose recognition
• Robust eye-tracking
• Gaze points, focus depth
• Emotion sensing/sharing
• Physiological sensing, emotion mapping

120. • Advanced displays
• Real time space capture
• Natural gesture interaction
• Robust eye-tracking
• Emotion sensing/sharing
Empathic
Tele-Existence

121. Empathic Tele-Existence
• Move from Observer to Participant
• Explicit to Implicit communication
• Experiential collaboration – doing together

122. The Metaverse
• Neal Stephenson’s “SnowCrash”
• VR successor to the internet
• The Metaverse is the convergence of:
• 1) virtually enhanced physical reality
• 2) physically persistent virtual space
• Metaverse Roadmap
• http://metaverseroadmap.org/

123. Metaverse Dimensions

124. CONCLUSIONS

125. Conclusions
• AR/VR/MR is becoming commonly available
• Significant advances over 50+ years
• In order to achieve Sutherland’s vision, need research in
• Display, Tracking, Input
• New MR technologies will enable this to happen
• Display devices, Interaction, Tracking technologies
• There are still significant areas for research
• Social Acceptance, Perception, Collaboration, Etc.

126. More Information
Billinghurst, M. (2021). Grand
Challenges for Augmented
Reality. Frontiers in Virtual Reality, 2, 12.

127. www.empathiccomputing.org
@marknb00
mark.billinghurst@auckland.ac.nz