Keynote speech given by Mark Billinghurst at the AWE Asia 2015 conference on October 18th 2015. The talk gives an outline of future developments in Augmented Reality

1. AUGMENTED REALITY:
THE NEXT 20 YEARS
Mark Billinghurst
mark.billinghurst@unisa.edu.au
October 18th 2015
AWE Asia 2015

2. 1977 – StarWars

3. Augmented Reality
1. Combines Real andVirtual Images
• Both can be seen at the same time
2. Interactive in real-time
• The virtual content can be interacted with
3. Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

4. 50 Years of Progress (1965-2015)
• Moving from lab to living room
• AR devices available in every pocket
1968: First AR HMD
1980’s: SuperCockpit
1997: Outdoor AR
2005: Mobile AR

5. Example: 1998 vs. 2008
CPU: 300 Mhz
HDD; 9GB
RAM: 512 mb
Camera: VGA 30fps
Graphics: 500K poly/sec
1998: SGI O2 2008: Nokia N95
CPU: 332 Mhz
HDD; 8GB
RAM: 128 mb
Camera: VGA 30 fps
Graphics: 2m poly/sec

6. AR in 2015
• Large growing market
• > $600 Million USD in 2014
• Many companies
• Many available devices
• HMD, phones, tablets, HUDs
• Robust developer tools
• Vuforia, ARToolKit, Unity, Wikitude, etc
• Large number of applications
• > 250K developers, > 200K mobile apps
• Strong research/business communities
• ISMAR, AWE conferences, AugmentedReality.org, etc

7. Looking to the Future
What’s Next?

8. Key Enabling Technologies
1. Combines Real andVirtual Images
Display Technology
2. Interactive in real-time
Interaction Technologies
3. Registered in 3D
Tracking Technologies

9. DISPLAY

10. • Past
• Bulky Head mounted displays
• Current
• Handheld, lightweight head mounted
• Future
• Projected AR
• Wide FOV see through
• Retinal displays
• Contact lens
Evolution in Displays

11. Projected AR (1-3 years)
• Use stereo head mounted projectors
• Rollable retro-reflective sheet
• Wide FOV, shared interaction
• Eg CastAR (http://castar.com)
• $400 USD, available Q4 2015

12. CastAR Demo

13. Wide FOV See-Through (3+ years)
• Waveguide techniques
• Wider FOV
• Thin see through
• Socially acceptable
• Pinlight Displays
• LCD panel + point light sources
• 110 degree FOV
• UNC/Nvidia
Lumus DK40
Maimone, A., Lanman, D., Rathinavel, K., Keller, K., Luebke, D., & Fuchs, H. (2014). Pinlight displays: wide
field of view augmented reality eyeglasses using defocused point light sources. In ACM SIGGRAPH 2014
Emerging Technologies (p. 20). ACM.

14. Pinlight Display Demo

15. Retinal Displays (5+ years)
• Photons scanned into eye
• Infinite depth of field
• Bright outdoor performance
• Overcome visual defects
• True 3D stereo with depth modulation
• Microvision (1993-)
• Head mounted monochrome
• MagicLeap (2013-)
• Projecting light field into eye

16. Contact Lens (10 – 15 + years)
• Contact Lens only
• Unobtrusive
• Significant technical challenges
• Power, data, resolution
• Babak Parviz (2008)
• Contact Lens + Micro-display
• Wide FOV
• socially acceptable
• Innovega (innovega-inc.com)
http://spectrum.ieee.org/biomedical/bionics/augmented-reality-in-a-contact-lens/

17. INTERACTION

18. Evolution of Interaction
• Past
• Limited interaction
• Viewpoint manipulation
• Present
• Screen based, simple gesture
• tangible interaction
• Future
• Natural gesture, Multimodal
• Intelligent Interfaces
• Physiological/Sensor based

19. Natural Gesture (2-5 years)
• Freehand gesture input
• Depth sensors for gesture capture
• Move beyond simple pointing
• Rich two handed gestures
• Eg Microsoft Research Hand Tracker
• 3D hand tracking, 30 fps, single sensor
• Commercial Systems
• Meta, MS Hololens, Occulus, Intel, etc
Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Leichter, D. K. C. R. I., ... & Izadi, S.
(2015, April). Accurate, Robust, and Flexible Real-time Hand Tracking. In Proc. CHI (Vol. 8).

20. Smart Glass Hand Interaction
• EnvisageAR + Phonevers
• RGB-D hand tracking on Android
• Natural gesture input for glasses

21. Multimodal Input (5+ years)
• Combine gesture and speech input
• Gesture good for qualitative input
• Speech good for quantitative input
• Support combined commands
• “Put that there” + pointing
• Eg HIT Lab NZ multimodal input
• 3D hand tracking, speech
• Multimodal fusion module
• Complete tasks faster with MMI, less errors
Billinghurst, M., Piumsomboon, T., & Bai, H. (2014). Hands in Space: Gesture Interaction with
Augmented-Reality Interfaces. IEEE computer graphics and applications, (1), 77-80.

22. HIT Lab NZ Multimodal Input

23. Intelligent Interfaces (10+ years)
• Move to Implicit Input vs. Explicit
• Recognize user behaviour
• Provide adaptive feedback
• Support scaffolded learning
• Move beyond check-lists of actions
• Eg AR + Intelligent Tutoring
• Constraint based ITS + AR
• PC Assembly (Westerfield (2015)
• 30% faster, 25% better retention
Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent Augmented Reality Training for
Motherboard Assembly. International Journal of Artificial Intelligence in Education, 25(1), 157-172.

24. TRACKING

25. Evolution of Tracking
• Past
• Location based, marker based,
• magnetic/mechanical
• Present
• Image based, hybrid tracking
• Future
• Ubiquitous
• Model based
• Environmental

26. Model Based Tracking (1-3 yrs)
• Track from known 3D model
• Use depth + colour information
• Match input to model template
• Use CAD model of targets
• Recent innovations
• Learn models online
• Tracking from cluttered scene
• Track from deformable objects
Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., & Navab, N. (2013).
Model based training, detection and pose estimation of texture-less 3D objects in heavily
cluttered scenes. In Computer Vision–ACCV 2012 (pp. 548-562). Springer Berlin Heidelberg.

27. Deformable Object Tracking

28. Environmental Tracking (3+ yrs)
• Environment capture
• Use depth sensors to capture scene & track from model
• InifinitAM (www.robots.ox.ac.uk/~victor/infinitam/)
• Real time scene capture on mobiles, dense or sparse capture
• Dynamic memory swapping allows large environment capture
• Cross platform, open source library available

29. InfinitAM Demo

30. Wide Area Outdoor Tracking (5+ yrs)
• Process
• Combine panorama’s into point cloud model (offline)
• Initialize camera tracking from point cloud
• Update pose by aligning camera image to point cloud
• Accurate to 25 cm, 0.5 degree over very wide area
Ventura, J., & Hollerer, T. (2012). Wide-area scene mapping for mobile visual tracking. In Mixed
and Augmented Reality (ISMAR), 2012 IEEE International Symposium on (pp. 3-12). IEEE.

31. Wide Area Outdoor Tracking

32. ENHANCED
EXPERIENCE

33. AR Business Today
• Around $600 Million USD in 2014 (>$2B 2015)
• > 80% Games and Marketing applications

34. Market Projections
cf. 2014 computer game market = $84 Billion USD

35. Gartner Hype Cycle 2015
http://www.gartner.com/newsroom/id/3114217

36. Crossing the Chasm - 5-10 years
http://www.gartner.com/newsroom/id/3114217

37. Getting from Here to There
• New markets
• Medical
• Education
• Industry
• Etc
• New applications enabled
• Training
• Collaboration
• Information Presentation
• Etc

38. Example: Commercial Systems
• Ngrain
• http://www.ngrain.com/
• Training authoring tool
• Model based AR tracking
• ScopeAR
• http://www.scopear.com/
• Remote assistance
• Image based tracking

39. Ngrain Example

40. Research Needed in Many Areas
• Social Acceptance
• Overcome social problems with AR
• Cloud Services
• Cloud based storage/processing
• AR Authoring Tools
• Easy content creation for non-experts
• Collaborative Experiences
• AR teleconferencing
• Etc..

41. Example: Social Panoramas
• Google Glass
• Capture live image panorama (compass + camera)
• Remote device (tablet)
• Immersive viewing, live annotation
Reichherzer, C., Nassani, A., & Billinghurst, M. (2014). Social panoramas using wearable
computers. In Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on
(pp. 303-304). IEEE.

42. Social Panorama Demo

43. Example:SocialAcceptance
• People don’t want to look silly
• Only 12% of 4,600 adults would be willing to wear AR glasses
• 20% of mobile AR browser users experience social issues
• Acceptance more due to Social than Technical issues
• Needs further study (ethnographic, field tests, longitudinal)

44. CONCLUSIONS

45. Conclusions
• AR is becoming commonly available
• In order to achieve significant growth AR needs to
• Expand into new markets
• Move onto new platforms
• Create new types of applications
• New AR technologies will enable this to happen
• Display, Interaction, Tracking technologies
• However there are still significant areas for research
• Social Acceptance, Cloud Services, AR authoring tools, Etc

46. www.empathiccomputing.org
@marknb00
mark.billinghurst@unisa.edu.au