1.
Tutorial:
Building Mobile AR Applications using
the Outdoor AR Library
Gun A. Lee
Mark Billinghurst
The Human Interface Technology Laboratory New Zealand
University of Canterbury, New Zealand
21 Nov 2013 09:00-10:15

2.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Welcome
 Mark Billinghurst
 Director, HIT Lab NZ
 PhD, Univ. Washington
 AR, Interaction Design
 Gun Lee
 Post Doc, HIT Lab NZ
 PhD, POSTECH
 AR, Android, Google Glass

3.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Schedule







Introduction (Mark)
Outdoor AR History (Mark)
Design Guidelines (Mark)
Example Applications (Mark)
Building a Mobile Outdoor AR Application (Gun)
Research Directions (Mark)
Conclusions (Gun + Mark)

4.
Introduction
Mark Billinghurst

5.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
What is Augmented Reality?
 Definition [Azuma 97]
 Combines real and virtual images
 Is interactive in real-time
 Content registered in 3D
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

6.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Outdoor AR
 Using mobile/wearable
systems to overlay AR content
outdoors
 Technical Requirements
 Tracking (GPS, compass, etc)
 Display (handheld,
headmounted)
 Input devices
 Processing, networking

7.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Applications
 Tourism, Gaming, Architecture, Engineering,
Etc

8.
Outdoor AR History
Mark Billinghurst

9.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Evolution of Mobile AR
Camera phone
Camera phone
- Thin client AR
Wearable
Computers
Wearable AR
Handheld
AR Displays
Camera phone
- Self contained AR
PDAs
-Thin client AR
PDAs
-Self contained AR
1995
1997
2001
2003
2004

10.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
MIT Wearable Computing
(1996)

11.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Mobile AR: Touring Machine (1997)
 University of Columbia
 Feiner, MacIntyre, Höllerer, Webster
 Combines





See through head mounted display
GPS tracking
Orientation sensor
Backpack PC (custom)
Tablet input
Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine:
Prototyping 3D mobile augmented reality systems for exploring the urban
environment. Personal Technologies, 1(4), 208-217.

12.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
MARS View
 Virtual tags overlaid on the real world
 “Information in place”

13.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Backpack/Wearable AR
1997 Backpack AR




Feiner’s Touring Machine
AR Quake (Thomas)
Tinmith (Piekarski)
MCAR (Reitmayr)
 Bulky, HMD based

14.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Mobile AR - Hardware
RTK correction Antenna
GPS
Antenna
HMD
Controller
Example self-built working
solution with PCI-based 3D graphics
PCI 3D Graphics Board
Tracker
Controller
PC104 Sound Card
DC to DC
Converter
Wearable
Computer
CPU
PC104 PCMCIA
Battery
GPS
RTK
correction
Radio
Hard Drive
Serial
Ports
Columbia Touring Machine

15.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
HIT Lab NZ Wearable AR (2004)
 Highly accurate outdoor AR
tracking system
 GPS, Inertial, RTK system
 HMD
 First prototype
 Laptop based
 Video see-through HMD
 2-3 cm tracking accuracy

16.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Image Registration
AR Stakeout Application

17.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Wearable AR Video

18.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Location Aware Phones (2008)
Motorola Droid
Nokia Navigator

19.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Outdoor Information Overlay (2009)
 Mobile phone based
 Tag real world locations
 GPS + Compass input
 Overlay graphics data on live video
 Applications
 Travel guide, Advertising, etc
 Wikitude, Layar, Junaio, etc..
 Android based, Public API released

20.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Google Glass (2013)

21.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS

22.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
View Through Google Glass
Always available peripheral information display
Combining computing, communications and content capture

23.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
 Hardware
 CPU TI OMAP 4430 – 1.2 Ghz
 16 GB SanDisk Flash,1 GB Ram
 570mAh Battery
 Input
 5 mp camera, 720p recording, microphone
 GPS, InvenSense MPU-9150 inertial sensor
 Output
 Bone conducting speaker
 640x360 micro-projector display

24.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Competitors
 Vuzix M100
 $999, profession
 Recon Jet
 $600, more sensors, sports
 Opinvent
 500 Euro, multi-view mode
 Motorola Golden-i
 Rugged, remote assistance

25.
Design Guidelines
Mark Billinghurst

26.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
AR UI Design
 Consider your user
 Follow good HCI principles
 Adapt HCI guidelines for AR
 Design to device constraints
 Design for perception/info presentation
 Design for evaluation

27.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Consider Your User
 Consider context of user
 Physical, social, emotional, cognitive, etc
 Outdoor AR User






Probably Mobile
One hand interaction
Short application use
Need to be able to multitask
Use in outdoor environment
Enhance interaction with real world

28.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Adapting Existing Guidelines
 Mobile Phone AR
 Phone HCI Guidelines
 Mobile HCI Guidelines
 HMD Based AR
 3D User Interface Guidelines
 VR Interface Guidelines

29.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS





Applying Principles to Mobile
AR
Clean
Large Video View
Large Icons
Text Overlay
Feedback

30.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
AR vs. Non AR Design
Characteristics
Non-AR Interfaces
AR Interfaces
Object Graphics
Mainly 2D
Mainly 3D
Object Types
Mainly virtual objects
Both virtual and physical objects
Object behaviors
Mainly passive objects
Both passive and active objects
Communication
Mainly simple
Mainly complex
HCI methods
Mainly explicit
Both explicit and implicit
 Design Guidelines
 Design for 3D graphics + Interaction
 Consider elements of physical world
 Support implicit interaction

31.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Maps vs. Junaio
 Google Maps
 2D, mouse driven, text/image heavy, exocentric
 Junaio
 3D, location driven, simple graphics, egocentric

32.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Design to Device Constraints
 Understand the platforms and design for limitations
 Hardware, software platforms
 Eg Outdoor AR game with handheld






Use large screen icons
Consider screen reflectivity
Support one-hand interaction
Consider the viewing angle
Do not tire users out physically
Don’t require accurate tracking

33.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
AR as Perception Problem
 Goal of AR to fool human senses – create
illusion that real and virtual are merged
 Depth





Size
Occlusion
Shadows
Relative motion
Etc..

34.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Which Object is Closest?

35.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Depth Cues
Pictorial: visual cues
• Occlusion, texture, relative brightness
Kinetic: motion cues
• Relative motion parallax, motion perspective
Physiological: motion cues
• Convergence, accommodation
Binocular disparity
• Two different eye images

36.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Use the Following Depth Cues
 Movement parallax
 Icon/Object size (for close objects)
 Linear perspective
 To add side perspective bar.
 Overlapping
 Works if the objects are big enough
 Shades and shadows
 Depends on the available
computation

37.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Provide Perspective Cue
 Eg ground plane grid

38.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS

39.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Depth Perception

40.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Information Presentation
• Amount of information
• Clutter, complexity
• Representation of information
• Navigation cues, POI representation
• Placement of information
• Head, body, world stabilized
• View combination
• Multiple views

41.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Twitter 360




www.twitter-360.com
iPhone application
See geo-located tweets in real world
Twitter.com supports geo tagging

42.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Wikitude – www.mobilizy.com
Blah
Blah
Blah
Blah
Blah
Blah Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah Blah
Blah
Blah
Blah
Blah
Blah
Blah

43.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Information Filtering

44.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Information Filtering

45.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Outdoor AR: Limited FOV

46.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Possible solutions
 Overview + Detail
 spatial separation; two views
 Focus + Context
 merges both views into one view
 Zooming
 temporal separation

47.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Zooming Views
 Zooming panorama, Zooming Map
Mulloni, A., Dünser, A., & Schmalstieg, D. (2010, September). Zooming interfaces for
augmented reality browsers. In Proceedings of the 12th international conference on Human
computer interaction with mobile devices and services (pp. 161-170). ACM.

48.
Example Applications
Mark Billinghurst

49.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
HIT Lab NZ Building Viewer
 Architectural Application
 Loads 3D models
 a OBJ/MTL format
 Positions content in space
 GPS, compass
 Intuitive user interface
 toolkit to modify the model
 Targeting museum guide/outdoor site
applications

50.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
2011 Christchurch Earthquake

51.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Christchurch Today

52.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
CityViewAR
 Using AR to visualize Christchurch buildings
 3D buildings, 2D images, text, panoramas
 AR View, Map view, List view
 Available on Android market, iOS App Store

53.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS

54.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
User Experience
 While walking in the real world people can
see text, 2D images and 3D content

55.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
List View
List of all assets

56.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Map View
Icons for buildings, viewpoints,
panoramas

57.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Building History Data

58.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Photographic Images

59.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Panorama Images
360 degree photo bubbles

60.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Augmented Reality View

61.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS

62.
Building a Mobile Outdoor AR
Application
Gun Lee

63.
Research Directions
Mark Billinghurst

64.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Looking to the Future
What’s Next?

65.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
What’s Next
 Key Research Problems
 Wide area tracking
 Input Methods/Displays
 Collaboration
 Emerging Trends




AR + Social Networking
AR Standards/Platforms
AR + Human Computing
Scaling up

66.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Robust Outdoor Tracking
 Hybrid Tracking
 Computer Vision, GPS, inertial
 Going Out
 Reitmayer & Drummond (Univ. Cambridge)

67.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Handheld Display
Reitmayr, G., & Drummond, T. W. (2006, October). Going out: robust model-based
tracking for outdoor augmented reality. In Mixed and Augmented Reality, 2006. ISMAR
2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.

68.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Meta Gesture Interaction
 Depth sensor + Stereo see-through

69.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Meta Video

70.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Contact Lens Display
 Babak Parviz
 University Washington
 MEMS components
 Transparent elements
 Micro-sensors
 Challenges
 Miniaturization
 Assembly
 Eye-safe

71.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Contact Lens Prototype

72.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Ego-Vision Collaboration
 Google Glass
 camera + processing + display +
connectivity

73.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Ego-Vision Research
 System
 How do you capture the user's environment?
 How do you provide good quality of service?
 Interface
 What visual and audio cues provide best
experience?
 How do you interact with the remote user?
 Evaluation
 How do you measure the quality of
collaboration?

74.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Massive Multiuser
 Handheld/Outdoor AR for the first time allows
extremely high numbers of AR users
 Requires
 New types of applications/games
 New infrastructure (server/client/peer-to-peer)
 Content distribution…

75.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
PERSONAL VIEW

76.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Augmented Reality 2.0
Infrastructure

77.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Leveraging Web 2.0
 Content retrieval using HTTP
 XML encoded meta information
 KML placemarks + extensions
 Queries
 Based on location (from GPS, image recognition)
 Based on situation (barcode markers)
 Syndication
 Community servers for end-user content
 Tagging
 AR client subscribes to data feeds

78.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
AR Standards + Markup Languages






KHARMA and Argon
ARML
AREL
Patterns if Interest
X3D+
KML vs. RDF + Multimedia Markup
Languages

79.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS

80.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
AR + Human Computation
 Human Computation
 Real people solving problems
difficult for computers
 Web-based, non real time
 Little work on AR + HC
 AR attributes
 Shared point of view
 Real world overlay
 Location sensing
What does this say?

81.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Human Computation
Architecture
 Add AR front end to typical HC platform

82.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Scaling Up
 Seeing actions of millions of users in the
world
 Augmentation on city/country level

83.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
AR + Smart Sensors + Social Networks
 Track population at city scale (mobile
networks)
 Match population data to external sensor
data
 medical, environmental, etc

84.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Scaling Up
 AR on a City Scale
 Using mobile phone as ubiquitous sensor
 MIT Senseable City Lab
 http://senseable.mit.edu/

85.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
WikiCity Rome
http://senseable.mit.edu/wikicity/rome/

86.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Orange Data for Development
 Orange made available 2.5 billion phone
records
 5 months calls from Ivory Coast
 > 80 sample projects using data
 eg: Monitoring human mobility for disease

87.
Conclusions
Mark Billinghurst, Gun Lee

88.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Conclusions
 Outdoor AR hardware available
 Handhelds (GPS, compass, camera), Wearables
 Many possible applications
 HIT Lab NZ Outdoor AR platform
 Easy for building AR applications
 Multi-view support, Client/Server interface
 Cross platform (handheld, Glass, etc)

89.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
Next steps
 More tutorials on our website
 http://www.hitlabnz.org/mobileAR
 Tags, LocationEvents, etc.
 Projects on Google Glass
 http://arforglass.org
 One week workshop in Feb 2014
 Stay tuned for server components
 Web-based Authoring Tool closed beta (Jan 2014)

90.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
http://www.hitlabnz.org/mobileAR

91.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
http://www.arforglass.org/

92.
SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS
More Information
 Website
 http://www.hitlabnz.org/mobileAR
 http://arforglass.org
 Gun Lee
 gun.lee@hitlabnz.org
 Mark Billinghurst
 mark.billinghurst@hitlabnz.org