Building Mobile AR Applications Using the Outdoor AR Library (Part 1)

Tutorial:
Building Mobile AR Applications using
the Outdoor AR Library
Gun A. Lee

Mark Billinghurst

The Human Interface Technology Laboratory New Zealand
University of Canterbury, New Zealand
21 Nov 2013 09:00-10:15

SYMPOSIUM ON MOBILE GRAPHICS
AND INTERACTIVE APPLICATIONS

Welcome
 Mark Billinghurst
 Director, HIT Lab NZ
 PhD, Univ. Washington
 AR, Interaction Design

 Gun Lee
 Post Doc, HIT Lab NZ
 PhD, POSTECH
 AR, Android, Google Glass


Schedule








Introduction (Mark)
Outdoor AR History (Mark)
Design Guidelines (Mark)
Example Applications (Mark)
Building a Mobile Outdoor AR Application (Gun)
Research Directions (Mark)
Conclusions (Gun + Mark)

Introduction
Mark Billinghurst


What is Augmented Reality?
 Definition [Azuma 97]
 Combines real and virtual images
 Is interactive in real-time
 Content registered in 3D

Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.


Outdoor AR
 Using mobile/wearable
systems to overlay AR content
outdoors
 Technical Requirements
 Tracking (GPS, compass, etc)
 Display (handheld,
headmounted)
 Input devices
 Processing, networking


Applications

 Tourism, Gaming, Architecture, Engineering,
Etc

Outdoor AR History
Mark Billinghurst


Evolution of Mobile AR
Camera phone

Camera phone
- Thin client AR
Wearable
Computers

Wearable AR

Handheld
AR Displays

Camera phone
- Self contained AR
PDAs
-Thin client AR
PDAs
-Self contained AR

1995

1997

2001

2003

2004


MIT Wearable Computing
(1996)


Mobile AR: Touring Machine (1997)
 University of Columbia
 Feiner, MacIntyre, Höllerer, Webster

 Combines






See through head mounted display
GPS tracking
Orientation sensor
Backpack PC (custom)
Tablet input

Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine:
Prototyping 3D mobile augmented reality systems for exploring the urban
environment. Personal Technologies, 1(4), 208-217.


MARS View

 Virtual tags overlaid on the real world
 “Information in place”


Backpack/Wearable AR
1997 Backpack AR





Feiner’s Touring Machine
AR Quake (Thomas)
Tinmith (Piekarski)
MCAR (Reitmayr)

 Bulky, HMD based


Mobile AR - Hardware
RTK correction Antenna
GPS
Antenna

HMD
Controller

Example self-built working
solution with PCI-based 3D graphics
PCI 3D Graphics Board

Tracker
Controller
PC104 Sound Card
DC to DC
Converter

Wearable
Computer

CPU

PC104 PCMCIA

Battery
GPS

RTK
correction
Radio

Hard Drive

Serial
Ports

Columbia Touring Machine


HIT Lab NZ Wearable AR (2004)
 Highly accurate outdoor AR
tracking system
 GPS, Inertial, RTK system
 HMD

 First prototype
 Laptop based
 Video see-through HMD
 2-3 cm tracking accuracy


Image Registration

AR Stakeout Application


Wearable AR Video


Location Aware Phones (2008)

Motorola Droid

Nokia Navigator


Outdoor Information Overlay (2009)
 Mobile phone based
 Tag real world locations
 GPS + Compass input
 Overlay graphics data on live video

 Applications
 Travel guide, Advertising, etc

 Wikitude, Layar, Junaio, etc..
 Android based, Public API released


Google Glass (2013)


View Through Google Glass

Always available peripheral information display
Combining computing, communications and content capture


 Hardware
 CPU TI OMAP 4430 – 1.2 Ghz
 16 GB SanDisk Flash,1 GB Ram
 570mAh Battery

 Input
 5 mp camera, 720p recording, microphone
 GPS, InvenSense MPU-9150 inertial sensor

 Output
 Bone conducting speaker
 640x360 micro-projector display


Competitors
 Vuzix M100
 $999, profession

 Recon Jet
 $600, more sensors, sports

 Opinvent
 500 Euro, multi-view mode

 Motorola Golden-i
 Rugged, remote assistance

Design Guidelines
Mark Billinghurst


AR UI Design
 Consider your user
 Follow good HCI principles
 Adapt HCI guidelines for AR
 Design to device constraints
 Design for perception/info presentation
 Design for evaluation


Consider Your User
 Consider context of user
 Physical, social, emotional, cognitive, etc

 Outdoor AR User







Probably Mobile
One hand interaction
Short application use
Need to be able to multitask
Use in outdoor environment
Enhance interaction with real world


Adapting Existing Guidelines
 Mobile Phone AR
 Phone HCI Guidelines
 Mobile HCI Guidelines

 HMD Based AR
 3D User Interface Guidelines
 VR Interface Guidelines








Applying Principles to Mobile
AR
Clean
Large Video View
Large Icons
Text Overlay
Feedback


AR vs. Non AR Design
Characteristics

Non-AR Interfaces

AR Interfaces

Object Graphics

Mainly 2D

Mainly 3D

Object Types

Mainly virtual objects

Both virtual and physical objects

Object behaviors

Mainly passive objects

Both passive and active objects

Communication

Mainly simple

Mainly complex

HCI methods

Mainly explicit

Both explicit and implicit

 Design Guidelines
 Design for 3D graphics + Interaction
 Consider elements of physical world
 Support implicit interaction


Maps vs. Junaio

 Google Maps
 2D, mouse driven, text/image heavy, exocentric

 Junaio
 3D, location driven, simple graphics, egocentric


Design to Device Constraints
 Understand the platforms and design for limitations
 Hardware, software platforms
 Eg Outdoor AR game with handheld







Use large screen icons
Consider screen reflectivity
Support one-hand interaction
Consider the viewing angle
Do not tire users out physically
Don’t require accurate tracking


AR as Perception Problem
 Goal of AR to fool human senses – create
illusion that real and virtual are merged
 Depth






Size
Occlusion
Shadows
Relative motion
Etc..


Which Object is Closest?


Depth Cues
Pictorial: visual cues
• Occlusion, texture, relative brightness

Kinetic: motion cues
• Relative motion parallax, motion perspective

Physiological: motion cues
• Convergence, accommodation

Binocular disparity
• Two different eye images


Use the Following Depth Cues
 Movement parallax
 Icon/Object size (for close objects)
 Linear perspective
 To add side perspective bar.

 Overlapping
 Works if the objects are big enough

 Shades and shadows
 Depends on the available
computation


Provide Perspective Cue

 Eg ground plane grid


Depth Perception


Information Presentation
• Amount of information
• Clutter, complexity

• Representation of information
• Navigation cues, POI representation

• Placement of information
• Head, body, world stabilized

• View combination
• Multiple views


Twitter 360






www.twitter-360.com
iPhone application
See geo-located tweets in real world
Twitter.com supports geo tagging


Wikitude – www.mobilizy.com
Blah

Blah
Blah

Blah
Blah

Blah Blah
Blah
Blah

Blah
Blah

Blah

Blah

Blah
Blah Blah
Blah
Blah

Blah
Blah

Blah

Blah


Information Filtering


Outdoor AR: Limited FOV


Possible solutions
 Overview + Detail
 spatial separation; two views

 Focus + Context
 merges both views into one view

 Zooming
 temporal separation


Zooming Views

 Zooming panorama, Zooming Map
Mulloni, A., Dünser, A., & Schmalstieg, D. (2010, September). Zooming interfaces for
augmented reality browsers. In Proceedings of the 12th international conference on Human
computer interaction with mobile devices and services (pp. 161-170). ACM.

Example Applications
Mark Billinghurst


HIT Lab NZ Building Viewer
 Architectural Application
 Loads 3D models
 a OBJ/MTL format

 Positions content in space
 GPS, compass

 Intuitive user interface
 toolkit to modify the model

 Targeting museum guide/outdoor site
applications


2011 Christchurch Earthquake


Christchurch Today


CityViewAR

 Using AR to visualize Christchurch buildings
 3D buildings, 2D images, text, panoramas
 AR View, Map view, List view
 Available on Android market, iOS App Store


User Experience

 While walking in the real world people can
see text, 2D images and 3D content


List View

List of all assets


Map View

Icons for buildings, viewpoints,
panoramas


Building History Data


Photographic Images


Panorama Images

360 degree photo bubbles


Augmented Reality View

Building a Mobile Outdoor AR
Application
Gun Lee

Research Directions
Mark Billinghurst


Looking to the Future

What’s Next?


What’s Next
 Key Research Problems
 Wide area tracking
 Input Methods/Displays
 Collaboration

 Emerging Trends





AR + Social Networking
AR Standards/Platforms
AR + Human Computing
Scaling up


Robust Outdoor Tracking

 Hybrid Tracking
 Computer Vision, GPS, inertial

 Going Out
 Reitmayer & Drummond (Univ. Cambridge)


Handheld Display

Reitmayr, G., & Drummond, T. W. (2006, October). Going out: robust model-based
tracking for outdoor augmented reality. In Mixed and Augmented Reality, 2006. ISMAR
2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.


Meta Gesture Interaction

 Depth sensor + Stereo see-through


Meta Video


Contact Lens Display
 Babak Parviz
 University Washington

 MEMS components
 Transparent elements
 Micro-sensors

 Challenges
 Miniaturization
 Assembly
 Eye-safe


Contact Lens Prototype


Ego-Vision Collaboration

 Google Glass
 camera + processing + display +
connectivity


Ego-Vision Research
 System
 How do you capture the user's environment?
 How do you provide good quality of service?

 Interface
 What visual and audio cues provide best
experience?
 How do you interact with the remote user?

 Evaluation
 How do you measure the quality of
collaboration?


Massive Multiuser
 Handheld/Outdoor AR for the first time allows
extremely high numbers of AR users
 Requires
 New types of applications/games
 New infrastructure (server/client/peer-to-peer)
 Content distribution…


PERSONAL VIEW


Augmented Reality 2.0
Infrastructure


Leveraging Web 2.0
 Content retrieval using HTTP
 XML encoded meta information
 KML placemarks + extensions

 Queries
 Based on location (from GPS, image recognition)
 Based on situation (barcode markers)

 Syndication
 Community servers for end-user content
 Tagging

 AR client subscribes to data feeds


AR Standards + Markup Languages







KHARMA and Argon
ARML
AREL
Patterns if Interest
X3D+
KML vs. RDF + Multimedia Markup
Languages


AR + Human Computation
 Human Computation
 Real people solving problems
difficult for computers

 Web-based, non real time
 Little work on AR + HC

 AR attributes
 Shared point of view
 Real world overlay
 Location sensing

What does this say?


Human Computation
Architecture

 Add AR front end to typical HC platform


Scaling Up

 Seeing actions of millions of users in the
world
 Augmentation on city/country level


AR + Smart Sensors + Social Networks

 Track population at city scale (mobile
networks)
 Match population data to external sensor
data
 medical, environmental, etc


Scaling Up

 AR on a City Scale
 Using mobile phone as ubiquitous sensor
 MIT Senseable City Lab
 http://senseable.mit.edu/


WikiCity Rome

http://senseable.mit.edu/wikicity/rome/


Orange Data for Development

 Orange made available 2.5 billion phone
records
 5 months calls from Ivory Coast

 > 80 sample projects using data
 eg: Monitoring human mobility for disease

Conclusions
Mark Billinghurst, Gun Lee


Conclusions
 Outdoor AR hardware available
 Handhelds (GPS, compass, camera), Wearables

 Many possible applications
 HIT Lab NZ Outdoor AR platform
 Easy for building AR applications
 Multi-view support, Client/Server interface
 Cross platform (handheld, Glass, etc)


Next steps
 More tutorials on our website
 http://www.hitlabnz.org/mobileAR
 Tags, LocationEvents, etc.

 Projects on Google Glass
 http://arforglass.org
 One week workshop in Feb 2014

 Stay tuned for server components
 Web-based Authoring Tool closed beta (Jan 2014)


http://www.hitlabnz.org/mobileAR


http://www.arforglass.org/


More Information
 Website
 http://www.hitlabnz.org/mobileAR
 http://arforglass.org

 Gun Lee
 gun.lee@hitlabnz.org

 Mark Billinghurst
 mark.billinghurst@hitlabnz.org

Building Mobile AR Applications Using the Outdoor AR Library (Part 1)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building Mobile AR Applications Using the Outdoor AR Library (Part 1)

Similar to Building Mobile AR Applications Using the Outdoor AR Library (Part 1) (20)

More from Mark Billinghurst

More from Mark Billinghurst (20)

Recently uploaded

Recently uploaded (20)

Building Mobile AR Applications Using the Outdoor AR Library (Part 1)

Editor's Notes