This document discusses various techniques for prototyping augmented reality interfaces, including sketching, storyboarding, wireframing, mockups, and video prototyping. Low-fidelity techniques like sketching and paper prototyping allow for rapid iteration and exploring interactions at early stages. Higher-fidelity techniques like interactive mockups and video prototypes communicate the look and feel of the final product and allow for user testing. A variety of tools are presented for different stages of prototyping, from sketching and interactive modeling in VR, to scene assembly using drag-and-drop tools, to final mockups using design software. Case studies demonstrate applying these techniques from initial concepts through to higher-fidelity prototypes. Overall the document
4. Different Types of AR Interaction
• Browsing Interfaces
• simple (conceptually!), unobtrusive
• 3D AR Interfaces
• expressive, creative, require attention
• Tangible Interfaces
• Embedded into conventional environments
• Tangible AR
• Combines TUI input + AR display
5. 1. AR Interfaces as Data Browsers
• 2D/3D virtual objects are
registered in 3D
• “VR in Real World”
• Interaction
• 2D/3D virtual viewpoint control
• Applications
• Visualization, training
6. Example: Google Maps AR Mode
• AR Navigation Aid
• GPS + compass, 2D/3D object placement
7. 2. 3D AR Interfaces
• Virtual objects displayed in 3D
physical space and manipulated
• HMDs and 6DOF head-tracking
• 6DOF hand trackers for input
• Interaction
• Viewpoint control
• Traditional 3D user interface
interaction: manipulation, selection,
etc.
Kiyokawa, et al. 2000
8. 3. Augmented Surfaces and Tangible Interfaces
• Basic principles
• Virtual images are projected
on a surface
• Physical objects are used as
controls for virtual objects
• Support for collaboration
Wellner, P. (1993). Interacting with paper on the
DigitalDesk. Communications of the ACM, 36(7), 87-96.
9. Tangible User Interfaces (Ishii 97)
• Create digital shadows
for physical objects
• Foreground
• graspable UI
• Background
• ambient interfaces
10. i/O Brush (Ryokai, Marti, Ishii) - 2004
Ryokai, K., Marti, S., & Ishii, H. (2004, April). I/O brush: drawing with everyday objects as ink.
In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 303-310).
11. Lessons from Tangible Interfaces
• Physical objects make us smart
• Norman’s “Things that Make Us Smart”
• encode affordances, constraints
• Objects aid collaboration
• establish shared meaning
• Objects increase understanding
• serve as cognitive artifacts
12. But There are TUI Limitations
• Difficult to change object properties
• can’t tell state of digital data
• Limited display capabilities
• projection screen = 2D
• dependent on physical display surface
• Separation between object and display
• ARgroove – Interact on table, look at screen
13. Orthogonal Nature of Interfaces
3D AR interfaces Tangible Interfaces
Spatial Gap No – Interaction is
Everywhere
Yes – Interaction is
only on 2D surfaces
Interaction Gap
Yes – separate
devices for physical
and virtual objects
No – same devices for
physical and virtual
objects
14. 4. Tangible AR: Back to the Real World
• AR overcomes display limitation of TUIs
• enhance display possibilities
• merge task/display space
• provide public and private views
• TUI + AR = Tangible AR
• Apply TUI methods to AR interface design
Billinghurst, M., Kato, H., & Poupyrev, I. (2008). Tangible augmented reality. ACM Siggraph Asia, 7(2), 1-10.
15. Tangible AR: Tiles (Space Multiplexed)
• Tiles semantics
• data tiles
• operation tiles
• Operation on tiles
• proximity
• spatial arrangements
• space-multiplexed
Poupyrev, I., Tan, D. S., Billinghurst, M., Kato, H., Regenbrecht, H., & Tetsutani, N. (2001,
July). Tiles: A Mixed Reality Authoring Interface. In Interact (Vol. 1, pp. 334-341).
16. Tangible AR: Time-multiplexed Interaction
• Use of natural physical object manipulations to control
virtual objects
• VOMAR Demo
• Catalog book:
• Turn over the page
• Paddle operation:
• Push, shake, incline, hit, scoop
Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., & Tachibana, K. (2000, October). Virtual object manipulation on a table-top AR
environment. In Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000) (pp. 111-119). IEEE.
17. 5. Natural AR Interfaces
• Goal:
• Interact with AR content the same
way we interact in the real world
• Using natural user input
• Body motion
• Gesture
• Gaze
• Speech
• Input recognition
• Nature gestures, gaze
• Multimodal input
FingARtips (2004)
Tinmith (2001)
18. Evolution of AR Interfaces
Tangible AR
Tangible input
AR overlay
Direct interaction
Natural AR
Freehand gesture
Speech, gaze
Tangible UI
Augmented surfaces
Object interaction
Familiar controllers
Indirect interaction
3D AR
3D UI
Dedicated
controllers
Custom devices
Browsing
Simple input
Viewpoint control
Expressiveness, Intuitiveness
20. Interaction Design
“Designing interactive products to support
people in their everyday and working lives”
Preece, J., (2002). Interaction Design
• Design of User Experience with Technology
21. Typical Interaction Design Cycle
Develop alternative prototypes/concepts and compare them, And iterate, iterate, iterate....
24. Tom Chi’s Prototyping Rules
1. Find the quickest path to experience
2. Doing is the best kind of thinking
3. Use materials that move at the speed of
thought to maximize your rate of learning
25. How can we quickly prototype
XR experiences with little or no
coding?
34. Advantages/Disadvantages
Prototype Advantages Disadvantages
Low-fidelity
prototype
- low developmental cost
- evaluate multiple
design concepts
- limited error checking
- navigational and flow
limitations
High-fidelity
prototype
- fully interactive
- look and feel of final
product
- clearly defined
navigational scheme
- more expensive to develop
- time consuming to build
- developers are reluctant to
change something they have
crafted for hours
35. From Sketch to Product
•Sketches:
- early ideation stages of design
•Prototypes:
- detailing the actual design
43. Buxton’s Key Attributes of Sketching
• Quick
• Work at speed of thought
• Timely
• Always available
• Disposable
• Inexpensive, little investment
• Plentiful
• Easy to iterate
• A catalyst
• Evokes conversations
48. AR Augmented Sketch
● Sketch on top of photos of real world – target workplaces
Luisa Vasquez - https://blog.prototypr.io/10-tips-for-rapid-prototyping-on-the-hololens-2-bda021e21743
52. Key Elements
1. Scenario: Storyboards are based on a scenario or a user story. The
persona or role that corresponds to that scenario is clearly specified
2. Visuals: Each step in the scenario is represented visually in a sequence.
The steps can be sketches, illustrations, or photos.
3. Captions: Each visual has a corresponding caption. The caption
describes the user’s actions, environment, emotional state, device, etc.
55. Low and High Fidelity Storyboards
Low fidelity
High fidelity
56. Storyboard Guidelines
Successful Storyboards:
§ Visually tell the story
§ Detail the concept, characters, setting, sequence, interactions
§ Reflect key ideas and POV
Describe the following:
§ Field-of-View: What surroundings can the viewer see?
§ Estimated POV: Where is viewer positioned in their surroundings?
§ Key Frames: What is the starting and ending points of the scene?
§ Interaction: How does the viewer interact in this scene?
57. Wireframes
It’s about
- Functional specs
- Navigation and interaction
- Functionality and layout
- How interface elements work together
- Defining the interaction flow/experience
Leaving room for the design to be created
68. Scene Assembly In AR
• Many tools for creating AR scenes
• Drag and drop your assets
• Develop on web, publish to mobile
• Examples
• Catchoom - CraftAR
• Blippar - Blipbuilder
• ARloopa - Arloopa studio
• Wikitude - Wikitude studio
• Zappar - ZapWorks Designer
69. CraftAR
•Web-based AR marker tracking
•Add 3D models, video, images to real print content
•Simple drag and drop interface
•Cloud based image recognition
•https://catchoom.com/augmented-reality-craftar/
73. Mock-up Guidelines
1. Generate final 2D/3D interface elements
2. Replace wireframe UI elements with high quality visuals
3. Use standard AR/VR UI elements
4. Simulate AR/VR views
5. Focus on visual/audio design
6. Collect feedback from target end-users
74. Sketch vs. Wireframe vs. Mock-up
Low Fidelity Low to Medium
Fidelity
Medium to High Fidelity
IDEATE FLOW VISUALIZE
80. AR Prototyping with Layers
● Separate world-stabilized and head stabilized
○ Draw world stabilized on background paper
○ Draw head stabilized on transparent plastic
● Simulate Field of View of AR HMD
Lauber, F., Böttcher, C., & Butz, A. (2014, September). Papar: Paper prototyping for augmented reality. In Adjunct Proceedings
of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (pp. 1-6).
FOV in Red
Head Stabilized - Foreground
World Stabilized - Background
81. Example: Mobile AR Prototyping
https://www.youtube.com/watch?v=Hrbflbct99o
96. Video Sketching
• Process
• Capture elements of real world
• Use series of still photos/sketches in a movie format.
• Act out using the product
• Benefits
• Demonstrates the product experience
• Discover where concept needs fleshing out.
• Communicate experience and interface
• You can use whatever tools you want, e.g. iMovie.
112. Project Goals
Make a mobile AR Navigational Aid
○ Easy to use, clear and understandable
○ Useful to visitors
○ Creating interaction between the visitor and the aid through the user interface
○ Engage the visitor
To ensure this…
○ It is necessary to understand the visitor of a navigational aid in Bunratty Folk Park
○ Identify visitor motives and goals while going through the Folk Park.
113. NeedsAnalysis
Understanding Technology and Related Work
• Literature
• Similar Technologies
• Electronic Tours in Museum Settings
• Interactions design in Outdoor Museums
Understanding the User over time
• Observations
• Interviews
114. Findings
• Most visitors do not use the map
• Most visitors have mobile phones
• Visitors want more information
• View the Folk Park at their own pace
• Information should be straight to the point
• Large social interaction within groups
Needs Analysis
115. Desired Functionality
• View Options
• CameraView
• MapView
• ListView
• Sub-Options
• Places
• Events
• Restaurants
• Augmented Reality Features
for navigation
• Text Information
• 3D Objects
• 3D Tour Guide
• 3D Placement of Buildings
117. ITERATIVE DESIGN PROCESS
Prototyping and UserTesting
• Low Fidelity Prototyping
• Sketches
• Paper Prototyping
• Post-It Prototyping
• PowerPoint Prototyping
• High Fidelity Prototyping
• Wikitude
118. INITIAL SKETCHES (Photoshop)
Pros:
• Good for idea generation
• Cheap
• Concepts seem feasible
Cons:
• Not great feedback gained
• Photoshop not fast enough for
making changes
120. POWERPOINT PROTOTYPING
Benefits
• Used for User Testing
• Interactive
• Functionalities work when following the
story of Scenario 1
• Quick
• Easy arrangement of slides
User Testing
• Participants found
• 15 minute sessions screen captured
• ‘Talk Allowed’ technique used
• Notes taken
• Post-Interview
121. WIKITUDE
• Popular augmented reality browser
for mobile devices
• Mapping
• Point of Interest abilities
• Multiplatform
• Shows the points of interest of
Bunratty Folk Park
• Markers can be selected in and an
information pop-up appears
122. VIDEO PROTOTYPE
¢ Flexible tool for capturing the use
of an interface
¢ Elaborate simulation of how the
navigational aid will work
¢ Does not need to be realistic in
every detail
¢ Gives a good idea of how the
finished system will work
126. XR Tools Landscape
Digital & Immersive Authoring
Proto.io, Tour Creator, ...
Tilt Brush, Blocks, Quill, …
Web-Based Development
THREE.js, Babylon.js, …
A-Frame, AR.js, …
Cross-Platform Development
Unity + SDKs
Unreal + SDKs
Native Development
Cardboard/Oculus/Vive/... SDK
Vuforia/ARCore/ARKit/… SDK
127. XR Tools Landscape
Digital & Immersive Authoring
Good for storyboarding but limited
support for interactions
Web-Based Development
Good for basic XR apps but often
interactions feel unfinished
Native Development
Good for full-fledged XR apps but
limited to a particular platform
Cross-Platform Development
Good for full-fledged XR apps but
usually high learning curve
130. Digital Authoring Tools for AR
Vuforia Studio
Lens Studio
• Support visual authoring of marker-
based and/or marker-less AR apps
• Provide default markers and support
for custom markers
• Typically enable AR previews
through emulator but need to deploy
to AR device for testing
131. Immersive Authoring Tools for AR
• Enable visual authoring of 3D
content in AR
• Make it possible to edit while
previewing AR experience in the
environment
• Provide basic support for interactive
behaviors
• Sometimes support export to
WebXR
Apple Reality Composer
Adobe Aero
132. Interactive Sketching
•Pop App
● Pop - https://marvelapp.com/pop
● Combining sketching and interactivity on mobiles
● Take pictures of sketches, link pictures together
135. Proto.io
• Web based prototyping tool
• Visual drag and drop interface
• Rich transitions
• Scroll, swipe, buttons, etc
• Deploy on device
• mobile, PC, browser
• Ideal for mobile interfaces
• iOS, Android template
• For low and high fidelity prototypes
139. CityViewAR Application
Goal: Create a mobile app that allowed
people to view immersive panoramas on
site, showing the 2011 damage
Key technology: Mobile phone, GPS, 360
panorama, map
User Experience: People walking the city
streets will be able to be immersed in the
earthquake damage from 10 years ago
140. CityViewAR Prototyping Process
1. Sketch Interaction Flow (Paper)
2. Create Wireframe (Powerpoint)
3. Sketch Screens (Paper)
4. Test Transitions (Pop)
5. Collect/Create Assets (Photoshop, various)
6. Create High Fidelity Prototype (Proto.io)
7. Code (Unity, Mapbox)
8. Make video demo (Various)
9. Publish (Android, iOS)
154. AR Visual Programming
• Rapid prototype on desktop
• Deliver on mobile
• Simple interactivity
• Examples
• Zapworks Studio
• https://zap.works/studio/
• Snap Lens Studio
• https://lensstudio.snapchat.com/
• Facebook Spark AR Studio
• https://sparkar.facebook.com/ar-studio/
156. Lens Studio
Author and preview AR prototypes
● Tool behind Snapchat Lenses, but
also a powerful AR prototyping tool
● Can do face (using front camera)
and world lenses (rear camera)
● Simulated previews using webcam
Deploy and use advanced AR features
● Can deploy to phone running
Snapchat app via Snapcode
● Has advanced AR tracking and
segmentation capabilities
Some specific features
● SnapML: train and use ML models
● Face landmarks: track 93 points
● Face expressions: detect winks etc.
● Eye tracking: get eye pos/rotation
● Behavior: set triggers & actions
● UI: add UI widgets without scripting
● Visual scripting: no code via graphs
● Templates: lots of templates and
tutorials (we’ll explore later)
157. Example: Museum AR Experience
A. Cardenas (2019): I Used Snapchat’s Lens Studio
to Prototype Augmented Reality Apps and This is
What I Learned
Original art piece by
Penelope Umbrico on
display at UMMA One of our
lenses
On-site observation &
Snapchat access statistics
159. Creating On Device
•Adobe Aero
•Create AR on mobile devices
•Touch based interaction and authoring
•Only iOS support for now
•https://www.adobe.com/nz/products/aero.html
161. Apple Reality Composer
• Rapidly create 3D scenes and AR experiences
• Creation on device (iPhone, iPad)
• Drag and drop interface
• Loading 2D/3D content
• Adding simple interactivity
• Anchor content in real world (AR view)
• Planes (vertical, horizontal), faces, images
168. Paths to Being an XR Developer
Web dev
background
Game dev
background
Mobile dev
background
“Easy”
“Hard”
WebXR
THREE.js AR.js
A-Frame ...
Unity /
Unreal
SteamVR
AR
Foundation
MRTK ...
Native
SDKs
Oculus
ARKit/
ARCore
VIVE ...
169. A-Frame
• Based on Three.js and WebGL
• New HTML tags for 3D scenes
• A-Frame Inspector (not editor)
• Asset management (img, video,
audio, & 3D models)
• ECS architecture with many open-
source components
• Cross-platform XR
170. Hello World in A-Frame
https://aframe.io/examples/showcase/helloworld/ https://aframe.io/docs/1.0.0/introduction/
171.
172.
173. Unity
• Started out as game engine
• Has integrated support for many
types of XR apps
• Powerful scene editor
• Asset management & store
• Basically all XR device vendors
provide Unity SDKs
174.
175.
176. Web vs. A-Frame
Structure & Content
HTML
Organization of page content
& hierarchy
Presentation
CSS
Definition of page
content presentation
Behavior
JavaScript
Specification of interactive
behavior
Web
A-Frame
Structure
Entities
Organization of 3D
scene hierarchy
Content & Presentation
Components
Definition of 3D scene
content & presentation
Behavior
Systems & Scripts
Specification of interactive
behavior
177. WebXR & A-Frame
Web
HTML
<div id><!--
… --></div>
CSS
#id { left: x; top:
y; z-index: z }
JS
function f () {
… }
content + structure presentation behavior
WebXR
navigator.xr
.requestSession
WebGL
<canvas class=“a-canvas” />
THREE.js
entity.object3D
A-Frame
<a-entity id position=“x y z”>
</a-entity>
178. WebXR
Enable XR applications on the web by
allowing pages to:
• Detect XR capabilities
• Query XR device capabilities
• Poll XR device and associated input
device state
• Display imagery on XR device at the
appropriate frame rate
Is XR mode supported?
Advertise XR mode to
user
AR VR
Enter XR mode is
triggered?
Request XR session
Render frame loop in
session
Exit XR mode is
triggered?
End XR session
↻
179. Unity vs. A-Frame
Unity is a game engine and XR dev
platform
● De facto standard for XR apps
● Increasingly built-in support
● Most “XR people” will ask you about
your Unity skills :-)
Support for all XR devices
● Basically all AR and VR device
vendors provide Unity SDKs
A-Frame is a declarative WebXR
framework
● Emerging XR app development
framework on top of THREE.js
● Good for novice XR designers with
web dev background
Support for most XR devices
● Full WebXR support in Firefox,
Chrome, & Oculus Browser
180. XR Platforms & Toolkits
AR
VR
A-Frame
SteamVR
MRTK
XR Interaction
Oculus
VIVE
Marker-based
AR.js
Vuforia
Marker-less
A-Frame
ARKit/ARCore
AR Foundation
MRTK
XR Interaction
202. Other Books
3D User Interfaces: Theory and Practice
- Joseph J. LaViola Jr.Ernst Kruijff, Ryan P.
McMahan, Doug Bowman, Ivan Poupyrev
Augmented Reality: Principles and Practice
- Dieter Schmalstieg, Tobias Hollerer
Creating Augmented and Virtual Realities: Theory
and Practice for Next-Generation Spatial
Computing
- Erin Pangilinan, Steve Lukas, Vasanth Mohan
203. Tools and Guidelines
The UX of VR
Low-fi prototyping tools for VR
3D design patterns for VR Virtual Reality Human Interface Guidelines
204. Useful Links
VR paper prototyping
https://blog.prototypr.io/vr-paper-prototyping-9e1cab6a75f3
Draw sketches for VR
https://medium.com/inborn-experience/vr-sketches-56599f99b357
Wireframing 3D UI
https://arvrjourney.com/lesson-3-4-story-flow-and-wireframe-3d-interactions-dc3c7d5cb79a