ISMAR 2014 AR Workshop: Designing Location Based Experiences

Location Based Experience
Design Workshop
Mark Melnykowycz
ISMAR 2014 Sept. 9th 2014
1

Story
Structure
Prototyping
User
Experience
Story Location
A/MR
Technology
Mark Melnykowycz mark@idezo.ch
Location
Hero’s
Journey
Changing
structures
Books,
movies,
ebooks,
games
IxD / UX
Fundamentals
of IxD
Consumption
Patterns
User Journey
Game
Mechanics
Metaio
SDK
Tracking /
Positioning
AR / MR
Teaching
Users How
to Play
Fundamental
differences
Technology
World
Building
Game
Space
Rules
2014 ISMAR Workshop:
Location based experience design
Ghosts of Venice
Mixed-reality film
Lost In Reality
Mobile storytelling
2

Location
Mark Melnykowycz mark@idezo.ch 3

Generic Location App Use Case
User goes to a
location
Content or
interaction is
triggered

Designing Location Experiences
Why is the
person in that
location?
How are they
interacting with
the location?
Why are they motivated
to engage in this
experience?
How are
experiences
planned?
What
technologies
are available?

Key Physical Elements
Location
Position /
Orientation
Interaction
Content

Sketching out user flows /
prototyping / product definition
Technology
trends
User motivations
/ immersion
Communication
patterns
Building interaction /
defining app world

Tech and Society Trends
IoT/M2M
Devices
Device Fragmentation Evolving Communication

Technology and Device
Trends

Connected Device Evolution
50 Billion!
By 2020*!
Location
Interaction
Connected City
Quantified User!
Quantified Self!
IoT!
Large
connected
ecosystem
http://share.cisco.com/internet-of-things.html

Location
Time
Heart Rate
Quantified User
Sensor Fusion User State

Defined UX in Closed Systems
Interaction Device
Interaction and consumption patterns are learned making the
transition to new devices easy

Fragmented UX Patterns
Distributed UX
Ubiquitous Computing
Defined UX
Space

Location as a Device Experience
How are users trained to interact with locations?
Maps

Determine orientation
in location using
navigation aids
Maps
Identify physical
anchor points to
define position
Use technology to
identify information
around you
Future actions based on
understanding where you are
and what you can explore

Communication Trends

Short Evolution of Communication
Face to Face
Copier
Printing
Writing
Web
Location Services
Email
Static HTML
Blogging
Instant Messaging
Fragmented Lines!
Push/Pull!
Transmedia
Social Networks
Telephone
Mobile SMS
MMS

Content Container Consumption Velocity
User Experience Design
The User
18
Consumption Concepts

Consuming Media
Content
The User
Information transferred between
containers and consumed by people

Physical Container
The User
A real, tangible thing used
to hold, and through which
content can be consumed
Has been designed, has
structure and fills up
physical space

The User
Virtual Container
A virtual thing used to
hold, and through which
content can be consumed
Has structure, fills up
space and can scale
Defined through graphic
design and UI engineering

Consumption
The User
Vocabulary
Content
Understand/Retain information
Make it part of your knowledge base
How fast is this?
Velocity

Consumption Patterns
Print Web Mobile
Landing Page Grid
Content Home
Linear
progression
Pull out bits
of information
Scan - look
for anchors
Save an idea
and move on

Push/Pull Consumption
Patterns
24

Digital Consumption
App/Webpage Social Media Video Augmented Reality
Push/Pull Curator
Audience

SnowFall
New York Times / http://www.nytimes.com/projects/2012/snow-fall/
Scroll page, easy to pull out bits of the story and be
immersed in graphics, story, characters.

Text, animations, videos, different
themes, consistent narrative
Pull-based consumption and
reading of content

Parallax, easy to explore
Pull consumption
Easy to design/prototype
https://www.scrollkit.com

Easy to integrate, drag/drop
images, embedded media (video/
audio/iframes)
Custom URL or publish on
scrollkit.com

Social Media Curation
Flow/steam Slideshow
More powerful than Tumblr, connect to and search
social media channels, present content in sequence

Connect to social media
channels
Arrange elements to
group together
Add text and links

Publish as landing page
on BrickFlow, share link to
social media channels,
remix
Embed ex ter nal l y as
iframe element, view as
slideshow experience

Connect to and search social
Add content, text, etc. media channels

Mixed Video/Interaction
Non-linear video
navigation
Clickable elements
Easy assembly
Combine multi-media
elements
View in video/slide format
Videos, audio, animations,
text, etc.

http://www.raptmedia.com/product
Add clickable elements to
videos, seamlessly jump
from one video to another.
Allow choices for jumping
between storylines/
elements

Interactive layer allows clicking/hyperlinks
Different video
clips can be
linked together
User can make
decisions at plot
points
Easy to create
alternative story
lines or allow
exploration

Connect to channels,
ability to upload
Arrange
elements
Share
www.zeega.com

Designing Experiences

Discovery High-Level Solution Product Solution
Empathy Define Ideation Prototype Test
Product/Experience Design Phases
User Needs
Focus / Flare
App Prototype
39
Experience Planning Interactivity Prototyping
Story/Location Technology

Story Consumption Container
What is the story,
the world it takes
place in, the
character, etc.
How will this story
world be consumed
by the audience or
users?
How will the
consumption be
realized, physically/
virtually designed, etc?
User Experience/Interaction
Know your User!
Wireframe/prototyping
Webpage/App
Video/ebook
Know your Audience!
Story boarding
Pre-Visualization

Creative Product
Creates stories,
art, movies,
etc.
Builds physical/
virtual products
User Experience/
Interaction
Find your
audience
The Audience The Users
Prototype/Test/Explore

How will the consumption
be realized, physically/
virtually designed, etc?
Wireframe/prototyping
Webpage
Video
ebook
Consumption Container
User Experience/Interaction
User Journey
Paper/Digital
Mockup
Interactive
Prototype
Final
Experience

How does the user get into your
story and engage in your
experience?
User Journey
User sees meme via
Facebook/Tumblr
Searches for
Zurich Dada
Game on
Google
Starts reading/
learning about
the Dada ghost
in Zurich
Need to have
good SEO base
on landing page
Need fast
consumption on
desktop/mobile
Parallax
page design
User is push/pull
fragmented
reader

Design representation of your app/container
to communicate and test the user experience
Paper/Digital
Mockup/Prototype
Communicate with
potential partners /
investors / users
Testing tool for fast
user feedback
before development
Communicate
between design /
development
Demo App Mockup Paper

Web/App Prototyping
Keynote
Powerpoint
OmniGraffle
GUIToolkits
AppCooker
BluePrint
iMockups
HotGloo
JetStrap
Stencils
Visio
Balsamiq
Axure
iRise MX
Photoshop
Illustrator
iPad
Desktop Online

Storytelling
46

Story
•Story, a recounting of a sequence of events*
•An account of imaginary or real people and events
told for entertainment: "an adventure story"**
Narrative
•A narrative (or story) is any account that presents
connected events***
•A spoken or written account of connected events; a
story: "a bare narrative of the details"**
*Wikipedia
**Google dictionary
***Oxford English Dictionary (online)

Emotions
Knowledge
Society
Story

Shared emotions help form the
fabric of society
...storytelling is a vehicle used to combine
emotions and information
...and communicate those feelings and information in a
structure which gives us comfort or instills feelings of fear,
optimism, etc.

Emotions
Memories
Connections
Structure

Storytelling gives emotional
context to our linear lives
51

Physical time is linear
Emotional memory is non-linear
Birth Death

Story
Connect current events to good feelings from the past
Hope in the future and our confidence to meet challenges
Play out emotions we can’t explore in reality

Die Hard
...makes me feel good about my chances of surviving a dramatic
terrorist attack
Die Hard, 20th Century Fox
...statistically it will probably never happen, but I’m sort of
mentally prepared if it does

What is traditional story structure?
Does it conflict with our fragmented communication
patterns?
55
Story Structure

Many different models of story
structure for different media
56

Linear Story Basics

Generic Story Ingredients
Villain Conflict

Challenge Transformation
Over-coming extreme odds...to succeed in
the end…

Linear Story Structure
http://www.musik-therapie.at/PederHill/Structure&Plot.htm
http://wandertones.deviantart.com/art/Standard-Story-Structure-178276496
60
Rising drama leading to an apex with essential
plot points and local challenges/resolutions

Joseph Campbell
“Campbell explores the
theory that important
myths from around the
world which have
survived for thousands
of years all share a
fundamental structure,
which Campbell called
the monomyth.”
https://en.wikipedia.org/wiki/The_Hero_with_a_Thousand_Faces

Joseph Campbell
THE HERO IS INTRODUCED IN HIS ORDINARY WORLD
THE CALL TO ADVENTURE
THE HERO IS RELUCTANT AT FIRST
THE HERO IS ENCOURAGED BY THE WISE OLD MAN OR WOMAN
THE HERO PASSES THE FIRST THRESHOLD
THE HERO ENCOUNTERS TESTS AND HELPERS
THE HERO REACHES THE INNERMOST CAVE
THE HERO ENDURES THE SUPREME ORDEAL
THE HERO SIEZES THE SWORD
THE ROAD BACK
RESURRECTION
RETURN WITH THE ELIXIR
http://www.cs.uu.nl/docs/vakken/b2go/docs/hero_journey.pdf

The Hero’s Journey repeated in
countless media…

Suspension of Disbelief
Suspension of disbelief or willing suspension of
disbelief is a term coined in 1817 by the poet
and aesthetic philosopher Samuel Taylor
Coleridge, who suggested that if a writer could
infuse a "human interest and a semblance of
truth" into a fantastic tale, the reader would
suspend judgement concerning the
implausibility of the narrative.
http://en.wikipedia.org/wiki/Suspension_of_disbelief

Why is the Hero’s Journey
structure so popular in movies
and media?
Fairy Tales, myths, Star
Wars, The Matrix, etc.
Do we like it because
we expect it and are
conditioned for it?
Is it inherent in our
minds and DNA?
Does it hold us back in
life? Must we be chosen
to do great things by
someone else?

Games and Story

Play
Confront a
challenge
Interact with other
Understand your
progress
players
Games
Interactive with the
world
Actions have
consequences
Conquer small
quests

Players
Story
Game Elements
Conflict /
Mechanics Challenge
Resources
Game world / field of play

Game Mechanics
Players overcome
challenges
Players understand
the rules
Players understand
their place in the
context of the game
Reason for the
experience to exist
Know how to interact with
the experience
Points System
Levels
Time

Game Enjoyment
70
The Challenge The Story The Game Play
http://en.wikipedia.org/wiki/Doom_(1993_video_game)
http://z2.com/game/metalstorm-online/
http://en.wikipedia.org/wiki/Puzzle_Bobble
http://en.wikipedia.org/wiki/World_of_Warcraft

World Building

A Generic Story
Character presented with challenge
Generic Threat
Finds courage... Triumphs!
Character evolves
Generic Hero
Character overcomes challenges

Character Change Through Conflict
Character evolves
A character evolves/
changes due to conflict
inherent in the story
Relationships
Environment

Who is the Main Character?
Is the user a participant or an outside viewer in a location game
or experience?

Story Development
Character Driven World Driven
Generic environment,
used as a setting for
character interaction to
take place.
The story of a character
in a given environment,
which forces action /
conflict
...become a different
person
How will the world affect character development?
...character traits stay the same

Contemporary
super hero
Traditional
action hero
Die Hard
Arnold Schwarzenegger
Ironman, Thor, Spiderman, etc.
Watchmen
Dark Night
Man of Steel
X-men
Simple dialogue, hero
never dies, always
triumphs
Question their actions
and place in the story,
supporting characters
die, like real life.
Good vs. Evil Good? vs. Grey Zone

World
Location Time Physics
Where is the
world? Is it a city,
a new planet, a
coffee house?
Take a character
from another time
or use a world
from a different
time?
If it’s a new world,
does it respect the
physics of Earth as
we understand them?

78
X | Media | Lab Switzerland 2013: Transmedia
Alex McDowell – Transmedia’s Renaissance Man: Narrative Designer for Minority Report; Fight Club; Man of
Steel; Associate Professor, USC – School of Cinematic Arts; Visiting Artist, MIT Media Lab (Hollywood)

79

80

How does a character
move around the world?
Character
Journey
Character
becomes sick
Must go to
hospital
How would she
die?
Transportation
Never invented
gasoline, all
hydrogen
logistics

Physical Virtual World Building /
Prototyping
Unity 3D!
• Multi-platform Game
development
• Terrain building
• Visualization
• Integration of buildings,
people, etc.

Sketch out worlds
Physical story worlds define logic of story progression
The Hobbit Map
http://tolkiengateway.net/wiki/File:Christopher_Tolkien_-_Map_of_Wilderland.jpg

Paint terrain and topography
features
!
Add textures for grass, rock,
water, etc.
!
Add buildings, people, tanks,
etc.

Unity User Interface
Paint on topography
Load new models
Use different brushes
Load textures

Augmented and Mixed
Reality Technologies
86

Generic AR Experience
App tracks feature
in reality
Marker
Marker-less
Point-cloud
Edge-based
Content positioning
User engaged with
app
Virtual content
87

Creator / Studio
SDKs
AR Software
AR Browsers
88

Depth Cameras
Desktop
Laptop
Mobile
Structure.io
Project Tango

90
Ghosts of Venice
ghostsofvenice.it

Augmented Reality Ghost
Stories of Venice
ghostsofvenice.it

92
Film Maker /
Director
Marco Gatti
AR Glass Expert
Story Tech
Sara Mautino
UX / AR coding
Mark
Melnykowycz
Visual Effects
Production
ghostsofvenice.it

Take users on a ghost story through Venice, mixing story elements and
93
Place Ghosts in the
Environment
ghostsofvenice.it
3D animations into the local environment

Film vs. App
94
C o n s u m e d b y
watching/listening
while stationary
L o c a t i o n a n d
content can be
mixed to provide
more immersive
experience
ghostsofvenice.it

Augmented Reality
95
Technology used to slightly
modify the perception of
the environment of the user
GPS location
Media
overlay
Envi ronment or user
triggers new actions in
the software
Image
recognition
Feature
tracking
4
Image
3
Video
2
Audio
Text 1
ghostsofvenice.it

Story - Technology
Development
96

Transmedia Narrative
Main point of contact between
user, story, and location
Online
Story
Hard Copy
Mobile App
Educate users of the
background stor y
before starting
Give users something
tangible, objects from
t h e gh o s t s , t h e i r
letters, etc.
97 ghostsofvenice.it

Story
Development
App Design
App
Development
Media
Production
How do all of these key elements
come together in the transmedia
development process?

Story/App Workflow
Test/Iterate
Story App Design
Prototype
App
Development
User Experience
User Interaction
Background
stories of ghosts
How stories are
delivered
Implement UX/UI in
App Development
Production
App

Story/Media Creation
Story Locations Ghosts
Choose locations
Choose trigger
elements
Background story
for each ghost
Visual Design
Casting
Filming
Post-production
Mobile
Media
Ready to
distribute on
mobile
device
Test App
Code into data
model for
viewing/testing
Production
App

Transmedia Landscape
A transmedia project integrates a larger narrative across different distribution
channels, we need to develop a plan for how this would be realized
Sets the stage, gives tangibility to the
story experience, being in the same
physical place as the ghosts
App is used to view the ghosts,
to connect with the “being” the
user has learned about
Interaction
How can the story and location be interactive
with the participants? This increases the
emotional connection to the experience
The impact of seeing the ghosts comes from the
emotional connection to the story behind them
Location
Mobile App
Story Channels
Real book/diary of ghosts
Wikipedia page each individual
Artifacts of ghosts in the city/museums...what else?
Establish emotional connection and empathy to ghost
101 characters ghostsofvenice.it

Augmented Reality
Technology Needs
102

User Walks
Through Area
User Flow
Location
Aware
Image
recognition
Event
Playing
User sees they are
in the area where
a trigger is
User searches
for physical
trigger
Location triggers
haptic/vibration on
device
Content plays
through device
ghostsofvenice.it

Alpha Mask
Overlay in video feed
Transparent background
ghostsofvenice.it 104

Image Recognition for
Location?
Software recognizes
specific features of the
environment
Image pre-programmed
int o the device or
queried from a server
Recognition may not be
automatic, depends on
hardware speed
ghostsofvenice.it
105

Install Junaio
Access GOV
channel
View content on
image trigger
106
ghostsofvenice.it

Changing light
conditions will impact
tracking ability
Image target not
reliable enough for
spatial location use

Feature Recognition
Software recognizes
specific features of the
environment
P o i n t - c l o u d m a p
created to aid in feature
recognition
CAD or edge-based
tracking can be tested
as well
ghostsofvenice.it 108

Metaio Edge-based
Tracking
https://dev.metaio.com/sdk/documentation/tracking-config/optical-tracking/cad-model/
CAD model
overlay on
environment
Better in low-light
tracking scenarios
Where do I get
building CAD?
www.metaio.com
109

App Development Flow
110
Initial Test Functional Prototype Initial Release
Layar/Junaio
Metaio SDK
Unity
Beta App
Test the concept
of playing audio
o r v i d e o fi l e s
based on GPS
location with AR
Browser
I m p l e m e n t v i d e o
overlay concept in a
nat i v e iOS/Androi d
application, testing with
users to understand the
UX better
With a developed
s t o r y l i n e / p l o t ,
producing the full
m i x e d r e a l i t y
exper ience and
broad testing with
users
ghostsofvenice.it

Mobile 3D Scanning
Use 3D scan of
objects to create
CAD for model
positioning
Experiment with
3D mapping for
model positioning

Combine Person and
Environment Models
Ghosts as model
scans
Match to
environmental
features

Film Maker /
Director
Marco Gatti
AR Glass
Expert
Sara Mautino
UX / AR
Development
Mark Melnykowycz
Visual
Effects
Team
www.ghostsofvenice.it
113

Lost In Reality
Mobile
App Web
App
Search
Consume
Create
Share
Create
Edit
Storytelling
Social
Network
115 lostinreality.net mark@lostinreality.net

Francisco Campelo José Rodrigues Mark Melnykowycz Ismail Cimen
App Developer
Partner and
Story
Development
Project Lead
UX/UI
Design
UX/UI
Team
Lisbon, Portugal Zurich, Switzerland

People
Loca8ons Emo8ons
How can we use story to add to the
information context of locations?

For Storytellers
Stories
saved
Authors
can
write
loca8on
stories
on
the
web
app
Users
find
stories
based
on
loca8on
and
tags
Users
can
write
stories
on
the
mobile
app
Story
server

Loca8on
used
as
integral
element
of
Tourists
have
more
unique
experiences
in
new
ci8es
For Authors
Authors
can
write
loca8on
stories
on
the
web
app
Users
can
download
stories
Locals
have
new
perspec8ves
on
their
city
stories

For Readers
Users
search
and
access
loca8on-‐stories
on
demand
with
mobile
app
Story
elements
triggered
based
on
loca8on
in
ci8es
Context
of
stories
gives
more
value
to
loca8ons

Location Stories vs. Social Media
Social Networks
Context?
Location Services
Social media evolved from
blogging as faster way to
publish ideas
Social media updates are
only useful for a short time
frame
Location stories give
structure and context to
information and combats
the social media noise
“Facebook/Twitter shows
people by who they are, what
they have done and said”
Location services give an
idea of what a person can do
in a place (irregardless of
when)
“Locat ion s tor ies show
people the experience
they’ve had”
“Location stories let people
follow experiences and have
their own”
lostinreality.net mark@lostinreality.net 121

Mobile App Flow
Focus on location-based
storytelling
Users can create, search,
and follow location stories
Emotional engagement to
location
Primary
Goals of LIR
Smart phone application
Dashboard Story Library
Local Story Search Story Element
Story elements delivered
to user at GPS location
(triggered)
lostinreality.net
mark@lostinreality.net
122

Maker Web App

125
User can follow
different stories
in a city as they
walk around.
lostinreality.net mark@lostinreality.net

Sketching out user flows /
prototyping / product definition
User motivations
/ immersion
Building interaction /
defining app world
Technology
trends
Communication
patterns
126

ISMAR 2014 AR Workshop: Designing Location Based Experiences

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a ISMAR 2014 AR Workshop: Designing Location Based Experiences

Similar a ISMAR 2014 AR Workshop: Designing Location Based Experiences (20)

Más de Mark Melnykowycz

Más de Mark Melnykowycz (10)

Último

Último (20)

ISMAR 2014 AR Workshop: Designing Location Based Experiences