Cross Manipulation in Mixed Reality BAsed on a Bridge Virtual Marker

1. Cross-Manipulation in Mixed Reality Based on a Bridge Virtual Marker
Raphael Grasset∗
¨ Julian Stadon † Mark Billinghurst‡
HIT Lab NZ
University of Canterbury
Private Bag 4800, Christchurch, NZ
A BSTRACT AR VR
In this paper we introduce a concept for transferring and manip- User
User
ulating objects between different interface contexts (AR and VR),
thus extending the capabilities of manipulation in transitional inter-
faces, which provide navigation between these different contexts.
To transport objects between AR and VR environments, we propose Shared Data
the use of a virtual marker proxy, which provides a bridge between
worlds. The concept is demonstrated with a prototype that allows
users to transfer objects between a Massively Multiplayer Online
World and a desktop Augmented Reality application. AR VR
Index Terms: H.5.1 [Information Interfaces and Presentation]:
Multimedia Information Systems—Artificial, augmented, and vir-
User
? User
tual realities; H.5.3 [Information Interfaces and Presentation]:
Group and Organization Interfaces—Collaborative computing
Data 1 Data 2
1 I NTRODUCTION
Hybrid collaborative systems supporting a combination of Aug-
mented Reality Environments, Virtual Reality Environments or
Web Interfaces have been previously explored. In this type of Figure 1: Cross-Manipulation Problem: In the top figure, two users
Mixed Reality collaboration (often highly asymmetric), users can in different worlds (Augmented Reality and Virtual Reality Environ-
experience a shared world such as that proposed in [1]. In this con- ments) can manipulate similar data, such as the purple element via
text, the information and the model are generally similar, the dif- a shared data mechanism. However if the data are separated, like in
fering element is indeed the interface and the representation of the the bottom figure, how can objects be transferred between worlds?
data (see the top of Figure 1 for a typical example).
Another range of configuration issues consist in cases where the
data is not shared between both worlds, each world is associated
with its own data (illustrated at the bottom of Figure 1). In this con- the creation of a unique mixed environment rather than dealing with
text manipulating and exchanging data and content between worlds two separate environments.
is intrinsically more challenging. It can be sometimes even more This brief literature survey shows that different solutions have
difficult when users have no control over the architecture of the sys- been proposed for symmetric environments, but hardly any research
tem used; for example with a Massively Multiplayer Online World has been conducted with an asymmetric solution (like between an
(MMO). Offering a way to combine a traditional AR system with a AR Context and a VR Context) where different type of data are
VR world like a MMO is therefore attractive; thus there is a need associated with each world. In the next section we introduce our
for techniques to transfer the data between both of these world (see approach to the problem. We focused on the scenario of an AR
bottom of Figure 1). environment coupled with a VR environment for a mixed-reality
Previous solutions have been proposed in the context of multiple collaboration.
virtual worlds in virtual reality. For example, Kiyokawa et al.[3]
suggested the use of a window portal metaphor for remote manipu- 2 C ONCEPT
lation, using the concept of a tunnel window. 2.1 Virtual Marker
For Augmented Reality, Butz et al. [2] proposed a drag and drop
technique between a 3D AR space to a screen space within the EM- Our approach has been largely inspired by previous works in Vir-
MIE system. Schmalstieg et al. [6] has explored the use of tangible tual Reality and interactive workspaces that use the concept of a
object and public locales to migrate applications between remote bridge. A bridge defines a dedicated space where the user can trans-
physical spaces in a context of a collaborative AR scenario. Fi- fer things between worlds, spaces, and contexts. Since we consider
nally, MacIntyre has recently modified a client of a MMO for creat- the possible inability to modify or redesign the Virtual Reality En-
ing mixed reality experiences [4]. However, his purpose was more vironment, it is necessary to define this bridge as part of the content
of the virtual world.
∗ e-mail:Raphael.Grasset@hitlabnz.org To do this, we introduce the notion of a virtual marker. Real
† e-mail:Julian.Staton@hotmail.com fiducial markers are generally used for tracking but they also sup-
‡ e-mail:Mark.Billinghurst@hitlabnz.org port association with virtual objects using tangible interface con-
cepts in an Augmented Reality Environment. Similarly, equivalent
Virtual Markers can be described in the virtual world. However, in
our case the presence of a virtual marker in the virtual world will
provide an anchor for an object in the world, and thus a bridge into

2. Screen (Window of VR) 2.3 Object Coherence State
The transfer of an object between both worlds implies the ability to
define the state of the object in both of these worlds. For example, if
Virtual Marker we transfer an object from SecondLife to the AR Base and modify
an object property like scale, what should happen with the object in
SecondLife, is it affected by this or is it in undefined state? If we
transfer the object back into SecondLife how will its properties be
Real
Marker affected?
To resolve this issue we consider three cases, mimicking the gen-
eral solution implemented in standard 3D Modeling software. A
transferred object can be in three states: (1) a copy, (2) an instance
or (3) the same unique object. The instance will keep a coherent
Figure 2: The user can visualize both real markers and virtual mark- state when manipulated in both of the worlds. The other two cases
ers in the Augmented Reality Environment. won’t.
3 D ESIGN AND I MPLEMENTATION
We have implemented this concept between SecondLife and a ded-
the virtual environment. icated AR prototype application. This prototype is part of an art
To support the transfer between both worlds, a window of the installation that will be presented for an exhibition in May, in Perth,
virtual world should be available in the real world. Having this Australia.
window, a user can observe the content of the virtual world and The AR system has been implemented using the osgART library
therefore visualize virtual markers. In the user’s view, this virtual [5]. The user is equipped with an E-Magin Z800 HMD, and a Log-
marker becomes equivalent to a real marker, integrated in the AR itech Quickcam 5000 USB camera. The VR World is displayed
Application (illustrated in Figure 2). on a standard LCD screen. A 3D mesh corresponding to the vir-
tual avatar of SecondLife is used to demonstrate the transfer. When
the user moves the transfer tool from the AR context base next to
2.2 Interactive Space the virtual marker on screen a 3D mesh is transferred from the AR
environment into the on-screen SecondLife environment. To show
Assuming the availability of a virtual marker, our system uses three
that we can change object state while doing this, when the user has
components to transfer objects from the virtual world to the AR
transferred the uncoloured 3D mesh from the transfer tool to the
world. We use the virtual marker, a transfer tool and a contextual
base, the object is coloured and textured (and can be also animated).
base. The Contextual Base provides a reference space for manip-
ulating the Augmented Reality Content, while the Transfer Tool We have been confronted with different design issues for the im-
provides the interface to transfer things between the Virtual World plementation of this prototype, including:
and the Augmented Reality World. Window: the visibility of the virtual marker is highly dependent
on the technology used to display the virtual marker. Thus the use
The user can transfer elements using proximity between two of a non-reflective desktop screen provided the best solution. A
markers. They can thus realize bidirectional transfers in this or- projected image is problematic since the use of the camera for AR
der: Virtual Marker ↔ Transfer Tool ↔ Base. To help the user to tracking requires good ambient lighting.
understand the relationship between them we augment the markers
Markers: the virtual marker can be easily designed with built-in
with basic visual feedback provided by a virtual torus around each
tools from SecondLife. Using the navigation mode of SecondLife,
of the components (shown in Figure 3).
placing a vertical marker in front of a clear background will im-
[You might want to put more here about the actual mechanism prove the robustness of the AR tracking.
of how the transfer is done - see what I wrote below] Coherence State: it remains challenging to support and maintain
a coherent object state with SecondLife. The SecondLife API is
limited, so we have only been able to implement a copy mode in
VR AR our application.
Window of VR
Context 4 C ONCLUSION
We have presented an implementation for transferring and manipu-
Virtual
lating objects from a VR MMO to an AR World based on using a
Bridge Marker virtual marker. Acting like a bridge, the user can have the benefit
of both worlds to visualize and modify different objects. We have
Transfer identified design issues with this type of solution and presented our
Tool initial approach to resolving them. In the future we hope to inves-
tigate further the problems related to maintaining a coherent data
AR Context state between both worlds.
Base
R EFERENCES
[1] B. Brown, I. MacColl, M. Chalmers, A. Galani, C. Randell, and
A. Steed. Lessons from the lighthouse: collaboration in a shared mixed
Figure 3: A Virtual transfer: the red torus is on the virtual marker, the reality system. In CHI ’03, pages 577–584, 2003.
green torus on the transfer tool, and the blue on the AR base marker. [2] A. Butz, T. H¨ llerer, S. Feiner, B. MacIntyre, and C. Beshers. Envelop-
o
ing users and computers in a collaborative 3d augmented reality. In
IWAR ’99: Proceedings of the 2nd IEEE and ACM International Work-
shop on Augmented Reality, page 35, Washington, DC, USA, 1999.

3. [3] K. Kiyokawa and H. Takemura. A tunnel window and its variations:
Seamless teleportation techniques in a virtual environment. In HCI
International 2005, 2005.
[4] T. Lang, B. MacIntyre, and I. J. Zugaza. Massively multiplayer online
worlds as a platform for augmented reality experiences. In IEEE VR
’08, 2008.
[5] J. Looser, R. Grasset, H. Seichter, and M. Billinghurst. Osgart - a prag-
matic approach to mr. In Industrial Workshop at ISMAR 2006, 2006.
[6] D. Schmalstieg and G. Hesina. Distributed applications for collabora-
tive augmented reality. In IEEE Virtual Reality 2002, 2002.