Presentazione tenuta da Pieter Jonker e Jolande Kolstee in occasione del Forum "Multimedia per la Cultura", tenutosi a Vigevano l'11 e 12 giugno 2010

1. Augmented Reality
Pieter Jonker
Professor of Vision based Robotics
Dept Bio-Mechanical Engineering, Faculty 3ME
Delft University of Technology, Delft
The Netherlands
p.p.jonker@tudelft.nl
Yolande Kolstee
Lector of Innovative Visualisation Techniques for Art & Design
Royal Academy of Art, The Hague
The Netherlands
Y.Kolstee@kabk.nl
01/07/2010 15:36 1

2. Augmented Reality vs Virtual Reality
Het volgen van een enkele hypothese (toegepast in AR)
01/07/2010 15:36 2

3. Sorts of Augmented Reality
Screen Based
Head-Set Based
01/07/2010 15:36 3

4. Sorts of Augmented Reality
Video see through AR Optical see through AR
2 cameras + 2 displays + 1 camera + 2 half translucent
video in video mirrors + 2 displays
01/07/2010 15:36 4

5. History of our headsets
01/07/2010 15:36 5

6. The Measurement Problem
How to hold the
virtual objects in a
perfect position and
orientation in the
physical world if I
move my head?
By searching and
tracking (camera +
image processing)
one or more known
landmarks!
01/07/2010 15:36 6

7. How to anchor onto a marker?
Select closed Simple corner
Grab an image Edge detection
contours detection
Intersect 4 lines Sub-pixel fit of Split contour Keep contours
for corners the 4 lines into 4 lines with 4 corners
Calculate pose of Determine the Calculate pose
marker ID of camera
1 2 4 8 16
32 64 128 256 512
1024 2048 4096 8192 16384
01/07/2010 15:36 7

8. Why not tracking at natural features?
This costs too much time fort now, but soon…
01/07/2010 15:36 8

9. Is only a camera & image processing enough?
No, we must know where the head will be in 30 msecs
because rendering a virtual image costs time. We have
to predict where the head will be bij measuring the
motion of the head.
That is why we have to fuse the data of the camera and
image processing with the date from a so called inertia
tracker to track fast head movements.
This is comparable with the yes and the inner ear /
vestibular organ of a human.
An inertia tracker has 3 accelerometers to measure linear
acceleration in 3 dimensions (x, y, z) and 3 gyroscopes
to measure 3 rotational velocities (roll, pitch, yaw).
This gives relative displacements at 100Hz that are
combined with the 25Hz absolute position and
orientation of the camera.
01/07/2010 15:36 9

10. Fusion of camera and inertia tracker data
Inertia measurement 100 Hz
Camera measurement Image acquisition Image processing 15 Hz
Data Fusion
Rendering Image 60 Hz
time
01/07/2010 15:36 10

11. So how accurate can one place virtual
objects in the physical world?
Statical misalignment
< 0.6 ° (< 0.03° = eye)
Dynamical Misalignment
< 2.3° (< 0.5° = eye)
With a 20 x 20 cm marker
at 5 m and a 1MPix
camera
Own and object motion
will camouflage this
01/07/2010 15:36 11

12. Future elektronics to loose the
backpack and headset
01/07/2010 15:36 12

13. Conclusions so far
• Augmented Reality is not Virtual Reality
• Augmented Reality is screen or headset based
• Headset based AR is video or optical see-through
• The AR problem is dominated by the the allignment
problem of virtual and real world objects
• This is solved by inertia and camera tracking
• Due to the imperfect technology one can still see a
bit alignment errors, but far better than GPS!
• Happily the human visual system compensates a bit
• Future hardware will make it as small as iPods.
• AR is a new and complex medium to discover!
01/07/2010 15:36 13

14. The Vision of Yolande Kolstee
Visualization in Art and Design
01/07/2010 15:36 14

15. Undergraduate / Bachelors
Design: Textile, Fashion, Interior,
Furniture, Graphic, Interactive Media
Fine Arts & ArtScience
Royal Academy of
Photography
Art (KABK)
Graduate / Masters
The Hague
Type and Media
The Netherlands ArtScience
www.ARLAB.nl Interior Architecture
Research in the Arts
Industrial Design
01/07/2010 15:36 15

16. Undergraduate, Graduate & Doctorate
Faculties:
Aerospace Engineering
Delft Applied Sciences
University of
Architecture
Technology
Civil Engineering and Geosciences
Delft Electrical Engineering, Mathematics & Computer Sc.
The Industrial Design Engineering
Netherlands Mechanical, Maritime & Materials Engineering
Technology, Policy and Management
01/07/2010 15:36 16

17. The fusion of perception of real and virtual
images
The AR Lab is for all students, also
outside students, who need help in the AR
domain.
Institutes of cultural heritage come to us,
like Museums:
•Van Gogh
•Boijmans van Beuningen
•Stedelijk Museum
•Kröller-Müller
•Escher
Manifestations of:
Multi-media
Modern-Art
01/07/2010 15:36 ICT 17

18. For example we used AR to show animated chairs
and tables in the 2008 exhibition Salone del Mobile in
Milan, Italy
01/07/2010 15:36 18

19. AR was used to demonstrate the construction process
01/07/2010 15:36 19

20. AR can be used to demonstrate the
possibilities of the product
Melanie Luchtenveld
01/07/2010 15:36 20

21. The virtual objects can be studied from all
viewpoints
01/07/2010 15:36 21

22. Transportation and duplication of furniture is quite
easy
01/07/2010 15:36 22

23. Connecting with the virtual
world, experiments with
interfaces
Audience likes AR & Interaction
Wii remote controller
RFID
Data glove: 5 flexor strips in fingers
Markers 'stand-alone' or incorporated
Fragments of pottery
01/07/2010 15:36 23

24. We use RFID technology to let (tagged) physical
objects influence virtual objects
01/07/2010 15:36 24

25. We use a data-glove to manipulate virtual
manipulated objects. Here to realize a virtual hand-
puppet
01/07/2010 15:36 25

26. One can interact with virtual objects in several
ways
One can us real physical
objects to change behavior
of virtual puppets e.g.
Bath-sponge towel and
foam
Pot-holder cook
Fay-wand blow fish is
crowned
Cigarettes blow fish
blows
01/07/2010 15:36 26

27. Interaction can also be performed by moving the
markers
01/07/2010 15:36 27

28. Interaction with a WII mote
Escher in the Palace:
Wii remote controller:
throw a visual to the
wall
01/07/2010 15:36 28

29. Headset Design
Students of the Royal Academy designed a new AR
headset based on the VR/AR headset of the Dutch
SME Cybermind who made a working version of the
student’s mock-up
01/07/2010 15:36 29

30. Design requirements:
More ease to wear
Don't isolate the user
Don't block the reality
Contact with the environment
Unique experience / attractive
01/07/2010 15:36 30

31. Headset 'George'
camera at the right side
inertia tracker left
open view below
wide screen
Side view
01/07/2010 15:36 31

32. Various other designs have been made, but not
yet prototyped
Design Niels Mulder
01/07/2010 15:36 32

33. What to display?
Man made:
There is a variety of 3D design programs and skilled and
talented people who master this:
Fine Art Artists, Animators, 3D Designers, Architects
Computer Generated:
3D imagery (industrial) machines
especially from the bio-medical research
CT scan generated 3D images, yield an
inexhaustible amount of CT images
Data visualization:
There are various huge databases, e.g. geo
01/07/2010 15:36
information, annual reports 33

34. Art: Drops of White
Marina de Haas
01/07/2010 15:36 34

35. Geo Info
Data Visualization of Acid
level in the first AR MAP
on behalf of GSDI 11
World Conference,
Rotterdam
June 16-20 2009
Global Spatial Data Infrastructure
association
01/07/2010 15:36 35

36. AR Art: first experiment for general public
Marina de Haas
01/07/2010 15:36 36

37. Cultural Heritage: Displaying computer
generated data:
Using bio-medical imaging equipment:
CT scans for visualizing sgraffito-pottery:
Pre industrial original (1250-1450)
Post industrial means
Scanning- Digitalizing
3D printing physical copy
3D Archiving virtual copy
01/07/2010 15:36 37

38. 01/07/2010 15:36 38

39. Lessons learned
Let the public view on a beamer what is seen by the HMD
The “empty” third dimension is very useful for information display and
interaction and detaches the application from gravity. One can have art or any
info in the air ( e.g. drops of white)
Motion of the virtual objects covers up for misalignment and jitter
Design packages such as Cinema 4D make design with animated figures
possible. For real 3D animated films with large plots, we are busy to incorporate
game engines like Unity
01/07/2010 15:36 39

40. Headset based AR is at its best when a full immersive experience is required
and people can walk around or within larger objects; buildings, DNA, airplanes to
be refurbished
For outdoor AR it is necessary that the ambient light intensity and the intensity of
the LCD displays on the headset is in balance (automatic sunglasses)
Augmented Reality can be fruitfully used to attract a broad public to displays of
cultural heritage. Its narrative power is huge.
The collaboration between researcher in the area of image processing with
artists, designers and curators appeared to be very fruitful and has lead to many
amazing productions and exhibitions
01/07/2010 15:36 40

41. ‘Why more realities?’
the old man in Milan said:
‘Live is already so complicated’.
“It begins with a blessing
and it ends with a curse;
making life easy,
by making it worse.”
01/07/2010 15:36 41