I made this presentation at the MPEG Multimedia Ecosystem 2013 in Incheon. It includes a summary of MPEG technologies related to Augmented Reality and is focuses on the separation between the AR creation and AR consuming. A system architecture for AR is also presented. Photo credits: Lisa Blum, Richard Wetzel, Veronica Scurtu Note: many pictures used in this presentation are downloaded from the Internet; I'll be happy to add credits to the original authors if they let me know

1. CONNECTING PHYSICAL AND DIGITAL WORLDS
MARIUS PREDA, MPEG 3DG CHAIR, INSTITUT MINES TELECOM
www.slideshare.net/MariusPreda

2. From centuries, we are building two different types of worlds
Physical
Informational
Immersion, from a human user point of view, is making abstraction of the Physical
world (remove all possible connection to it) and have stimuli only from the
Informational one
Immersion, presence, stimuli, perception, …
stories
music
films
knowledge

3. In which world our users are?
The stimuli from physical worlds
are familiar or at very low
intensities

4. We intentionally block our
interfaces with the
physical world
Improving immersion feeling

5. Strategies for blocking
the interfaces with the
Physical world
Improving immersion feeling

6. Strategies for extending the space of
the Informational world stimuli
Improving immersion feeling

7. How far we’ve gone with
this strategy?
Relatively far …
Improving immersion feeling

8. Probably too far …
Improving immersion feeling
How far?

9. It’s the time to connect our worlds

10. Why now?
• For the first time, smartphones enable
“everyone” to access Informational world
while sensing also the Physical world
• They change our view of the environment
• Enhance the rich history
and meaningfulness of
the real world with
powerful digital
elements

11. Physical & Informational Augment The Reality
An invitation: become architects of a new
kind of bridges

12. In AR, almost everything remains to be invented
Las Vegas
This place some years ago

13. In AR, almost everything remains to be invented

14. • Marketing
• Tourism
• Culture
• Education
• Serious and less serious games
• …
• Hardware manufacturers
• Software vendors
In AR, almost everything remains to be invented

15. We (and few others) already started and invite
you to be part of
The AR Revolution
In AR, almost everything remains to be invented

16. Tidy City

17. Portal Hunt

18. Elements

19. ARQuiz

20. Augmented Books

21. Summer School games

22. TECHNOLOGY INSIGHTS
MPEG-V AND MPEG ARAF

23. MPEG technologies related to AR
MPEG-1/2
(AV content)
1992/4
VRML
1997
• Part 11 - BIFS:
-Binarisation of VRML
-Extensions for streaming
-Extensions for server command
-Extensions for 2D graphics
- Real time augmentation with
audio & video
• Part 2 - Visual:
- 3D Mesh compression
- Face animation
1998
• Part 2 – Visual
- Body animation
1999
MPEG-4 v.1
MPEG-4 v.2
First form of broadcast signal augmentation

24. MPEG technologies related to AR
MPEG-4
2003
•AFX 2nd Edition:
- Animation by
morphing
- Multi-texturing
2005
• AFX 3rd Edition
- WSS for terrain
and cities
- Frame based
animation
2007
MPEG-4
MPEG-4
• Part 16 - AFX:
- A rich set of 3D
graphics tools
- Compression of
geometry,
appearance,
animation
• AFX 4th Edition
- Scalable complexity
mesh coding
2011
MPEG-4A rich set of 3D Graphics
representation and
compression tools

25. MPEG technologies related to AR
MPEG-4
2003
•AFX 2nd Edition:
- Animation by
morphing
- Multi-texturing
2005
• AFX 3rd Edition
- WSS for terrain
and cities
- Frame based
animation
2007
MPEG-4
MPEG-4
• Part 16 - AFX:
- A rich set of 3D
graphics tools
- Compression of
geometry,
appearance,
animation
• AFX 4th Edition
- Scalable complexity
mesh coding
2011
MPEG-4
2009
• Part 25
- Compression of
third-party XML
(X3D, COLLADA)
MPEG-4
2004
• Part 16
- X3D
Interactive
Profile
MPEG-4

26. MPEG technologies related to AR
MPEG-V - Media
Context and Control
2011
• 2nd Edition:
- GPS
- Biosensors
- 3D Camera
201x
• Compression
of video +
depth
201x
MPEG-V
- 3D Video
• 1st Edition
- Sensors and
actuators
- Interoperability
between Virtual
Worlds
• Feature-point based
descriptors for image
recognition
201x
CDVS
MPEG-U –
Advanced
User Interface
2012
A rich set of sensors and
actuators
- 3D Audio
MPEG-H

27. MPEG technologies related to AR
MPEG-V – Media Context and Control

28. Actuators
Light
Flash
Heating
Cooling
Wind
Vibration
Sprayer
Scent
Fog
Color correction
Initialize color correction parameter
Rigid body motion
Tactile
Kinesthetic
Global position command
Sensors
Light
Ambient noise
Temperature
Humidity
Distance
Atmospheric pressure
Position
Velocity
Acceleration
Orientation
Angular velocity
Angular acceleration
Force
Torque
Pressure
Motion
Intelligent camera type
Multi Interaction point
Gaze tracking
Wind
Dust
Body height
Body weight
Body temperature
Body fat
Blood type
Blood pressure
Blood sugar
Blood oxygen
Heart rate
Electrograph
EEG , ECG, EMG, EOG , GSR
Weather
Facial expression
Facial morphology
Facial expression characteristics
Geomagnetic
Global position
Altitude
Bend
Gas
MPEG technologies related to AR
MPEG-V – Media Context and Control

29. • All AR-related data is available from MPEG standards
• Real time composition of synthetic and natural objects
• Access to
• Remotely/locally stored BIFS/compressed 2D/3D mesh objects
• Streamed real-time BIFS/compressed 2D/3D mesh objects
• Inherent object scalability (e.g. for streaming)
• User interaction & server generated scene changes
• Physical context
• Captured by a broad range of standard sensors
• Affected by a broad range of standard actuators
Main features of MPEG AR technologies

30. MPEG vision on AR: the MPEG AR Browser
Point to a URL – no need to download new applications for each context
The browser
• Retrieves scenario from the internet
• Starts video acquisition
• Tracks objects
• Recognizes objects from visual signatures
• Recovers camera pose
• Gets streamed 3D graphics
• Composes new scenes
• Gets inputs from various sensors
• Offers optimal AR experience by constantly adapting interaction possibilities and
objects from a remote server.
Industry
• Maximize number of customers through MPEG-compliant authoring tools and
browsers
• No need to develop a new application for each use case and device platform

31. MPEG vision on AR
MPEG-4 + MPEG-7 + MPEG-21 +
+ MPEG-U+ MPEG-V
= ARAF
MPEG Player
Compression
Authoring Tool
Produce
Download

32. Technology insights
Modeling the content chain
– Creation
– Consumption
– The pivot : a representation formalism

33. Technology insights: the chain
ARAF
Browser
Media
Servers
Service
Servers
User
Local
Sensors &
Actuators
Remote
Sensors &
Actuators
MPEG
ARAF
Local
Real World
Environment
Remote
Real World
Environment
Auth. Tools

34. Technology insights: Infrastructure
ARAF Browser
Media
Servers
Service
Servers
User
Local
Sensors &
Actuators
Remote
Sensors &
Actuators
MPEG
ARAF
Local
Real World
Environment
Remote
Real World
Environment
Auth. Tools

35. The ARAF browser
ARAF Browser
• Full scene graph representation (2D and 3D)
• Static and dynamic objects and scenes
• Programmable behavior (JS)
• Access to sensors and cameras
• Access to remote resources and services (http
request and web sockets)
• Support for user interaction
• Support for compressed resources

36. The ARAF browser
ARAF Browser
ARAF Browser
JavaScript Engine
MPEG Totem Services
WebSockets
Server
Communication
service
WebSockets
Client
ARAF Browser
SC3DMC
Decoder
BBA Decoder
ARAF File
BIFS Stream
SC3DMC
Stream
BBA Stream
BIFS Decoder
MPEG-4
Scene
IndexedFaceSet
SBVCAnimation
Camera
Image data
Reference
Images
Image data
Image data
Image data
AFAR Browser
Reference
Signal
Detection
Detected
Images

37. MyMultimediaWorld.com
• Convert media assets in a common format
• Describe, index, organize, search digital assets
• Compressed assets, ready for transmission
A multimedia repository and processing platform
supporting both creation and consumption Media
&
Service
Servers

38. Technology insights: the formalism
ARAF Player
Media
Servers
Service
Servers
User
Local
Sensors &
Actuators
Remote
Sensors &
Actuators
MPEG
ARAF
Local
Real World
Environment
Remote
Real World
Environment
Auth. Tools

39. ARAF Description Language
ARAF : an XML representation
Player
Compression
Authoring Tool
Produce
Download
ARAF

40. A set of scene graph nodes defined in MPEG-4
• Audio, image, video, graphics, programming, communication, user
interactivity, animation
• Map, MapMarker, Overlay, ReferenceSignal, ReferenceSignalLocation, Cam
eraCalibration, AugmentedRegion
Connection to sensors defined in MPEG-V
• Orientation, Position, Angular
Velocity, Acceleration, GPS, Geomagnetic, Altitude, Local camera(s)
Compressed media
• Image, sound, video, 2D/3D graphics
Augmented Reality Application Format
ARAF

41. Technology insights: creation
ARAF
Browser
Media
Servers
Service
Servers
User
Local
Sensors &
Actuators
Remote
Sensors &
Actuators
MPEG
ARAF
Local
Real World
Environment
Remote
Real World
Environment
Auth. Tools

42. Authoring ARAF
• Specific tools
• Generic tools
– TOTEM.Designer
– TOTEM.Scout
Auth. Tools

43. Specific Authoring Tools
• Scenario driven approach
– Tidy City, Portal Hunt, ARQuiz, AR Book, Treasure
Hunt
Auth. Tools

44. Towards a generic Designer
• Create templates for app. objects
• Fill structure with data
• Create GUI
• Create application logic
• Export data

45. On site designer

46. Conclusions
• AR – the next digital revolution
• For making revolutions, one needs weapons:
– MPEG technologies represent an efficient and
powerful set of weapons
• Current MPEG activities
– Improve and enrich the weapons
– Package them in consistent and easy to use tools