Recent advances in quality of experience in multimedia communication

IMTC 20th Anniversary Forum – Porto, Portugal1
Recent advances in
quality of experience in
multimedia
communication
Touradj Ebrahimi
Touradj.Ebrahimi@epfl.ch
International Multimedia Telecommunications Consortium

•What
is
“quality”?
•Origins
of
“quality”
evalua7on
in
telecommunica7ons
•How
is
“quality”
measured
in
mul7media
communica7ons

today
?
•What
are
the
trends
on
“quality”
in
mul7media

communica7on?
•
What
are
the
challenges
ahead?
Today
we
will
talk
about…

Quality: a simple yet difficult concept
• Like many human sensations, quality is easy to
understand but difficult to define
• Quality according to Wikipedia:
– A quality (from Latin - qualitas) is an attribute or a
property
– Some philosophers assert that a quality cannot be
defined

A fundamental, ancient, but largely under investigated concept
Aristotle classified every object of human
apprehension into 10 Categories
– Substance
– Quantity
– Quality
– Relation
– Place
– Time
– Position
– State
– Action
– Affection

Qualinet white paper
• White Paper produced by COST Action
IC1003 (Qualinet):
– http://www.qualinet.eu/images/stories/
whitepaper_v1.1_dagstuhl_output_corrected.pdf
• Several definitions of quality in multimedia
systems and services and other related
concepts

Quality is like an elephant …
The blind men and the elephant: Poem by John Godfrey Saxe

Quality in
telecommunications

Quality
in
telecommunica4ons
Network Quality
Capacity
Coverage
Handoff
Link Quality
Bitrate
Frame/Bit/Packet loss
Delay
User Quality
Speech fidelity
Audio fidelity
Image fidelity
Video fidelity

Quality in telecommunications
Objective: input = output

QoS in telecommunications
• Quality of Service (QoS)
– Resource reservation control mechanisms
– Ability to provide different priority to different
applications or data flows
– Guarantee a certain level of performance (quality) to
a data flow
• Requires measuring or estimating quality
• (Service) Provider-centric concept
– Tightly related to the concept of Mean Opinion Score
(MOS)

Mean Opinion Score (MOS)?
• Widely used in many fields:
– Politics/Elections
– Marketing/Advertisement
– Food industry
– Multimedia
– …
• The likely level of satisfaction of a service or product as
appreciated by an average user in a typical use
• Should be performed such that it generates reliable and
reproducible results
– Subjective evaluation methodology
– More complex and difficult that it seems

What is behind a MOS?

What is wrong with MOS?
• Ground truth
– It is assumed to be the optimal solution (ground truth)
• Defies user preference
– It is assumed to be independent of specific subjects
• Ignores key parameters
– It is often assumed to be independent of content and
context
• Misuse as a comparison approach
– It is often used to compare performance between
alternative tools for a same task/operation
• Impractical or expensive
• Statistical assumptions not always realistic

Ground truth

Defies user preference

Content
p01 $$$$$$p06 $$$$$$$$$$p10 $$$$$$$$$$$$$$$bike$$$$$$$$$$$$$$$$$$cafe$$$$$$$$$$$$$$$woman$

Context

Objec4ve
quality
metrics
• Subjec7ve
tests
are
7me
consuming,
expensive,
diﬃcult
to

design,
and
cannot
be
used
in
real-‐7me
systems
• Objec7ve
algorithms,
i.e.
metrics,
es7ma7ng
MOS
with
high

level
of
correla7on
are
desired
• Full
reference
metrics
• No
reference
metrics
• Reduced
reference
metrics

Objec4ve
quality
metrics
based
on
ﬁdelity
• Full
Reference
scenario
• Metrics
which
look
at
the
ﬁdelity
of
the
signal
when

compared
to
an
explicit
reference:

processed
signal
=
perfect
quality
reference
signal
+
error
signal

Objec4ve
quality
metrics
based
on
ﬁdelity
• Examples
of
ﬁdelity
metrics
for
image
quality
quality

assessment
− Mean
Square
Error
(MSE)
− Peak
Signal
to
Noise
Ra7o
(PSNR)
− Maximum
Pixel
Devia7on
(Linf)
− Weighted
PSNR

− Masked
PSNR
− Structural
SIMilarity
(SSIM)
− Mul7scale
Structural
Similarity
(MSSIM)
− Visual
Informa7on
Fidelity
(VIF)
− etc…

What is wrong with objective quality metrics ?
• Poor performance in predicting MOS
– Especially in real life situations
• How to deal with color components?
– Even in PSNR!
• How to deal with video?
– Time versus spatial dimension
• How to take into account context?
– Display size/type
– Environmental illumination
– …
• How to deal with user preference/expectation?
• ...

IMTC 20th Anniversary Forum – Porto, Portugal
MVC assessment using PSNR as metric
22 Vetro: Update on MPEG 3D Video Coding Activity
MVC Coding Performance
Ballroom
31
32
33
34
35
36
37
38
39
40
0 200 400 600 800 1000 1200 1400 1600 1800
Bitrate (Kb/s)
PSNR(db)
Simulcast
MVC
Race1
32
33
34
35
36
37
38
39
40
41
42
0 200 400 600 800 1000 1200 1400 1600
Bitrate (Kb/s)
PSNR(db)
Simulcast
MVC
Sample comparison of simulcast vs inter-view prediction
8 views (640 480), consider rate for all views
~25% bit rate savings over all views

MVC assessment by subjective evaluation
23
Vetro: Update on MPEG 3D Video Coding Activity
MVC: Subjective Performance
• Main finding: MVC achieves comparable quality to
simulcast with as little as 25% rate for dependent view
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
O
riginal
Sim
ulcast
(AVC+AVC
)
12L_50Pct
12L_35Pct
12L_25Pct
12L_20Pct
12L_15Pct
12L_10Pct
12L_5Pct
MeanOpinionScore
Base view fixed at 12Mbps
Dependent view at varying percentage of base view rate

Evolving Quality Paradigms

User centered evaluation
25
• It is becoming increasing important to evaluate quality from a user
perspective rather than from a provider/system perspective
• Constrains the tests to potential users and stimuli and evaluates the
acceptability with respect to the task and the context
• Identify underlying quality factors beside the overall quality

What people mean when they say QoE ?
• “The degree of fulfillment of an intended experience
on a given user”
as defined by Touradj Ebrahimi, 2001
• “The overall acceptability of an application or service
… by the end user”
as defined by the ITU-T
• “The degree of delight ... of the user of an
application or service ... user’s personality and
current state.”
as defined in white paper by Qualinet
The term ‘experience’ promises individual engagement …

Quality
of
Service
vs
Quality
of
Experience
• Quality
of
Service:
Value
of
the
average
user’s

experience
es7mated
by
a
service/product/content

provider

• Quality
of
Experience:
Value
(es7mated
or
actually

measured)
of
a
speciﬁc
user’s
richness
of
experience
• Quality
of
Experience
is
the
dual
(and
extended)
view
of

QoS
problem
QoS=provider-‐centric
QoE=user-‐centric

A simple model for QoE
• User attributes
– individual attributes – expectation, age, sex, personality, background…
– sensorial attributes – including limitations and deficiencies
– perceptual attributes
– emotional attributes
• System attributes
– technical attributes (as in QoS)
• Contextual attributes
– environmental attributes
– device attributes
– service attributes
– content attributes
user
QoS
QoE
context

Personas and scenarios [Strohmeier2009]
• Personas (user preference)
– Archetypical user representing the needs, behaviors
and goals of a particular group of users
– Not real people but fictional characters derived from
user requirements
• Scenarios (context)
– Realistic usage environment
29

Quality of Experience in Mobile Multimedia
• The influence of context in mobile multimedia
user experience and QoE
– Various probes to monitor user behavior
– Specific scenario to be followed by subjects
30
needed to
plexity of
was nec-
ur of the
quired to
ondly, as
environ-
around,
d robust
k system
he above
ontent to
eo player
k system
the video
he audio
ical user
following
Figure 1: Test equipment system installed on a user.
(a) (b)
The test equipment selected for the experiment needed to
satisfy two main constraints. Firstly, given the complexity of
the measurement task, a range of different devices was nec-
essary to capture the video consumption behaviour of the
users during the experiment. The devices were required to
work and interact with each other in real time. Secondly, as
the experiment was conducted in a non-laboratory environ-
ment and the users needed to carry the equipment around,
the devices were constrained to be lightweight and robust
enough to work under moving conditions.
The test equipment consisted of a video playback system
and a recording system selected with respect to the above
constraints. The former was used to show the content to
the user and capture information regarding the video player
status and the user location. The video playback system
consisted of the following items:
• a HTC Tattoo mobile phone used to display the video
content and an external headset to listen to the audio
• a standard button based video player graphical user
interface, as illustrated in Figure 2, with the following
functionalities:
– pause/play
– next channel
– previous channel
– menu to change channel
– standard volume controls
• a logging software to keep track of the controls acti-
vated at anytime by the user
• a GPS integrated in the mobile phone used to deter-
mine the position and the mobility scenario of the user.
The recording system facilitated the capture and storage
of video information recorded to determine the user’s visual
focus of attention. The recording system consisted of the
following items:
• a head mounted Logitech Webcam Pro 9000 (cam1) to
record the scene in front of the viewer
• a mobile device mounted Logitech Webcam Pro 9000
(cam2) to record the scene in front of the display of
Figure 1: Test equipment system installed on a user.
(a) (b)
(c) (d)
Figure 2: Video GUI for the following modi: (a)
play, (b) pause, (c) menu, and (d) volume control.
4.3 Data processing
The data acquired for each subject during the experiment
has been processed offline. The user attention has been de-
termined by applying an object detection algorithm [25], to
detect the markers in the two sequences. Particularly, the
markers were placed on cam1 and cam2 in such a way that,
when the user is watching at the mobile phone screen, the
scene recorded from cam1 will include the marker on top
of the mobile phone (Figure 3(a)). Vice versa, the scene
recorded from cam2 will include the frontal user’s face as
well as the marker on user’s head (Figure 3(b)). Addition-
(a) (b)
Figure 3: Sample frames of the video sequences
recorded by (a) cam1 and (b) cam2, when the user
is looking at the phone screen.
(a) (b)
Figure 4: Sample frames of the video sequences
recorded by (a) cam1 and (b) cam2, when the user
is not looking at the phone screen.
phone at that instant.
It is worth mentioning that we also tested a face detection
algorithm [26] to detect the face of the user in the video
recorded by cam2. Unfortunately, too many false positive
and false negative face detections were present in the results.
Particularly, false positive occurred when the face was not
frontal, even if the frontal face cascade classifier has been
used. Some false negative frontal face detections can be
explained by the bad quality of the video frames, due to
user’s motion and abrupt scene illumination changes.
Such frame quality degradations were found to create prob-
lems to the marker detection algorithm as well, however,
they were considerably fewer compared to the face detec-
tion. Furthermore, the marker detection did not show any
false positives. The false negative marker detections were
removed by applying a correction algorithm that is based
on the assumption that it is impossible for the user to look
away from the phone for a very short instance in time. To
be precise, if the marker was not detected for a succession of
k frames but it was detected in its m previous and n follow-
Figure 5: Path of the experiment reconstructed us-
ing the GPS data from one test session.
lecting information regarding the focus of attention of the
users.
The information regarding the detection of the marker
on the frames captured at each time instance from the two
cameras, together with the information regarding the video
player status at the same time instance, facilitates to identify
in which of the three consumption states (see Section 3.1)
the user is during the entire duration of the experiment. The
GPS data additionally allows to distinguish between the mo-
bility states (static or dynamic) based on a speed estimation
of the user. The path followed during the experiment, re-
constructed on the Earth map using the acquired GPS data
of one user, is shown in figure 5.

3D QoE in mobile multimedia
• Comparison of different methods
to produce 3D on mobile phones
- 2D
- Anaglyph
- Motion Parallax - Wiggle
- Autostereoscopic
31
(a) Overall quality (b) Perceived depth
Fig. 9. MOS/CI of the individual test conditions.
cessing Magazine, IEE
jan. 2011.
[3] M. Rerabek, L. Goldm
“Motion parallax bas
legacy consumer mob
nal Processing (MMSP
Workshop on. IEEE, 20
[4] P. Benzie, J. Watson
K. Hopf, H. Urey, V. S
survey of 3dtv display
Circuits and Systems fo
Fig. 2. Motion parallax based 3D restitution.
2.2.1. Multi-view motion parallax
The idea of portable device restitution method based on mo-
tion parallax employing the multi-view image data-set is il-
lustrated in Fig. 2. By rotating the portable device, the viewer
controls the view which is rendered on the display and so
he/she can observe the scene from different positions. This
way the viewer imitates a process similar to what he/she does
when observing the real world.
When the number of images in a stereoscopic or multi-
scopic image set is too small to achieve a smooth restitution,
intermediate images are generated using depth image based
rendering [9]. The depth estimation and view synthesis tools
of the 3D video coding (3DV) framework [10] developed by
MPEG are used for the content generation process within
our experiments. The depth estimation reference software
(DERS) uses three camera views (left, center, right) together
with the intrinsic and extrinsic camera parameters to estimate
the depth map of the center view. From the three operation
modes (automatic, segmentation, semi-automatic) of DERS,
the automatic mode has been used. Once the depth maps
were obtained the view synthesis reference software (VSRS)
was used to synthesize intermediate views. A virtual view is
generated based on two reference views with the correspond-
ing depth maps as well as the intrinsic and extrinsic camera
parameters. This is achieved by depth and texture mapping
and hole ﬁlling for each of the reference views followed by
Fig. 3. Content restitu
play.
corresponds to anothe
of a view depends sol
moves the display.
3. QU
3.1. Image data prep
A subset of the multi
Video (3DV) data-set
More speciﬁcally 4 s
Mobile) from the clas
one was used for train
Suitable frames h
each video. Using the
reference software (D
reference software (V
views were synthesize
mobile phone does no
MPEG tools, the resu
ther converted to high
tial resolution of the
match that of the por
Android based smart
Fig. 4. Creation of the individual test conditions from multi-
scopic data-set.
(a) Ballons

Trends in QoE
• Digital world has (re-)discovered the notion of
quality
– Lower quality content is increasingly less tolerated by
end-users
– Digital technology have surpassed barriers of
attainable quality, while remaining cost effective
• Increasing interest in QoE
– Extending from device-centric and system-centric
quality optimization to end-to-end and especially
user-centric optimization

Trends in QoE community building
• Increased interest in workshops and conferences
around the notion of quality assessment and metrics
– QoMEX: International Workshop on Quality of Multimedia
Experience (http://www.qomex.org)
– VPQM: International Workshop on Video Processing and Quality
Metrics for Consumer Electronics (http://www.vpqm.org)
– …
• QoE is one of the issues referred to in research
programs funded by the EC
– Various calls in the recent year include QoE issues
– QoE is one of the top topics in future framework program
– Already ongoing initiatives such as COST Action IC1003
(Qualinet)

Trends in standardization
• Standardization efforts in quality assessment and metrics
– Video Quality Experts Group (VQEG)
– ITU-T SG 12 (Performance, QoS and QoE)
– JPEG (Advanced Image Coding - AIC, HDR image compression)
– MPEG (HEVC, 3DVC)
– …

Challenges ahead
• Some key issues in QoE:
– Content-dependent quality assessment methods and metrics
– Context-dependent quality assessment methods and metrics
– Quality assessment methods and metrics beyond AV (haptics, …)
– Multi-modal quality assessment methods and metrics (AV, …)
– 3D quality assessment methods and metrics (3D sound, 3D video, …)
– HDR content quality assessment methods and metrics
– Interaction quality metrics (closely related to usability)
– Presence/immersion quality metrics
– …
• Need for Quality Certification Mechanisms of multimedia
services and products
– Similar in idea to ISO 9000 series

What does this all mean to you?
• Era of user-centric multimedia has already started
– It is not anymore sufficient to merely add new features and
functionalities to multimedia systems
– True added value in terms of impact on user experience of such
features and functions should be demonstrated
– Quality of Experience plays a central role in this new game
• Opportunities in research, technology, business, art and
entertainment

Thank you for your attention
Questions?

Recent advances in quality of experience in multimedia communication

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (6)

Destacado

Destacado (20)

Similar a Recent advances in quality of experience in multimedia communication

Similar a Recent advances in quality of experience in multimedia communication (20)

Más de IMTC

Más de IMTC (20)

Último

Último (20)

Recent advances in quality of experience in multimedia communication