Quality of Experience Past, Present and Future Trends

1

Quality of Experience
Past, Present
and Future Trends
Touradj Ebrahimi
Touradj.Ebrahimi@epfl.ch

Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt
in Multimedia Systems and Services 23rd November 2012, Austria

Today
we
will
talk
about… 2

•What
is
“quality”?
•Origins
of
“quality”
evalua7on
in
telecommunica7ons
•How
is
“quality”
measured
in
mul7media
communica7ons

today
?
•What
are
the
trends
on
“quality”
in
mul7media

communica7on?
•
What
are
the
challenges
ahead?


3
Quality: a simple yet difficult concept

• Like many human sensations, quality is easy to
understand but difficult to define

• Quality according to Wikipedia:
– A quality (from Latin - qualitas) is an attribute or a
property
– Some philosophers assert that a quality cannot be
defined


A fundamental, ancient, but largely under investigated concept 4

Aristotle classified every object of human
apprehension into 10 Categories
– Substance
– Quantity
– Quality
– Relation
– Place
– Time
– Position
– State
– Action
– Affection


5
Quality According to ISO 9000

• ISO 9000: a family of standards for quality
management systems
• Quality of something can be determined by
comparing a set of inherent characteristics with
a set of requirements
– High quality: if characteristics meet requirements
– Low quality: if characteristics do not meet all
requirements
• Quality is a relative concept
– Degree of quality


6
Qualinet white paper

• White Paper produced by COST Action
IC1003 (Qualinet):
– http://www.qualinet.eu/images/stories/
whitepaper_v1.1_dagstuhl_output_corrected.pdf
• Several definitions of quality in multimedia
systems and services and other related
concepts


7
Quality is like an elephant …

The blind men and the elephant: Poem by John Godfrey Saxe

8

Quality in
telecommunications


9
Quality
in
telecommunica4ons

Network Quality Link Quality
Capacity Bitrate
Coverage Frame/Bit/Packet loss
Handoff Delay

User Quality
Speech fidelity
Audio fidelity
Image fidelity
Video fidelity


10
Quality in telecommunications

Objective: input = output


11
QoS in telecommunications

• Quality of Service (QoS)
– Resource reservation control mechanisms
– Ability to provide different priority to different
applications or data flows
– Guarantee a certain level of performance (quality) to
a data flow
• Requires measuring or estimating quality
• (Service) Provider-centric concept
– Tightly related to the concept of Mean Opinion Score
(MOS)


12
Mean Opinion Score (MOS)?

• Widely used in many fields:
– Politics/Elections
– Marketing/Advertisement
– Food industry
– Multimedia
– …
• The likely level of satisfaction of a service or product as
appreciated by an average user in a typical use
• Should be performed such that it generates reliable and
reproducible results
– Subjective evaluation methodology
– More complex and difficult that it seems


13
What is behind a MOS?


14
What is wrong with MOS?
• Ground truth
– It is assumed to be the optimal solution (ground truth)
• Defies user preference
– It is assumed to be independent of specific subjects
• Ignores key parameters
– It is often assumed to be independent of content and
context
• Misuse as a comparison approach
– It is often used to compare performance between
alternative tools for a same task/operation
• Impractical or expensive
• Statistical assumptions not always realistic

15
Ground truth


16
Defies user preference


17
Content

p01 $$$$$$p06 $$$$$$$$$$p10 $$$$$$$$$$$$$$$bike$$$$$$$$$$$$$$$$$$cafe$$$$$$$$$$$$$$$woman$

18
Context


Objec4ve
quality
metrics 19

• Subjec7ve
tests
are
7me
consuming,
expensive,
diﬃcult
to

design,
and
cannot
be
used
in
real-‐7me
systems
• Objec7ve
algorithms,
i.e.
metrics,
es7ma7ng
MOS
with
high

level
of
correla7on
are
desired
• Full
reference
metrics
• No
reference
metrics
• Reduced
reference
metrics


Objec4ve
quality
metrics
based
on
ﬁdelity 20

• Full
Reference
scenario
• Metrics
which
look
at
the
ﬁdelity
of
the
signal
when

compared
to
an
explicit
reference:

processed
signal
=
perfect
quality
reference
signal
+
error
signal


Objec4ve
quality
metrics
based
on
ﬁdelity 21

• Examples
of
ﬁdelity
metrics
for
image
quality
quality

assessment
− Mean
Square
Error
(MSE)
− Peak
Signal
to
Noise
Ra7o
(PSNR)
− Maximum
Pixel
Devia7on
(Linf)
− Weighted
PSNR

− Masked
PSNR
− Structural
SIMilarity
(SSIM)
− Mul7scale
Structural
Similarity
(MSSIM)
− Visual
Informa7on
Fidelity
(VIF)
− etc…


22
What is wrong with objective quality metrics ?

• Poor performance in predicting MOS
– Especially in real life situations
• How to deal with color components?
– Even in PSNR!
• How to deal with video?
– Time versus spatial dimension
• How to take into account context?
– Display size/type
– Environmental illumination
– …
• How to deal with user preference/expectation?
• ...

MVC Coding Performance
23
MVC assessment using PSNR as metric
Sample comparison of simulcast vs inter-view prediction

8 views (640 480), consider rate for all views
~25% bit rate savings over all views

Ballroom Race1
40 42

39 41

40
38

39
37
38

PSNR (db)
PSNR (db)

36
37
35
36

34
35

33
Simulcast 34 Simulcast
32 MVC 33 MVC

31 32
0 200 400 600 800 1000 1200 1400 1600 1800 0 200 400 600 800 1000 1200 1400 1600

Bitrate (Kb/s) Bitrate (Kb/s)

CORESA
24-25 mai 2012, Lille, MPEG 3D Video Coding Activity
Vetro: Update on
France

MVC:assessment by subjective evaluation
MVC Subjective Performance 24

• Main finding: MVC achieves comparable quality to
simulcast with as little as 25% rate for dependent view

4.50
Mean Opinion Score

4.00

3.50

3.00

2.50

2.00

1.50

1.00

al C
) c t c t c t c t c t c t c t
in 0P 5P 5P 0P 5P 0P 5P
r ig AV _
O + _5 _3 _2 _2 _1 _1 L
VC 12
L
12
L
12
L
12
L
12
L
12
L 12
(A
st
ca
ul
m
Si
Base view fixed at 12Mbps
Dependent view at varying percentage of base view rate

CORESA Vetro: Update on MPEG 3D Video Coding Activity
24-25 mai 2012, Lille, France

25
Evolving Quality Paradigms


26
User centered evaluation
• It is becoming increasing important to evaluate quality from a user
perspective rather than from a provider/system perspective
• Constrains the tests to potential users and stimuli and evaluates the
acceptability with respect to the task and the context
• Identify underlying quality factors beside the overall quality

CORESA

27
What people mean when they say QoE ?

• “The degree of fulfillment of an intended experience
on a given user”
as defined by Touradj Ebrahimi, 2001

• “The overall acceptability of an application or service
… by the end user”
as defined by the ITU-T

• “The degree of delight ... of the user of an
application or service ... user’s personality and
current state.”
as defined in white paper by Qualinet

The term ‘experience’ promises individual engagement …

Quality
of
Service
vs
Quality
of
Experience 28

• Quality
of
Service:
Value
of
the
average
user’s

experience
es7mated
by
a
service/product/content

provider

• Quality
of
Experience:
Value
(es7mated
or
actually

measured)
of
a
speciﬁc
user’s
richness
of
experience

• Quality
of
Experience
is
the
dual
(and
extended)
view
of

QoS
problem

QoS=provider-‐centric
QoE=user-‐centric

29
A simple model for QoE

• User attributes
– individual attributes – expectation, age, sex, personality, background…
– sensorial attributes – including limitations and deficiencies
– perceptual attributes
– emotional attributes

• System attributes
– technical attributes (as in QoS) QoE
• Contextual attributes user
QoS
context
– environmental attributes
– device attributes
– service attributes
– content attributes


Personas and scenarios [Strohmeier2009] 30

• Personas (user preference)
– Archetypical user representing the needs, behaviors
and goals of a particular group of users
– Not real people but fictional characters derived from
user requirements
• Scenarios (context)
– Realistic usage environment

CORESA

31
Trends in QoE

• Digital world has (re-)discovered the notion of
quality
– Lower quality content is increasingly less tolerated by
end-users
– Digital technology have surpassed barriers of
attainable quality, while remaining cost effective
• Increasing interest in QoE
– Extending from device-centric and system-centric
quality optimization to end-to-end and especially
user-centric optimization


32
Trends in QoE community building

• Increased interest in workshops and conferences
around the notion of quality assessment and metrics
– QoMEX: International Workshop on Quality of Multimedia
Experience (http://www.qomex.org)
– VPQM: International Workshop on Video Processing and Quality
Metrics for Consumer Electronics (http://www.vpqm.org)
– …
• QoE is one of the issues referred to in research
programs funded by the EC
– Various calls in the recent year include QoE issues
– QoE is one of the top topics in future framework program
– Already ongoing initiatives such as COST Action IC1003
(Qualinet)


33
Trends in standardization

• Standardization efforts in quality assessment and metrics
– Video Quality Experts Group (VQEG)
– ITU-T SG 12 (Performance, QoS and QoE)
– JPEG (Advanced Image Coding - AIC)
– MPEG (HEVC, 3DVC)
– …


34
Challenges ahead
• Some key issues in QoE:
– Content-dependent quality assessment methods and metrics
– Context-dependent quality assessment methods and metrics
– Quality assessment methods and metrics beyond AV (haptics, …)
– Multi-modal quality assessment methods and metrics (AV, …)
– 3D quality assessment methods and metrics (3D sound, 3D video, …)
– HDR content quality assessment methods and metrics
– Interaction quality metrics (closely related to usability)
– Presence/immersion quality metrics
– …
• Need for Quality Certification Mechanisms of multimedia
services and products
– Similar in idea to ISO 9000 series


35
What does this all mean to you?

• Era of user-centric multimedia has already started
– It is not anymore sufficient to merely add new features and
functionalities to multimedia systems
– True added value in terms of impact on user experience of such
features and functions should be demonstrated
– Quality of Experience plays a central role in this new game

• Opportunities in research, technology, business, art and
entertainment


Thank you for your attention 36
Questions?


Quality of Experience Past, Present and Future Trends

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (10)

Similar to Quality of Experience Past, Present and Future Trends

Similar to Quality of Experience Past, Present and Future Trends (20)

More from Förderverein Technische Fakultät

More from Förderverein Technische Fakultät (20)

Recently uploaded

Recently uploaded (20)

Quality of Experience Past, Present and Future Trends