Quality of Experience in Multimedia Systems and Services: A Journey Towards the Quality of Life

Quality of Experience in
Multimedia Systems and Services:
A Journey Towards the Quality of Life
Christian Timmerer(AAU Klagenfurt, Austria)
Fernando Pereira (IST-IT, Portugal)
Touradj Ebrahimi (EPFL, Switzerland)
IEEE International Conference on Multimedia& Expo (ICME)
11th July 2016, Seattle, WA, USA

Outline
1. Quality of Experience for Multimedia
Systems and Services
Fernando Pereira
2. Applications of QoE: Adaptive Video
Streaming and Sensory Experience
Christian Timmerer
3. Towards the Concept of Quality of Life
Touradj Ebrahimi

Fernando Pereira: About Me …
• Associate Professor at University of Lisbon, Portugal
• Senior Researcher at Instituto de Telecomunicações, Lisbon, Portugal
• More than 250 publications in international journals and conferences
• One of the designers of the MPEG-4 and MPEG-7 standards
• ICIP, PCS, VCIP, WIAMIS, QoMEX General or Technical Program Chair
• Associate Editor of several journals
• Editor-in-Chief of the IEEE Journal of Selected Topics in Signal Processing (2013-2015)
• ISO/IEC Award for contributions to the MPEG-4 Visual Standard
• SPS Distinguished Lecturer
• IEEE Fellow in 2008 for “contributions to object-based digital video representation
technologies and standards”
• EURASIP Fellow in 2013 for “contributions to digital video representation technologies
and standards”
• IET Fellow in 2015
• IEEE SPS Board of Governors and EURASIP Board of Directors
• Several Excellence Teaching Awards
• JPEG (currently) and MPEG (past) Requirements Chair

1. Quality of Experience for Multimedia
Systems and Services
A. What is Quality
B. Quality of Service (QoS)
C. Quality of Experience (QoE)
D. Trends in QoE

Quality: a Simple yet Difficult Concept
• Like many human sensations, quality is easy to
understand but difficult to define.
• According to Wikipedia:
– A quality (from Latin - qualitas) is an attribute or a property.
– Some philosophers assert that a quality cannot be defined.
– In contemporary philosophy, the idea of qualities and
especially how to distinguish certain kinds of qualities from
one another remains controversial.

An Old, Largely Under-Investigated Concept
Aristotle classified every object of
human apprehension into 10 Categories
– Substance
– Quantity
– Quality
– Relation
– Place
– Time
– Position
– State
– Action
– Affection
Aristotle, 384 BC – 322 BC, Greece

Quality: Some Definitions from the Dictionary (1)
• Definition 1
– General : Measure of excellence or state of being free from
defects, deficiencies, and significant variations.
– ISO 8402-1986 standard defines quality as “the totality of
features and characteristics of a product or service that
bears its ability to satisfy stated or implied needs”.
• Definition 2
– Manufacturing : Strict and consistent adherence to
measurable and verifiable standards to achieve uniformity
of output that satisfies specific customer or user
requirements.

Quality: Some Definitions from the Dictionary (2)
• Definition 3
– Objective : Measurable and verifiable aspect of a thing or
phenomenon, expressed in numbers or quantities, such as
lightness or heaviness, thickness or thinness, softness or
hardness.
• Definition 4
– Subjective : Attribute, characteristic, or property of a thing
or phenomenon that can be observed and interpreted, and
may be approximated (quantified) but cannot be
measured, such as beauty, feel, flavor, taste.

Quality According to ISO 9000
• ISO 9000: a family of standards for quality management
systems.
• Quality of something can be determined by comparing a set of
inherent characteristics with a set of requirements
– High quality: if characteristics meet requirements
– Low quality: if characteristics do not meet all requirements
• Quality is a relative concept
– Degree of quality

Quality is like an Elephant …
The blind men and the elephant, poem by John Godfrey Saxe

Quality in QUALINET
• Quality: Is the outcome of an individual’s comparison and judgment
process. It includes perception, reflection about the perception, and the
description of the outcome.
• In contrast to definitions which see quality as “qualitas”, i.e. a set of
inherent characteristics, QUALINET considers quality in terms of the
evaluated excellence or goodness, of the degree of need fulfillment, and in
terms of a “quality event” (see Martens & Martens, 2001, and Jekosch,
2005).
• Event: An observable occurrence. An event is determined in space (i.e.
where it occurs), time (i.e. when it occurs), and character (i.e. what can be
observed).
from “Qualinet White Paper on Definitions of Quality of Experience”, March 2013

What is QUALINET ?
• Group of institutions and companies interest in multimedia quality.
• Coordination of multidisciplinary QoE research in Europe and beyond.
• Strengthening dissemination efforts through already established, and new
initiatives, e.g. QoMEX, special events, books, journals.
• Strengthening interaction between academia and industry (industrial
forum, STSM, …).
• Strengthening educational efforts in QoE, e.g. summer schools, PhD
events, exchange of young researchers.
• Coordinated contribution to international standardization bodies, e.g.
ISO/IEC, ITU-T, VQEG, MPEG, JPEG.
In summary, group of researchers interested in (multimedia) QoE
issues, both theoretical and practical …
Open to new researchers … http://www.qualinet.eu/

Quality of Service (QoS): in Theory
“[The] Totality of characteristics of a telecommunications service
that bear on its ability to satisfy stated and implied needs of
the user of the service.”
ITU-T Rec. E.800, 2008
• QoS is focused on telecommunications services.
• The context of usage and the user characteristics are not
comprehensibly addressed by QoS as defined by the ITU.

Quality of Service (QoS): De-Facto
• The QoS de-facto definition deals mostly with physical, measurable
performance factors of networks and delivery platforms in general.
• Sometimes, also application-level factors, such as encodings, and
their effect on the underlying network's performance are addressed,
e.g. by taking more of the available bandwidth.
Quality of Service (QoS) refers to a collection of networking
technologies and measurement tools that allow for the
network to guarantee delivering predictable results.
partly from “Qualinet White Paper on Definitions of Quality of Experience”, March 2013

Quality in QoS Framework: Several Dimensions
Network Quality
Capacity
Coverage
Handoff
Link Quality
Bitrate
Frame/Bit/Packet loss
Delay
User Quality
Speech fidelity
Audio fidelity
Image fidelity
Video fidelity
The multimedia signal processing community is already
often using concepts such as the Mean Opinion Score
(MOS) which directly involves users …

QoS in Computer Networks and Communications
• Quality of Service (QoS)
– Resource reservation control mechanisms
– Ability to provide different priority to different
applications, users, or data flows
– Guarantee a certain level of performance (quality) to a
data flow
• (Service) Provider-centric concept

QoS Boundaries
• Scope: QoS typically focuses on telecommunications services.
• Focus: QoS deals with performance aspects of physical
systems.
• Methods: QoS has a very technology-oriented approach, and it
relies on analytic approaches and empirical or simulative
measurements.

User Quality: Mostly Signal Fidelity
• Subjective
Evaluation
• Objective
Evaluation

Subjective Evaluation
• Subjective tests aim at producing User Opinion Scores as a
delicate mixture of ingredients and choices:
– Test & lab environment
– Test material
– Test methodology
– Test subjects
– Analysis of the data

What is Mean Opinion Score (MOS)?
• Widely used in many fields:
– Politics/Elections
– Marketing/Advertisement
– Food industry
– Multimedia
– …
• The likely level of satisfaction of a specific service/product
dimension, e.g. visual quality, as appreciated by an averageuser
(from a provider point of view).
• Should be performed such that it generates reliable and
reproducible results
– Subjective evaluation methodology
– More complex and difficult that it a priori seems
– Much used for (and limited to) video and audio subjective qualities

Objective Evaluation
• Subjective tests are time consuming, expensive, and difficult
to design …
• Objective algorithms, i.e. metrics, estimating subjective MOS
with high level of correlation are desired
– Full reference metrics
– No-reference metrics
– Reduced reference metrics

FR, RR and NR Scenarios
• Full Referenceapproach:
• Reduced Reference approach:
• No-Reference approach:
Input/Reference
signal
Output/Processed
signal
Signal
processing
Input/Reference
signal
Output/Processed
signal
Signal
processing
FR METRIC
NR METRIC
Input/Reference
signal
Output/Processed
signal
Signal
processing
Features
extraction
RR METRIC

Automatic MOS Predictors are Essential …
Full Reference scenario
• Most automatic MOS predictors are based on fidelity measures
• Metrics look at the fidelity of the signal when compared to an
explicit ´perfect’ reference:
processed signal = perfect quality reference signal + error signal
• Examples:
– Mean Square Error (MSE)
– Peak Signal to Noise Ratio (PSNR)
– Weighted PSNR
– Masked PSNR
– Structural SIMilarity (SSIM)
– Multiscale Structural SIMilarity (MSSIM)
– Visual Information Fidelity (VIF)

UHD, 4K
HDR
HFR
3D
Light fields
Point clouds
…

Many Events ... Building Experiences ...
• Event: An observable occurrence. An event is determined in
space (i.e. where it occurs), time (i.e. when it occurs), and
character (i.e. what can be observed).
– Sensation refers to the responses of sensory receptors and sense organs to
environmental stimuli.
– Perception is a process which involves the recognition and interpretation of
stimuli which register our senses.
– Emotion is any relatively brief conscious experience characterized by intense
mental activity and a high degree of pleasure or displeasure.
• Experience: An experience is an individual’s stream of
perception and interpretation of one or multiple events.
partly from “Qualinet White Paper on Definitions of Quality of Experience”, March 2013

So, Users are More than Perception Engines …

Many Services Sell Emotions ...

Multimedia Nowadays …
• Multimediais about sharing experiences
(real or imaginary) with others.
• In a way, it all started with story telling and
wall drawing around the fire in the caves
of early men.
• Modern multimedia systemsare evolved
versions of the good old story tellingand
wall drawing, which hopefullyoffer
increasingly richer experiences.
• The degree of richness of the experience
may be measured by Quality of Experience
(QoE).

What do People Talk about when they Talk about QoE ?
• “The degree of fulfillment of an intended experienceon a given
user.”
by Touradj Ebrahimi, 2001
• “perceived user experienceis psychological in nature and changes in
different environmental conditions and with different multimedia
devices.”
from QoMEX 2009 Call for Papers
• “The overall acceptability of an application or service, as perceived
subjectively by the end user.”
as defined by the ITU
The term ‘experience’ is appealing because it implicitly
promises individual engagement … Look good, sound good,
and feel good !

QUALINET QoE Definition
• Quality of Experience (QoE) is the degree of delight or
annoyance of the user (persona) of an application or service. It
results from the fulfillment of his or her expectations with
respect to the utility and/or enjoyment of the application or service
in the light of the user’s personality and current state (context).
• Experience: An experience is an individual’s stream of perception and
interpretation of one or multiple events.
• QoE feature: A perceivable, recognized and namable characteristic of the
individual’s experience of a service which contributes to its quality.
In the context of communication services, QoE can be influenced by factors
such as service, content, network, device, application, and context of use.

Moving to Quality of Experience
• Quality of Service: Value of the average user’s service
richness estimated by a service/product/contentprovider
• Quality of Experience: Value (estimated or actually measured)
of a specific user’s experience richness
Quality of Experience is the dual (and extended) view of
Quality of Service !
QoS=provider-centric
QoE=user-centric

Factors Impacting Quality of Experience
Context
• System/Techni
cal Influence
Factors
• Human/User
Influence
Factors
• Context
Influence
Factors
• Content
Influence
Factors
• Social and
Psychological
Influence
Factors

Experiences are Individual !
• Applications and Services may have to be designed to provide
individual experiences ...
• This involves capabilities allowing the user to gain control, e.g.
interaction, personalization, recommendation, etc.
• In fact, the user contributes to build is own experience ... If
the system/service allows ...

How Shall a Multimedia User Experience Be ?
Depending on the specific application, it may have to be
• Faithful - accuracy
• Truthful – realistic if relevant, synchronization
• Immersive – natural, multimodal consistency
• Contextual - adaptive
• Engaging – fun, intense, emotional
• Effective – fast, recognition
• Useful – task performing
• Interactive – natural, short delay
• Intuitive, Easy – interfaces
• …

QoE Modeling
QoE modelingmay consider more or less
influence factors dependingon the
service/application, each with a different
weight on the overall assessment.
QoE is multi-dimensional, multi-modal and
multi-sensorial.
User centered influence factors are expected
to be dominating.
• System factors
– technical properties (as in
QoS)
• Human/User factors
– individual properties
– sensorial properties
– perceptual properties
– emotional properties
• Context factors
– environmental/physical
properties
– temporal properties
– service properties
– economic properties
– social properties
• Content factors
• …

A Practical QoE Model Example: IPTV
• Video quality
• Audio quality
• Audiovisual syncronization
• Stall occurence
• Error resilience
• Random access
• Channel hoppingdelay
• Interface usability
• Navigation capabilities
• Personalization capabilities
• Metadata quality
• Immersion effectiveness
• …

QoE: Not an Easy Target … Why Should it be ?
QoS
QoP
QoE
QoS/P/E: Quality of Service/Perception/Experience

Experiences are multisensorial ...

Building Multisensorial Immersion ...
• To insert Marianna’s movie ...
Feel-around, from Kentucky Fried Movie

QoE Assessment: Again Subjective and
Objective

QoS versus QoE
• Scope: QoS typically focuses on telecommunications services, whereas
QoE covers a much broader domain, which sometimes does not even
involve telecommunications.
• Focus: QoS deals with performance aspects of physical systems,
whereas QoE deals with the users' assessment of system
performance, as colored by context, culture, the users' expectations
with respect to the system or service and their fulfillment, socio-
economic issues, and psychological profiles, among other factors.
• Methods: QoS has a very technology-oriented approach, whereas QoE
requires a multi-disciplinary and multi-methodological approach for its
understanding.
• But it is also important to remember that QoE is, in a large part of
instances, highly dependent on QoS.

QoE is like a (Bigger) Elephant …
The blind men and the elephant, Poem by John Godfrey Saxe

QoE Related Standardization Efforts
• Standardization efforts in quality assessment and metrics
– ITU-T SG 12 (Performance, QoS and QoE)
– MPEG/ITU-T (High Efficiency Video Coding, HEVC)
– MPEG (3D video coding, FTV, HDR)
– Video Quality Experts Group (VQEG)
– JPEG (Advanced Image Coding, AIC)
– …
QUALINET established links and deep collaborations with all of
them !

QoE is Becoming Inevitable …
• Digital world has (re-)discovered the notion of quality
– Lower quality content is less and less tolerated by end-users in
some environments
– However, other environments seem to accept much lower
quality and still be successful
• Increasing interest in QoE
– Extending from device-centric and system-centric quality
optimization to end-to-end and especially user-centric
optimization

QoE Holistic Approach
• Marketing
• Business model, e.g. prices, fidelization
• System factors
• Context factors
• Human factors
• Personalization
• Content (and metadata)
• Interface
• Client support
• ...

NOS UMA: an Example
• Ultra HD 4K
• Portability accross terminals, i.e. follows you
• Voice control (voice recognition ?), i.e. recognizes you
• User profiles within same family, i.e. individualizes you
• Recommendations based on user characterization, i.e.
targets you
• Complementary content for the favourite series, i.e. thinks
on you
• Time warping, i.e. helps you
• ...
A TV that
knows you !

QoE in Industry
• QoE is becoming mainstream.
• Many companies now speak about QoE.
• Personalization, interaction and recommendation capabilities
empower the user to create more individual experiences!
• However, QoE has a budget impact in terms of network and
system design, dimensioning, operation, maintenance, etc.
• But QoE is becoming more affordable in many application
domains …
• Embracing QoE principles may bring revenue, e.g. by
increasing viewing times and reducing churn.

Challenges Ahead
• Content-dependentqualityassessmentmethods and metrics
• Context-dependentqualityassessmentmethods and metrics
• Quality assessmentmethods and metrics beyond AV (haptics, smell,
…)
• Multi-modal quality assessmentmethods and metrics (AV, …)
• 3D quality assessmentmethods and metrics (3D sound, 3D video, …)
• New modalities content quality assessmentmethods and metrics
• Interaction qualitymetrics (closely related to usability)
• Presence/immersion qualitymetrics
• Role of emotions
• Virtual reality immersive experiences
• …

A Light Field Image …
Behind each microlens, a micro-image (MI) is formed …

Light Field Photography: Array of Cameras

New Displays
Microsoft Hololens
Oculus Rift
Microsoft
holographic display
InnoVision Diamond Series
holographic projector
USC light field display
Holografika HoloVizio
light field display

QoE for Virtual Reality
• Compelling immersive and realistic visual experiences !
• Provides visual depth cues, such as stereopsis, binocular occlusions,
vergence, full motion parallax and natural view-dependent lighting.
• High resolution and high frame rate.
• Low latency spatial random access.
• Low motion-to-photon latency.
On current HMDs, the closest
depth for an object of interest
is recommended to be at
0.75m without causing
excessive eyestrain.

Holoportation: Virtual 3D Teleportation
Courtesy of P.Chou, Microsoft

What Does this all Mean ?
• Era of user-centric multimedia has already started …
User is King/Queen !
• It is not anymore sufficient to merely add new features
and functionalities to multimedia systems.
• True added value in terms of impact on user’s
experience of such features and functions should be
evaluated and demonstrated.
• Quality of Experience plays a central role in this new
game ! Already targeting revenue …

Assessing Quality of Experience … A Bit
Like Measuring ‘Happiness’ …

Take-Home Messages
• QoE is user-centric !
• QoE is individual, multidimensional and multisensorial.
• Services and systems are increasingly designed to
allow the users to maximize its QoE.
• Industry is increasingly embracing QoE principles
because they may bring revenue.
• QoE assessment is costly and risky but worth doing it.
• Int’l Conference on Quality of Multimedia Experience
(QoMEX): http://qomex.org/

2. Applications of QoE: Adaptive Video
Streaming and Sensory Experience
A. Adaptive Video Streaming Principles and QoE
B. Quality of Sensory Experience (QuaSE)

Christian Timmerer: About Me …
• Associate Professor at Alpen-Adria-Universität Klagenfurt, Austria
(blog.timmerer.com, dash.itec.aau.at)
• Chief Innovation Officer | Head of Standardization and Research at
Bitmovin Inc., bitmovin.com
• Geschäftsführer Förderverein Technische Fakultät, ftf.or.at
• Lecturer | Carinthia University of Applied Sciences, www.fh-kaernten.at
• Research interest: immersive multimedia communication, streaming, adaptation, Quality
of Experience, and sensory experience
• More than 170 publications in international journals and conferences
• General chair: WIAMIS’08, QoMEX’13, QCMan’14, MMSys’16
• Associate editor/editorial board: IEEE Computer, IEEE Trans. on Multimedia, Signal
Processing: Image Communication, MTAP, IEEE Computing Now, ACM SIGMM Records,
ACM SIGMM OSSC
• Vice chair of IEEE ComSoC MMTC, WG leader in QUALINET
• Research projects: FP6-IST-DANAE (2004-2006), FP6-IST-ENTHRONE (2006- 2008), FP7-
ICT-P2P-Next (2008-2012), FP7-ICT-ALICANTE (2010-2013), FP7-ICT-SocialSensor (2010-
2014), COST-IC1003-Qualinet (2010-2014), FFG-AdvUHD-DASH (2014-2016), and FP7-ICT-
ICoSOLE (2013-2016)
• MPEG: MPEG-21, MPEG-M, MPEG-V, MPEG-DASH
• IEEE Senior member; ACM member

Applications of QoE: Adaptive Video Streaming and
Sensory Experience
Priv.-Doz. Dr. Christian Timmerer
[Ack: Ali C. Begen, MediaMelon, Inc., OzyeginUniversity]
Alpen-Adria-Universität Klagenfurt (AAU) w Faculty of Technical Sciences (TEWI) w Department of Information
Technology (ITEC) w Multimedia Communication (MMC) w Sensory Experience Lab (SELab)
http://blog.timmerer.com w http://selab.itec.aau.at/ w http://dash.itec.aau.at w christian.timmerer@itec.aau.at
Chief Innovation Officer (CIO) at bitmovin GmbH
http://www.bitmovin.comw christian.timmerer@bitmovin.com
Tutorial @ ICME 2016, July 2016
http://www.slideshare.net/christian.timmerer

Importance of Multimedia Delivery
• Multimedia is predominant on
the Internet
• Real-time entertainment
– Streaming video and audio
– More than 70% of Internet traffic
at peak periods
• Popular services
– YouTube (17.85%), Netflix
(37.05%), Amazon Video (3.11%),
Hulu (2.58%)
– All delivered over-the-top (OTT)
July 2016 ICME 2016 Tutorial, C. Timmerer 2
Global Internet Phenomena Report: Dec 2015

Open Digital Media Value Chain
Create
Content
Aggregate
Monetize
Distribute
Content
Consume
Content
Any Content Any Storefront Any Network Any Device
CDNsMedia
Protocols
Internet
Transport
DRM
Encoding
Encapsulation
Dynamic
Ads
Clients
Happy User

Common Annoyances in Streaming
• Wrong format
• Wrong protocol
• Plugin requirements
• DRM issues
• Long start-up delay
• Poor quality
• Frequent stalls
• Quality oscillations
• No seeking features

Over-The-Top – Adaptive Media Streaming
• In a nutshell…
Adaptation logic is within the
client, not normatively
specified by the standard,
subject to research and
development

Multi-Bitrate Encoding and Representation Switching
Contents on the Web Server
Movie A – 200 Kbps
Movie A – 400 Kbps
Movie A – 1.2 Mbps
Movie A – 2.2 Mbps
. . .
. . .
Movie K – 200 Kbps
Movie K – 500 Kbps
Movie K – 1.1 Mbps
Movie K – 1.8 Mbps
. . .
. . .
Time (s)
Start quickly
Keep requesting
Improve quality
Loss/congestion detection
Revamp quality
. . .
. . .
Segments

Adaptive Streaming over HTTP
…
…
…
…
HTTPGETs
Client
Buffer
Media
Player
HTTP
Server

Scope of DASH: what is specified?
Media Presentation on
HTTP Server
DASH-enabled ClientMedia Presentation
Description
.
.
.
Segment
…
.
.
.Segment
…
.
.
.
Segment
…
.
.
.Segment
…
…
Segments located
by HTTP-URLs
DASH Control Engine
HTTP/1.1
HTTP
Client
MPD
Parser
Media
Engine
On-time HTTP
requests to
segments

DASH Data Model
MPD
Period id = 1
start = 0 s
Period id = 3
start = 300 s
Period id = 4
start = 850 s
Period id = 2
start = 100 s
Adaptation Set 0
subtitle turkish
Adaptation Set 2
audio english
Adaptation Set 1
BaseURL=http://abr.rocks.com/
Representation 2
Rate = 1 Mbps
Representation 4
Rate = 3 Mbps
Representation 1
Rate = 500 Kbps
Representation 3
Rate = 2 Mbps
Resolution = 720p
Segment Info
Duration = 10 s
Template:
3/$Number$.mp4
Segment Access
Initialization Segment
http://abr.rocks.com/3/0.mp4
Media Segment 1
start = 0 s
Media Segment 2
start = 10 s
Adaptation Set 3
audio german
Adaptation Set 1
video
Period id = 2
start = 100 s
Representation 3
Rate = 2 Mbps
Selection of
components/tracks
Well-defined
media format
Selection of
representations
Splicing of arbitrary
content like ads
Chunks with addresses
and timing

type=static typically,
for on demand content
Base URL of the
segments
Subtitles
Audio adaptation set
with different
representations (bw)
Video adaptation set
with different
representations (bw)
Different codecs
(profiles)
Segment URL constructed
with template and base
URL

http://www.dash-player.com/demo/

Adaptive Streaming Content Workflow
Source Transcoding Encapsulation Encryption
Origin
Server
HelperDistribution
Client
Linear: Multicast
VoD: FTP, RTMP, HTTP, etc.
Unicast HTTP (PUSH), FTP, etc.
HTTP GET small objects
Single highest-bitrate
stream
Multiple streams at
target bitrates
Multiple streams at
target encapsulation formats
Large video/virtual
files and manifests

Adaptive Streaming Content Workflow Simplified
Standard Delivery
Infrastructure (CDN)
Source Transcoding Encapsulation Encryption
Multiple streams: video [bitrate (32000-20000000), profile (baseline,
main, high), preset (standard, professional, premium), height (128-
7680), width (96-4320), frame rate (1-120), codec (h264, hevc)],
audio: [bitrate (8000-256000), sample rate (0, 8000, 11025, 12000,
16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000)]
Single highest-bitrate stream:
HTTP, FTP, RTMP; mp4, ts; AVC,
AAC, Subtitles
Multiple streams at target
encapsulation formats: DASH
(MPD + mp4), HLS (m3u8, ts)
Multiple streams with multiple DRM
formats: MPEG-CENC, Widewine,
PlayReady, PrimeTime, Fairplay
Player
Heterogeneous Clients e.g.
Bitmovin HTML5 Adaptive Player
DASH, HLS, HTML5, MSE, EME

QoE for DASH Services
• Different application domains have different QoE
requirements
– Need to provide specializations of the general QoE
definition
– Take into account requirements formulated by means of
influence factors and features of QoE
• QoE influence factors for DASH
– Initial/start-up delay (low)
– Buffer underruns, stalls, freezes (zero)
– Quality switches (low)
– Media throughput (high)
– …

!
h t t p s : / / b i t m o v i n . c o m /
QoE Evaluation for DASH-based Services
• Test sequence
– Many datasets available
– Adopted Big Buck Bunny & DASHed it
• Players
– bitdash
– Proprietary solutions (smooth, HLS, HDS)
– YouTube, dash.js, DASH-JS
– …and compare it with ten different adaptation algorithms
• Objective evaluation
– Common test setup using network emulation & bandwidth shaping
– Predefined bandwidth trajectory (or real network traces)
• Subjective evaluation
– Lab [ITU-T B.500 / P.910] vs.
crowdsourcing with special
platforms or social networks

Crowdsourced QoE Evaluation
• Quality of Experience …
– Mean Opinion Score [0..100]
– [other objective metrics:
start-up time, throughput, stalls]
• … Web-based Adaptive HTTP Streaming Clients …
– HTML5+MSE: DASH-JS (dash.itec.aau.at), dash.js (DASH-IF, v1.1.2), YouTube
• … Real-World Environments …
– DASH-JS, dash.js hosted at ITEC/AAU (~ 10Gbit/s)
– YouTube hosted at Google data centers
– Content: Tears of Steel @ 144p (250 kbit/s), 240p (380 kbit/s), 360p (740 kbit/s), 480p (1308 kbit/s), and
720p (2300 kbit/s); segment size: 2s
– Users access content over the open Internet (i.e., real-world environment)
• … Crowdsourcing
– Campaign at Microworker platform (others also possible: Mechanical Turk, social networks) limited to
Europe, USA/Canada, India
– Screening Techniques: Browser fingerprinting, stimulus presentation time, QoE ratings and pre-
questionnaire
B. Rainer, C. Timmerer, “Quality ofExperience ofWeb-based Adaptive HTTP Streaming Clients in Real-World Environments using Crowdsourcing”,
Proceedings of International Workshop on VideoNext: Design,Quality andDeployment of Adaptive Video Streaming, Sydney, Australia,Dec. 2014.

MOS and Average Bitrate
• 288 microworkers, 33 screened (Fingerprinting: 20, presentation time: 6, QoE
ratings and pre- questionnaire: 7), 175 male and 80 female, majority (80%) is aged
between 18 and 37

Startup Time and Number of Stalls

Results Summary
• DASH-JS
– High start-up time
– Low number of stalls
– Good throughput, QoE
• dash.js
– Low start-up time
– High # stalls
– Low throughput
– Low QoE
• YouTube
– Low start-up time
– Low number of stalls
– Best throughput, QoE

Now, 10 different
adaptation logics …
Adaptation logics well-known in
research literature
Predefined bandwidth trajectory and
test setup
Different segment sizes, RTTs, HTTP/2,
etc.
C. Timmerer, M. Maeiro, B. Rainer, “Which Adaption
Logic? An Objective and Subjective Performance
Evaluation of HTTP-basedAdaptiveMedia Streaming
Systems”, arXiv cs.MM, June 2016,
http://arxiv.org/abs/1606.00341.

DASH-JS vs. bitdash
C. Timmerer, D. Weinberger, C. Mueller, and S. Lederer, “Ultra-High-Definition-Quality of Experience with MPEG-DASH”,
Proceedings of the Broadcast EngineeringConference (BEC), NAB2015, Las Vegas, NV, USA, April 2015.

Objective Evaluations
Stalls (lower is better)Average Bitrate (higher is better)

Stalls are really bad…
Conviva: Viewer Experience Report. 2014

Conclusions (1)
• MPEG-DASH defines formats only
– Media Presentation Description (MPD)
– Segment format: mp4, ts
• MPEG-DASH is not
– System, protocol, presentation, codec, interactivity, DRM, client specification
– Other standards required for a complete ecosystem: e.g., DASH-IF, WAVE,
HMTL5, MSE, EME
• Do we need MPEG-DASH? (for adaptive media streaming)
– Not necessarily: e.g., WebM + VPx + manifest & control end-to-end
– Required to address heterogeneous environments
• Role of standards sometimes overrated but often underestimated

Conclusions (2)
• QoE for DASH-based services (a rule of thumb)
– Startup delay (low [but live vs. on-demand & short vs. long-tail
content])
– Buffer underrun / stalls (zero)
– Quality switches (low) and media throughput (high)
– Energy- and cost-awareness (data plan)
• No general applicable QoE model for DASH
– (Too) many factors influencing / features of QoE for DASH-based
services
– Methodology for reproducible research is in place and well established
– Ample research opportunities
Main QoE
factors for DASH
Come up with our own QoE factor and design, conduct, analyze a small-scale experiment

Quality of Sensory Experience
• Consumption of multimedia content may stimulate also other senses
– Vision or hearing
– Olfaction, mechanoreception, thermoception, …
• Annotation with metadata providing so-called sensory effects that steer
appropriate devices capable of rendering these effects
… giving her/him the sensation of being part
of the particular mulsemedia
worthwhile, informative user experience

General Principle – Outline
• General principle: there is a need for a scientific
framework to capture, measure, quantify, judge,
and explain the quality of (sensory) experience
• Outline
– [How to create, delivery, consume?]
– How to capture and measure?
– How to quantify?
– How to judge and explain?

How to create, delivery, consume?
• Sensory Effect Description Language (SEDL)
– Basic building blocks to describe, e.g., light, wind, fog, vibration, scent
– MPEG-V Part 3, Sensory Information: Effects, GroupOfEffects
– Adopted MPEG-21 DIA tools for adding time information (synchronization)
• Description conforming to SEDL :== Sensory Effect Metadata (SEM)
– Can be associated to any kind of multimedia content (e.g., movies, music, Web
sites, games)
– Support to be included in file (MP4) and transport (M2TS) formats
• Tool support for creating (annotation tools) and consumption (players,
Web plugins) selab.itec.aau.at
• Devices: e.g., amBX (Ambient Experience) system + SDK,
Gameskunk, Scentscape, etc.

How to capture and measure?
• Subjective quality assessments
– Methodology: based on standard methods
– Test content: different genres, manually annotated (cf. QUALINET DB)
• Experiment I
– Aim: Demonstrate sensory effects as a vital tool for enhancing the quality of experience
depending on the actual genre
• Experiment II
– Aim: investigate the relationship of the QoE to various video bit-rates of multimedia contents
annotated with sensory effects.
– Subjective quality gap between video resources annotated with and without sensory effects at
different bit-rates
• [Experiment III] ambient lights & different color calculation settings
• Experiment IV
– Aim: investigate the enhancement of the QoE and how users’ emotions are elicited and
influenced by Web videos annotated with and without sensory effects

Experiment II: Results
Sequence Babylon A.D. Earth
Duration 35s 21s
Resolution 1280 x 544 1280 x 720
Motion High Low
Nr. of Effects W: 7; V: 9 W: 8; V: 1
Bit-rates Kbit/s PSNR Kbit/s PSNR
Low Quality 2154 38.93 2204 38.11
Medium Quality 3112 41.27 3171 40.65
High Quality 4044 42.95 4116 42.27
Highest Quality 6315 N/A 6701 N/A
Test Sequences
MOS vs. PSNR/bit-rate for Earth.

How to quantify?
• Experiment V
– Aim: towards a quality/utility model for QuaSE
• Stimuli with all combinations
of sensory effects
– Vibration higher impact than
light & wind
– Highest QoE with all effects
present
• General QuaSE model

How to judge and explain?
• Experiment VI
– Aim: understand QuaSE
• Biosensor-based QoE evaluation system
J. Donley, C. Ritz, M. Shujau, "Analysing the Quality of Experience
of Multisensory Media from Measurements of Physiological
Responses,” QoMEX2014, Singapore, Sep. 2014.

• Experiment VII
• EEG Correlates of Pleasant and Unpleasant Odor Perception
E. Kroupi, A. Yazdani, J.-M. Vesin, T. Ebrahimi, "EEG Correlates of Pleasant and
Unpleasant Odor Perception," ACM TOMM, vol. 11, no. 1s, Sep. 2014.

• Experiment VIII
• Multiple-Scent Enhanced Multimedia Synchronization
N. Murray, B. Lee, Y. Qiao, and G.-M. Muntean, "Multiple-Scent Enhanced Multimedia Synchronization," ACM TOMM, vol. 11, no. 1s, Sep. 2014.
General temporal boundaries:
-10s to +15s are “in-sync”, skew values beyond are “out-of-sync”

Conclusions
• From the need for a scientific framework to capture,
measure, quantify, judge, and explain the quality of
experience
• To …
– How to create, delivery, consume?
– How to capture and measure?
– How to quantify?
– How to judge and explain?
• Open issues?
Many!

Open Issues / Challenges
• QoE assessment is a delicate mixture of ingredients and choices
– Test & lab environment
– Test content
– Test methodology
– Data analysis
• (Semi-)Automatic content creation/annotation
• Towards large scale deployment
– Lessons learnt from 3D (disaster)
– 4D, 5D, xD – adding another dimension does not guarantee success
• Holistic approach not feasible
– Need for much more specialized QuaSE models
• QUALINET Task Force: "Immersive Media Experiences (IMEx)”
– https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:imex

Assessing Quality of Experience … A Bit
Like Measuring ‘Happiness’ …
© F. Pereira, Instituto Superior Técnico, Univ. Lisboa,
PortugalJuly 2016 ICME 2016 Tutorial, C. Timmerer 40

3. Towards the Concept of Quality of Life
A. Beyond Quality of Experience
B. The age of wearables
C. Ingredients of a modern assessment of Quality
of Life

Touradj Ebrahimi: About Me …
• Professor of multimedia signal processing at EPFL
• Active in image/video compression, media interpretation
(segmentation, annotation, search, retrieval, quality assessment,
brain computer interface, affective computing, etc.) and media
security (privacy protection, copyright protection, media integrity
verification, etc.)
• Member of MPEG standardization committee since 1992 and active
in many of its video standardization activities: MPEG-4, H.264/AVC,
MVC, H.265/HEVC, MV-HEVC, 3D-HEVC, SCC, HDR extensions.
• Member of JPEG standardization committee since 1994 and active in
many of its image standardization activities: JPEG 2000, JPSearch,
JPEG XR, JPEG AIC, JPEG XT, JPEG XS, JPEG PLENO.
• Member of the Steering Committee of QoMEX and chair of its first
edition in 2009
• Convener of JPEG Standardization Committee since 2014
• Chair of COST Action IC1003 Qualinet
• First coined in February 2001 the term Quality of Experience (QoE) as
a user-centric alternative to Quality of Service (QoS)

A. Beyond Quality of
Experience

What future for Quality of Experience
• Prediction is very difficult, especially about the future
→ Niels Bohr (1885-1962): Physics Nobel Prize Winner 1922

Trends in user-centric multimedia
• Consequences of Moore’s law
- Better and richer content
- Larger bandwidth networks
- Bigger storage capacities
- More sophisticated codecs/processing
• Better integration
- Art, design
- Psychology, psychophysics, neuroscience
- Sociology, humanities
- Technology and engineering

Trends in user-centric experiences
• New media experiences
• Personal well-being and personal health
• Big data and social media

New media experiences
• UHD, HDR, HFR, 3D, …
• Light field imaging
• Integral imaging
• Holographic imaging
• Haptics
• Virtual, Augmented, Mixed reality
• Immersive media
• Multi-sensory media
• …

Major trends in multimedia
New Media
Wearables
Internet of Things
Big data
Social media
ContentSensor
Processing
Multimedia
experiences

Major trends in multimedia
New Media
Wearables
Internet of Things
Big data
Social media
ContentSensor
Processing
Multimedia
experiences
Life

Generation-0 (smart) wearables

Not any wearable should be smart!

Other wearables for user sensing

Other wearables for environment sensing

Other variants of smart glasses

Sony’s recently announced smart glasses

C. Ingredients of a
modern assessment
of Quality of Life

Quality of Life
• Quality of life (QoL) is the general well-being
of individuals and societies. QoL has a wide
range of contexts, including the fields of
international development, healthcare,
politics and employment.

When past becomes future
• Quality of Life: Meaning, Measurement, and
Models
– Elyse W. Kerce, Navy Personnel Research and Development Center
(U.S.)
– Navy Personnel Research and Development Center, 1992
• Origins of the concept date back to 1725!
– Francis Hutcheson

Ingredients of a modern QoL assessment
• User sensing
• Environment sensing
• Context extraction
• Big data analytics
Users
Context
Content

Data is the King!
• (A lot of ) Data from users
and their environments is
needed to carry out
research on wearables:
– Publicly available
– Reliable
– Rich data
– Generated via crowdsourcing

Big data and social media
1
billion
monthly
active
users5million
photos
added
every day
175k
tweets
posted
every
second

Devices and sensors are Queens!
• (Easily available) Wearable
devices and sensors are
needed to generate the
data:
– Affordable components and
sensors to be purchased by
interested individuals
– Reliable
– Easy to configure and
calibrate
– User friendly

Software is the President!
• (Efficient) Software
to control wearables
and sensors
– Open source
– Extensible
– Reliable
– Easy to install on a wide
variety of platforms

Data management is a must!
• (Distributed) server
architecture for data
synchronization, storage
and access:
– Cloud based
– Reliable
– Scalable
– Respectful of Privacy and Ethical
issues

Interoperability is essential!
• (Open) Standard solutions
– Standard components
– Standard data syntax
– Standard Interface
– Compliance/Certification

Compelling use cases are important!
• Concrete use cases
– Dietary assessment
– Life log
– …

International consortium is needed!
• A seed consortium from US, Japan and Swiss
universities in place and has initiated work on
this topic around Multimedia Dietary
Assessment Use case

Consumer Health Applications
• Mobile device as a data collection tool for dietary assessment
• Increasing demand in applications for mobile devices

Consumer Health Applications
• Manual intake data entry
– Tap & Track
• Barcode based intake entry
– Fooducate
• Image based intake entry
– MealSnap

Where do people look when they eat?
Need to cover a wide angle

Did you finish the plate?
Need to consider temporal aspects
Before After

360/omnidirectional video camera

Creation of a new database
+ …

Food recognition
Food
Deep learning
Omnidirectional
image

Swallowing detection
Use EMG to detect swallowing and enable/disable camera
• The sound of mastication (food crushing) has relation to
physical properties of the food, but little relevance to
energy content
Chewing sensors
O. Amft, M. Stäger, and G. Tröster, “Analysis of chewing sounds for dietary monitoring,” UbiComp 2005, pp. 56–72, 2005.
S. Päßler, M. Wolff, and W.-J. Fischer, “Food intake monitoring: an acoustical approach to automated food intake activity detection
and classification of consumed food,” Physiol. Meas., vol. 33, no. 6, pp. 1073–1093, 2012.
• The sound of mastication (food crushing) has relation to
physical properties of the food, but little relevance to
energy content
Chewing sensors
O. Amft, M. Stäger, and G. Tröster, “Analysis of chewing sounds for dietary monitoring,” UbiComp 2005, pp. 56–72, 2005.
S. Päßler, M. Wolff, and W.-J. Fischer, “Food intake monitoring: an acoustical approach to automated food intake activity detection
and classification of consumed food,” Physiol. Meas., vol. 33, no. 6, pp. 1073–1093, 2012.

Integrate other contextual aspects
Time
Place
Context
User personality

Take-Home Messages
• QoL is the natural step beyond QoE !
• QoL not a new concept but it can take advantage of
modern technologies.
• A federating project is needed in order to create the
necessary critical mass (especially in data).
• Multimedia Dietary Assessment is a compelling use
case.

Quality of Experience in Multimedia Systems and Services: A Journey Towards the Quality of Life

Quality of Experience in Multimedia Systems and Services: A Journey Towards the Quality of Life

Recommended

Recommended

More Related Content

What's hot

What's hot (7)

Viewers also liked

Viewers also liked (20)

Similar to Quality of Experience in Multimedia Systems and Services: A Journey Towards the Quality of Life

Similar to Quality of Experience in Multimedia Systems and Services: A Journey Towards the Quality of Life (20)

More from Alpen-Adria-Universität

More from Alpen-Adria-Universität (20)

Recently uploaded

Recently uploaded (20)

Quality of Experience in Multimedia Systems and Services: A Journey Towards the Quality of Life