Video is by far the predominant consumer of network capacity. Yet, the Internet is a ‘video repellent’ machine, one that can transfer data but has no notion of deadlines. So what are we getting from modern video services? How can we measure quality of experience? And can we predict the quality perceived by the user, starting from simple network measurements? In this talk I give a critical perspective on conventional QoE assessment, ending up with a controversial proposition.
This talk is based on the following material:
V. Menkovski, A. Liotta
'Adaptive Psychometric Scaling for Video Quality Assessment'
Journal of Signal Processing: Image Communication (Elsevier, 2012)
http://bit.ly/JSP-2012
V. Menkovski, G. Exarchakos, A. Liotta
'The Value of Relative Quality in Video Delivery'
Journal of Mobile Multimedia. Vol.7(3), pp. 151-162 (Sept. 2011)
http://bit.ly/JMM-2011
V. Menkovski, G. Exarchakos, A. Liotta
'Online Learning for Quality of Experience Management'
The annual machine learning conference of Belgium and The Netherlands, Leuven, Belgium, 2010
http://bit.ly/BENELEARN-2010
A. Liotta, G. Exarchakos
'Networks for Pervasive Services: Six Ways to Upgrade the Internet'
Springer (2011)
http://bit.ly/pervasive-networks
1. Can networks deliver
Quality of Experience?
Prof. Antonio Liotta
Eindhoven University of Technology, NL
http://bit.ly/autonomic_networks
Twitter: a_liotta
2. Four questions about Quality of Experience
• Why is QoE so important?
• How good are we at scoring QoE?
• How can we use Machine Learning to manage QoE?
• Can networks learn about QoE too?
Prof. A. Liotta 2
3. Why we should NEVER ignore QoE in streaming services
Sending rate: 2048 Kbps
HIGHER
39.48 dB
- 23.85 dB
Received quality
LOWER
15.65 dB
Prof. A. Liotta 3
4. Why we should NEVER ignore QoE in streaming services
Sending rate: 768 Kbps
LOWER
35.15 dB
- 16.03 dB
Received quality
HIGHER
19.12 dB
Prof. A. Liotta 4
5. QoE ≠ Σ QoS
Video
stream
Overprovisioning !!
Prof. A. Liotta 5
6. QoE results from a diversity of factors
Transport Environmental
Encoding
Prof. A. Liotta 6
7. Understanding QoE is essential for video delivery
(Internet not designed for QoS nor QoE)
4 3 2 1 2 1
Server A 1
4
3
4 3 2 4 3
Client B
No guarantees about:
• Delivery of all packets
• Delivery order
• Delivery time
Further details:
A.Liotta, G.Exarchakos, “The network as we know it”
Prof. A. Liotta (Springer, 2011) – http://bit.ly/pervasive-networks 7
8. Without ‘edge tricks’ the Internet wouldn’t operate
(MPEG-4 video, 1% packet loss, no buffering)
J. Okyere-Benya, M. Aldiabat, V. Menkovski, G. Exarchakos, A. Liotta
Video Quality Degradation on IPTV Networks, IEEE ICNC‟12, USA, Feb. 2012
http://bit.ly/ICNC-12
Prof. A. Liotta 8
9. Trends: 1. Traffic explosion
Anything on video – more producers than watchers?
(*)
Source: Cisco data
Estimated IP traffic in 2015: 1.3 Zettabytes (1021 bytes) (*)
forecast, Feb 2011
Prof. A. Liotta 9
10. Trends: 2. extremely mobile video
(mobile video responsible for majority of mobile traffic)
Prof. A. Liotta 10
11. Trends: 3. M2M video
(new communication patterns and ‘volumes’)
Building
ICT
autom.
Security
Energy
& Safety
Consu-
mer & Retail
Home
Health- Transpor
care Indust. tation
Autom.
(*)
Source: Cisco data
forecast, Feb 2011
Prof. A. Liotta 11
12. Trends: 4. opportunistic communications
(even more ‘video repellent’ than ordinary IP network)
A.Liotta, G.Exarchakos, “Spontaneous networks”
Prof. A. Liotta (Springer, 2011) – http://bit.ly/pervasive-networks 12
13. Trends: 4. opportunistic communications
(even more ‘video repellent’ than ordinary IP network)
A.Liotta, G.Exarchakos, “Spontaneous networks”
Prof. A. Liotta (Springer, 2011) – http://bit.ly/pervasive-networks 13
14. QoE: the missing link towards closed-loop delivery
Video
stream
N-QoS
A-QoS Objective Subjective
inference
QoE QoE
Some Some control
Expensive process
actuators techniques
Prof. A. Liotta 14
15. Subjective QoE: so far a helpless effort
Annoying
Expensive
Inaccurate
Impossible to capture
complexity and
variability of the
delivery context
Prof. A. Liotta 15
16. Absolute rating comes with huge variability and bias
(mean opinion score)
35%
VQEG HD5
30%
25%
STDEV[%scale]
20% 17,94% 1
15%
10%
5%
0%
0% 50% 100% 0
MOS [%scale]
Further details:
“Report on the Validation of Video Quality Models for High Definition Video Content”
by the Video Quality Experts Group, Jun. 2010.
Prof. A. Liotta 16
17. Human perception is biased
(the same video format gives different perceptions)
K. Seshadrinathan et al. “Study of subjective and objective quality assessment of video”
Prof. A. Liotta 17
IEEE Trans. Image Processing, Vol.29(6). June 2010.
18. Both videos are encoded exactly in the same way but
the “pedestrian” one is “perceived” as worse
• Difference from unimpaired reference is greater in ‘pedestrian’ video
• But the overlapping Gaussians indicate that many subjects had opposite
perceptions
Differential scoring is more
accurate than absolute MOS
BUT IT’S CLEAR THAT
Prof. A. Liotta
humans are not good at ranking 18
19. Can we use ‘machine learning’ to model QoE on a
QoE domain continuous scale?
Cumulative
Gaussian
(2 parameters)
QoS domain
Psychophysics quantitatively investigates the relationship
Prof. A. Liotta between physical stimuli and the sensation of perception 19
20. A more effective question
(Maximum Likelihood Different Scaling)
Which one of these two pairs has bigger difference?
Prof. A. Liotta 20
21. Responses fit a psychometric curve
(human perception)
DEVIATION OF RESPONSES
BETWEEN 1 AND 10%
DEPENDING ON VIDEO TYPE
V. Menkovski, G. Exarchakos, A. Liotta,
The Value of Relative Quality in Video Delivery,
Journal of Mobile Multimedia. Vol.7(3), pp. 151-162 (Sept. 2011)
http://bit.ly/JMM-2011
Prof. A. Liotta 21
22. Use of psychometric curves for QoE management
(resource optimization zone)
ZONE 1
QoS deltas
don’t produce
delta QoEs
364 Kbps
Prof. A. Liotta 512 Kbps 22
23. Use of psychometric curves for QoE management
(quality optimization zone)
ZONE 2
strong
non
linearity
64 Kbps
Prof. A. Liotta 256 Kbps 23
24. MLDS is more accurate than conventional QoE rating
but still unscalable
• Must consider all combinations of samples
• A full round of tests including 10 levels of stimuli requires
10
210 tests
4
• The test matrix explodes as we consider more parameters
Can we speed up the prediction-model
learning process?
Prof. A. Liotta 24
25. Active learning helps eliminating the redundant tests
• After the first few test we can start estimating
the answers of the remaining tests
• The estimation of the unanswered test uses
the characteristics of the psychometric curve
to reduce the problem domain
River bed
Tractor
Blue sky
Prof. A. Liotta 25
26. Learning convergence varies for different videos but
always leads to improved scalability
V. Menkovski, A. Liotta, Adaptive Psychometric Scaling for Video Quality Assessment
Journal of Signal Processing: Image Communication (Elsevier, 2012)
http://bit.ly/JSP-2012
Prof. A. Liotta 26
27. Reinforcement Learning to realize
‘trial & error’ network loops
„Sport over mobile phone‟
QoS probe
actuators
Optimizing QoE Machine QoE measure
QoS prediction Learning or inference
Prof. A. Liotta 27
28. Networks quickly learn how to deal with new conditions
(problem domain is constrained to psychometric function)
100
95
90
85
Accuracy
80
75
70
Old conditions New conditions
65
60
55
50 1030
1090
1150
1020
1080
1140
10
70
130
190
250
310
370
430
490
550
610
670
730
790
850
910
970
60
120
180
240
300
360
420
480
540
600
660
720
780
840
900
960
New ‘trial & error’ samples
V. Menkovski, G. Exarchakos, A. Liotta, Online Learning for Quality of Experience Management
The annual machine learning conference of Belgium and The Netherlands, Leuven, Belgium, 2010
http://bit.ly/BENELEARN-2010
Prof. A. Liotta 28
29. Take-home messages
• Existing QoE methods are
– annoying, expensive, inaccurate, ineffective
• What is the ‘right’ question?
– we are good at spotting difference of differences
• Learning how to deal with new perturbations is more
promising than brute-force subjective studies
– human perception is a moving target
– ML works with incomplete information, extrapolates non-
obvious patterns and handles the unknown via trial&error
Prof. A. Liotta 29
30. Thank you !
More about my work http://bit.ly/autonomic_networks
In the press http://bit.ly/press_articles
“All of YouTube through “Cognitive “Networks for
a 40-year-old funnel” Interconnections” pervasive services”
http://bit.ly/Volkskrant-EN http://bit.ly/booklet-antonio http://bit.ly/pervasive-networks
Prof. A. Liotta 30