Slide deck that explains the rationale behind perceptual speed index, and the collaborative work that happened over last year.

1. Parvez Ahammad
Principal Engineer / Head of DS&ML Team
Instart Logic Inc.
July 2016
Perceptual Speed Index (PSI)
for Measuring above-fold visual
Quality of Experience of webpages

2. Acknowledgments
• WPO-Foundation/visualmetrics · GitHub
• Pat Meenan (Google & WebPageTest)
• DS&ML team @ Instart Logic
– Clark Gao
– Matthew Mok
• Karan Kumar (Instart Logic)

3. Why we (@ InstartLogic) care about metrics for end-user QoE
● Towards a data-driven ‘Control System’ for improving web performance
● Accurate measurement of end-user experience is key!

4. Page
Load
Idle
Response
to Input
Animation
&
Scrolling
Reference: How Users Perceive the Speed of The Web (2015): Paul Irish / Google
Four phases of interaction: end-user’s perception

5. The space of end-to-end perf. metrics
• PA’s taxonomy of web performance measurements
A. Milestone timings (of various kinds)
B. Byte size counts (relative to various milestones)
C. Request counts
D. Metrics on above-fold visual QoE
E. ...

6. Quick Refresher on Speed Index
● Popular public metric on above-fold visual QoE
● Patrick Meenan (WPT / Google Inc.)
● Aggregate function on quickness of visual completion above-fold
● Goes beyond milestone timer based approaches

7. Measurement of visual progress in Speed Index
● Frame-by-frame VC progress is computed from pixel-histogram comparisons
● Pixel-wise similarity (mean histogram difference / MHD) doesn’t capture visual
perception !
● Perception of Shape / Color / Object similarity

8. Pixel-wise similarity doesn’t capture shape similarity
Black / White :: 50 / 50
MHD (Mean Histogram Difference) = 0

9. Pixel-wise similarity doesn’t capture color similarity
WWW
Edward Adelson / MIT

10. Pixel-wise similarity doesn’t capture object similarity
1 2
3
4
5
6
7
Antonio Torralba / MIT

11. Proposal for a perceptually oriented visual QoE metric
● Update: Frame-by-frame VC progress computation using SSIM
Original, MHD = 0; SSIM = 1 MHD = 144, SSIM = 0.988 MHD = 144, SSIM = 0.913
MHD = 144, SSIM = 0.840 MHD = 144, SSIM = 0.694 MHD = 142, SSIM = 0.662
Wang / Bovik / Simoncelli - 2004

12. pSI (perceptual Speed Index)
● Frame-by-frame VC progress computation using SSIM
perceptual Via SSIM

13. Experiment #1
● Key question: Does the MHD  SSIM change capture “real” website
behavior?
● Experiments with IR-100 websites (April 2015)
● Presented to Pat Meenan in May 2015
● Let’s look at a few illustrative examples

14. Tracking Perceptual Change: SI vs. pSI – RalphLauren (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)

15. Tracking Perceptual Change: SI vs. pSI – ChicosFas (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)

16. Tracking Perceptual Change: SI vs. pSI – Etsy (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)

17. Tracking Perceptual Change: SI vs. pSI – Express (2015)
Green: % visual completion via SSIM (PSI)
Red: % visual completion via MHD (SI)

18. Recap: Experiment #1
● Does the “MHD  SSIM change” capture “real” website behavior?
● Experiments with IR-100 websites (April 2015)
● Presented to Pat Meenan in May 2015
● Preliminary conclusions
● SSIM based visual progress measurements match human
perception more closely than MHD
● Key observation: PSI appears higher when visual jitter exists
● Pop-up ads / large lay-out changes / etc.
● SSIM / MHD swap doesn’t affect websites without visual jitter
● ChicosFas / Etsy examples

19. Experiment #2
● How do SI/PSI compare on a larger sample of Alexa top-K websites?
● Experiments with Alexa top mobile websites (March/April 2016)
● Dataset of 500+ mobile above-fold videos
● Dataset provided by Pat Meenan
● All analysis complete (May 2016)
● Results sent to Pat
● What did we find out?

20. #1: SI and PSI are highly (linearly) correlated (ρ = 0.91)

21. #2: Median PSI (5020) is slightly higher than median SI (4857)

22. Experiment #2
● How do SI/PSI compare on a larger sample of Alexa top-K websites?
● Experiments with Alexa top mobile websites (March/April 2016)
● Dataset of 500+ mobile above-fold videos
● Dataset provided by Pat Meenan
● All analysis complete (May 2016)
● Results sent to Pat
● What did we find out?
● SI and PSI are linearly correlated
● Key observation: PSI is much higher than SI for webpages with
visual jitter / layout thrashing behavior

23. Summary (1)
● Metrics on above-fold visual QoE of web application delivery must
take human end-user perception into account.
● Speed Index
● Primarily focused on progress of above-fold loading
● Does not account for layout stability
● Perceptual Speed Index
● A perceptually oriented metric to measure above-fold visual QoE
● Designed to account for visual jitter (layout stability)
● Complementary to SI
● PSI implementation complete
● Open sourced  GitHub: WPO-Foundation/visualmetrics

24. Summary (2)
● For more details and explanations:
● http://www.parvez-ahammad.org/blog/perceptual-speed-index-
psi-for-measuring-above-fold-visual-performance-of-webpages