Streaming Video Quality & User Engagement Whitepaper from July, 2011. In an industry first, IDC conducted a statistical analysis of the server log files of six major 2010 sports events that were streamed live to consumers in both North America and Western Europe with a total of more than2 million users. The analysis found that both user engagement (measured as session length) and, consequently, unique user numbers were influenced by video quality. Several factors were shown to have an impact: --Higher bit rates do increase user engagement. For each event, after a certain bit rate threshold, a further increase of bit rates had no additional positive effect on user engagement anymore. --An important factor negatively impacting user engagement was the number of rebuffering events per hour. --Other, less influential negative factors were the share of time the video player spent rebuffering during users' sessions and the number of dropped frames per hour. --Our research suggests that measuring and monitoring key performance indicators (KPIs) for video quality is of critical importance for publishers because they affect user engagement and audience reach and therefore publishers' revenue and competitiveness. Visit http://www.akamai.com/html/solutions/sola_analytics.html for more information about Akamai's solutions.

1. WHITE P APER
Streaming Video Quality and User Engagement
Sponsored by: Akamai
Karsten Weide
July 2011
IDC OPINION
www.idc.com
In an industry first, IDC conducted a statistical analysis of the server log files of six
major 2010 sports events that were streamed live to consumers in both North
America and Western Europe with a total of more than 2 million users.
The analysis found that both user engagement (measured as session length) and,
F.508.935.4015
consequently, unique user numbers were influenced by video quality. Several factors
were shown to have an impact:
 Higher bit rates do increase user engagement. For each event, after a certain bit
rate threshold, a further increase of bit rates had no additional positive effect on
P.508.872.8200
user engagement anymore.
 An important factor negatively impacting user engagement was the number of
rebuffering events per hour.
 Other, less influential negative factors were the share of time the video player
Global Headquarters: 5 Speen Street Framingham, MA 01701 USA
spent rebuffering during users' sessions and the number of dropped frames per
hour.
Our research suggests that measuring and monitoring key performance indicators (KPIs)
for video quality is of critical importance for publishers because they affect user
engagement and audience reach and therefore publishers' revenue and competitiveness.
METHODOLOGY
Akamai tasked IDC with a research project to explore the impact of different aspects
of online streaming video quality on user engagement. To that end, Akamai provided
IDC with the server log files of six major 2010 sports events that were streamed live
to consumers via Akamai's HD Network employing HTTP streaming, using adaptive
bit rate technology as outlined in Table 1.
For all of the following analyses, keep in mind that the bit rates offered by publishers
were vastly different between events. The FIFA World Cup soccer events in particular
were offered at comparatively low bit rates because of the massive crowds expected
to watch.

2. TABLE 1
Event Overview
Event Region Number of Users Bit Rates Served
Soccer A: FIFA World Cup 2010 North America 235,052 400, 750, 1000, 1300, 1800 kbps
Soccer B: FIFA World Cup 2010 Western Europe 76,843 700, 1300, 2200, 3000 kbps
Soccer C: FIFA World Cup 2010 Western Europe 125,244 400, 800, 1200, 1600 kbps
Sports Event A: Major 2010 sports event North America 1,376,727 564, 1064, 1564, 2200 kbps
Sports Event B: Major 2010 sports event North America 25,644 564, 1064, 1564, 2200 kbps
Sports Event C: Major 2010 sports event North America 469,876 Four bit rates under 2000 kbps
Source: IDC, 2011
The log files were cleaned up before statistical analysis as follows:
 Sessions were consolidated by user ID. Each log file entry originally represented
one viewing session. Where there were two or more separate sessions for the
same user ID, these sessions were consolidated so that log entries represented the
complete viewing experiences for each user ID for each event.
 Logs were cleaned up. We removed any entry where it was clear from the data
that it was either impossible for the respective user to have seen the video
(average playback bit rate was zero, number of average frames per second
[FPS] was zero) or where more than 20% of total session time was spent
rebuffering (with the picture frozen), making it unlikely for the user to have
endured that bad of a viewing experience. We also ignored cases where the total
aggregated session time was less than one minute, assuming that shorter
sessions could not be counted as "viewing" a live video.
We also assumed that each user ID related to one person, even though several
persons or different persons at different times may have watched the video.
Furthermore, we assumed that users had spent the entire total aggregated session
time watching the video. In practice, users might have walked away from their PC or
could have had the video run in the background. For both assumptions, there was no
way for us to determine from the log files whether they held true.
IDC then conducted a statistical analysis of the remaining cases using the statistical
software package SPSS. The approach was to correlate user engagement (measured
as the total aggregated session time per user ID per event [short: session length or
session duration]) with certain measurements of streaming video quality for that
user's session during that event (see the Correlation section for details on the
statistical method):
 Average playback bit rate: The average bit rate at which the video was rendered
on the user's screen as reported to the server by the user's video player
2 #229083 ©2011 IDC

3.  Rebuffering events per hour: The number of times the buffer ran out of data
and had to be replenished, possibly with the picture frozen if the rebuffering
event was long enough to be noticeable by the user
 Percent of time spent rebuffering: The share of the total aggregated session
time that was spent rebuffering
 Dropped frames per hour: The number of frames that the user's video player
did not show
(The project did not analyze the impact of video start-up times on user engagement
because the log files did not include that information.)
The hypotheses were that:
 Where positive KPIs such as playback bit rate or average FPS were higher (i.e.,
video quality was better), user engagement would also be higher (i.e., session
times would be longer). The expected correlation coefficient would be > 0.
 Where positive KPIs were lower (i.e., video quality was worse), we expected user
engagement to also be lower (i.e., session times would be shorter). The
expected correlation coefficient would be < 0.
 Conversely, where negative KPIs were higher (i.e., video quality was worse), we
expected user engagement to also be lower (i.e., session times would be
shorter). The expected correlation coefficient would be < 0.
 Where negative KPIs were lower (i.e., video quality was better), user
engagement would also be higher (i.e., session times would be shorter). The
expected correlation coefficient would be > 0.
All correlation coefficients reported in this document were significant at the 0.01 level.
This means that mathematically, there is only a 1% likelihood that the reported
correlation occurred by chance.
Correlation
Correlation is a statistical method that analyzes the relationship between two sets of
data and expresses the closeness of their relation in a "correlation coefficient," a
single number between 1 and -1.
For instance, we compared the average bit rates at which thousands of users
watched a video and the total time they spent watching the video.
 If higher bit rates in each case translate into longer viewing times in a certain
proportion, the correlation coefficient would be 1.
 If there was no relation at all between bit rates and viewing times, the coefficient
would be 0.
 If higher bit rates in each case translate into shorter viewing times in a certain
proportion, the correlation coefficient would be -1.
 Values between 0 and 1 and 0 and -1 would express varying degrees of relationship.
©2011 IDC #229083 3

4. Correlation does not necessarily indicate causation (i.e., two sets of data might be
shown to relate to each other statistically even though there is no relation between
the two data sets in the real world).
IN THIS WHITE P APER
This IDC white paper explores the impact of different aspects of online streaming
video quality on user engagement based on the statistical analysis of server log files
of six major 2010 sports events that were streamed live to consumers.
SITUATION OVERVIEW
Statistical Analysis of the Impact of
Streaming Video Quality on User Engagement
IDC's statistical analysis of the server log files of six major sports events that were
streamed live to users found that user engagement was influenced by video quality.
We found the following two factors had a positive impact on session durations (i.e.,
they tended to improve user engagement):
 Higher playback bit rates (which of course are based on higher transferred bit
rates) had the greatest impact in terms of extending session lengths, but only up
to a certain optimal bit rate threshold. If bit rates were further increased beyond
that threshold, session durations were not further increased, or they were not
increased as much.
 Higher frame rates (frames per second) also had a positive impact on session
lengths, but to a lesser extent than higher bit rates.
The following factors had a negative impact on session lengths — that is, they tended
to worsen user engagement (in sequence of their level of impact):
 The number of rebuffering events
 The share of the session time spent rebuffering
 The number of dropped frames
Of the preceding factors negatively impacting user engagement, one of the most
important was the number of rebuffering events. The share of viewing time spent
rebuffering and the number of dropped frames/s had less of an impact.
Our research suggests that measuring and monitoring KPIs for video quality is of
critical importance for publishers because they affect user engagement and audience
reach and therefore publisher revenue and competitiveness.
4 #229083 ©2011 IDC

5. Playback Bit Rate
Online video publishers all adopt high-quality or high-definition video for competitive
and branding purposes, also based on the experience in cable TV, where higher
resolutions translated into greater user engagement.
There has been a lot of discussion in the industry about whether increasing the bit
rate available to the user and thereby improving video resolution has a positive
impact on user engagement. Therefore, we began our analysis by correlating users'
average playback bit rates and session lengths. We expected a positive correlation
(i.e., that higher bit rates come with longer sessions).
Our statistical analysis showed that users did watch the video streams for a longer
time if they watched the event at higher playback bit rates (i.e., at higher video
resolutions) — but only up to a certain bit rate. That is, for each event, if we analyzed
only the cases up to that event's optimal bit rate threshold, correlation between
playback bit rates and session length was positive, which means that higher bit rates
tended to go with longer sessions (see Figure 1). The impact was slight, but
statistically significant.
FIGURE 1
Correlation Between Session Time and Playback Bit Rates for
Cases up to the Optimal Bit Rate Threshold for Each Event
Soccer A
Soccer B
Soccer C
Sports Event A
Sports Event B
Sports Event C
-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4
(Correlation coef f icient)
Note: For the optimal bit rate threshold (i.e., the playback bit rates up to which cases were
analyzed for each of the above events), see Table 2.
Source: IDC, 2011
©2011 IDC #229083 5

6. After that threshold, there was no additional positive effect on session times, or
the effect decreased. For the six events analyzed, the threshold was at different
levels (see Table 2). For publishers, this means that it is necessary to carefully
measure and monitor the impact of bit rate on session lengths to establish the optimal
bit rate range.
TABLE 2
Maximum Playback Bit Rate Level Showing Positive Impact on Session Length
Maximum Bit Rate (kbps) up to Which
Higher Bit Rates Further Improved
Impact on Session Lengths Correlation Coefficient
Soccer A 1000 0.120
Soccer B 2500 0.148
Soccer C 1000 0.031
Sports Event A 2000 0.021
Sports Event B 3500 0.048
Sports Event C 1500 0.023
Source: IDC, 2011
It is difficult to arrive at a formula that would express how much user engagement
(i.e., session lengths) increases as bit rate increases given the many factors that have
an impact on video performance (see next paragraph). But based on the kind of
performance increases we have seen in the data, we would expect to see, as a rule
of thumb, an increase of 10% in session lengths per 500 kbps increase in average
playback bit rate.
Keep in mind that the bit rates offered by publishers were vastly different between
events. This may be one reason why the cutoff is at different levels for different
events. We also theorize that there may be other effects at work as well. For instance,
those users who watch video at the highest bit rates also must have the infrastructure
(e.g., a high broadband access speed) in place to be able to watch at these rates.
Those users are also more likely to have incomes and busier lives, which could
explain why they are more likely to watch for shorter periods of time. More research is
needed, taking into account cultural, social, and situational factors.
Rebuffering
Rebuffering events are among the most frustrating experiences when watching a live
video stream. We analyzed the impact of rebuffering events per hour. These are
incidents where the buffer of the user's video player runs out of data and must be
6 #229083 ©2011 IDC

7. replenished by the server while the picture freezes. Rebuffering is caused either by a
connection slowdown or by bad heuristics (i.e., when the player waits too long to
switch to a lower bit rate).
We expected a negative correlation between the number of rebuffering events and
session durations (i.e., for more rebuffering events to go with shorter sessions)
because with more interruptions of the video stream, users would become more
frustrated and more likely to stop watching it.
That is precisely what we found. The number of rebuffering events turned out to be
one of the worst factors impacting user engagement (see Figure 2).
FIGURE 2
Correlation Between Sessi on Time and Rebuffering Events per
Hour
Soccer A
Soccer B
Soccer C
Sports Event A
Sports Event B
Sports Event C
-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4
(Correlation coef f icient)
Source: IDC, 2011
Then we looked at the impact of the total share of the session time that users had to
spend waiting for the rebuffering to complete and the video to resume (aggregating
the waiting time incurred by all rebuffering events) on session lengths.
Again, we expected a negative correlation (i.e., for higher rebuffering time shares to
go with shorter session durations). And again, we found that to be the case (see
Figure 3). Rebuffering duration was the second most important factor negatively
impacting user engagement. This also means the negative impact of rebuffering time
was smaller than that of the number of rebuffering events. Apparently, viewers find
that disruption as such is worse than waiting for it to end.
©2011 IDC #229083 7

8. FIGURE 3
Correlation Between Session Time and Percent of Time Spent
Rebuffering
Soccer A
Soccer B
Soccer C
Sports Event A
Sports Event B
Sports Event C
-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4
(Correlation coef f icient)
Source: IDC, 2011
Video Frames
The number of frames per second (FPS) or frame rate expresses the number of
consecutive images shown in a video transmission per second that create the illusion
of motion. Higher FPS numbers translate into better video quality because the video
plays more smoothly; lower FPS numbers conversely result in worse video streams.
Dropped frames are images that are not displayed by the user's video player, either
because local resources (CPU, graphics adapter, memory, etc.) are not sufficient or
because there is a disruption in the video transmission. From the user's perspective,
dropped frames translate into choppy video.
For the number of dropped frames per hour, we expected a negative correlation with
session durations (i.e., for more dropped frames to coincide with lower user
engagement) because dropped frames disrupt the viewing experience. This is what
we found in the numbers, too. Dropped frames per hour were the third most important
factor negatively impacting session lengths. However, the impact was fairly minimal
(see Figure 4).
8 #229083 ©2011 IDC

9. FIGURE 4
Correlation Between Session Time and Average Number of
Dropped Frames per Hour
Soccer A
Soccer B
Soccer C
Sports Event A
Sports Event B
Sports Event C
-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4
(Correlation coef f icient)
Source: IDC, 2011
FUTURE OUTLOOK
This research has established that there is an impact of streaming video quality on
user engagement (i.e., session durations) and, therefore, on audience reach. As
consumers embrace online video distribution as a viable alternative to cable and
broadcast TV, their expectations of video quality will continue to increase. This will be
even more so as online video makes its way into consumers' living rooms, where it
ends up on high-definition television sets and will have to compete with the quality
that cable routinely provides. Here, a reliable online transmission at a resolution of
720p is only the beginning.
Publishers will need to embrace measuring and monitoring video quality such as
buffering, drop-off, and bit rate consumption on an ongoing, routine basis to tune the
experience and avoid dips in video quality and the resulting drop in user engagement
in order to protect their financial performance and competitiveness.
Given the wide range of bit rates offered in the six events analyzed, and the different
infrastructures given for them, it is difficult to arrive at a universal bit rate benchmark.
Two of the three soccer events in particular offered comparatively low bit rates
because of the expected huge numbers of viewers. If one looked only at the United
States, recommended bit rates would have to be set quite a bit higher.
Based on the given events, for the bit rate provided by publishers, IDC suggests
maintaining a level of at least 1200 kbps to attract the kind of audience numbers we
saw in the events analyzed. If one wanted to increase audience reach and user
engagement beyond that level, it would be prudent to increase the average bit rate to
1500 kbps.
©2011 IDC #229083 9

10. The single most negative impact on engagement was the number of rebuffering
events. We believe the best practice is not about keeping rebuffering events to a
certain exact number; rather, it is about ensuring that the largest share of your
audience experiences no buffering at all.
Of course, only part of the occurrence of rebuffering events can be controlled by
publishers. Again, an optimized distribution technology and measuring and monitoring
rebuffering events are key to an optimized user engagement.
The second most important negative impact on viewer engagement is the amount of
time spent rebuffering. IDC recommends, as a rule of thumb, maintaining a level of
video quality at which users experience rebuffering for a maximum of 1% of the time.
This lines up with the experiences that publishers have had in practice. In an
interview with IDC, Glenn Goldstein, MTV's VP, Video Technology Strategy, said,
"Once rebuffering time hits 1% of the playback time, we know we're in trouble."
More research is needed regarding the impact of start-up times on user engagement
(which was not explored in this research) and the influence of demographic,
psychographic, and situational factors on video consumption.
Copyright Notice
External Publication of IDC Information and Data — Any IDC information that is to be
used in advertising, press releases, or promotional materials requires prior written
approval from the appropriate IDC Vice President or Country Manager. A draft of the
proposed document should accompany any such request. IDC reserves the right to
deny approval of external usage for any reason.
Copyright 2011 IDC. Reproduction without written permission is completely forbidden.
10 #229083 ©2011 IDC