Integrating content delivery networks into your application infrastructure can offer many benefits, including major performance improvements for your applications. So understanding how CDNs perform — especially for your specific use cases — is vital. However, testing for measurement is complicated and nuanced, and results in metric overload and confusion. It's becoming increasingly important to understand measurement techniques, what they're telling you, and how to apply them to your actual content. In this session, we'll examine the challenges around measuring CDN performance and focus on the different methods for measurement. We'll discuss what to measure, important metrics to focus on, and different ways that numbers may mislead you. More specifically, we'll cover: Different techniques for measuring CDN performance Differentiating between network footprint and object delivery performance Choosing the right content to test Core metrics to focus on and how each impacts real traffic Understanding cache hit ratio, why it can be misleading, and how to measure for it

1. Measuring
CDN
Performance
Hooman
Beheshti,
VP
Technology

2. Why
this
matters
• Performance
is
one
of
the
main
reasons
we
use
a
CDN
• Seems
easy
to
measure,
but
isn’t
• Performance
is
an
easy
way
to
comparison
shop
• Nuanced
• Metric
overload

3. Common
mistakes
• Getting
lost
in
data
• Focusing
on
one
thing
and
one
thing
alone
– We
like
numbers!
• Forgetting
about
our
applications
• Letting
vendors
influence
us
– (I’m
fully
aware
of
the
irony!)

4. Goals
• Share
measurement
experiences
• What
does
the
measurement
landscape
look
like
• Help
guide
towards
good
testing
plan
– Avoid
pitfalls

5. Background

7. CDNs:
delivery
platforms
• We
won’t
talk
about
– Extra
stuff:
security,
GSLB,
TLS
offload,
etc
– We
also
won’t
talk
about
page-­‐level
optimizations
• We
will
focus
on
the
delivery
side
– Delivering
HTTP
objects

8. Delivery:
static/cached
objects
Client
CDN
Node
Origin

9. Delivery:
dynamic/uncached
objects

10. What
we’ll
be
focusing
on
• How
we
measure
• Metrics
to
measure
• What
to
measure
• Some
gotchas,
misconceptions,
and
common
mistakes

11. Measurement
Techniques
(how
we
measure)

12. Measurement
techniques
• Pretend
Users
– Synthetic
tests
– Not
actual
users
• Real
Users
– In
the
browser
– Actual
users

13. Synthetic
testing

14. Synthetic
testing
• Usually
a
large
network
of
test
nodes
all
over
the
globe
• Highly
scalable,
can
do
lots
of
tests
at
once
• Many
vendors
that
have
this
model
– Examples:
Catchpoint,
Dynatrace(Gomez),
Keynote,
Pingdom,
etc

15. Synthetic
testing
• Built
to
do
full
performance
and
availability
testing
– Lots
of
“monitors”
–
emulating
what
real
users
do
– DNS,
Traceroute,
Ping,
Streaming,
Mobile
– HTTP
• Object
• Browser
• Transactions/Flows
• Tests
set
up
with
some
frequency
to
repeatedly
test
things
– Aggregates
reported

16. Backbone
nodes
• Test
machines
sitting
in
datacenters
all
around
the
globe
• Terrible
indicators
of
raw
performance
– No
latency
– Infinite
bandwidth
• But
really
good
at:
– Availability
– Scale
– Backend
problems
– Global
reach

17. https://www.flickr.com/photos/stars6/4381851322/

18. Backbone
nodes
• Test
machines
sitting
in
datacenters
all
around
the
globe
• Terrible
indicators
of
raw
performance
– No
latency
– Infinite
bandwidth
• But
really
good
at:
– Availability
– Scale
– Backend
problems
– Global
reach

19. Last
mile
nodes
• Test
machines
sitting
behind
a
real
home-­‐like
internet
connection
• Much
better
at
reporting
what
you
can
expect
from
users,
but
sometimes
unreliable
• Also
not
as
dense
in
deployment

20. backbone
last
mile

21. Synthetic
testing
• Pros
– Geographic
distribution
– Lots
of
options
for
testing
– Really
good
for
on-­‐the-­‐spot
troubleshooting
– Last-­‐mile
nodes
can
be
pretty
good
proxies
for
performance
• Cons
– Not
real
users!
– Backbone
nodes
can
be
misleading

22. Real
users
(RUM)

23. RUM
• Use
javascript
to
collect
timing
metrics
• Can
collect
lots
of
things
through
browser
APIs
– Page
metrics,
asset
metrics,
user-­‐defined
metrics

24. Use
test
assets
• Use
this
model
to
initiate
tests
in
the
browser
• Some
vendors:
– Cedexis,
TurboBytes,
CloudHarmony,
more…
– Usually,
this
isn’t
their
business,
but
the
data
drives
their
main
business
objectives
• You
can
build
this
yourself
too

25. Use
real
assets
in
the
page
• Collect
timings
from
actual
objects
– Resource
timing
• Vendors
– SOASTA,
New
Relic,
most
synthetic
vendors
– Boomerang
(open
source)
– Google
Analytics
User
Timings

26. DATA,
DATA,
DATA
• For
either
RUM
technique,
we
need
A
LOT
of
data
• Too
many
variances
• Most
vendors
don’t
use
averages
– Medians
and
percentiles

27. Real
user
measurements
• Pros
– Real
users,
real
browsers,
real
world
conditions
– If
you
use
your
own
content,
could
be
close
to
what
your
users
experience
– With
enough
data,
great
for
granular
analysis
• Cons
– We
need
a
lot
of
data
– If
you
do
it
yourself,
data
infrastructures
aren’t
trivial

28. Measurement
Metrics

29. Client
Server

30. Client
Server
1
x
RTT

31. Client
Server
DNS
DNS

32. TCP
Client
Server
DNS
DNS

33. TCP
Client
Server
DNS
DNS
(TLS)

34. TCP
Client
Server
DNS
DNS
(TLS)
HTTP

35. TCP
Client
Server
DNS
DNS
(TLS)
HTTP
Download

36. DNS
TCP
(TLS)
TTFB
Download
(TTLB-­‐TTFB)
Time

37. DNS
TCP
(TLS)
TTFB
Download
(TTLB-­‐TTFB)
Time
DNS
RTT
to
DNS
server,
DNS
iterations,
DNS
caching
and
TTLs

38. DNS
TCP
(TLS)
TTFB
Download
(TTLB-­‐TTFB)
Time
DNS
TCP
RTT
to
DNS
server,
DNS
iterations,
DNS
caching
and
TTLs
RTT
to
cache
server
(CDN
footprint
&
routing
algorithms)

39. DNS
TCP
(TLS)
TTFB
Download
(TTLB-­‐TTFB)
Time
DNS
TCP
(TLS)
RTT
to
DNS
server,
DNS
iterations,
DNS
caching
and
TTLs
RTT
to
cache
server
(CDN
footprint
&
routing
algorithms)
RTT
to
cache
server
(or
RTTs
depending
on
TLS
False
Start),
efficiency
of
TLS
engine

40. DNS
TCP
(TLS)
TTFB
Download
(TTLB-­‐TTFB)
Time
DNS
TCP
(TLS)
TTFB
RTT
to
DNS
server,
DNS
iterations,
DNS
caching
and
TTLs
RTT
to
cache
server
(CDN
footprint
&
routing
algorithms)
RTT
to
cache
server
(or
RTTs
depending
on
TLS
False
Start),
efficiency
of
TLS
engine
RTT
to
where
the
object
is
stored
+
storage
efficiency
(different
for
requests
to
origin);
lower
bound
=
network
RTT

41. DNS
TCP
(TLS)
TTFB
Download
(TTLB-­‐TTFB)
Time
DNS
TCP
(TLS)
TTFB
TTLB-­‐TTFB
RTT
to
DNS
server,
DNS
iterations,
DNS
caching
and
TTLs
RTT
to
cache
server
(CDN
footprint
&
routing
algorithms)
RTT
to
cache
server
(or
RTTs
depending
on
TLS
False
Start),
efficiency
of
TLS
engine
RTT
to
where
the
object
is
stored
+
storage
efficiency
(different
for
requests
to
origin);
lower
bound
=
network
RTT
Bandwidth,
congestion
avoidance
algorithms
(and
RTT!)

42. Core
object
metrics
• Not
every
request
experiences
every
metric:
– DNS:
once
per
domain
– TCP/TLS
once
per
connection
– HTTP/Download
for
every
object
(not
already
in
browser
cache)
• All
techniques/tools
measure
and
report
these
metrics

43. Resource
timing
http://www.w3.org/TR/resource-­‐timing/

44. Resource
timing
window.performance.getEntries()

45. object
metrics
or
page
metrics

48. Download:
15Mbps
Upload:
5Mbps
Latency:
10
ms,
25
ms

49. 10
msec
25
msec

50. 10
msec
25
msec

51. onload
Speed
Index
Start
Render
10
msec
25
msec

52. What
the…???
• We
always
assume
“all
things
equal”
• Too
many
factors
affect
page
load
time
– 3rd
parties
(sometimes
varying),
content
form
origin,
layout,
JS
execution,
etc
• Too
much
variance

53. 3rd
parties
Source:
httparchive.org

54. To
be
clear…
• Always
use
webpagetest
(or
something
like
it)
to
understand
your
application’s
performance
profile
• Continue
to
monitor
application
performance,
and
always
spot
check
• Be
extremely
careful
when
using
it
to
gage/compare
CDN
performance,
it
can
mislead
you
– If
using
RUM
to
measure
page
metrics,
with
lots
of
data,
things
become
a
little
more
meaningful
(data
volume
handles
variance)

55. What
to
measure
(plus
the
right
metrics)

56. Most
commonly…
• Pick
a
“normal
object”
– e.g.
some
object
on
the
home
page
• Set
up
testing
from
multiple
places
(usually
with
a
synthetic
vendor)
– And
hopefully
not
backbone!
• Compare
either
overall
load
time,
or
some
object
metrics

57. Totally
application-­‐dependent

58. Example:
web
application

59. Web
application:
objects
• Ratio
of
objects
coming
from
CDN
cache
vs
those
coming
from
origin
(through
CDN)
should
determine
objects
to
test
• If
HTML
is
from
origin,
we
must
measure
it
– Essential
to
critical
page
metrics

60. Web
application:
object
sizes

61. Web
application:
metrics
• On
any
page
– DNS
queries
only
happen
a
small
number
of
times
– 6
TCP
connections
per
domain
– Many
many
many
HTTP
fetches
• Core
metrics
– TTFB
– Download
(TTLB-­‐TTFB)
if
important
large
objects
– Should
have
a
good
idea
of
DNS/TCP/TLS,
but
less
critical

62. Web
application
• If
CDN
only
for
static/cacheable
objects:
– One
or
two
representative
assets
– TTFB
and
maybe
download
most
important
Client
CDN
Node

63. X-Cache: HIT

64. Web
application
• If
CDN
also
for
whole
site
– Sample
of
key
HTML
pages,
delivered
from
origin
– TTFB
will
show
efficiency
of
routing
(and
connection
management)
to
origin
– TTLB
will
show
efficiency
of
delivery
Web
Server
Client
CDN
Node

65. Web
application
• If
CDN
also
for
whole
site
– Sample
of
key
HTML
pages,
delivered
from
origin
– TTFB
will
show
efficiency
of
routing
(and
connection
management)
to
origin
– TTLB
will
show
efficiency
of
delivery
Web
Server
Client
CDN
Node
CDN
Node

66. Example:
software
download

67. Software
download:
objects
• Pick
a
standard
file
that
users
will
be
downloading
– Representative
file
size
• Also
pick
something
you
expect
to
be
on
the
CDN
but
not
fetched
all
that
often
– More
on
this
later….

68. Software
download:
metrics
• Ratio
of
TCP-­‐to-­‐HTTP
is
closer
to
1-­‐1
– Especially
if
you
have
a
dedicated
download
domain
– Could
mean
the
same
for
DNS
• For
large
files,
we
care
about
download
time
• Core
metrics:
– TTFB
(+
TCP
and
maybe
DNS,
if
applicable)
– TTLB
– TTLB-­‐TTFB
will
usually
be
a
larger
component

69. Bandwidth/Download
measurements

70. Download
time
• Careful
about
where
you
expect
this
download
to
happen
in
the
lifetime
of
a
TCP
connection
• In
the
beginning
of
the
connection
– Function
of
init_cwnd
and
TCP
slow
start
(and
RTT)
• Later
in
the
connection
– Function
of
congestion
avoidance
and
bandwidth
• Large
files
will
experience
both

71. time
bytes

73. Download
time
• Careful
about
where
you
expect
this
download
to
happen
in
the
lifetime
of
a
TCP
connection
• In
the
beginning
of
the
connection
– Function
of
init_cwnd
and
TCP
slow
start
(and
RTT)
• Later
in
the
connection
– Function
of
congestion
avoidance
and
bandwidth
• Large
files
will
experience
both

74. Cache
hit
ratios

75. Cache
hit
ratio:
traditional
calculation
1
-­‐
Requests
to
Origin
Total
Requests

76. Origin

77. Origin
Cache

78. TCP
Origin
Cache

79. HTTP
Origin
Cache

80. Origin
Cache
HTTP

81. Origin
Cache
HTTP

82. Origin
Cache
HTTP

83. Origin
Cache
HTTP

84. Origin
Cache
HOT
COLD

85. Origin
Cache
cache
“hit”

86. Cache
hit
ratio:
traditional
calculation
1
-­‐
Requests
to
Origin
Total
Requests

87. Isn’t
this
better?
Hits
Total
Requests
@edge

88. Isn’t
this
better?
Hits
Hits
+
Misses
@edge

89. Cache
hit
ratio
vs.
1
-­‐
Requests
to
Origin
Total
Requests
Hits
Hits
+
Misses
@edge

90. Cache
hit
ratio
vs.
1
-­‐
Requests
to
Origin
Total
Requests
Hits
Hits
+
Misses
@edge
Offload

91. Cache
hit
ratio
vs.
1
-­‐
Requests
to
Origin
Total
Requests
Hits
Hits
+
Misses
@edge
Offload
Performance

92. Effect
on
long
tail
content

93. Effect
on
long
tail
content
(long
tail:
Cacheable
but
seldom
fetched)

94. Popular
Medium
Tail
(1hr)
Long
tail
(6hr)

95. Popular
Medium
Tail
(1hr)
Long
tail
(6hr)
Connect
(median)
Popular
14msec
1hr
Tail
15msec
6hr
Tail
16msec

96. Popular
Medium
Tail
(1hr)
Long
tail
(6hr)
Connect
(median)
Popular
14msec
1hr
Tail
15msec
6hr
Tail
16msec
6,400+
measurements
77,000+
measurements
38,000+
measurements

97. Popular
Medium
Tail
(1hr)
Long
tail
(6hr)
Connect
(median)
Popular
14msec
1hr
Tail
15msec
6hr
Tail
16msec
6,400+
measurements
77,000+
measurements
38,000+
measurements

98. Popular
Medium
Tail
(1hr)
Long
tail
(6hr)
Connect
(median)
Wait
(median)
Popular
14msec
19msec
1hr
Tail
15msec
26msec
6hr
Tail
16msec
32msec
6,400+
measurements
77,000+
measurements
38,000+
measurements

99. Popular
Medium
Tail
(1hr)
Long
tail
(6hr)
Isn’t
this
better?

100. Now…

101. Does
all
this
really
matter?

102. Does
all
this
really
matter?
Yes,
but…

105. The
bigger
picture
• It’s
really
easy
to
lock
in
on
a
metric
• Performance
absolutely
matters
• True
performance
isn’t
always
as
easy
to
measure

106. Storage
model
and
long
tail
content

107. Cache
hit
ratios
for
offload
and
performance

108. Footprint

109. HTTP
vs
TLS
footprint

110. Caching
something
you
didn’t
think
you
could
cache

111. Serve
stale
content
if
necessary

112. Key
takeaways
• Everything
is
application-­‐dependent
– Evaluate
how
your
application
works
and
what
impacts
performance
the
most
• Don’t
get
locked
into
a
single
number
• Always
know
your
application
performance
and
bottlenecks
• Be
mindful
of
the
bigger
picture!

113. Thank
you!
hooman@fastly.com