Presented at Streaming Media West. Techniques for evaluating CDN providers for video streaming. Tips and tricks for using serverless computing tools like AWS Lambda and GOAD for quick and reliable testing. Including questions like: * How to evaluate new CDN partners in a highly commoditized and undifferentiated industry? * How long can I keep origin request cached on the CDN? (eg; 1 month, 6 months, 1 year?) * What are the costs of that data retention on your provider? * What are my time to first frame values? * IF all player conditions are equal, how long did it take to…. make a player request download the manifest; queue into a player download a chunk; seek to a chunk begin playback * How does this improve my QoS experience for my clients for my consumers? * Should I make my chunk size small? * Should I invest in re-encoding my library with a new codec?

1. Zype
CDN Performance & Cost
Economics
Streaming Media West
11/19/2019
Check us out at
https://www.zype.com

2. Agenda
About Zype
CDN Evaluation Background
Testing Methodology & Approach
Evaluating KPIs & Results
Learnings & Opportunities

3. About Zype
● Full stack video content management and
distribution infrastructure.
● Dashboards, APIs, & publishing tools to build
video products and services at scale
● Publish apps & video services across web,
social, mobile, and OTT / connected TV
● On-demand, live, linear-live, and linear playout
products
● Support for subscription, transaction, and ad
monetization
● Founded in 2014, HQ in New York

4. VideoMeta™ CMS
Easily import, manage
and curate video and
manage subscriber data
App Distribution
Distribute and monetize
content across all web,
mobile and CTV/OTT devices
and MVPD partners
Monetization
Full support for all monetization
models - Subscriptions,
Transactional, Ad-Based or any
hybrid.
Analytics
Powerful analytics dashboard
providing valuable insight into
engagement, streaming, and
revenue
Streaming
CDN-agnostic delivery with
performance and analytics in
mind
Zype Live
Deliver and monetize premium
live video to any destination with API
enabled encoding for live, linear and
social multicasted events
Embeddables
Extend the power of the Zype
platform to your own website
with easy-to-use video embeds
Integration Hooks
Integrate Zype with all core
SaaS products you use to
power your business
4
Social & Syndication
Easily publish content to key
social platforms to build
awareness, drive adoption or
syndicated revenue streams
Full stack Video Content Management
& Distribution infrastructure

5. Aren’t CDNs all the same?
How to evaluate new CDN
partners in a highly
commoditized and
undifferentiated industry?

6. Big 3: CDN Evaluation Rubric
Find 3 well
known, top tier
providers to
evaluate
Providers Core Criteria Integration
Evaluate
performance,
costs, and
feature sets
How easy is it
to integrate into
our delivery
pipeline

7. Primary performance metrics we evaluated
● Time to first frame
● Caching policies
● Cache-hit ratios
● Reducing origin hits by
advertised features
● Origin egress
● Player performance
(Sorry, we didn’t care about
the Weissman scores)

8. Performance & Cost KPIs
Cost Performance● What are the cache hit ratios over time
○ How long can we keep origin requests
cached on the CDN before they expire
○ Understand the benefits and cost of
Hyper/Mid-Tier/Origin Shield Caches
● We need to understand what my egress traffic
(and costs) would be for origin request
○ How much transfer costs would be incurred
over time (day/month/year)
○ Understand how CDN caching policies will
affect origin requests over time

9. Features we actually care about
Origin shields /
hypercaching
Mid-tier
caching
architecture
and
implications
Security
features
Delivery cost
Origin egress
and routing
05
01
02 03
04

10. Testing methodology / criteria
Origin & Transfer
Considerations
Load &
Performance
Geographic
Distribution
Endpoints &
Players

11. Using Goad to Build a Test Framework
What is a goad?
AWS Lambda powered, highly
distributed, load testing tool written
Go
GOAD is a nodeJS application that
executes within AWS lambda.
Given the nature of it’s architecture,
it’s highly distribute, relatively simple
to implement, and lightweight.
Requiring very little engineering
engagement
Check it out > https://goad.io/

12. Using Goad to Simulate Globally
Analyze and Compare the results
Leverage AWS Athena for analyzing the data
Storage the resultant data in S3
SQS queues for centralizing results
Build a Goad microservice; leveraging
regionalized Lambda functions for generating
global requests and playback execution
Build staging environments; each configured to
a CDN provider
Distributed Player Requests at
Scale
Testing Framework & Methodology

13. Test Execution Strategy Using Goad
Regions
Assets
Concurrency
Tests
• US EAST, US WEST, CANADA,UK, INDIA,
SOUTHEAST ASIA
• 10 unique & content specific video assets; per
CDN
• A mix of consistent and varied content types
(50/50)
• Identically encoded
• 2 live streams
• How many concurrent, sustained requests do
we want to test for
•1 per minute
•5 per minute
•10,000 per 5 minutes
•100,000 per 15 minutes
• Tests were executed over the course of a
month + 1 week
• A typical test would span several days, some
as long as a week; continuous request load

14. Analysis Architecture
● Goad microservice using regionalized
Lambda cloud functions for generating
player requests and playback
● Goad sends those requests directly to
a bucket (us-east-1)
● Where a predefined AWS Glue model
is leveraged against the JSON files in
S3 by Athena
Results

15. Understanding our Test Results: Performance
● How long can I keep origin request cached on the CDN?
○ (eg; 1 month, 6 months, 1 year?)
○ what are the costs of that data renition on your provider (if applicable)
● What are my time to first frame values per provider
○ IF all player conditions are equal, how long did it take to….
■ make a player request
■ download the manifest; queue into a player
■ download a chunk; seek to a chunk
■ begin playback
○ How does this improve my QoS experience for my clients for my consumers
■ should I make my chunk size small
■ should I invest in re-encode my library with a new codec

16. Results
Current CDN CDN 2 CDN 3 CDN 4
April 2019 VOD Testing to Date Testing to Date Testing to Date
Total = 565.86 TB Total = 632.8.4GB Total = 5.20TB Total = 1.17GB
Origin Egress = 367.44 TB Origin Egress = 2.4GB Origin Egress = 29.7GB Origin Egress = 1.01GB
Total Requests = 251.73 million Total Requests = 1.22 million Total Requests = 1.55million Total Requests = 700,360
Hits = 91.06 million Hits = 1.21 million Hits = 1.37million Hits = 294,520
Misses = 159.23 million Misses = 3810 Misses = 180k Misses = 405,840
Errors(all) = 1.44 million Errors(all) = 0 Errors(all) = 900 Errors(all) = too many
Avg Efficiency for April = 63% Avg Efficiency = 98.9 % Avg Efficiency = 88.7% Avg Efficiency = 42%

17. AVG CDN Latency North Americatime(ms)
CDN 4
CDN 1
CDN 3
CDN 2
April - May 2019

18. Understanding our Test Results: Cost
● Breaking down results by KPI
○ Calculate the following (for 1 day, 1 week, 1 month)
■ How content was delivered?
■ How much did that delivery cost?
■ How many origin requests were made?
■ How much did I pay in origin transfer cost for those request?
■ What are my fixed CDN platform fees?

19. Evaluating cost (don’t forget origin egress!!!)
Delivery
Costs (per
GB),
Platform
Origin
Egress
Codecs &
Advanced
Features
Total Cost of Ownership

20. Real VOD Use Case Post-
Optimization

21. Conclusions
● Caching policies: the longevity of assets sitting on the CDN is important
○ Decreases overall origin requests
○ Decreases overall cdn transfer costs (if applicable)
● As a provider, origin egress (particularly when operating origins within
the cloud) can make or break your delivery margins
○ Understanding origin transfer cost is imperative
● Routing and first-hop considerations
○ Origin egress transfer costs are directly related to outbound
traffic egressing. To optimize routing, understand the network
topology between your cloud origin and your CDN of choice (if
not all of them). There can be hidden cost savings there
● Negotiate a good delivery rate

22. https://www.zype.com
New York City
115 Broadway, FL5
New York, NY 10006
Thanks!
Check us out at
https://www.zype.com