FOX uses Optimizely's full platform solutions to support experimentation across its product and marketing teams. FOX relies on experimentation to drive key business metrics for its brands like FOX Now, FOX Sports, and FOX News. The Director of Digital Strategy and Optimization at FOX's Growth Marketing team oversees experimentation efforts to improve subscription signups, video and content engagement, and pay-per-view purchases. FOX Product teams also leverage Optimizely to test product updates, inform decisions, and validate hypotheses. The Director of Product Management leads an Optimization Guild to help FOX product teams design, run and analyze experiments.

1. 1
How FOX Tests Everything
from Mobile, Web, to Living
Room Devices
Sara Miller & Paul Tongyoo
January 2020

2. Housekeeping
● Customize the widgets on your page to
your preference
● This webinar is recorded and you will
receive the link with the slides in the next
few days
● We will have time for questions! Please
submit them in the Q&A box on the left
side of the screen

3. 3
Paul Tongyoo
Director, Product Management
Sara Miller
Director, Optimization and Digital Strategy
Introductions
Mei Luo
Strategic Customer Success Manager

4. optimizely.com/digital-change-agents/

5. Agenda
● Introduction
● About Optimizely
● FOX’s Journey
● Q&A

6. Revenue, share of wallet, funnel conversion, risk
mitigation, ops efficiency
What
We Do Next gen “Test and Learn” platform for enterprise-
wide digital experience optimization
Over 1B impressions daily
Replaces Digital Guesswork
With Evidence-based Optimization
Digital Experimentation
SaaS Platform
Apply the scientific method to “at-scale”
business decision making.
20X increase in Yield

7. Experimentation: Testing variations of digital experiences
with real users and deploying what wins
+21%
-5%

8. The digital-centric
customer journey: end-to-
end, multi-device
engagement
Marketing Product & Engineering Support
Views ad
at Checkout
Signs up for
7 day trial
Tracks
packages
Manages
account
Converts to
subscriber
Downloads
App

9. Frontend UI Backend Business
Logic & Data
The anatomy of an experience
i.e. navigation,
search location
& visual treatment
Copy
Images &
Colors
Layout Search algorithms
Personalize content
based on previous
behavior
Recommendations
Make your headlines
more personal

10. “Our success is a function of
how many experiments we do
per year, per month, per week,
per day.”
Jeff Bezos
“Our aim is to create the best
product for our customers, and
we do that through constant
innovation and testing.”
Gillan Tans, CEO
“Our company culture
encourages experimentation
and free flow of ideas.”
Larry Page
“We use experimentation and
testing to inform as much of the
business as we possibly can” -
Gregory Peters, CPO
Today’s Digital Leaders Win By
Using Experimentation At-Scale

11. Spent with Digital Media
19 hr
week
2018
Digital-Centric Customer
Spent with Digital Media
41
$396m
Mobile commerce Mobile commerce
$1 8t.
2008
Digital-Convenient Customer
hr
week
Your customer has fundamentally
changed how they engage

12. What
We Enable
1500+
+51%
+$21M
+19%
+28%
Improved subscription
acquisition
Users Scale and agility
of experiments; users
incremental
in revenue
Improved Driver
Acquisition
increase in cross-sell
performance

13. We’re Proud to Work With Great Global Enterprises
24 of the Fortune 100 have chosen Optimizely to drive their digital experience

14. FOX Leverages Optimizely to
Foster a Data-Driven Culture
FOX relies on Optimizely’s full platform solutions
to support experimentation in product and
marketing teams among brands like:
● FOX Now
● FOX Sports
● FOX News
To drive key business metrics like:
● Video & Content Engagement
● Subscription Signups
● Pay-Per-View Purchases

15. How Growth Marketing at
FOX Experiments Everything
JANUARY 15, 2020

16. Topics
● About Me
● My Experimentation Philosophy
● Plans to Grow and Scale
● Advice to Other Marketers
● How Marketing and Product
Experimentation Works Together

17. ABOUT ME
17
Director of Digital
Strategy and
Optimization
Experience in Marketing
and Product
8+ yrs CRO and
Experimentation
Experience
Avid sports fan
Experimented with a
variety of tools

18. My Experimentation
Philosophy

19. Experiment EVERYTHING*
*where it makes sense unless you have unlimited resources

20. My Experimentation Philosophy
20
If decisions are not going to be impacted by results of
the experiment, focus resources elsewhere

21. Optimization should be
scientifically rigorous*
*but be flexible to meet the needs and realities of the business

22. My Experimentation Philosophy
22
The real world isn’t a perfectly controlled environment,
so you may have give up certain things in order to move
at the pace the business needs.

23. Pre-Experiment analysis is more
important than post analysis*
*Post experiment analysis communication is the most important of all

24. My Experimentation Philosophy
24
If you do the work in pre-experiment analysis, your post
experiment analysis is easy and you can avoid
inconclusive results and wasted resources.

25. Being wrong is a good thing*
*The more wrong you are in the beginning, the more right you become later

26. My Experimentation Philosophy
26
One of the key benefits of experimentation is preventing
incorrect assumptions from being fully implemented.

27. How we plan to grow
and scale

28. FOX Growth Marketing continues to grow...
28
Continue to
optimize key
User Journeys
More Personalization
to support array of
FOX digital users
Capitalize on Major
Events to drive key
business objectives
Expand Optimization
onto Mobile and
Connected Devices

29. How other Marketers
can grow and scale

30. How other marketers can grow and scale
30
Start with Low-
Hanging Fruit
Focus on
Business
Objectives for
Prioritization
Insist on dedicated
resources for
Optimization
Cast a wide net
for brainstorming
and ideation

31. How Marketing and Product Experimentation Work Together
31
Ideation (Marketing)
Exploratory experiments to
validate research/analysis
based hypotheses
Example
Product/Feature
Development
Cycle
Definition/Planning
(Product)
Exploratory experiments for
direction and to inform
prioritization
Development (Product)
QoS experiments to prevent unintended
consequences of feature design
Release (Product)
Validation experiments before
full launch to validate expected
impact
Monitor/Optimization
(Marketing)
Optimization experiments
to personalize experiences,
optimize user journeys,
increase adoption, etc.

32. HOW FOX PRODUCT
TEAMS USE OPTIMIZELY
January 15, 2020
Process, Examples, Advice

33. 3 3
WHO AM I?
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y

34. 3 4
Who am I?
• A Director of Product Management focusing on Optimization
• Leads Optimization Guild to help anyone at FOX use experimentation
• Software Engineer before Product Manager
• About 7 years at FOX managing Web, Mobile, Analytics, Optimization products
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Paul Tongyoo

35. 3 5H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
PRODUCT TEAM OVERVIEW

36. 3 6
How to help
all these PMs
learn from
testing?
Web
Mobile
OTT (Native)
OTT (Web-based)
Sports Web
Sports Mobile
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Pilot Product Team: Los Angeles
Commerce & Customer Care
Profile & Personalization
Video
SDK
CMS
and more…

37. 3 7H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
PROCESS DISCOVERY

38. EXPERIMENTATION PROCESS
PRODUCT
MANAGER
EV ERY THING =
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Strategy #1: Enablement

39. EXPERIMENTATION PROCESS
PRODUCT
MANAGER
EV
Product Managers couldn’t prioritize time for:
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Strategy #1: Enablement
What I Learned
• Design & planning
• Test configuration & management
• Info sharing across organization
• Impression governance

40. EXPERIMENTATION PROCESS
PRODUCT
MANAGER
OPTIMIZATION
GUILD
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Strategy #2: Service
PROJECT
TIMEFRAME
WRITE BRIEF
WRITE DEV
GUIDE
DEV & QA
CONFIG/RUN
EXPERIMENT
KEEP ORG INFORMED
MONITOR IMPRESSIONS
DEFINE 4
W’s*
DERIVE
LEARNING
S
*What, Why, Who, and What Metrics

41. EXPERIMENTATION PROCESS
PRODUCT
MANAGER
OPTIMIZATION
GUILD
PROJECT
TIMEFRAME
WRITE BRIEF
WRITE DEV
GUIDE
DEV & QA
CONFIG/RUN
EXPERIMENT
KEEP ORG INFORMED
MONITOR IMPRESSIONS
DEFINE 4
W’s*
DERIVE
LEARNING
S
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Strategy #2: Service
*What, Why, Who, and What Metrics

42. EXPERIMENTATION PROCESS
PRODUCT
MANAGER
OPTIMIZATION
GUILD
PROJECT
TIMEFRAME
WRITE BRIEF
WRITE DEV
GUIDE
DEV & QA
CONFIG/RUN
EXPERIMENT
KEEP ORG INFORMED
MONITOR IMPRESSIONS
DEFINE 4
W’s*
DERIVE
LEARNING
S
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Strategy #2: Service
*What, Why, Who, and What Metrics

43. EXPERIMENTATION PROCESS
PRODUCT
MANAGER
OPTIMIZATION
GUILD
PROJECT
TIMEFRAME
WRITE BRIEF
WRITE DEV
GUIDE
DEV & QA
CONFIG/RUN
EXPERIMENT
KEEP ORG INFORMED
MONITOR IMPRESSIONS
DEFINE 4
W’s*
DERIVE
LEARNING
S
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Strategy #2: Service
*What, Why, Who, and What Metrics

44. EXPERIMENTATION PROCESS
PRODUCT
MANAGER
OPTIMIZATION
GUILD
PROJECT
TIMEFRAME
WRITE BRIEF
WRITE DEV
GUIDE
DEV & QA
CONFIG/RUN
EXPERIMENT
KEEP ORG INFORMED
MONITOR IMPRESSIONS
DEFINE 4
W’s*
DERIVE
LEARNING
S
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Strategy #2: Service
*What, Why, Who, and What Metrics

45. EXPERIMENTATION PROCESS
PRODUCT
MANAGER
OPTIMIZATION
GUILD
PROJECT
TIMEFRAME
WRITE BRIEF
WRITE DEV
GUIDE
DEV & QA
CONFIG/RUN
EXPERIMENT
KEEP ORG INFORMED
MONITOR IMPRESSIONS
DEFINE 4
W’s*
DERIVE
LEARNING
S
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Strategy #2: Service
*What, Why, Who, and What Metrics

46. EXPERIMENTATION PROCESS
PRODUCT
MANAGER
OPTIMIZATION
GUILD
PROJECT
TIMEFRAME
WRITE BRIEF
WRITE DEV
GUIDE
DEV & QA
CONFIG/RUN
EXPERIMENT
KEEP ORG INFORMED
MONITOR IMPRESSIONS
DEFINE 4
W’s*
DERIVE
LEARNING
S
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Strategy #2: Service
*What, Why, Who, and What Metrics

47. 4 7
EXPERIMENTATION PROCESS
PRODUCT
MANAGER
OPTIMIZATION
GUILD
PROJECT
DURATION
WRITE BRIEF WRITE DEV GUIDE
DEV & QA
CONFIG/RUN
EXPERIMENT
KEEP ORG INFORMED
MONITOR IMPRESSIONS
DEFINE 4
W’s
DERIVE
LEARNINGS
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Service Strategy Observations
• Increased engagement
• Increased test-informed decisions
• Increased experimentation expertise
• Foundation for future enablement

48. 4 8H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
2020 GOALS

49. 4 9H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
2020 Goals
• "Experiment Running on Every Device”
• Enable Engineering Team Experimentation
• Build Pre-Experiment Analysis Tools
• Expand beyond Los Angeles

50. 5 0H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
EXAMPLE EXPERIMENTS

51. 5 1
New DesignControl What: Test new Series Page design
Why: Current UI “not clear”, Low CVRs
Who: All Series Page Traffic
What Metrics: CVRs for Notification Opt-in,
Favoriting, Starting Video
How: Implemented one Optimizely “Live
Variable” to toggle visibility of new design
What We Learned: New Design resulted in stats-
sig CVR increases across all Metrics:
• Global Notifications: +28%
• Series Notifications: +64%
• Favoriting Series: +15%
• Starting Video: +7%
H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Test to Validate Product Updates

52. 5 2H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Test to Inform Decisions

53. 5 3H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Test to Inform Decisions

54. 5 4H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Add
Apple Sign In
Test to Inform Decisions

55. 5 5H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Add
Apple Sign In
Remove
Facebook
Test to Inform Decisions

56. 5 6H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
What effect does removing Facebook
have on sign up rate?
Test to Inform Decisions

57. 5 7H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
KPI
Effect of Removing
Facebook
Decision
Sign Up Rate Stays Flat or Increases Remove Facebook
Sign Up Rate Decreases Add Apple Sign In
Test to Inform Decisions

58. 5 8H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Control No Facebook
Sign Up Form
Test to Inform Decisions

59. 5 9H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Control
No
Facebook
Sign Up
Form
Sign In
Form
How to
implemen
t this?
“New”
Users
Test to Inform Decisions
“Existing”
Users
Experiment

60. 6 0H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Sign Up Form
2. Call
activate(…)
method
Sign In Form
3. Call
getVariation(…)
method
1. Integrate
Optimizely
Full Stack
SDK
Test to Inform Decisions

61. 6 1H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
ADVICE TO PRODUCT MANAGERS

62. 6 2H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
Advice to Product Managers
• Changing the UI is not enough. Validate your test’s data and statistical power as well.
• Evaluate the credibility, accessibility, and usability of your data. Can you improve?
• New to statistics? Pick up a book! Will motivate, rationalize, and refine your testing efforts.

63. 6 3H O W F O X P R O D U C T T E A M S U S E O P T I M I Z E L Y
THANKS!