1.
The Always-On
Approach
How to Continually Improve Your
Streaming Advertising with Incrementality
FEATURING:

2.
Today’s Logistics
Kerry Mallett
Senior Content Specialist,
Webinar
Recording & slides will be in your inbox
shortly after the session
Download relevant resources
Join our chat and submit questions for live Q&A
Log back anytime with same link
Join our poll questions - including if you want to talk to
our teams after!

3.
WE’VE MASTERED THE
MEDIA THAT
MATTERS MOST
Tinuiti is the largest independent performance
marketing firm across Streaming TV and the
Triopoly of Google, Facebook, Amazon.
Employees
Media managed
Award-winning
culture*
$3B
#1
1000+

4.
Some of
Our Clients
Our
Recognition
For Commerce Leaders Of Today And The Future

5.
Our Speakers
PHILIP RUNDALL
Senior Manager of
Economics
TINA MOFFETT
Principal Analyst
LAUREN WOLFEN
VP of Client Strategy &
Analytics
WITH GUEST SPEAKER

6.
Agenda
● Improve Your Streaming With Incrementality Testing
● Introduction To Randomized Controlled Experiments
● Incrementality Case Study
● Traditional And Updated Incrementality Testing
● What Is Incrementality & Always-On Incrementality?
● What This Means For Modern Marketers
6

7.
What is your comfort level with
experimental methods from other
channels?
● Very comfortable
● Somewhat comfortable
● Not comfortable
POLL

8.
How much of a priority is building
experiments to measure incrementality?
● A big priority
● Somewhat a priority
● Not a priority
POLL

9.
Improve Your Streaming With
Incrementality Testing
Tina Moffett
Principal Analyst
B2C Marketing Executives

10.
10
© 2022 Forrester. Reproduction Prohibited.
Let me introduce two macro trends that are reshaping
business.
Consumer hyper adoption Digital disruption
Consumers who are always willing to try
new products or services

11.
11
11
© 2017 FORRESTER. REPRODUCTION PROHIBITED.
B2C Marketing Executives have competing priorities
25% Improve
ROI effectiveness.
24% Improve
marketing
alignment and
collaboration with
other
departments.
23% focus on
post-sale
customer
engagement.

12.
12
© Forrester Research, Inc. All rights reserved.
Businesses have a
lot to worry about.
Data integrity
Geopolitical fluctuations
Empowered customers
Supply chain stability
Climate change
Economic uncertainty
Human health and wellness

13.
13
© Forrester Research, Inc. All rights reserved.
Massive hurdles prevent data driven marketing decisions
Source: Forrester Global Marketing Survey, 2022 (B2C)
Base: 638 B2C Marketing decision-makers whose do not rate their marketing org's use of measurement and analytics as optimized
Question: Please select the top challenges that prevent your organization from making use of measurement and analytics.
19%
Data materializes too
fast for us to manage.
21%
Inconsistent levels of
data quality.
18%
Buy in on the value of
measurement and
analytics.
“The biggest challenge our marketing measurement
is the speed at which we're analyzing data. The lag
time between getting the data and evaluating the
results is too long.”
--Automotive Marketing Analytics Executive

14.
14
14
© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Unified measurement blend strategic and tactical marketing
measurement to enable customer obsessed strategies

15.
15
© Forrester Research, Inc. All rights reserved.
Experimental design is the process
of carrying out research in an
objective and controlled fashion so
that precision is maximized, and
specific conclusions can be drawn
regarding a hypothesis statement.
Experimentation helps us decipher cause and effect
Source: Science Direct

16.
16
© Forrester Research, Inc. All rights reserved.
B2C marketing executives use experimentation to understand
the incremental impact of marketing
Marketers use experimental designs
to analyze how a marketing campaign
incrementally influences customer
behaviors.
• Provide simplicity and clarity.
• Measures impact of multiple
marketing variables, like creative
and messaging.

17.
17
© Forrester Research, Inc. All rights reserved.
Building experiments require careful development of test
and control cells
Total
Populatio
n
Test Group
Control Group
Test Group
Results
Control Group
Results
Incremental
Lift
Measurement

18.
18
© Forrester Research, Inc. All rights reserved.
Source: Forrester Global Marketing Survey, 2022 (B2C)
Base: 638 B2C Marketing decision marketers
2021
What advanced marketing measurement approaches does your organization use to understand
marketing efficacy?

19.
19
© Forrester Research, Inc. All rights reserved.
Source: Forrester Global Marketing Survey, 2022 (B2C)
Base: 638 B2C Marketing decision marketers
2022
2021
What advanced marketing measurement approaches does your organization use to understand
marketing efficacy?

20.
20
© Forrester Research, Inc. All rights reserved.
Naked Wines used incrementality testing to boost brand
awareness and revenue
• Summary: Wanted to increase
brand awareness in new
Australian market.
• Approach: Developed testing
strategy across key brand specific
channels.
• Results: Increase in brand
awareness 59%, from 40%.
̶ Aggregate market payback
increased.

21.
21
© Forrester Research, Inc. All rights reserved.
Apply core best practices to upstart your incrementality measurement
Build basic statistics and
data literacy.
1
Use your analytics team to
help construct or guide your
incrementality tests.
2
Split test cells across
homogenous populations,
like segment level and
ensure audience and
geo-location areas
consistency for precise
incrementality results.
3

22.
Putting Incrementality
into Practice in Streaming

23.
Experimental Methods in OTT Can Provide More Efficient
Decisions
23
● As omnichannel marketers, we
need to understand
incrementality throughout the
funnel to drive optimal
investment decisions.
● OTT can serve as a full-funnel
marketing tool, broadening
reach and filling the top of the
funnel while producing efficient
acquisition in its own right.
Awareness
Acquisition

24.
There Are A Few Techniques To Understand
Incrementality Throughout The Funnel
Experimental
Methods
Quasi-Experimental
Methods
Observational
Methods

25.
Experimental Methods Are The Gold Standard
25
OTT Incrementality Tests benefit from being experiments.
● Test designed to meet requirements.
● Proper randomization is done.
● Accurately measures incrementality on an OTT publisher.
The basic principle of the test
involves creating two distinct
impression groups: a control group,
and an exposed group.
TV lift tests can often be run with a
conservative budget and a relatively
short time frame to produce
statistically significant results that
equate to a measured lift in response.
Measurement
To understand the incrementality of
TV, we measure and compare the
responses of the control group vs. the
group that saw the commercials over
a distinct time frame and for a
particular KPI.
Test Structure Implementation

26.
Test Overview
Client industry: Wedding
services
KPI: Signups
Test duration: 3 weeks
Property: Hulu Reality Cluster
Total test spend: $30k
Spend ratio: 80% / 20%
(exposed / control)
Findings
Based on the difference in
response rates, the exposed
group was ~6x more likely to
register for an account than
the control group.
Control Group Exposed Group
N= 159k
N= 102 signups
N= 36k
N= 4 signups
4.0
3.8 x 10-5
t-Stat
p-Value
CASE STUDY
Streaming Audience Holdout
Audience Sample
Sample Response
Response Rate
1.1 x
10-4
6.4 x 10-4

27.
Experiments Have Their Practical Limitations
27
Truth in
the test
Truth in
real life
Generalization
Internal
Validity
External
Validity
● Money spent on placebo ads.
● Limited OTT partner capabilities.
● Incrementality changes over time.
● Results to not apply to other publishers.
Incrementality Test Drawbacks Problems With External Validity

28.
Quasi-Experimental Methods Solve Drawbacks And
Maintain Validity
28
● Quasi-experimental methods are used
when randomization is not possible.
● Results are still valid even if
randomization is not used.
● Commonly used in the social sciences.

29.
Always-On Incrementality Allows For Costless,
Continuous, and Campaign-Wide Insights
29
Incrementality: Measuring a funnel
step or action that would not have
occurred without a specific
interaction, such as seeing an
advertisement.
Always-On Incrementality: Constantly
being able to measure the
incrementality of a marketing
campaign. Not just a static
measurement based on a test that
has been run in the past.
Incremental
Conversions
Converted because
they saw the OTT
impression
Background
Responses
Would have
converted
regardless of
exposure to
impression
Video Impressions That Converted

30.
How Always-On
Incrementality Works
30
● Impressions are served on
the same publishers by
multiple clients.
● Behaviorally match
impressions to create
control groups.
● Quantify incrementality.
Daily Measurement Process

31.
What Does This Mean For A Marketer?
31
● With always-on
incrementality, we can
take the guesswork out of
driving a campaign as a
marketer.
● No longer need to apply a
haircut to the results to
get the “real” metrics, now
know the full picture
based on always-on
incrementality.

32.
Key Takeaways Tinuiti/Bliss Point Media:
1. Quasi-experimental methods in OTT can lead
more efficient decisions.
2. Always-on incrementality allows for costless,
continuous, and campaign-wide insights.
Forrester:
1. Create a testing culture in your organization by
sharing success stories and the economic value
of marketing experiments.
2. Build a baseline of analytics literacy. Start with the
constructs of what makes a good test: statistical
significance, population representation, and clear
testing parameters.

33.
Schedule Your
Consultation
with a Streaming
Expert

34.
Q&A
PHILIP RUNDALL
Senior Manager of
Economics
TINA MOFFETT
Principal Analyst
LAUREN WOLFEN
VP of Client Strategy &
Analytics
WITH GUEST SPEAKER

35.
35
Stay informed
on the future of
digital marketing
Visit our content hub
➜

36.
Thank you!