14. Giulia Panozzo - @SequinsNSearch #brightonseo
However, with Causal Impact…
X
15. Giulia Panozzo - @SequinsNSearch #brightonseo
Causal Impact gives you the
confidence to leverage
statistically significant results and
drive changes at scale
16. Giulia Panozzo - @SequinsNSearch #brightonseo
‘But will this help
bring more traffic
in?’
25. Giulia Panozzo - @SequinsNSearch #brightonseo
What is Causal Impact
And how it can help your
strategy
26. Giulia Panozzo - @SequinsNSearch #brightonseo
What is Causal Impact?
27. Giulia Panozzo - @SequinsNSearch #brightonseo
Powerful package to analyse
data and infer the
cumulative impact of a change
in a time series
28. Giulia Panozzo - @SequinsNSearch #brightonseo
Based on BSTS
(Bayesian Structural Time
Series)
statistical model
29. Giulia Panozzo - @SequinsNSearch #brightonseo
Uses past data to predict the
outcome in the absence of the
treatment
(the counterfactual)
30. Giulia Panozzo - @SequinsNSearch #brightonseo
Defines the impact by
measuring the deviation of the
actual VS predicted outcome
31. Giulia Panozzo - @SequinsNSearch #brightonseo
What is Causal Impact?
32. Giulia Panozzo - @SequinsNSearch #brightonseo
How can it help us in marketing?
33. Giulia Panozzo - @SequinsNSearch #brightonseo
How can it help us in marketing?
Example of a clear winner from a title tag change
Clicks: +58% CTR: +38% Position: -15%
34. Giulia Panozzo - @SequinsNSearch #brightonseo
It can validate proposed strategy
changes in case of any doubts
35. Giulia Panozzo - @SequinsNSearch #brightonseo
It’s great to clearly show
stakeholders
the impact of our team’s work
36. Giulia Panozzo - @SequinsNSearch #brightonseo
It can help forecast the
direction of changes at scale and
help
make a case for more resources
37. Giulia Panozzo - @SequinsNSearch #brightonseo
How can it help us in marketing?
Example of a clear loser from a title tag change
38. Giulia Panozzo - @SequinsNSearch #brightonseo
By clearly identifying a winner or
loser, we can understand
what works and doesn’t work
for our audience
39. Giulia Panozzo - @SequinsNSearch #brightonseo
How to run your analysis
with Causal Impact on
R Studio
40. Giulia Panozzo - @SequinsNSearch #brightonseo
1. Download R Studio
Download R first
https://cran.r-project.org/
Download RStudio
https://www.rstudio.com/products/rstudio/download/
41. Giulia Panozzo - @SequinsNSearch #brightonseo
1. Download R Studio
43. Giulia Panozzo - @SequinsNSearch #brightonseo
2. Prepare the data
44. Giulia Panozzo - @SequinsNSearch #brightonseo
2. Prepare the data
45. Giulia Panozzo - @SequinsNSearch #brightonseo
The first column is always
your test group.
Other columns can be used as
control groups if they are a good
fit
46. Giulia Panozzo - @SequinsNSearch #brightonseo
The pre-period should be at least
twice as long as the post-period,
to allow the model to plot the
actual VS predicted outcome
47. Giulia Panozzo - @SequinsNSearch #brightonseo
Any column with 0 should be
either removed or corrected
48. Giulia Panozzo - @SequinsNSearch #brightonseo
Isolated 0 in data set
Multiple 0s
VS
49. Giulia Panozzo - @SequinsNSearch #brightonseo
3. Run the script!
50. Giulia Panozzo - @SequinsNSearch #brightonseo
Choose file to import
56. Giulia Panozzo - @SequinsNSearch #brightonseo
Now give it a go!
Request access to this
script here
57. Giulia Panozzo - @SequinsNSearch #brightonseo
What I’ve learned from (several)
trials and errors…
58. Giulia Panozzo - @SequinsNSearch #brightonseo
The date column should always
be removed
when using this script
59. Giulia Panozzo - @SequinsNSearch #brightonseo
Column titles can error out if
they contain special characters,
spaces, capitalised letters
60. Giulia Panozzo - @SequinsNSearch #brightonseo
Start small, then expand your
datasets with additional controls
and features once you’re
comfortable with the script
66. Giulia Panozzo - @SequinsNSearch #brightonseo
Create a document to map
internal changes & external events
This will allow you to take into account any
other known factors and isolate the treatment
in the analysis
67. Giulia Panozzo - @SequinsNSearch #brightonseo
2. Mind the Outliers!
82. Giulia Panozzo - @SequinsNSearch #brightonseo
Sometimes, your test will be inconclusive, or might
be a loser even when you thought it’d be an easy
winner
84. Giulia Panozzo - @SequinsNSearch #brightonseo
In that case, you can run the test a little longer,
or repeat the test with bigger groups
85. Giulia Panozzo - @SequinsNSearch #brightonseo
If it’s still inconclusive or a loser, it’s probably best
to revert the change and focus on other tests
86. Giulia Panozzo - @SequinsNSearch #brightonseo
References and useful resources
• How we use causal impact analysis to validate campaign success - Part and Sum
• Measuring No-ID Campaigns with Causal Impact - Remerge & Alicia Horsch
• Causal Impact – Data Skeptic
• R Studio on GitHub
• The Comprehensive R Archive Network
• Causal Impact for App Store Analysis - William Martin