Se ha denunciado esta presentación.
Se está descargando tu SlideShare. ×

Benchmark_Winner or Loser_GiuliaPanozzo.pdf

Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Anuncio
Cargando en…3
×

Eche un vistazo a continuación

1 de 96 Anuncio

Más Contenido Relacionado

Más reciente (20)

Anuncio

Benchmark_Winner or Loser_GiuliaPanozzo.pdf

  1. 1. Giulia Panozzo StockX @SequinsNSearch
  2. 2. Giulia Panozzo - @SequinsNSearch Winner or Loser? Understanding the effect of your SEO changes with Causal Impact Agenda: 1. What does Causal Impact do? 2. Why should you care about statistics and Causal Impact? 3. What is it really – and how can it help your strategy? 4. Demo: how to run an analysis with Causal Impact 5. Statistics VS the world
  3. 3. Giulia Panozzo - @SequinsNSearch What does Causal Impact do?
  4. 4. Giulia Panozzo - @SequinsNSearch Example: Title change ‘Let’s add a price point to the title!’
  5. 5. Giulia Panozzo - @SequinsNSearch Example: Title change ‘Let’s add a price point to the title!’
  6. 6. Giulia Panozzo - @SequinsNSearch Change implemented
  7. 7. Giulia Panozzo - @SequinsNSearch
  8. 8. Giulia Panozzo - @SequinsNSearch Pre- implementation data Post- implementation data
  9. 9. Giulia Panozzo - @SequinsNSearch
  10. 10. Giulia Panozzo - @SequinsNSearch
  11. 11. Giulia Panozzo - @SequinsNSearch Why should you care about Causal Impact (and Statistics?)
  12. 12. Giulia Panozzo - @SequinsNSearch Because Testing is Hard!
  13. 13. Giulia Panozzo - @SequinsNSearch
  14. 14. Giulia Panozzo - @SequinsNSearch
  15. 15. Giulia Panozzo - @SequinsNSearch
  16. 16. Giulia Panozzo - @SequinsNSearch Example ‘Let’s add a price point to the title tag!’
  17. 17. Giulia Panozzo - @SequinsNSearch Example ‘Let’s add a price point to the title tag!’
  18. 18. Giulia Panozzo - @SequinsNSearch ‘But will this help bring more traffic in?’
  19. 19. Giulia Panozzo - @SequinsNSearch ‘Well…’
  20. 20. Giulia Panozzo - @SequinsNSearch
  21. 21. Giulia Panozzo - @SequinsNSearch However, with Causal Impact… X
  22. 22. Giulia Panozzo - @SequinsNSearch Causal Impact gives you the confidence to leverage statistically significant results and drive changes at scale
  23. 23. Giulia Panozzo - @SequinsNSearch ‘But will this help bring more traffic in?’
  24. 24. Giulia Panozzo - @SequinsNSearch YES (Most likely) NO (Most likely)
  25. 25. Giulia Panozzo - @SequinsNSearch You can use Causal Impact on a number of domains, not only on SEO tests!
  26. 26. Giulia Panozzo - @SequinsNSearch Impact of feature changes on app installs
  27. 27. Giulia Panozzo - @SequinsNSearch Impact of influencer campaigns
  28. 28. Giulia Panozzo - @SequinsNSearch Impact of offline events and campaigns
  29. 29. Giulia Panozzo - @SequinsNSearch And almost any time series data
  30. 30. Giulia Panozzo - @SequinsNSearch If you run Causal Impact on R Studio, it’s free and open-source
  31. 31. Giulia Panozzo - @SequinsNSearch
  32. 32. Giulia Panozzo - @SequinsNSearch What is it really – and how can it help your strategy?
  33. 33. Giulia Panozzo - @SequinsNSearch What is Causal Impact?
  34. 34. Giulia Panozzo - @SequinsNSearch Powerful package to analyse data and infer the cumulative impact of a change in a time series
  35. 35. Giulia Panozzo - @SequinsNSearch Based on BSTS (Bayesian Structural Time Series) statistical model
  36. 36. Giulia Panozzo - @SequinsNSearch Uses past data to predict the outcome in the absence of the treatment (the counterfactual)
  37. 37. Giulia Panozzo - @SequinsNSearch Defines the impact by measuring the deviation of the actual VS predicted outcome
  38. 38. Giulia Panozzo - @SequinsNSearch What is Causal Impact?
  39. 39. Giulia Panozzo - @SequinsNSearch How can it help us in marketing?
  40. 40. Giulia Panozzo - @SequinsNSearch How can it help us in marketing? Example of a clear winner from a title tag change Clicks: +58% CTR: +38% Position: -15%
  41. 41. Giulia Panozzo - @SequinsNSearch It can validate proposed strategy changes in case of any doubts
  42. 42. Giulia Panozzo - @SequinsNSearch It’s great to clearly show stakeholders the impact of our team’s work
  43. 43. Giulia Panozzo - @SequinsNSearch It can help forecast the direction of changes at scale and help make a case for more resources
  44. 44. Giulia Panozzo - @SequinsNSearch How can it help us in marketing? Example of a clear loser from a title tag change
  45. 45. Giulia Panozzo - @SequinsNSearch By clearly identifying a winner or loser, we can understand what works and doesn’t work for our audience
  46. 46. Giulia Panozzo - @SequinsNSearch Demo: how to run a Causal Impact analysis
  47. 47. Giulia Panozzo - @SequinsNSearch 1. Download R Studio Download R first https://cran.r-project.org/ Download RStudio https://www.rstudio.com/products/rstudio/download/
  48. 48. Giulia Panozzo - @SequinsNSearch 1. Download R Studio
  49. 49. Giulia Panozzo - @SequinsNSearch 1.1 Install Causal Impact
  50. 50. Giulia Panozzo - @SequinsNSearch 2. Prepare the data
  51. 51. Giulia Panozzo - @SequinsNSearch 2. Prepare the data
  52. 52. Giulia Panozzo - @SequinsNSearch The first column is always your test group. Other columns can be used as control groups if they are a good fit
  53. 53. Giulia Panozzo - @SequinsNSearch The pre-period should be at least twice as long as the post-period, to allow the model to plot the actual VS predicted outcome
  54. 54. Giulia Panozzo - @SequinsNSearch Any column with 0 should be either removed or corrected
  55. 55. Giulia Panozzo - @SequinsNSearch Isolated 0 in data set Multiple 0s VS
  56. 56. Giulia Panozzo - @SequinsNSearch 3. Run the script!
  57. 57. Giulia Panozzo - @SequinsNSearch Choose file to import
  58. 58. Giulia Panozzo - @SequinsNSearch Check preview
  59. 59. Giulia Panozzo - @SequinsNSearch Set pre and post periods
  60. 60. Giulia Panozzo - @SequinsNSearch It’s a winner!
  61. 61. Giulia Panozzo - @SequinsNSearch
  62. 62. Giulia Panozzo - @SequinsNSearch
  63. 63. Giulia Panozzo - @SequinsNSearch Now give it a go! Request access to this script here
  64. 64. Giulia Panozzo - @SequinsNSearch What I’ve learned from (several) trials and errors…
  65. 65. Giulia Panozzo - @SequinsNSearch The date column should always be removed when using this script
  66. 66. Giulia Panozzo - @SequinsNSearch Column titles can error out if they contain special characters, spaces, capitalised letters
  67. 67. Giulia Panozzo - @SequinsNSearch Start small, then expand your datasets with additional controls and features once you’re comfortable with the script
  68. 68. Giulia Panozzo - @SequinsNSearch Statistics VS the world
  69. 69. Giulia Panozzo - @SequinsNSearch
  70. 70. Giulia Panozzo - @SequinsNSearch
  71. 71. Giulia Panozzo - @SequinsNSearch 1. External events can impact your data
  72. 72. Giulia Panozzo - @SequinsNSearch Google algo updates
  73. 73. Giulia Panozzo - @SequinsNSearch Tools tracking failures
  74. 74. Giulia Panozzo - @SequinsNSearch Engineering releases
  75. 75. Giulia Panozzo - @SequinsNSearch Create a document to map internal changes & external events This will allow you to take into account any other known factors and isolate the treatment in the analysis
  76. 76. Giulia Panozzo - @SequinsNSearch 2. Mind the Outliers!
  77. 77. Giulia Panozzo - @SequinsNSearch
  78. 78. Giulia Panozzo - @SequinsNSearch Outliers can originate from…
  79. 79. Giulia Panozzo - @SequinsNSearch New product launches (within the test group)
  80. 80. Giulia Panozzo - @SequinsNSearch Holidays and seasonal events
  81. 81. Giulia Panozzo - @SequinsNSearch Results only from one page Page 1 Page 2 Page 3 Page 4 Page 5
  82. 82. Giulia Panozzo - @SequinsNSearch Tracking bugs
  83. 83. Giulia Panozzo - @SequinsNSearch You can spot outliers and rule them out by…
  84. 84. Giulia Panozzo - @SequinsNSearch Increasing the size of the test group
  85. 85. Giulia Panozzo - @SequinsNSearch Adding control groups
  86. 86. Giulia Panozzo - @SequinsNSearch Pre-processing your raw data
  87. 87. Giulia Panozzo - @SequinsNSearch
  88. 88. Giulia Panozzo - @SequinsNSearch
  89. 89. Giulia Panozzo - @SequinsNSearch
  90. 90. Giulia Panozzo - @SequinsNSearch 3. Beware of confirmation bias
  91. 91. Giulia Panozzo - @SequinsNSearch Sometimes, your test will be inconclusive, or might be a loser even when you thought it’d be an easy winner
  92. 92. Giulia Panozzo - @SequinsNSearch
  93. 93. Giulia Panozzo - @SequinsNSearch In that case, you can run the test a little longer, or repeat the test with bigger groups
  94. 94. Giulia Panozzo - @SequinsNSearch If it’s still inconclusive or a loser, it’s probably best to revert the change and focus on other tests
  95. 95. Giulia Panozzo - @SequinsNSearch References and useful resources • How we use causal impact analysis to validate campaign success - Part and Sum • Measuring No-ID Campaigns with Causal Impact - Remerge & Alicia Horsch • Causal Impact – Data Skeptic • R Studio on GitHub • The Comprehensive R Archive Network • Causal Impact for App Store Analysis - William Martin
  96. 96. Thank you! Giulia Panozzo @SequinsNSearch

×