This document discusses A/B testing and how companies like Netflix, Airbnb, and Globo use it. A/B testing involves experimenting with changes by exposing different groups of users to variations and analyzing metrics to see what works best. It's important to have a hypothesis, trust dashboards and data scientists for analysis, and continue tests long enough to account for seasonal behaviors. Multivariate testing (MVT) can test multiple variables at once but has higher statistical requirements. Multi-armed bandit algorithms can help optimize experiments by minimizing exposure to weaker options over time through exploration vs exploitation of results.

1. with a/b testing

2. A GUY WHO
LIKES
DATA ON
GLOBO.COM
@renan_oliveira

3. What is A/B?

5. Experimentwith a group
of users who will notbe
impactedby change and
other groups is impacted
withit

8. Who use?

10. Netflix

11. When you have highly talented
individuals who have lots of
great ideas, it’s important to have a
framework where any new idea
can be developed and tested,
and data, notopinion, is used to
make decisions.
https://medium.com/netflix-techblog/a-b-testing-and-beyond-improving-the-netflix-streaming-experience-with-experimentation-and-data-5b0ae9295bdf

14. AirBnb

15. The outsideworld often has a much larger
effectonmetrics than product changes
do. Users can behave very differently
depending on the day of week, the time of
year, the weather, or whether they learned
about the website through an online ad or
found the site organically. Controlled
experimentsisolatetheimpact of the
product change while controlling for the
aforementioned external factors.
https://medium.com/airbnb-engineering/experiments-at-airbnb-e2db3abf39e7

17. globo.com

18. We use it to validate our
products both in the UX area and
in the APIs. One of the steps we
are seeking and increasingly
integrate our A/B solution with
the platforms developed by other
teams.

19. A/B vs

20. First step to
make your
A/B

23. So… What's a
hypothesis?

24. Aquestion! After
observing a behavior create
a question to try to guess
the answer with the
experiment.

25. When I get 99%
confidence can I stop
my experiment?

26. NO! This test says that the
difference between the
alternative and the control is
real. If you do not include the
time axis you lose the
seasonal behavior.

27. How long do I leave
my test running?

28. IDON’TKNOW! Each
product has its own user
seasonal behavior. You'll
know it better than I do.

29. How do I analyze
this?

30. Trust thedashboardsand
consult a datascientist

31. Statistic? Data
Science?

32. It is a skill that will notbeableto
escape, everyone should at
some time know how to metrify
their products in a correct way.
Until there, look for specialists
toguideyou.

33. More than one
test on the
same page?

35. The default answer is
no.

36. No solution at all?

38. MVT - multivariate testing

40. For pages with a lotof
audience all the
experiments and
alternatives of your product
can be combined.

41. Think of settheory and the
intersection of it, so we know
what the bestcombination
of experiments and
alternatives for your product.

42. IsolatedMVT

44. If we build all the
combination then we have
one where the useronly
sawthisexperiment.

45. AB vs MVT?

46. AB MVT
High statistical
confidence
Does not depend
on too much
volume
Non-visible noise
may occur
Best combination
Population between
experiments must be
significant
Intercessions are
seen

47. https://blog.acolyer.org/2017/09/27/an-efficient-bandit-algorithm-for-real-time-multivariate-optimization/
27%
MVT + MAB
conversion in 1 week

48. Multi-Armed
Bandit

49. Minimize the
consumption of the
worst

51. The best options may vary over time
ARMsmostdeliveryperhour

52. Exploration vs
Exploitation

55. Quem é Díaz-Canel, indicado para suceder Raúl Castro em Cuba?
Indicado para suceder Raúl Castro nasceu após a Revolução Cubana
https://g1.globo.com/mundo/noticia/quem-e-miguel-diaz-canel-o-discipulo-predileto-de-raul-castro-que-assume-o-poder-em-cuba.ghtml
TITLE1
TITLE2
TITLE1 >15%

56. Viewability
Use of an optimization algorithm (MAB)
to set the best time to start
advertising loading on the page and
distribute users accordingly
ALTERNATIVA
A
ALTERNATIVA
B
ALTERNATIVA
C
200px
250px
300px

58. THANK YOU
@renan_oliveira
www.renanoliveira.net