AB Testing used to be complex experiments only companies like Amazon or Google used to run. The tools available now are simple and enable this strategy even for one-man apps. But, what makes a meaningful experiment that can make your app better? how can failed experiments still bring value? We want to talk about the science on AB tests in a way that everybody understands and enjoys. We have some case studies from Spotify to analyze and share and we have a clear call to action for the audience : these are the basics, start AB-testing!

3. Spotify Numbers
60 markets Over 30 million Songs100 millionActive Users
(40 million Subscribers)

4. iOS Client Numbers
50 monthly contributors 0.5 million lines of code +100AB Tests Running

5. Hector Zarate (@ChocoChipset)
Software Engineer at Spotify
Boxun Zhang
Data Scientist at Spotify

6. 1.What?

8. 😱

9. 😱

10. YOUARE
PARTOFAN
EXPERIMENT

11. AB
Testing

13. Samson, 3

14. Max, 30Samson, 3
+5%
matches

15. Share of Matches
20%
23.75%
27.5%
31.25%
35%
Week 0 Week 1 Week 2 Week 3 Week 4
With Sunglasses
Sunglasses

16. Share of Matches
20%
23.75%
27.5%
31.25%
35%
Week 0 Week 1 Week 2 Week 3 Week 4
With Sunglasses Control
Sunglasses

17. Share of Premium Conversion
20%
23.75%
27.5%
31.25%
35%
Week 0 Week 1 Week 2 Week 3 Week 4
Variation A Control
Sunglasses

18. User Retention
20%
23.75%
27.5%
31.25%
35%
Week 0 Week 1 Week 2 Week 3 Week 4
Variation A Control
Sunglasses

20. spotify:user:chocochipset92b2976bb15d26c9008

21. 1 2 3 4 5 6 7 8 9 10
p ( x )
x
1 / n

22. 1 2 3 4 5 6 7 8 9 10
p ( x )
x
1 / n

23. 1 2 3 4 5 6 7 8 9 10
p ( x )
x
1 / n 0% 60%

24. Markets

25. Demographics

26. UserAttributes

28. Login
resolveABFlags()
response(ABFlags)
ABBA
Cache AB Values
Load Cached or
Default AB Flags

29. Example
button.color = [UIColor spotifyGreen];
BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]);
if (isButtonPink) {
// alternate path here:
button.color = [UIColor spotifyPink];
}
Key Value
charts Enabled
pink-buttons 0
buffer-quality low
gallery-artist Control

30. Key Value
charts Enabled
pink-buttons 0
buffer-quality low
gallery-artist Control
Example
button.color = [UIColor spotifyGreen];
BOOL isButtonPink = ([abFlags[@“pink-buttons"] isEqual:@"1"]);
if (isButtonPink) {
// alternate path here:
button.color = [UIColor spotifyPink];
}
Key Value
charts Enabled
pink-buttons 1
buffer-quality low
gallery-artist Control
pink-button 1
VariationA Control

31. Analytics
impressions and
interactions
ControlVariation

32. * just a personal preference

33. 2.How?

34. 1. Hypothesis 2. Design 3. Run 4.Analysis

35. 1. Formulate
a Hypothesis

36. 2. Design theTest

37. #
shuffle
plays
2.1Target Metrics

38. # matches
Samson, 3
# replies
# walks in the
park

39. 2.2Test Group
more test bandwith, less confidence, smaller effect
in business
less test bandwith, more confidence, bigger effect in
business
Small
Large

40. 2.3 Duration

41. Watch out for seasonality:
i.e. Music habits are different on
weekdays and weekends.
Purchasing habits are different near
paydays.

42. 3. Run the
test

43. 1 2 3 4 5 6 7 8 9 10
p ( x )
x
1 / n
TestA
TestA
TestB
TestB
TestA/TestB
Watch out for
conflicting tests!

44. Don’t cut them
short!

45. 0m 90m
Standard Match Duration
0m 20m
Inconclusive
0m +95m
Unnecessary

46. 0m +95m
Bruno
C
esar48’:0
-1

47. 0m +95m
Bruno
C
esar48’:0
-1
Ronaldo
89’:1-1

48. 0m +95m
Bruno
C
esar48’:0
-1
Ronaldo
89’:1-1
M
orata
90’+
4’:2
-1
2 - 1

49. 0
25
50
75
100
D1 D2 D3 D4 D5 D6 D7 D8
Don’t cut them
short!

50. 4.Analyze
the results

51. 1. Formulate
a Hypothesisnew

52. 3.CaseStudies

53. 3.1. Losing
Calories

60. Test: Tab Bar Navigation
Hypothesis:
By switching to TBN, we
expect an increase in the
share of users who click
at least one menu item.
1

61. Test: Tab Bar Navigation
Hypothesis:
By switching to TBN, we
expect an increase in the
share of users who click
at least one menu item.
2.1

62. 10%
New Users
1%
Existing Users
2.2

63. 2.3
1 week period

64. 4
Clicks on Menu Items
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
Control Tab Bar
130%
100%

65. 4
Clicks Overall
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
110%
120%
130%
Control Tab Bar
109%
100%

66. 4
1. Increased clicks in
Tab Items
2. Decreased clicks
in non-tab items.

67. WHY AB TEST?
QUANTIFY THE IMPACT
OF A SPECIFIC CHANGE

68. 3.2. Home
in Mexico

69. My Rock Your Rock

70. My Home Your Home

71. My Home Your Home

72. Test: Home
Hypothesis:
We will measure an
increased second week
retention by using the
new ‘Home’ as start
page. 1

73. Test: Home
Hypothesis:
We will measure an
increased second week
retention by using the
new ‘Home’ as start
page. 2.1

74. 10%
New Users
1%
Existing Users
2.2

75. 2.3
1 week period

76. US UK Germany Austria Mexico
4

77. 1. Is the localization good?
2. Are recommendations relevant for
the market?
3. Technical restrictions we are not
aware of?
4

78. 12.34 Mbps
US
13.70 Mbps
UK
13.42 Mbps
Germany
15.48 Mbps
Austria
7.4 Mbps
Mexico
* 3.7 Mb
4

79. 1. Hypothesis 2. Design 3. Run 4.Analysis

80. Test: Home
Hypothesis:
By compressing the data
sent for Home, more
users will play on day
one, two and be active
during their second
week. 1

81. 1. Is the localization good?
2. Are recommendations relevant for
the market?
3. Technical restrictions we are not
aware of?
4

82. 1. Is the localization good?
2. Are recommendations relevant for
the market?
3. Technical restrictions we are not
aware of?
4

83. 1. Is the localization good?
2. Are recommendations relevant for
the market?
3. Technical restrictions we are not
aware of?
4

84. 4.Pitfalls

85. ANALYSIS
PARALYSIS

86. NO
DATA

87. Max, 30
TOO MANY
SMALL CHANGES

88. Max, WOLF AT WALL STREET
TOO MANY
SMALL CHANGES

89. DONT
LISTEN TO
YOUR
HEART
(LISTEN TO THE DATA)

90. 5.WrapUp

96. KNOWLEDGE
ISYOUR
ROI

97. Atest is an investment
Design to maximize learning

98. NEGATIVE
RESULTS ARE
STILL POSITIVE
(DON’T GIVE UP!)

99. TRY AB
TESTING

101. Вопросов?
Hector Zarate
@ChocoChipset
spotify.com/jobs

102. Спасибо!
Hector Zarate
@ChocoChipset
spotify.com/jobs