My "flash" talk in Product Excellence Conference

1. Orly Amrany
General Manager, Wix
When the question ‘why?’ is not relevant
The Unexpected Facts of Life

2. Hi,
I’m Orly Amrany
General Manager, Head of Premium Services @ Wix

3. Question #01

4. • Assume you have 3 Million Subscription (Monthly/Yearly)
• Each recurring charge is done on it’s purchase date
• Re-try upon failure is done every 7 days
• Success rate of recurring charge is ~90%
• Which day do we have the best changes to charge ?
24th or 25th?

5. 85.25%
81.94%
84.14%
79.80%
82.79%
85.67%
85.13%
83.24%
81.59%
86.80%
83.55%
85.48%
86.49%
86.96%
79.00%
80.00%
81.00%
82.00%
83.00%
84.00%
85.00%
86.00%
87.00%
AU BR CA ES FR GB US
24th Vs. 25th, initial recurring

6. 85.25%
81.94%
84.14%
79.80%
82.79%
85.67%
85.13%
83.24%
81.59%
86.80%
83.55%
85.48%
86.49%
86.96%
79.00%
80.00%
81.00%
82.00%
83.00%
84.00%
85.00%
86.00%
87.00%
AU BR CA ES FR GB US
24th Vs. 25th, initial recurring
+0.5%increase in approval rate

7. 85.25%
81.94%
84.14%
79.80%
82.79%
85.67%
85.13%
83.24%
81.59%
86.80%
83.55%
85.48%
86.49%
86.96%
79.00%
80.00%
81.00%
82.00%
83.00%
84.00%
85.00%
86.00%
87.00%
AU BR CA ES FR GB US
24th Vs. 25th, initial recurring
+0.5%increase in approval rate
+15ksaved subscriptions

8. 85.25%
81.94%
84.14%
79.80%
82.79%
85.67%
85.13%
83.24%
81.59%
86.80%
83.55%
85.48%
86.49%
86.96%
79.00%
80.00%
81.00%
82.00%
83.00%
84.00%
85.00%
86.00%
87.00%
AU BR CA ES FR GB US
24th Vs. 25th, initial recurring
+0.5%increase in approval rate
+15ksaved subscriptions
+1.5M

9. And the winner is…

10. The 24th!

11. You’ve won 1,500,000
million dollars

12. Question: Why checking
the 24th vs. 25th?

13. Question: Why checking
the 24th vs. 25th?
Answer: Because we could
• We had data of million of transactions
• We actually checked all days of the month
• A lot of other assumptions failed..(15th Vs 16th)
• Should we AB test? (NO)

14. On the 24th there is more changes for charges to succeed
We gained +1.5M USD with a simple change!
When you have tones of data, you can find out a lot of interesting
and unexpected facts
You should do it as soon as possible: Time is money
Make all recurring charges on the 24th ? (NO) Improve re-try logic (Which Tool)
So what did we learn?

15. So what did we learn?
Don’t ask yourself
‘why?’
Ask yourself
‘why didn’t I check this before?’

16. Some more fun examples

17. Question #02

18. • Same data
• Which day do we have the best changes to charge?
Weekdays or
weekends?

19. And the winner is…

20. Weekday!

21. 4.41%
-1.85%
6.36%
2.42%
5.38%
8.00%
-0.59%
1.18%
5.67%
-4.84%
6.39%
0.45%
2.97%
5.94%
-6.67%
0.10%
-0.62%
5.21%
7.62%
1.50%
2.59%
3.22%
-5.88%
3.41%
0.66%
11.80%
6.81%
-0.35%
8.05%
7.64%
2.39%
-10.00%
-5.00%
0.00%
5.00%
10.00%
15.00%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Approval rate difference - weekdays vs. weekends
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

22. 4.41%
-1.85%
6.36%
2.42%
5.38%
8.00%
-0.59%
1.18%
5.67%
-4.84%
6.39%
0.45%
2.97%
5.94%
-6.67%
0.10%
-0.62%
5.21%
7.62%
1.50%
2.59%
3.22%
-5.88%
3.41%
0.66%
11.80%
6.81%
-0.35%
8.05%
7.64%
2.39%
-10.00%
-5.00%
0.00%
5.00%
10.00%
15.00%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
Approval rate difference - weekdays vs. weekends
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
+1.3%increase in approval rate

23. • Same data
• Which hour of the day would be the best? (in Brazil for example)
Time or Day?

24. And the answer is…

25. 8am – 10pm

26. 20.00%
25.00%
30.00%
35.00%
40.00%
45.00%
50.00%
55.00%
60.00%
65.00%
70.00%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Approval Time, Recurring/UTC time
all

27. Thanks