How Developers Can Take Experimentation to New Levels

1. opticon2017
How Developers can take
Experimentation to New Levels
Michael Ng
Software Eng, Optimizely
Ali Rizvi
Software Eng, Optimizely
Josh Wang
Software Eng, Optimizely

2. opticon2017
A story about
experimentation

6. opticon2017
$100 million
$50 million
Mike Ng
Business Impact

7. Technical Impact

8. opticon2017
How Developers can take
experimentation to new levels with Full
Stack
• Architecture that allows developer flexibility
• Experiment across platforms
• Experiment deep in the stack

9. opticon2017opticon2017
Full Stack
Experimentatio
n

10. opticon2017
How experimentation works with Full Stack
Your servers
SDKs
Results
Experiment
management
datafile
events
Flexible SDKs Optimizely Dashboard
A
B
Node
Java
Ruby
Python
PHP
C#
Javascript
iOS
Android
User devices

11. opticon2017
Search algorithms
Redesigns
Feature rollouts
Sorting algorithms
Price changes
Backend changes
Use cases for Full Stack
experimentation

12. opticon2017

13. opticon2017
SDK architecture:
Built for flexibility
• Stateless
• Consistent implementation across SDKs
• Non-blocking API calls
• Replaceable components

14. opticon2017
Creating an SMS
experiment with Full Stack

15. opticon2017
SMS experiment
“Are you interested in learning
more about Optimizely Full
Stack?”
Variation B
“Want to learn how to
experiment easily with
Optimizely Full Stack? Reply
"YES" for a guide to best
practices!”
Variation A
Hypothesis: Having a clear call to action will increase
positive response rate.

16. opticon2017
Text “Hello” to
(415)-980-8918
to participate in our live experiment

17. opticon2017
Optimizely on
Optimizely

18. opticon2017

19. opticon2017opticon2017
Collaborator
Email Experiment

20. opticon2017

21. opticon2017
Problem
• Email is being sent, but customers are not changing their password.
• Potential risk because if email is compromised, then Optimizely account can be compromised.
Potential solutions
• Send a reminder email.
• Have a call to action in the email.
• Link to the right place to make it easier for the user to change the password.
Hypothesis
More people will change password.

22. opticon2017
Original email Variation email

23. opticon2017

24. opticon2017opticon2017
Tool Face-off

26. opticon2017
VS
Tools at play

27. opticon2017
Hypothesis
We can use performance as a way to determine which tool to use.
Problem
Pick best tool to do cross-browser testing.

29. opticon2017opticon2017
Feature
Management

30. opticon2017
• Need to do code deploy to rollout feature.
• Need to do code deploy to roll back feature.
• Not easy to do feature randomization
amongst your users.
• No easy way to figure out feature access
amongst users.
• Generally, no access control.
• No one source of truth.
How we do feature management today

32. opticon2017
Takeaways
• Full Stack enables experimentation across platforms
• The Full Stack architecture allows for greater developer flexibility
• Experimentation can happen in any part of the stack

33. opticon2017
Q&
A

34. opticon2017
THANK YOU

35. opticon2017