Continuous delivery entails many concerns that you need to take care of; from changing the company culture to development methodologies. You have to follow Test Driven Development, perform A/B tests on every new feature you deploy to production; The product managers' is feature centric with feature toggles to turn features on and off; You have to perform gradual deployments, and more. In this talk Aviran will discuss some of the strategies we use at Wix to address these concerns, letting Wix to deploy hundreds of times every month. You will get a behind the scene look of how Wix manages A/B tests, how they changed their development methodology and the impact it had on their company culture. You will also learn what are feature toggles and how they use them to solve real life problems, and of course how they manage to do hundreds of deployments and evolve their system with no downtime. Speaker: Aviran Mordo, Wix Aviran Mordo leads Wix's back-end group. He has over 20 years of experience in the software industry and has filled many engineering roles and leading positions, from designing and building the US national Electronic Records Archives (http://www.archives.gov/era/) to building the infrastructure that powers indeed.com, the leading job search engine in the world. Aviran is a tech-savvy and a technology blogger since 2000, with vast knowledge of internet technologies, software development and continuous delivery. Twitter: @aviranm LinkedIn: http://www.linkedin.com/in/aviran Blog: http://www.aviransplace.com"

1. Strategies In Continuous Delivery
Learn the “magic” behind the scenes
DEVOPS Days Tel Aviv 2013
Aviran Mordo
Head Of Back-End Engineering @ Wix
@aviranm
http://www.linkedin.com/in/aviran
03:14

2. About Wix
03:14

3. Wix in Numbers
• Over 39,000,000 users
– Adding over 1,000,000 new users each month
• Static storage is over 150TB of data
– Adding over 1TB of files every day
• 3 Data centers + 2 Clouds (Google AE, Amazon)
– Around 300 servers
• Over 100,000,000 Server API calls per day
• ~450 people work at wix
– ~ 150 people in R&D
03:14

4. Do You Have The Guts To
Deploy 10 Times A Day?
03:14

5. From Jan’ 2013 – Jun’ 2013
• 1500
• 470
• 200
03:14

6. From Jan’ 2013 – Jun’2013
• 1500 Deployments
• 470 A/B Tests
• 200 Feature Toggles
03:14

7. Continuous Delivery – Key points
• Abandon the “VERSION” paradigm – move to a
feature centric methodology
• Make small and frequent release as soon as
possible
• Automate everything – TDD/CI/CD
• Measure Everything
– A/B test every new feature
– Monitor real KPIs (business, not CPU)
03:14

8. Test Driven Development
• No new code is pushed to Git without being fully tested
– We currently have around 10,000 automated tests
• Before fixing a bug first write a test to reproduce the bug
• Cover legacy (untested) systems with Integration tests
03:14

9. Test Driven Development
• Actual impact on development
– We develop products faster
– Removes fear of change
– Easier to enter some one else’s project
– Do we really need QA? (Yes, they code tests)
– Writing a feature is 10-30% slower, 45-90% less bugs
– 50% faster to reach production.
– Considerably less time to fix bugs
03:14

10. Feature Toggles
03:14

11. Feature Toggles
• Everyone develops on the Trunk
• Every piece of code can get to production at
anytime
03:14

12. Code branch
Yes
New Code
03:14
FT
Opene
d
No
Old Code

13. Usage example
Simple “if” statement in your code
03:14

14. DB Schema Changes Without
Downtime
• Adding columns
– Use another table link by primary key
– Use blob field for schema flexibility
• Removing fields
– Stop using. Do not make any DB schema
changes
03:14

15. New DB schema with data migration
• Plan a lazy migration path controlled by feature toggle
1.
2.
3.
4.
5.
6.
03:14
Write to old / Read from old
Write to both / Read from old
Write to both / Read from new, fallback to old
Write to new / Read from new, fallback to old
Eagerly migrate data in the background
Write to new / Read from new

16. Feature Toggle Strategies (gradual expose users)
•
•
•
•
•
•
•
•
03:14
Company employees
Specific users or group of users
Percentage of traffic
By GEO
By Language
By user-agent
User Profile based
By context (site id or some kind of hash on site id)

17. Feature Toggle Override
• By specific server
– Used to test system load
– New database flows/migration
– Refactoring that may affect performance and memory usage
• By Url parameter
– Enable internal testing
– Product acceptance
– Faking GEO
• By FT cookie value
– Testing
– When working with API on a single page application
03:14

18. Feature Toggles Management
03:14

19. A/B Tests
03:14

20. A/B Test
• Every new feature is A/B tested
• We open the new feature to a % of users
– Define KPIs to check if the new feature is
better or worse
– If it is better, we keep it
– If worse, we check why and improve
– If we find flaws, the impact is just for % of our
users (kind of Feature Toggle)
03:14

21. An interesting site effect on product
• How many times did you have the conversion
“what is better”?
– Put the menu on top / on the side
• Well, how about building both and A/B Testing?
03:14

22. Marking users with toss value in a
cookie
• Anonymous user
– Toss is randomly determined
– Can not guarantee persistent experience if changing browser
• Registered User
– Toss is determined by the user ID
– Guarantee toss persistency across browsers
– Allows setting additional tossing criteria (for example new users
only)
03:14
– Only use this for sections that a user has to be authenticated

23. • Do not mix anonymous and registered tests
• AB test parentage of users with optional filters
– New Users Only (Registered users only)
– By language
– By GEO
– By Browser
– user-agent
– OS
– Any other criteria you have on your users
03:14

24. A/B Test Features
• A/B Test Override
– Allows to set a value of a test for validation
– Helps support experience what users experiencing
• Override methods
– Via URL parameter
– Via cookie
• Start/Stop Test
• Pause tests
• Bots always get “A”
03:14

25. Manage your tests
03:14

26. Self Test / Post Deployment Test
After each server deployment run a self test before deploying the
next server.
• Checking server configuration and topology
– Make sure database is accessible (DB connection string)
– Is the schema the one I expect
– Access required local resources (data files, other config files, templates,
etc’)
– Access remote resources
– RPC / REST endpoints reachable and operational
• Server will refuse requests unless it passes the self test
03:14
• Allow a way to skip self test (and continue deployment)

27. Tools - App-info
03:14

28. 03:14
(blank)
6/16/2013
6/11/2013
6/7/2013
6/4/2013
6/1/2013
5/29/2013
5/26/2013
5/22/2013
5/19/2013
5/13/2013
5/9/2013
5/6/2013
5/1/2013
4/28/2013
4/23/2013
4/19/2013
4/15/2013
4/10/2013
4/7/2013
4/3/2013
3/24/2013
3/19/2013
3/15/2013
3/12/2013
3/7/2013
3/4/2013
2/28/2013
2/25/2013
2/21/2013
2/18/2013
2/13/2013
2/10/2013
2/6/2013
2/3/2013
1/30/2013
1/27/2013
1/22/2013
1/18/2013
1/15/2013
1/11/2013
1/8/2013
1/4/2013
1/1/2013
Where are we today?
• We have re-written our flash editor product as an
HTML 5 editor
– In just 4 months
• Introduced Wix 3rd party applications (developers API)
– In just 6 weeks
• We are easily replacing significant parts of our
infrastructure
Total
• And we are doing ~10 releases a day!
35
30
25
20
15
10
Total
5
0

29. Aviran Mordo
@aviranm
http://www.linkedin.com/in/aviran
http://www.aviransplace.com
http://www.slideshare.net/aviranwix/strategies-incontinuous-delivery
Read more: The Road To Continuous Delivery:
http://goo.gl/K6zEK
03:14