A story about the improvement of continuous delivery capability for a team in 5 months.

1. CONTINUOUS DELIVERY
in Action
by Hu Zhenbo (Andy) ThoughtWorks
Friday, April 27, 2012

2. History
Friday, April 27, 2012

3. 4
years
Friday, April 27, 2012

4. Page Views/Day
Millions
100
75
50
25
0
2008 2009 2010 2011
Friday, April 27, 2012

5. Team Size
70+
70
52.5
35
17.5
0
2008 2009 2010 2011
Friday, April 27, 2012

6. Features Delivered Days between major releases
2008 2009 2010 2011
Friday, April 27, 2012

7. 3
major releases per year
Friday, April 27, 2012

8. 20
days to deploy
Friday, April 27, 2012

9. 30
people involved in deployment
Friday, April 27, 2012

10. Angry Customer
Friday, April 27, 2012

11. Sad Team
Friday, April 27, 2012

12. Why?
Friday, April 27, 2012

13. Requirement
Friday, April 27, 2012

14. Fragile System
Friday, April 27, 2012

15. Deploy to Death
Friday, April 27, 2012

16. Features Delivered Days between major releases
2008 2009 2010 2011 2012
Friday, April 27, 2012

17. every 1 or 2 week
delivery frequency
Friday, April 27, 2012

18. 0.5
days to deploy
Friday, April 27, 2012

19. 1
person dedicated for deployment
Friday, April 27, 2012

20. How’d we do it?
Friday, April 27, 2012

21. One Team
Friday, April 27, 2012

22. Collaboration
Friday, April 27, 2012

23. Friday, April 27, 2012

24. Friday, April 27, 2012

25. Decode 4 types
of data
21 pts
Incremental Developing
Friday, April 27, 2012

26. Design
5 pts
Decode 4 types
Framework
of data
21 pts 8 pts
Decode Data
8 pts
Incremental Developing
Friday, April 27, 2012

27. Decode Data A Decode Data B
8 pts 5 pts
Decode 4 types
of data
21 pts
Decode Data C Decode Data D
5 pts 3 pts
Incremental Developing
Friday, April 27, 2012

28. Feature Toggle
Friday, April 27, 2012

29. Feature Toggle
Friday, April 27, 2012

30. Page File
Config File #if
( $nonRealTimeRecharge )
# Feature Toggle <div>非即时到帐</div>
nonRealTimeRecharge = true <div> ...
realTimeRecharge = false #end
#if( $realTimeRecharge )
<div>即时到帐</div>
<div> ...
#end
Feature Toggle
Friday, April 27, 2012

31. Increasing confidence in build’s production readiness
Environment become more production-like
User Acceptance
Testing
Commit Stage exploratory test
compile ...
Acceptance Test
analysis Production
Stage
unit test Non-Functional
... Testing
performance test
...
Slower Feedback
Deployment Pipeline
Friday, April 27, 2012

32. Deployment Pipeline
Friday, April 27, 2012

33. Code Commit Update
Jar Portal ZIP Upload
Change Jar
Dependen
Stop Create Start
Backup Unzip Config
Server Log File Server
Auto-Deploy
Friday, April 27, 2012

34. Code Commit Update
Jar Portal ZIP Upload
Change Jar
Dependen
Stop Create Start
Backup Unzip Config
Server Log File Server
Auto-Deploy
Friday, April 27, 2012

35. Code Commit Update
Jar Portal ZIP Upload
Change Jar
Dependen
Stop Create Start
Backup Unzip Config
Server Log File Server
One-click Deploy
Auto-Deploy
Friday, April 27, 2012

36. Payment
Mobile
Account
Portal
Order
Common Core
Admin
Components & Dependencies
Friday, April 27, 2012

37. Payment
Mobile
Account
Portal
Order
Common Core
Admin
Components & Dependencies
Friday, April 27, 2012

38. Payment
Mobile
Account
Portal
Order
Common Core
Admin
Components & Dependencies
Friday, April 27, 2012

39. Payment
Mobile
Account
Portal
Order
Common Core
Admin
Components & Dependencies
Friday, April 27, 2012

40. Payment
Mobile
Account
Portal
Order
Common Core
Admin
Components & Dependencies
Friday, April 27, 2012

41. Payment
Mobile
Account
Portal
Order
Common Core
Admin
Components & Dependencies
Friday, April 27, 2012

42. Test Quadrants
Friday, April 27, 2012

43. Manual
Session
Based
Testing
Automated
GUI Tests
API Tests
Integration Tests
Component Tests
Automated Unit Tests
Test Pyramid
Friday, April 27, 2012

44. Deployment Test
system configuration Acceptance
component communication
service is up and running Test
...
Friday, April 27, 2012

45. Performance Test
Friday, April 27, 2012

46. Web App DB
Server Server Server
Router
Blue-Green Deploy
Friday, April 27, 2012

47. Web App DB
Server Server Server
Router
v5.1 v5.1 v5.1
Blue-Green Deploy
Friday, April 27, 2012

48. Web App DB
Server Server Server
Router
v5.1 v5.1 v5.1
Blue-Green Deploy
Friday, April 27, 2012

49. Web App DB
Server Server Server
Router
v5.1 v5.1 v5.1
v5.2 v5.2 v5.2
Blue-Green Deploy
Friday, April 27, 2012

50. Web App DB
Server Server Server
Router
v5.1 v5.1 v5.1
v5.2 v5.2 v5.2
Blue-Green Deploy
Friday, April 27, 2012

51. migrate
- 001_create_products_table.sql
- 002_add_column_to_products.sql
- 003_update_data_for_products.sql
rollback
- 001_create_products_table.sql
- 002_add_column_to_products.sql
- 003_update_data_for_products.sql
Data Migration
Friday, April 27, 2012

52. v5.1
Green Database
Friday, April 27, 2012

53. v5.1
readonly
Green Database
Friday, April 27, 2012

54. Backup
v5.1 v5.1
readonly
Green Database
Friday, April 27, 2012

55. Backup Migrate
v5.1 v5.1 v5.2
readonly
Green Database
Friday, April 27, 2012

56. compatible with db 5.1 & 5.2 compatible with db 5.2 & 5.3
v5.2 v5.3
v5.1 v5.2 v5.3
Decouple App & Database
Friday, April 27, 2012

57. Voltage
DB
Fan Speed
HTTP
Temperature Hardware
OS
Middleware SMTP
CPU Application
DNS
Disk
Memory
Production Monitoring
Friday, April 27, 2012

58. Production Monitoring
Friday, April 27, 2012

59. Cross Functional Team
Continuous Integration
Incremental Developing
Feature Toggle
Automate Deployment
Weekly Release
Deployment Pipeline
Blue-Green Deployment
Database Migration
Production Monitoring
...
Friday, April 27, 2012

60. Good Enough?
Friday, April 27, 2012

61. http://code.flickr.com/
A Long Way to Go
Friday, April 27, 2012

62. Big Release
Friday, April 27, 2012

63. Continuous Delivery
Friday, April 27, 2012

64. Version Control
acceptance
System
test
blue-green
deployment
virtualization
canary
releases build pipeline
data
management automatic
deployment
dark
launch
branch by
abstraction
configuration
management
DevOps
infrastructure as code
could computing
Friday, April 27, 2012

65. Q&A
weibo: @HuZhenbo
Friday, April 27, 2012