1. The Challenge of
Continuously Delivering
Julian Fitzell, Cincom
ESUG, Edinburgh
August 24, 2011
Image © Petr Vins: http://www.sxc.hu/photo/1054507

2. ChannelStream

3. Document Output
Management

4. Control

5. Enrichment
Barc
ode Datamatrix
O MR
Ma rketing

6. Output

7. Why?
• Postal contract savings
• Saves licking envelopes
• Personalization and cross-marketing
• Avoid IT/supplier for customization
• Enhance legacy output

8. Challenges
Image © Tim Regan: http://www.flickr.com/photos/dumbledad/3225255407/

9. Difficulty
tracking versions Manual process
STRESS! Infrequent
integration
Building
images slow +
error-prone
Integration is Slow
+ risky/hard
Frequent development
releases External
dependencies
One fragile
developer
integrating Test Test data
environment not unversioned
easily reproducible
Reduced
confidence from
automated tests False positives Configuration
files unversioned
Tests slow/
hard to run
Unclear
Tests not Tests not separation between
always run always updated unit/functional

10. So...
• Make building images easier
• Make running tests easier
• Make integration easier

11. con·tin·u·ous de·liv·er·y
• putting the release schedule in the
hands of the business
• ensuring your software is always
production ready throughout its
entire lifecycle
• making any build potentially
releasable to users at the touch of a
button
Jez Humble, http://continuousdelivery.com/2010/08/continuous-delivery-vs-continuous-deployment/

13. Goals
• Reduce cycle time between an idea
and its availability to users
• Deliver high-quality software reliably
and efficiently

14. Frequent, automated releases:
• Decrease stress
• Eliminate errors and improve
auditability
• Make feedback more immediate
• Protect against reliance on individual
expertise
• Reduce wasted time on repetitive tasks

15. Make releases boring
• Automate (almost) everything
• Version everything
• Deploy everywhere the same way
• If it hurts, do it more often
• Everyone is responsible for delivery

16. Development
+
Integration

17. Integration
Problems
• Contention over “head”
• One person doing most of the integration
• Delayed integration => harder integration
• Poor visibility of changes
• Hard to predict/analyze failure

18. Integration
Issue tracker + email

19. Integration
Task/merge tools

20. Integration
Integrate
often
• Hide functionality
(feature toggles)
• Incremental, releasable
changes
• “Branch by abstraction”
for large-scale changes
• Use components to
decouple parts
Image © Lars Sundstrom: http://www.sxc.hu/photo/1082530

21. Integration
Remaining Issues
• Restructurings can still be tricky when
branches are outstanding
• “Integration token” still just informal
conversation

22. Building

23. Building
Problems
• Manual builds on developer machines
• Wasted developer time
• “Here, take my image”
• Taking shortcuts
• “Works on my machine”

24. Building
Helper scripts + docs

25. Building
Fully automated scripts
Builder Target
-filein foo-wrapper.ws
foo.ws
foo-wrapper.ws

26. Building
Builds run (at least)daily

27. Building
Archive built images

28. &'%9"'4+$")+J35&V4G+$HH$")%9+9'+E'K9+9)$EK*+$%(+9=3K+K9$"9+$+K)$"&=+L'"+#)99)"+$49)"%$957)K-+
Building
Image © 2009 by Urbancode
+
+
Run automatically on ALL code changes
P=)+L5"K9+K9)HK+L'"+E$93"5%I+9=)+#354(+$")+K9$%($"(5X5%I+9=)+#354(+H"'&)KK+$%(+H)"L'"E5%I+'LL5&5$4+#354(K+
R()7)4'H)"+E$&=5%)+L'"+9=)+#354(+E)$%K+&=$%I)K+5%+'%)+
Avoid versioning split between SVN + Store
5K+")9"5)7)(+L"'E+K'3"&)+&'%9"'4+9'+3K)+5%+#354(K-+P=5K+E$G+#)+9=)+4$9)K9+L"'E+9=)+95H+'L+$+#"$%&=*+'"+
4$#)4)(+K'3"&)+&'()*+'"+K'E)9=5%I+)4K)-+P=)+5EH'"9$%9+H$"9+5K+9=$9+9=)+&='K)%+&'%7)%95'%+#)+$HH45)(+
Rationalize build/version numbering
&'%K5K9)%94G-+N59=+9=)K)+H"$&95&)K+5%+H4$&)*+9=)+9)$E+=$K+")$&=)(+$%+!"&'+),-&+'.+4)7)4+'L+#354(+
&'EH)9)%&)-+
6%G+9)$E+3K5%I+K9$%($"(+;'%95%3'3K+,%9)I"$95'%+D544+9$V)+9=5K+$+K9)H+L3"9=)"+$%(+$39'E$9)+)Y)&395'%+'L+
9=)+#354(+K9)HK-+P=)+#354(+K)"7)"+D544+(5&9$9)+E$&=5%)K*+K'3"&)R&'()+"34)K*+$%(+"3%+9='K)+#354(+K9)HK*+
!"#$%&'()*+,%&-+.+/011+23&45(+67)%3)+8359)+:00+.+;4)7)4$%(*+<=5'+11>>?+.+@A+/>:-B?B-C000++.+DDD-3"#$%&'()-&'E+.+F+/00C+#G+!"#$%&'()+

29. http://www.urbancode.com/html/resources/white-papers/

30. Testing

31. Testing
Problems
• Tests took up to 16 hours to run
• Test environment hard to create
• Failing tests sometimes pass when re-run
• Not everyone could run tests or be sure of
results
• High cost of test maintenance
• Sometimes quicker to test manually than fix
the test environment

32. Testing
Done
• Optimize tests (now run in ~ 2 hours)
• Separate unit/regression from
functional/integration tests
• Store test data in a central location

33. Testing
Run tests automatically

34. &'%O5()%&)-+
Testing
Image © 2009 by Urbancode
Image © 2009 by Urbancode
+
+
Run regression and smoke tests as part of
S'I9+9)$EI+=$7)+I'E)+O'"E+'O+$39'E$9)(+9)I95%K+5%+J4$&)-+@)"=$JI+$+IE$44+3%59+9)I9+I359)+'"+I'E)+
build and notify of errors by email
I&"5J9)(+9)I9I+)%I3")+9=$9+9=)+#$I5&+O3%&95'%$459G+'O+9=)+$JJ45&$95'%+D'"PI-+H=)I)+#$I5&+$39'E$9)(+
")K")II5'%+9)I9I+J"'75()+)$IG*+)$"4G+()9)&95'%+'O+O3%($E)%9$4+J"'#4)EI-+H)$EI+$9+9=)+!"&'()*+&(',+
I9$K)I+'O+2%9)"J"5I)+;'%95%3'3I+,%9)K"$95'%+$")+K)%)"$44G+L3I9+#)&'E5%K+$&&45E$9)(+9'+$39'E$9)(+
Run functional tests after build
9)I95%K-+
H'+")$&=+9=)+-(./+%+4)7)4+'O+E$93"59G*+9=)")+I='34(+#)+$+I)9+'O+O$I9+9)I9I+9=$9+$")+"3%+D59=+)7)"G+#354(-+
Version control test data
H=)I)+9)I9I+J"'75()+9=)+9)$E+&'%O5()%&)+9=$9+9=)+$JJ45&$95'%+D544+#$I5&$44G+D'"P+75"93$44G+$44+'O+9=)+95E)-+
!"#$%&'()*+,%&-+.+/011+23&45(+67)%3)+8359)+:00+.+;4)7)4$%(*+<=5'+11>>?+.+@A+/>:-B?B-C000++.+DDD-3"#$%&'()-&'E+.+F+/00C+#G+!"#$%&'()+
Automate creation of testing environment

35. Releasing

36. Releasing
Problems
• One person is:
• best person for build
• best person for integration
(so, one can delay the other)
• Forced to branch when close to release

37. Releasing
Done
• Helper scripts

38. ()H4'GE)%9+D544+#)+J3&&)JJK34+5%+9=)+%)L9+)%75"'%E)%9-+
Releasing
Image © 2009 by Urbancode
+
+
6+9)$E+9=$9+=$J+E'7)(+$D$G+K"'E+$+K344G+E$%3$4+H"'&)JJ+9'+3J5%I+$+%3E#)"+'K+=)4H)"+J&"5H9J*+'"+$+K344G+
Fully automate runtime image build
J&"5H9)(+H"'&)JJ*+=$J+E$()+5EH'"9$%9+5EH"'7)E)%9J-+6&"'JJ+9=)+5%(3J9"G*+E'J9+9)$EJ+=$7)+J'E)+=)4H)"+
J&"5H9J+#39+K$44+J='"9+'K+K344G+J&"5H9)(+()H4'GE)%9J+ +)JH)&5$44G+9'+&'%9"'44)(+)%75"'%E)%9J-+
Automate installer creation
!"#$%&$'()#$+9)$EJ+$")+I''(+$9+()H4'G5%I+K'"+9)J95%I+H3"H'J)J-+M=)G+D544+=$7)+H39+9=)5"+K344G+J&"5H9)(+
()H4'GE)%9J+#)=5%(+9''45%I+9=$9+$44'DJ+K'"+H3J=N#399'%+()H4'GE)%9J+9'+J'E)+'"+$44+'K+9=)5"+9)J9+
Automate installation into test environments
)%75"'%E)%9J-+M=5J+9$O)J+$+I")$9+()$4+'K+4'$(+'KK+'K+9=)+()H4'GE)%9+)%I5%))"J+D=54)+")(3&5%I+9=)+
('D%95E)+'K+9)J9+9)$EJ+D$595%I+K'"+$+()H4'GE)%9-+P3J9+$J+&'%95%3'3J+#354(J+$")+$+&=$"$&9)"5J95&+'K+
!"#$%&$'()#$+#354(+9)$EJ*+$39'E$95&+()H4'GE)%9+9'+9=)+K5"J9+9)J9+)%75"'%E)%9+5J+$+=$44E$"O+'K+
Changelog generation
!"#$%&$'()#$+E$93"59G+5%+()H4'GE)%9-++Q)H)%(5%I+'%+9)$E+(G%$E5&J*+9=5J+J='34(+=$HH)%+$9+9=)+)%(+'K+
$%G+J3&&)JJK34+&'%95%3'3J+#354(+'"+$9+")I34$"+5%9)"7$4J+9="'3I=+9=)+($G+D=)%+9)J9)"J+$")+3%45O)4G+9'+#)+
5%9)""3H9)(-+

39. Julian Fitzell
jfitzell@cincom.com
Twitter: @jfitzell
CINCOM and the Quadrant Logo are registered trademarks of Cincom Systems, Inc. © 2011 Cincom Systems, Inc.
All other trademarks belong to their respective companies. All rights reserved