Presentation on Continuous Delivery I gave at BBWorld/DevCon 2013 in Las Vegas

1. Introduction to
Continuous Delivery
Mike McGarr, Director of Learn DevOps
Blackboard, Inc.

2. ABOUT ME
Mike McGarr
Director of Learn DevOps
Blackboard Inc.
michael.mcgarr@blackboard.com
I have been at Blackboard for 8 months.
Software Engineer for 12+ years
Founder of the DC Continuous
Integration, Delivery and Deployment Meetup

3. WHAT WE ARE GOING
TO LEARN TODAY
Overview of Continuous Delivery
Why everybody should care
Where to get started
Sneak Peek at some tools/techniques

4. THE DREADED
RELEASE

5. WHY ARE RELEASES
HARD?
They break things
This is production, so we have
to be careful
They don’t always work
A lot changed, so we need to
verify everything is ok
In-Frequent
Releases

6. THE FAMILY
VACATION

7. VACATION DISASTER

8. A DAY AT THE BEACH
Releases
should be a
Non-Event

9. THE LAST MILE
Manual Deployments
Insufficient Configuration Management
Infrequent, Error Prone Deployments

10. WHAT IS CONTINUOUS
DELIVERY?
“Continuous delivery is a set
of practices and principles
aimed at, building, testing
and releasing software
faster and more frequently”
cycle time

11. CYCLE TIME
“How long would it take your organization to
deploy a change [to production] that involves
just one single line of code? Do you do this
on a repeatable, reliable basis?”
- Mary and Tom Poppendieck,
Implementing Lean Software Development

12. Quality or Speed

13. FREQUENT AUTOMATED
DEPLOYMENTS

14. FASTER = LESS RISK

15. ALWAYS
PRODUCTION READY

16. APPLYING CONTINUOUS
DELIVERY
Principles:
• Improve continuously
• Releasing software MUST be
repeatable and reliable
• Automate everything!
• If it hurts, do it more often
• Keep everything in version
control
• Done means “released”
• Build quality in!
• Everybody has responsibility for
the release process Agile (Lean)
Configuration
Management
Continuous
Integration
Testing
Deployment Pipelines
Continuous Deployment
Deployment Automation
Practices:

17. AGILE (LEAN)
Agile (Lean)
Configuration
Management
Continuous
Integration
Testing
Deployment Pipelines
Continuous Deployment
Deployment Automation

18. CULTURE

19. PROCESS

20. HACKING CULTURE
Lean
Kanban
Scrum
XP
Lean Startup
Change Management

21. VISUALIZE FLOW

22. CONFIGURATION
MANAGEMENT
Agile (Lean)
Configuration
Management
Continuous
Integration
Testing
Deployment Pipelines
Continuous Deployment
Deployment Automation

23. VERSION CONTROL
23
• Version Everything
• Tool should stay out of the
way
• Good Branching Strategy
• Don’t check-in binary files

24. 24
AUTOMATE THE BUILD
• No IDE Builds!
• KISS
• Fail Fast (but smartly)
• No Wiki’s to Build
• Pick an appropriate Technology
• Keep it organized
• Dependency Management

25. 25
BUILD ONCE, DEPLOY MANY

26. BINARY REPOSITORIES
26

27. TESTING
Agile (Lean)
Configuration
Management
Continuous
Integration
Testing
Deployment Pipelines
Continuous Deployment
Deployment Automation

28. 28
TESTING IS NOT A PHASE
http://flic.kr/p/6bcg

29. 29
TYPES OF TESTS

30. 30
TESTING PYRAMID

31. 31
UNIT TESTING
Fast
Independent
Repeatable
Self-Verifying
Timely

32. 32
TEST DRIVEN DEVELOPMENT
http://reddevnews.com/articles/2007/11/01/testdriven-development-tdd.aspx

33. 33
SPECIFICATION BY EXAMPLE

34. CONTINUOUS
INTEGRATION
Agile (Lean)
Configuration
Management
Continuous
Integration
Testing
Deployment Pipelines
Continuous Deployment
Deployment Automation

35. CONTINUOUS
INTEGRATION
• Check-in Daily
• Commit to Trunk
• Automate the Build
• Keep the Build Fast
• Every Commit results in Build
• Test in Clone of Production
• Automate Deployment
35

36. HOW TO GET
STARTED
Continuous Integration on a Dollar a Day by James Shore
(http://bit.ly/13CUiVn)
Setup:
1. Old Computer
2. Rubber Chicken
3. Desk Bell
4. Automated Build
5. Drink the Kool-Aid
6. Launch
How it works:
1. Run build locally
2. Take Rubber Chicken
3. Get latest from SCM
4. Check-in code
5. Check-out on Old Computer
6. Run build on old computer
• If it fails, revert changes
7. Ring bell if passes

37. CI SERVERS
• Human Proof
• Build per commit
• Notifications (Email alerts, RSS, etc.)
• Dashboards & Charts
• Plugins and Integrations
• Historical Data
• Web Accessible

38. DEPLOYMENT
AUTOMATION
Agile (Lean)
Configuration
Management
Continuous
Integration
Testing
Deployment Pipelines
Continuous Deployment
Deployment Automation

39. CODE DEPLOYMENTS
39
1. Push New Binary to Server
2. Stop Server
3. Archive Old Binary
4. Deploy New Binary
5. Start Server

40. VERSION YOUR DATABASE
40
Version
Control
DB
Change
DB
Change
DB
Change
Build
Server
Production
Test
Dev
.sqlHTML
DBA

41. LIQUIBASE EXAMPLE

42. INFRASTRUCTURE
AS CODE

43. DEPLOYMENT
PIPELINES
Agile (Lean)
Configuration
Management
Continuous
Integration
Testing
Deployment Pipelines
Continuous Deployment
Deployment Automation

44. 44
DEPLOYMENT PIPELINES
A Deployment Pipeline is an automated
manifestation of your process for getting
software from version control into the
hands of your users.

45. 45
DEPLOYMENT PIPELINES
(AKA BUILD PIPELINES)

46. PIPELINES IN
JENKINS

47. CONTINUOUS
DEPLOYMENT
Agile (Lean)
Configuration
Management
Continuous
Integration
Testing
Deployment Pipelines
Continuous Deployment
Deployment Automation

48. 48
CONTINUOUS WHAT?
Continuous
Delivery
Continuous
Deployment
Continuous
Integration
commit =
a build
commit =
shippable
product
commit =
production
release

49. 49
CONTINUOUS DEPLOYMENT

50. CONTINUOUS
DEPLOYMENT
• Dark Releases
• Deploy != Software Release
• Test in Production
• Feature Flags
• Branch by Abstraction
• Canary Releases
• A/B Testing

51. 51
METRICS AND ALERTING
https://github.com/monitoringsucks

52. THANK YOU!
Mike McGarr
Director of Learn DevOps
Blackboard Inc
michael.mcgarr@blackboard.com
http://earlyandoften.wordpress.com
@jmichaelmcgarr

53. 53
FURTHER READING
• Continuous Delivery: Reliable Software Releases through Build, Test and
Deployment Automation, by Jez Humble and David Farley -
http://www.amazon.com/Continuous-Delivery-Deployment-Automation-
Addison-Wesley/dp/0321601912
• Continuous Integration, By Paul M Duvall -
http://www.amazon.com/Continuous-Integration-Improving-Software-
Reducing/dp/0321336380
• Refactoring Databases, By Scott Ambler and Pramod Sadalage -
http://www.amazon.com/Refactoring-Databases-Evolutionary-paperback-
Addison-Wesley/dp/0321774515
• Refactoring, By Martin Fowler - http://www.amazon.com/Refactoring-
Improving-Design-Existing-Code/dp/0201485672
• Growing Object-Oriented Software, Guided by Tests, by Steven Freeman and
Nat Pryce - http://www.amazon.com/Growing-Object-Oriented-Software-
Guided-Tests/dp/0321503627

54. 54
FURTHER READING
• Extreme Programming Explained, by Kent Beck -
http://www.amazon.com/Extreme-Programming-Explained-Embrace-
Change/dp/0201616416
• Release It!, by Michael Nygard - http://pragprog.com/book/mnee/release-it
• xUnit Test Patterns, by Gerard Meszaros - http://www.amazon.com/xUnit-
Test-Patterns-Refactoring-Code/dp/0131495054
• Working Effectively with Legacy Code, by Michael Feathers -
http://www.amazon.com/Working-Effectively-Legacy-Michael-
Feathers/dp/0131177052
• Agile Testing, by Lisa Crispin and Janet Gregory -
http://www.amazon.com/Agile-Testing-Practical-Guide-
Testers/dp/0321534468

55. 55
FURTHER READING
• Test Driven Development (TDD) – http://en.wikipedia.org/wiki/Test-
driven_development
• Introducing BDD, by Dan North – http://dannorth.net/introducing-bdd/
• Agile Manifesto – http://agilemanifesto.org/
• Scrum – http://www.scrumalliance.org/learn_about_scrum
• Continuous Integration, by Martin Fowler –
http://martinfowler.com/articles/continuousIntegration.html
• Specification by Example, by Gojko Adzic -
http://specificationbyexample.com/
• Build Pipelines - http://www.magpiebrain.com/2009/12/13/a-brief-and-
incomplete-history-of-build-pipelines/
• Implementing Lean Software Development, by Mary and Tom Poppendieck -
http://www.amazon.com/Implementing-Lean-Software-Development-
Concept/dp/0321437381

56. 56
FURTHER READING
• Maven Releases on Steriods, by Axel Fontaine –
http://www.axelfontaine.com/2011/01/maven-releases-on-steroids-
adios.html
• What is in a Name? Usually a version number, actually., by James Betteley -
http://jamesbetteley.wordpress.com/2011/07/07/what-is-in-a-name-usually-
a-version-number-actually/
• Build Once, Deploy Many -
http://earlyandoften.wordpress.com/2010/09/09/build-once-deploy-many/
• Evolutionary Design - http://martinfowler.com/articles/designDead.html
• Continuous Deployment -
http://timothyfitz.wordpress.com/2009/02/08/continuous-deployment/
• Sonar’s Technical Debt Calculation - http://www.sonarsource.org/evaluate-
your-technical-debt-with-sonar/
• Gherkin - https://github.com/cucumber/cucumber/wiki/Gherkin

57. 57
FURTHER READING
• Branch by Abstraction, by Paul Hammant
http://paulhammant.com/blog/branch_by_abstraction.html
• Making Large Scale Changes Incrementally with Branch by Abstraction, by Jez
Humble - http://continuousdelivery.com/2011/05/make-large-scale-changes-
incrementally-with-branch-by-abstraction/
• What is DevOps?, by Damon Edwards - http://dev2ops.org/2010/02/what-is-
devops/
• Monitoring Sucks - https://github.com/monitoringsucks

58. 58
TOOLS
• Git - http://git-scm.com/
• Subversion - http://subversion.tigris.org/
• Mercurial - http://mercurial.selenic.com/
• Rational ClearCase - http://www-01.ibm.com/software/awdtools/clearcase/
• Serena Dimensions CM - http://www.serena.com/products/dimensions-
cm/index.html
• Ant - http://ant.apache.org/
• Ivy - http://ant.apache.org/ivy/
• Maven - http://maven.apache.org/
• Gradle - http://gradle.org/
• JUnit – http://www.junit.org/
• Mockito – http://code.google.com/p/mockito/
• Hamcrest – http://code.google.com/p/hamcrest/
• Spock – http://code.google.com/p/spock/
• dbUnit – http://www.dbunit.org/
• Unitils – http://unitils.org/summary.html

59. 59
TOOLS
• Findbugs – http://findbugs.sourceforge.net/
• PMD – http://pmd.sourceforge.net/
• Checkstyle – http://checkstyle.sourceforge.net/
• JIRA – http://www.atlassian.com/software/jira/overview
• GitHub – https://github.com/
• Jenkins - http://jenkins-ci.org/
• TeamCity – http://www.jetbrains.com/teamcity/
• Nexus – http://www.sonatype.org/nexus/
• Artifactory – http://www.jfrog.com/products.php
• Sonar – http://www.sonarsource.org/
• FitNesse – http://fitnesse.org/
• Concordion – http://www.concordion.org/
• Cucumber – http://cukes.info/
• easyb – http://www.easyb.org/
• jBehave - http://jbehave.org/
• geb - http://www.gebish.org/

60. 60
TOOLS
• Liquibase – www.liquibase.org/
• Flyway – http://code.google.com/p/flyway/
• Escape – http://code.google.com/p/escservesconfig/
• Capistrano - https://github.com/capistrano/capistrano
• Deployinator - https://github.com/etsy/deployinator
• Puppet – http://puppetlabs.com/
• Chef – http://www.opscode.com/chef/
• Ansible - http://ansible.cc/
• Vagrant – http://vagrantup.com/
• JMeter – http://jmeter.apache.org/
• Nagios - http://www.nagios.org/
• Logstash - http://logstash.net/
• StatsD – https://github.com/etsy/statsd
• Metrics – http://metrics.codahale.com/
• Ganglia – http://ganglia.sourceforge.net/
• Graphite - http://graphite.wikidot.com/

61. 61
TESTING PYRAMIDS
• http://fabiopereira.me/blog/2012/03/05/testing-pyramid-a-case-study/
• http://www.takadevelop.com/2013/02/09/testing-pyramid-for-agile-
development/
• http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid/
• http://martinfowler.com/bliki/TestPyramid.html
• http://mifos.org/contributors/qa-testing/future-quality-improvements-mifos
• http://www.ibm.com/developerworks/java/library/j-aopwork11/