3. Agenda
• What is Continuous Delivery?
• I thought I knew CI, Agile, Quality and DevOps!
• Continuous development, integration and the ability to continuously release
– The CD Pipeline
• Key aspects and players
• What’s my role in CD and why should I care?
• Work, Gamified!
• CD Tools, Products and Solutions out there!
• Wrap-up: Case Studies, What’s Virtusa doing on CD?
• Coming up next…
3
5. The Big Question - What does it take to get a
feature/hotfix/patch to your customer?
• What’s involved in doing a release?
• How long does it take?
• How many steps are involved?
• How successful is the process?
• How often do you do them?
• Do you have “heroes”?
• What would happen if a particular environment disappeared/melted down?
• Do your environments match?
• How quickly can you build an environment?
• How hard is it to do a patch? A major release?
5
8. Continuous Delivery
• Continuous Delivery is a software development discipline where
you build software in such a way that the software can be
released to production at any time.
• Your software is deployable throughout its lifecycle
• Your team prioritizes keeping the software deployable over working on
new features
• Anybody can get fast, automated feedback on the production readiness of
their systems any time somebody makes a change to them
• You can perform push-button deployments of any version of the software
to any environment on demand
Martin Fowler - http://martinfowler.com/bliki/ContinuousDelivery.html
8
9. I thought I knew CI, Agile, Quality and
DevOps!
11. Audience Poll - 1
Who drives Continuous integration and
delivery in your project?
•DEV
•Operations
11
12. Audience Poll - 2
On average how frequently are changes integrated
into the release branch/trunk? (not the developer
feature/promo/task branch, but the main code line
from which a release candidate is built)
•daily
•weekly
•with each sprint
•only once feature development is complete
12
13. Audience Poll - 3
What latent code strategies have you used
within the last 12 months?
•None...SCM feature branching
•Hidden code
•Abstraction
•Feature toggles
13
14. Audience Poll - 4
Merge problems?
•None – We don’t do multiple releases and our single
release process has minimal merges whatsoever
•We don’t do multiple releases but our single release
process has merge issues
•Multiple releases – Merge issues
•Automated Merge process using SCM Streams and CI
automation
14
15. Quick Case Study
Etsy (Online retailer):
22+ million members
800,000+ active shops
18+ million items currently for sale
400+ employees
Business Problem - Our site is so successful, how
can we move fast enough to keep up with feature
demand?
15
24. Etsy - Continuous Deployment Math!
• N = # of deploys
• P = probability of site degradation
• S = average severity of degradation
• T = time to detect/resolve
Expected Downtime = N*P*S*T
24
25. Etsy - Release!!!
N = # of deploys
S = avg. severity of degradation
ED = Expected Downtime
P = prob. of degradation
T = time to detect/resolve
Before CD
N=1
P = 0.5
S = 0.7
T = 100
After CD
N = 250
P = 0.1
S = 0.05
T=5
E.D = 35
E.D = 6.25
25
*Arbitrary Numbers
29. Design
3 inevitabilities we design for:
• Things break, unexpectedly
• What we’re building changes
• We don’t get to start over
29
30. Architectural Principles
Don’t bet against the future.
Our customers are humans.
Simplicity always wins, in the end.
Ambiguity kills momentum.
Make failure cheap.
Technical debt is an inevitable by-product of
shipping code.
Optimize for change.
30
33. Continuous Integration (Build Pipeline)
•
•
•
•
•
•
•
•
Check-in Daily
Make Code Self Testing
Commit to Trunk
Automate the Build
Keep the Build Fast
Every Commit results in Build/Validate
Test in Clone of Production
Automate Deployments
33
35. Testing Recommendations
• Automated Acceptance Tests
•
•
•
•
•
Fast Feedback
Reduce Tester workload
Testers can do Exploratory Testing
Regression Tests
Requirements can be generated from Tests
• Acceptance Test Approaches
• Acceptance Test Driven Development (ATDD)
• Behavior Driven Development (BDD)
Given [some precondition] When [some event] Then [some result]
35
42. Gamified development!
• Gamifying (Agile) Development –
• Stories, Narratives, Spikes
• Project initiation (iteration 0), technology discovery thru games
• Hackathons
• Gamify feature development [motivation, social, engaged, velocity, quality]
• Develop from Cradle to Grave – rewards for tdd, automation from dev to prod (making life easier,
process improvements)
• Progress, Rules and Social impact–
• TDD (rules), Kanban (visual), CI build monitors, Card Wall (visual)
• Socialize – Yammer, Project Dashboards, Leaderboard (Jenkins CI Game Plugin)
42
43. Metrics and Rewards!
• An integrated view of the development efficiency can be derived from usual metrics all along • SVN Metrics
• JIRA Metrics
• Code review metrics
• CI Metrics
• Deployment Metrics
• Automated unit/acceptance test metrics
• Is there a tool (or a set of tools integrated) that pulls out and then co-relates this information to
derive meaningful metrics?
43
45. Metrics and Rewards…
Description
Reward Points
Check-in and build passes (with unit tests)
1
One or more unit tests failed
-10
Compiler error (come on now)
-20
Big check-in caused build to remain in a broken
state for hours
-40
Code review (complexity, re-use, leaks, copypaste)
-40
OthersJIRA metrics
Stack Exchange like metrics (Forums, QnA, tech discovery)
Continuous Improvement metrics (e.g. Maven build – time, accuracy, adherence to standards, Deployment
Engine metrics)
Jenkins - https://wiki.jenkins-ci.org/display/JENKINS/The+Continuous+Integration+Game+plugin
45
“Software development providers can’t deliver new services at the rate business leaders want” - Forrester Consulting “Continuous Delivery: A Maturity Assessment Model”, March 2013
Key Pain Points: Painful releasesInability to predict delivery affecting business commitments Deployment to environmentsCode quality Environment configuration & management Expensive, time consuming manual QA (unable to build lab) -Partial Agile implementation (Scrum but no technical processes)
“Software development providers can’t deliver new services at the rate business leaders want” - Forrester Consulting “Continuous Delivery: A Maturity Assessment Model”, March 2013 Delivery vs. Release •Delivery: integrated, passing all stages of pipeline validation •Release: accessible by customers and market •Deliver continuously, release when business decides •Requires latent code strategy (inherently latent code, abstraction, feature toggles, etc.)
“Software development providers can’t deliver new services at the rate business leaders want” - Forrester Consulting “Continuous Delivery: A Maturity Assessment Model”, March 2013
Throwing Code off the wall - QA testing and Deployment
“Software development providers can’t deliver new services at the rate business leaders want” - Forrester Consulting “Continuous Delivery: A Maturity Assessment Model”, March 2013
“Software development providers can’t deliver new services at the rate business leaders want” - Forrester Consulting “Continuous Delivery: A Maturity Assessment Model”, March 2013
“Software development providers can’t deliver new services at the rate business leaders want” - Forrester Consulting “Continuous Delivery: A Maturity Assessment Model”, March 2013
“Software development providers can’t deliver new services at the rate business leaders want” - Forrester Consulting “Continuous Delivery: A Maturity Assessment Model”, March 2013
Discuss – CIAgileQualityDevOps
Complex systems and change Distributed systems are inherently complex. The outcome of change in complex systems is hard to predict.The outcome of small, frequent, measurable changes are easier to predict, easier to recover from, and promote learning
CD is more difficult to achieve in large, enterprise organizations due to unique challenges; some of them are – - Scale of parallel activity- Complexity of software and environment - Variance in skills/experience of teams/individuals across the organization - Variance in the ability to integrate “work in progress” across the organization - Often no CI across team boundaries
Best Practices - Extreme Programming (XP)Fail a Build for Design BreachesFail a Build for Slow TestsFail a Build for Warnings & Code Style BreachesBuild Pipeline Practices –Build Once, Deploy ManyDeploy the Same Way to All EnvironmentsSmoke Test your DeploymentsDeploy into a Copy of ProductionEach Change should Propagate InstantlyIf any part fails, Stop the line
Use Appropriate Technology – In-House Vs. External Deployment automation productsApp deployment - Scripting Approaches Jenkins driven deployment Script logs (ssh) into each box and runs deployment commands Script runs locally, each remote machine has deployment agent to execute script Package app using platform’s packaging technology (rpm, gems) and use infrastructure management tools to push new versions App syncs-up with an online store and automatically pulls latest updates (artifacts, db, properties)Database DeploymentsVersion your DatabaseEvolve Your Database Incrementally41Environment ManagementVersion your Application ConfigurationVersion your Operating System42
CD is more difficult to achieve in large, enterprise organizations due to unique challenges; some of them are – - Scale of parallel activity- Complexity of software and environment - Variance in skills/experience of teams/individuals across the organization - Variance in the ability to integrate “work in progress” across the organization - Often no CI across team boundaries
Why should we gamify work? Adapt work for incoming, younger workforce Intrinsic rewards are renewable resource Develop leadership in teams More-satisfying work = better productivity Develops people by pointing them forward to clear goals Fosters teamwork and accountability Key to greater innovation (through imagination)
Dangers- You can’t just spawn a new project after failing Could depersonalize rather than personalize No one game can please all Gaming and misuse of metricsApproach-Take an agile approach (Increment, Iterate, Improve) Focus on the intrinsic first (extrinsic will come) Voluntary participation Give control and teams design their own games (and fail) Lightweight, both in implementation and tone (i.e., fun)
The Development Game
Jenkins CI Game Plugin –-10 points for breaking a build0 points for breaking a build that already was broken+1 points for doing a build with no failures (unstable builds gives no points)-1 points for each new test failures+1 points for each new test that passeshttp://www.badgeville.com/products/gamificationhttp://www.redcrittertracker.com/