Analyst Keynote: Continuous Delivery: Making DevOps Awesome

1. World®
’16
Analyst Keynote:
Continuous Delivery: Making DevOps
Awesome
Nicole Forsgren - PhD – CEO - Chief Scientist
DORA: DevOps Research & Assessment
DO5T14S
DEVOPS

2. 2 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type of
warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of this Presentation

3. 3 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Abstract
For years, there have been stories of continuous delivery making teams awesome… but can
continuous delivery make all teams awesome? And how?
Dr. Nicole Forsgren will present data from over 25,000 technical professionals showing the central
role that continuous delivery plays in software development and delivery. She will show you how
doing continuous delivery can drive key organizational outcomes like profitability, productivity,
and market share. Nicole also presents the key aspects of continuous delivery that make your
DevOps awesome, like trunk-based development, test data, and test automation, and provides
examples of success from teams undergoing their own technology transformations.
The presentation also includes other important drivers of DevOps success, like lean product
management and team culture. At the end of this talk, you will have the information to help you
prove your case (to management or even yourself) about why continuous delivery and DevOps
are essential to winning, as well as great stories and examples to really bring these concepts to
life. You’ll leave with tips you can take back to get started on your own DevOps initiative.
Nicole
Forsgren, PhD
DevOps Research &
Assessment
CEO, Chief Scientist

4. 4 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Agenda
INTRODUCTION
IT DOES MATTER
THERE’S MORE…
DEVOPS IS
CONTINUOUS DELIVERY
IT PERFORMANCE DRIVERS
1
2
3
4
5
6

5. Continuous Delivery:
Making DevOps Awesome
Nicole Forsgren, PhD
CEO and Chief Scientist, DevOps Research and Assessment (DORA)
Research Affiliate, Clemson University, Florida International University

6. “IT doesn’t matter.”
-- Nicholas Carr, 2003
@nicolefv

7. IT Does Matter
• Times – and IT – have changed
• DevOps is good for Organizations
• DevOps is good for IT
• And then some detail: What drives this change?
• Technical practices (hint: Continuous Delivery)
• Management practices (hint: from Lean Management)
• Culture and identity
@nicolefv

8. @nicolefv

9. @nicolefv

10. DevOps is
Technical practices
seen in Continuous Delivery,
Management practices
seen in Lean Management principles, and
Organizational Culture
@nicolefv
Research shows that these drive Organizational Performance
and IT Performance

11. DevOps is good for organizations
@nicolefv

12. High Performing IT organizations
2x
More likely to exceed
Profitability,
Market share, and
Productivity goals
50%
Higher market cap
growth over 3 years*
The 2014 and 2015 State of DevOps Report
@nicolefv

13. Devops is good for IT
Measuring DevOps and IT Performance
- Deploy frequency (when business demands)
- Lead Time for Changes
- Mean Time to Recover (MTTR)
- Change Fail Rate
@nicolefv

14. High Performing DevOps teams
More agile
The 2016 State of DevOps Report
@nicolefv
200x
200x more frequent
deployments
2,555x
2,555x shorter lead times

15. High Performing DevOps teams
More reliable
The 2016 State of DevOps Report
@nicolefv
24x
24x faster
recovery from failures
3x
3x lower
change fail rage

16. DevOps promises – and delivers
More throughput
More stability
In tandem. Without the tradeoffs that ITIL calls
for.
Let’s talk about what this means for us
@nicolefv

17. High Performing DevOps teams
More agile What does this mean for:
200x
More frequent
deployments
The 2016 State of DevOps Report
New content delivery
Value/savings around A/B testing
Value around speed to market
Compliance / regulatory
Security2,555x
Faster lead times
@nicolefv

18. Evaluating well-designed and executed
experiments that were designed to
improve a key metric, only about 1/3
were successful at improving the key
metric!
Online Experimentation at Microsoft, Kohavi et al http://stanford.io/130uW6X
@nicolefv

19. High Performing DevOps teams
More reliable What does this mean for:
3x
Fewer deploy
failures
The 2016 State of DevOps Report
Value/savings around reliability
Value/savings around uptime
Compliance
Security
Reputation around uptime,
compliance & security
24x
Faster MTTR
@nicolefv

20. @nicolefv

21. "Fixing it as soon as possible or
having compensating controls in place
days before could have saved this
entire breach from occurring in the
first place."

22. We know:
• IT Performance is comprised of throughput and stability, and
both are possible without tradeoffs
• IT Performance contributes to org performance
(profitability, productivity, market share)
So:
What drives IT and Organizational Performance?
@nicolefv

23. IT Performance
Org
Performance???
@nicolefv

24. IT Performance
Org
Performance???
1.
@nicolefv

25. IT Performance
Org
Performance???
2. 1.
@nicolefv

26. IT Performance
Org
Performance???
2. $$$1.
@nicolefv

27. IT Performance
Org
Performance
Continuous
Delivery
@nicolefv
Continuous Delivery

28. IT Performance
Org
Performance
Continuous
Delivery
@nicolefv
Continuous Delivery
• Effective test data
management
• Test & Deployment
automation
• Trunk-based
development
• Continuous
integration
• Version control
• Shifting left on
security

29. IT Performance
Org
Performance
Continuous
Delivery
@nicolefv
Continuous Delivery
• Effective test data
management
• Test &
Deployment
automation
• Trunk-based
development
• Continuous
integration
• Version control
• Shifting left on
security

30. Test Data Management
Analysis showed that data was
creating bottlenecks in the process.
Key activities in TDM include:
• Test data masking and
subsetting
• Test data automation
• Integration with tooling
Outcomes:
• Faster time to market
• Improved quality and customer
experience
• Significant productivity improvements
and savings
• Improved testing efficiencies
@nicolefv

31. Test & Deployment Automation
“By getting new releases and services out the door quickly, we can provide a better
experience to millions of car buyers and sellers and continue to differentiate ourselves in
a competitive market.”
-- Adam Mills, Senior Manager of Application Development, AutoTrader.com
Using a DevOps approach to unify teams, they have been able to automate
their test, build, and deployment
Using these principles, Autotrader.com has been able to:
• Reduce test, build, and deploy times from
weeks to just minutes
• Decrease software defects by 25%

32. IT Performance
Org
Performance
Continuous
Delivery
@nicolefv
Continuous Delivery also makes our work feel better
• Effective test data
management
• Deployment
automation
• Test automation
• Trunk-based
development
• Continuous
integration
• Version control
• Shifting left on
security Decreases
burnout and
deploy pain

33. Microsoft Engineering: DevOps Lessons
Thiago Almeida -- DevOps Days London, 2016
@nicolefv
Work/Life Scores
Before CD
38%
After CD:
75%
Source: https://vimeo.com/165184757

34. But what else drives IT Performance?
The 2015 DevOps Survey of Practice and its resulting database are the property of Puppet Labs, Inc. and Gene Kim and Associates, LLC. All rights reserved.
@nicolefv

35. IT Performance
Org
Performance
Lean
Practices
@nicolefv
Lean Management Practices

36. IT Performance
Org
Performance
Lean
Practices
@nicolefv
Lean Management Practices
• Effective WIP
limits
• Visual displays
• Monitoring tools
• Gathering,
montoring,
implementing
customer feedback
• Small batches
• Making work
visible through the
value stream

37. IT Performance
Org
Performance
Lean
Practices
@nicolefv
Lean Management Practices
• Effective WIP
limits
• Visual displays
• Monitoring tools
• Gathering,
montoring,
implementing
customer feedback
• Small batches
• Making work
visible through the
value stream

38. @nicolefv
Visual Displays
“If it moves, graph it.”
- Michael Rembetsy, Former VP Operations, Etsy

39. Making Work Visible Through the Value
Stream
“One of the greatest effects… is that
we are now asked much earlier by the
business to participate in the
requirements gathering process.”
-- Project Manager at a.s.r.
By creating flowcharts to show work as it moved throughout
the value stream, a.s.r. was able to leverage insights from all
stakeholders including business, test, and IT. Leveraging
these tools, teams could immediately understand changes
made to requirements and their impact.
Using this principle, a.s.r. was been able to:
• Reduce test time preparation time
significantly
• Release software with no incidents
• Deliver on time and
on budget

40. IT Performance
Org
Performance
Lean
Practices
@nicolefv
Lean Management Practices also make work
feel better
• Effective WIP
limits
• Visual displays
• Monitoring tools
• Gathering,
montoring,
implementing
customer feedback
• Small batches and
making work
visible
Decreases
burnout

41. “I was trying to figure out why my team was working themselves
to death but not getting anything done… By implementing WIP
limits, we were able to focus on our work. Finishing work feels
better than sprinting and feeling like a hero in the moment,
because that’s only a moment.”
- Julia Wester,
Development Manager for
Turner Sports, Turner
Broadcasting
@nicolefv

42. But wait… there’s more

43. IT Performance
Org
Performance
Continuous
Delivery &
Lean
Practices
@nicolefv
CD and Lean Management practices also improve our
culture and identity, which improve IT & Org
Performance
Westrum Org
Culture
Identity

44. Westrum Organizational Culture
@nicolefv
Pathological
Power-oriented
Bureaucratic
Rule-oriented
Generative
Performance-oriented
Low cooperation Modest cooperation High cooperation
Messengers shot Messengers neglected Messengers trained
Responsibilities shirked Narrow responsibilities Risks are shared
Bridging discouraged Bridging tolerated Bridging encouraged
Failure leads to scapegoating Failure leads to justice Failure leads to inquiry
Novelty crushed Novelty leads to problems Novelty implemented

45. Identity
• I am glad I chose to work for this organization rather than another company.
• I talk of this organization to my friends as a great company to work for.
• I am willing to put in a great deal of effort beyond what is normally expected to
help my organization to be successful.
• I find that my values and my organization's values are very similar.
• In general, the people employed by my organization are working toward the
same goal.
• I feel that my organization cares about me.
Adapted from Atreyi Kankanhalli, Bernard C.Y. Tan, and Kwok-Kee Wei (2005), “Contributing Knowledge to
Electronic Knowledge Repositories: An Empirical Investigation,“ MIS Quarterly, 29, 113-143.
@nicolefv

46. Google Team
Performance
Surprise! No magic
formula for what
makes the perfect
team
@nicolefv

47. Intuit
“By installing a rampant innovation culture, we performed 165
experiments in the peak three months of tax season.
Our business result? Conversion rate of the website is up 50%.
Employee result? Everyone loves it, because their new ideas
can make it to market. ”
- Scott Cook, Intuit founder
@nicolefv

48. Amazon
“I think building this culture is the key to innovation.
Creativity must flow from everywhere. Whether you are a
summer intern or the CTO, any good idea must be able to seek
an objective test, preferably a test that exposes the idea to real
customers. Everyone must be able to experiment, learn, and
iterate.”
- Greg Linden
@nicolefv

49. IT Does Matter
• Times – and IT – have changed
• DevOps is good for Organizations
• DevOps is good for IT
• And then some detail: What drives this change?
• Technical practices (hint: Continuous Delivery)
• Management practices (hint: from Lean Management)
• Culture and identity
@nicolefv

50. For more information:
@nicolefv
Sign up for our ROI
whitepaper & get peer-
reviewed research
devops-research.com

51. Thank you
nicole@devops-research.com
@nicolefv
nicolefv.com

52. 52 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Recommended Sessions
SESSION # TITLE DATE/TIME
DO5T18S
Manheim's Automated Testing Strategy With CA Test
Data Manager
11/17/2016 at 11:30 am
DO5T04S
American Family Insurance leverages CA Service
Virtualization to accelerate continuous integration in
development.
11/17/2016 at 12:45 pm
DO5T22S
ING Delivers Unprecedented Global Continuous Delivery
as a Service
11/17/2016 at 03:00 pm

53. 53 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Must See Demos
Modernize
App Delivery
Integrated CD
Theater 5 - DOV501P
Deliver Test
Data Faster
CA Test Data Manager
Theater 5 - DOV511P
Deliver Better
Apps
CA Service
Virtualization
Theater 5 - DOV507P
Orchestrate
Your Release
CA Release Automation
Theater 5 - DOV513P

54. 54 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
Stay connected at communities.ca.com
Thank you.

55. 55 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
DevOps – Continuous Delivery
For more information on DevOps – Continuous Delivery, please
visit: http://cainc.to/PiTFpu