ROI & Business Value of CI, CD, DevOps, DevSecOps, & Microservices

Business Value of
CI, CD, & DevOpsSec
Scaling Up to Billion User Global Systems of
Systems Using END-TO-END AUTOMATION &
CONTAINERIZED DOCKER UBUNTU IMAGES
Twitter: @dr_david_f_rico
Website: http://www.davidfrico.com
LinkedIn: http://www.linkedin.com/in/davidfrico
Agile Capabilities: http://davidfrico.com/rico-capability-agile.pdf
Agile Cost of Quality: http://www.davidfrico.com/agile-vs-trad-coq.pdf
DevOps Return on Investment (ROI): http://davidfrico.com/rico-devops-roi.pdf
Dave’s NEW Business Agility Video: http://www.youtube.com/watch?v=hTvtsAkL8xU
Dave’s NEWER Scaled Agile Framework SAFe 4.5 Video: http://youtu.be/1TAuCRq5a34
Dave’s NEWEST Development Operations Security Video: http://youtu.be/X22kJAvx44A
DoD Fighter Jets versus Amazon Web Services: http://davidfrico.com/dod-agile-principles.pdf
Principles of Collaborative Contracting: http://davidfrico.com/collaborative-contract-principles.pdf
DR. DAVID F. RICO, PMP, CSEP, EBAS, BAF, FCP, FCT, ACP, CSM, SAFE, DEVOPS

Author Background
 Gov’t contractor with 36+ years of IT experience
 B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys.
 Large gov’t projects in U.S., Far/Mid-East, & Europe
2

 Career systems & software engineering methodologist
 Lean-Agile, Six Sigma, CMMI, ISO 9001, DoD 5000
 NASA, USAF, Navy, Army, DISA, & DARPA projects
 Published seven books & numerous journal articles
 Intn’l keynote speaker, 260 talks to 105,000 people
 Specializes in metrics, models, & cost engineering
 Cloud Computing, SOA, Web Services, FOSS, etc.
 Professor at 7 Washington, DC-area universities

3
Internet of Things—Dinosaur Killer
IoT is an Extinction Level Event
• 25-50B Devices on IOT
• 5-10B Internet Hosts
• 4-8B Mobile Phones
• 2-3B End User Sys
• Mass Business Failure

4
DevOps—Scotty from Star Trek
We’re Gonna Need Some Really Big Warp Engines
to Move the Enterprise at Speed of Light Captain!

5
DevOps—The Rock
Don’t Show Up for a Gunfight
Without a Kevlar Vest

 Dev-Ops (dĕv′ŏps) Early, iterative, & automated combo
of development & operations; Incremental deployment
 An approach embracing principles & values of lean
thinking, product development flow, & agile methods
 Early, collaborative, and automated form of incremental
development, integration, system, & operational testing
 Design method that supports collaboration, teamwork,
iterative development, & responding to change
 Mult-tiered automated framework for TDD, Continuous
Integration, BDD, Continuous Delivery, & DevOps
 Maximizes BUSINESS VALUE of organizations, portfolios,
& projects by enabling buyers-suppliers to scale globally
6

Crispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.
Crispin, L., & Gregory, J. (2015). More agile testing: Learning journeys for the whole team. Boston, MA: Addison-Wesley.
DevOps—What is it?

Network
Computer
Operating System
Middleware
Applications
APIs
GUI
 Agile requirements implemented in slices vs. layers
 User needs with higher business value are done first
 Reduces cost & risk while increasing business success
7Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.
Agile Traditional
1 2 3 Faster
 Early ROI
 Lower Costs
 Fewer Defects
 Manageable Risk
 Better Performance
 Smaller Attack Surface
Late 
No Value 
Cost Overruns 
Very Poor Quality 
Uncontrollable Risk 
Slowest Performance 
More Security Incidents 
Seven Wastes
1. Rework
2. Motion
3. Waiting
4. Inventory
5. Transportation
6. Overprocessing
7. Overproduction
MINIMIZES MAXIMIZES
 JIT, Just-enough architecture
 Early, in-process system V&V
 Fast continuous improvement
 Scalable to systems of systems
 Maximizes successful outcomes
 Myth of perfect architecture
 Late big-bang integration tests
 Year long improvement cycles
 Breaks down on large projects
 Undermines business success

DevOps—How it works?


 



8
Traditional vs. Agile Cumulative Flow
Work(Story,Point,Task)orEffort(Week,Day,Hour)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Work(Story,Point,Task)orEffort(Week,Day,Hour)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
TRADITIONAL Cumulative Flow
 Late big bang integration increases WIP backlog
 Agile testing early and often reduces WIP backlog
 Improves workflow and reduces WIP & lead times
Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.
Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.

DevOps—Workflow Results
 
DEVOPS Cumulative Flow

9
 Methods to “scope” project, product, or system
 “Key” is smallest possible scope with highest value
 Reduces cost, risk, time, failure, & tech. obsolescence
INCREASES TESTABILITY, QUALITY, RELIABILITY, SECURITY, MORALE, MAINTAINABILITY, & SUCCESS
Denne, M., & Cleland-Huang, J. (2004). Software by numbers: Low-risk, high-return development. Santa Clara, CA: Sun Microsystems.
Ries, E. (2011). The lean startup: How today's entrepreneurs use continuous innovation. New York, NY: Crown Publishing.
Patton, J. (2014). User story mapping: Discover the whole story, build the right product. Sebastopol, CA: O'Reilly Media.
Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.
Krause, L. (2014). Microservices: Patterns and applications. Paris, France: Lucas Krause.

MINIMUM
MARKETABLE FEATURE
- MMF -
 Advantage
 Difference
 Revenue
 Profit
 Savings
 Brand
 Loyalty
MINIMUM
VIABLE PRODUCT
- MVP -
 Goal
 Process
 Features
 Priorities
 Story Map
 Architecture
STORY MAP
OR IMPACT MAP
- SM or IM -
 Goal
 Actors
 Impacts
 Deliverables
 Measures
 Milestones
VISION
STATEMENT
- VS -
 For <customer>
 Who <needs it>
 The <product>
 Is a <benefit>
 That <customer>
 Unlike <other>
 Ours <different>
MICRO-
SERVICE
- MS -
 Purpose
 Automated
 Unique
 Independent
 Resilient
 Ecosystem
 Consumer
DevOps—MMF, MVP, MVA, etc.



10

 Lightweight, fast, disposable virtual environments
 Set of isolated processes running on shared kernel
 Efficient way for building, delivering, & running apps
Monolithic Applications Just-Enough Applications Containerized Apps
Minimal - Typically single process entities
Declarative - Built from layered Docker images
Immutable - Do exactly same thing from run to kill
• Small autonomous services that work together
• Self-contained process that provides a unique capability
• Loosely coupled service oriented architecture with bounded contexts
• Small independent processes communicating with each other using language-agnostic APIs
• Fined-grained independent services running in their own processes that are developed and deployed independently
• Suite of services running in their own process, exposing APIs, and doing one thing well (independently developed and deployable)
• Single app as a suite of small services, each running in its own process and communicating with lightweight mechanisms (HTTP APIs)
Krause, L. (2014). Microservices: Patterns and applications. Paris, France: Lucas Krause.
DevOps—Microservices




 
OS’s Have
UNREPORTED
30-50 CVEs


11

TDD
- 2003 -
CI
- 2006 -
BDD
- 2008 -
CD
- 2011 -
DEVOPS
- 2012 -
DEVOPSSEC
- 2014 -
 User Story
 Acc Criteria
 Dev Unit Test
 Run Unit Test
 Write SW Unit
 Re-Run Unit Test
 Refactor Unit
 Building
 Database
 Inspections
 Testing
 Feedback
 Documentation
 Deployment
 Analyze Feature
 Acc Criteria
 Dev Feat. Test
 Run Feat. Test
 Develop Feature
 Re-Run Feature
 Refactor Feat.
 Packaging
 Acceptance
 Load Test
 Performance
 Pre-Production
 Certification
 Deployment
 Sys Admin
 Config. Mgt.
 Host Builds
 Virtualization
 Containerization
 Deployment
 Monitor & Supp
 Sec. Engineer.
 Sec. Containers
 Sec. Evaluation
 Sec. Deploy.
 Runtime Prot.
 Sec. Monitoring
 Response Mgt.
Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.
Barker, K., & Humphries, C. (2008). Foundations of rspec: Behavior driven development with ruby and rails. New York, NY: Apress.
Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
Huttermann, M. (2012). Devops for developers: Integrate development and operations the agile way. New York, NY: Apress.
Bird, J. (2016). Devopssec: Delivering secure software through continuous delivery. Sebastopol, CA: O'Reilly Media.
 Numerous models of lean-agile testing emerging
 Based on principles of lean & agile one piece flow
 Include software, hardware, system, & port. testing
DevOps—Evolution

STAGE 1—Test Driven Development
 Term coined by Kent Beck in 2003
 Consists of writing all tests before design
 Ensures all components are verified and validated
12Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.



 Agile TDD consists of seven broad practices
 Document test criteria, tests, software units, etc.
 Include refactoring, verification, optimization, etc.
13
Practice
User Story
Acc Criteria
Dev Test
Run Test
Dev Unit
Rerun Test
Refactor Unit
Description
Read story, analyze meaning, ask questions, and clarify understanding
Identify, verify, and document acceptance criteria for each user story
Design, develop, code, and verify automated unit test for user story
Run automated unit test to verify that it fails the first time (sanity check)
Design, develop, code, and verify the software unit to satisfy user story
Rerun automated unit test to see if code satisfies automated unit test
Refine, reduce, and simplify code to remove waste and optimize performance
STAGE 1—Test Driven Develop.

Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.

STAGE 2—Behavior Driven Develop.
 Term coined by Dan North in 2006
 Consists of writing feature tests before design
 Ensures all system features are verified and validated
14Smart, J. F. (2014). BDD in action: Behavior-driven development for the whole software lifecycle. Shelter Island, NY: Manning Publications.



 Agile BDD consists of seven broad practices
 Document test criteria, tests, syst. features, etc.
 Include refactoring, verification, optimization, etc.
15
Practice
Feature
Acc Criteria
Dev Test
Run Test
Dev Feature
Rerun Test
Refac Feature
Description
Read feature, analyze meaning, ask questions, and clarify understanding
Identify, verify, and document acceptance criteria for each feature
Design, develop, code, and verify automated feature test for feature
Run automated feature test to verify that it fails the first time (sanity check)
Design, develop, code, and verify the feature software to satisfy feature
Rerun automated feature test to see if code satisfies automated feature test
Refine, reduce, and simplify code to remove waste and optimize performance
STAGE 2—Behavior Driven Dev.

Smart, J. F. (2014). BDD in action: Behavior-driven development for the whole software lifecycle. Shelter Island, NY: Manning Publications.

 Term coined by Martin Fowler circa 1998
 User needs designed & developed one-at-a-time
 Changes automatically detected, built, & fully-tested
16Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.

STAGE 3—Continuous Integration
Thousands of Tests
Continuously Executed
No More Late Big
Bang Integration
Build
Integration
Server
Version
Control
Server
Build
Scripts
UsesWatches
Build
Status
Provides
Developer A
Developer B
Developer C
Commits
Changes
Commits
Changes
Commits
Changes
Builds
Database
Analysis
Testing
Reporting
Documentation
Deployment
Early, Automated, Fast,
Efficient, & Repeatable
Constant Readiness
State & CM Control
Lean, Waste Free, Low WIP,
No Deadlocked Test Queues
Rapidly & Successfully
Dev. Complex Systems
  
  
  
 ALL DEVELOPERS RUN PRIVATE BUILDS
 DEVELOPERS COMMIT CODE TO VERSION CONTROL
 INTEGRATION BUILDS OCCUR SEVERAL TIMES PER DAY
 100% OF SYSTEM TESTS MUST PASS FOR EVERY BUILD
 A SHIPPABLE PRODUCT RESULTS FROM EVERY BUILD
 FIXING BROKEN BUILDS IS OF THE HIGHEST PRIORITY
 REPORTS AUTOMATICALLY GENERATED & REVIEWED

 Agile CI consists of seven broad practices
 Automated build, database, inspection, tests, etc.
 Include reporting, documentation, deployment, etc.
17
Practice
Building
Database
Inspections
Testing
Feedback
Documentation
Deployment
Description
Frequently assembling products and services to ensure delivery readiness
Frequently generating/analyzing database schemas, queries, and forms
Frequently performing automated static analysis of product/service quality
Frequently performing automated dynamic product and service evaluation
Frequently generating automated status reports/messages for all stakeholders
Frequently performing automated technical/customer document generation
Frequently performing automated delivery of products/services to end users
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
STAGE 3—Continuous Integration


 Created by Jez Humble of ThoughtWorks in 2011
 Includes CM, build, testing, integration, release, etc.
 Goal is one-touch automation of deployment pipeline
18
Ohara, D. (2012). Continuous delivery and the world of devops. San Francisco, CA: GigaOM Pro.


CoQ
• 80% MS Tst
• 8/10 No Val
• $24B in 90s
• Rep by CD
• Not Add MLK
STAGE 4—Continuous Delivery
Source Code
Control
Build
Automation
Test
Automation
Continuous
Integration
Release
Automation
Continuous
Delivery

 Agile CD consists of seven broad practices
 Automated acceptance, load, performance, etc.
 Include packaging, pre-production test, C&A, etc.
19
Practice
Packaging
Acceptance
Load Test
Performance
Pre-Production
Certification
Deployment
Description
Frequently generating system images for pre-production testing & checkout
Frequently performing automated system & user acceptance testing
Frequently performing automated system load, stress, & capacity testing
Frequently performing automated system user & technical performance testing
Frequently performing automated pre-production tests prior to final deployment
Frequently performing automated system certification & accreditation tests
Frequently generating product images for pre-deployment testing & checkout
Mukherjee, J. (2015). Continuous delivery pipeline: Where does it choke. Charleston, SC: CreateSpace.
Swartout, P. (2014). Continuous delivery and devops: A quickstart guide. Birmingham, UK: Packt Publishing.
STAGE 4—Continuous Delivery


 Created by Patrick Debois of Jedi BVBA in 2007
 Collaboration of developers & infrastructure people
 Goal to automate the deployment to end-user devices
20
Bass, L., Weber, I., & Zhu, L. (2015). Devops: A software architect's perspective. Old Tappan, NJ: Pearson Education.
Gruver, G., & Mouser, T. (2015). Leading the transformation: Applying agile and devops at scale. Portland, OR: IT Revolution Press.
Humble, J., Molesky, J., & O'Reilly, B. (2015). Lean enterprise: How high performance organizations innovate at scale. Sebastopol, CA: O'Reilly Media.

STAGE 5—Development Operations

Collaboration

 Agile DevOps consists of seven broad practices
 Automated sys admin, CM, building, monitor, etc.
 Include virtualization, containerize, deployment, etc.
21
Practice
Sys Admin
Config. Mgt.
Host Builds
Virtualization
Containerize
Deployment
Monitor & Supp
Description
Frequently performing automated system administration tasks, i.e., scripting
Frequently performing automated infrastructure config. mgt./version control
Frequently performing automated system and server host builds and config.
Frequently performing automated system, server, & net virtualization services
Frequently performing automated software and Microservices containerization
Frequently generating final end-user system & software images for distribution
Frequently performing automated metrics collection & deployment monitoring
Duffy, M. (2015). Devops automation cookbook: Over 120 recipes coverying key automation techniques. Birmingham, UK: Packt Publishing.
Farcic, V. (2016). The devops 2.0 toolkit: Automating the continuous deployment pipelines with containerized microservices. Victoria, CA: LeanPub.

STAGE 5—Development Operations


STAGE 6—Development Ops Sec
 DevOpsSec coined by Shannon Lietz in 2014
 Rugged devops, devsecops, secdevops, devopssec
 Microservices, security engineering & operations keys
Secure Microservices
• Docker App
• Docker Bins
• Docker Files
• Docker Images
• Docker Scanning
• Docker Registry
• Docker Host
• Docker Hub
• Docker Monitoring
Secure Engineering
• Security Champions
• Security Planning
• Security Training
• Security Requirements
• Security Architecture
• Security Analysis
• Security Testing
• Security Review
• Security Response
Secure Operations
• Activity Logging
• Event Monitoring
• Configuration Mgt.
• Patch Management
• User Access Control
• Privilege Management
• Vulnerability Mgt.
• Response Mgt.
• Performance Mgt.
22


 DevOpsSec consists of seven broad practices
 Automated secure build, analysis, & deployment
 Includes containerization, engineering & operations
23
Practice
Engineering
Containers
Evaluation
Deployment
Protection
Monitoring
Responses
Description
Frequently performing “baked-in” lean and agile security engineering practices
Frequently performing automated microservices containerization practices
Frequently performing automated static and dynamic vulnerability analysis
Frequently performing automated digitally signed security deployment practices
Frequently performing automated real-time self-security protection practices
Frequently performing automated real-time security monitoring practices
Frequently performing automated trigger-based rollback response practices

STAGE 6—Development Ops Sec

 SE framework by Dean Leffingwell of Rally in 2007
 Newest version leaner, meaner, lighter, and simpler
 Experimental bottoms-up DevOps-based innovation
Leffingwell, D. (2007). Scaling software agility: Best practices for large enterprises. Boston, MA: Pearson Education.

STAGE 7—Enterprise DevOpsSec

24

 Simple example of a DevOps reference architecture
 Includes CM, continuous integration, & deployment
 Code automatically built/tested/deployed to users
25Morris, B., & Cassatt, C. (2015). Devops for the rest of us. Proceedings of the Agile DC Conference, Washington, DC, USA.
Weeks, D. E. (2014). Devops and continuous delivery reference architectures (volume 1 & 2). Fulton, MD: Sonatype.

DevOps—Basic Example

26Juengst, D. (2015). Deliver better software faster: With the cloudbees jenkins platform. San Francisco, CA: CloudBees.

DevOps—Tools Ecosystem
 Numerous tools to automate DevOps pipeline
 People can piece together toolset along with hubs
 Tools include version control, testing, & deployment

27XeniaLabs. (2016). Periodic table of devops tools. Retrieved April 11, 2016, from https://xebialabs.com/periodic-table-of-devops-tools.
DevOps—Periodic Table

28Tesauro, M. (2016). Taking appsec to 11: Appsec pipelines, devops, and making things better. Denver, CO: SnowFROC 2016.

DevOps—Security Tools Ecosystem
 Many tools emerging for DevOps application security
 Begins-ends with microservices—tiny attack surface
 Includes containers, testing, & real-time monitoring


DevOps—Automation Metrics
30Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.

Defects
DecreaseIntegrations
Increase
Increase
Coverage
Increase
Automation

31
 DevOps metrics gaining in widespread popularity
 Hybrid of development & IT operations measures
 Includes code, deployment & e-business analytics
Velasquez, N. F. (2014). State of devops report. Portland, OR: Puppet Labs, Inc.

DevOps—Deployment Metrics

Activity Def CoQ DevOps Economics Hours ROI
Development Operations 100 0.001 100 Defects x 70% Efficiency x 0.001 Hours 0.070 72,900%
Continuous Delivery 30 0.01 30 Defects x 70% Efficiency x 0.01 Hours 0.210 24,300%
Continuous Integration 9 0.1 9 Defects x 70% Efficiency x 0.1 Hours 0.630 8,100%
Software Inspections 3 1 2.7 Defects x 70% Efficiency x 1 Hours 1.890 2,700%
"Traditional" Testing 0.81 10 0.81 Defects x 70% Efficiency x 10 Hours 5.670 900%
Manual Debugging 0.243 100 0.243 Defects x 70% Efficiency x 100 Hours 17.010 300%
Operations & Maintenance 0.073 1,000 0.0729 Defects x 70% Efficiency x 1,000 Hours 51.030 n/a
32
 Agile testing is orders-of-magnitude more efficient
 Based on millions of automated tests run in seconds
 One-touch auto-delivery to billions of global end-users
Rico, D. F. (2016). Devops cost of quality (CoQ): Phase-based defect removal model. Retrieved May 10, 2016, from http://davidfrico.com

DevOps—Cost of Quality

Under 4
Minutes
4,500 x Faster
than Code
Inspections

 Traditional testing finds a defect in 10 hours
 Manual code inspections find a defect in 1 hour
 DevOps finds a defect in 4 min/test (4,500x faster)
33
Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.

DevOps—Economics

34
 Hewlett-Packard is a major user of CI, CD, & DevOps
 400 engineers developed 10 million LOC in 4 years
 Major gains in testing, deployment, & innovation
Gruver, G., Young, M. & Fulghum, P. (2013). A practical approach to large-scale agile development. Upper Saddle River, NJ: Pearson Education.

TYPE METRIC MANUAL DEVOPS MAJOR GAINS
CYCLE TIME
IMPROVEMENTS
Build Time 40 Hours 3 Hours 13 x
No. Builds 1-2 per Day 10-15 per Day 8 x
Feedback 1 per Day 100 per Day 100 x
Regression Testing 240 Hours 24 Hours 10 x
DEVELOPMENT
COST EFFORT
DISTRIBUTION
Integration 10% 2% 5 x
Planning 20% 5% 4 x
Porting 25% 15% 2 x
Support 25% 5% 5 x
Testing 15% 5% 3 x
Innovation 5% 40% 8 x
DevOps—HP Case Study



 Assembla went from 2 to 45 releases every month
 15K Google developers run 120 million tests per day
 30K+ Amazon developers deliver 136K releases a day
35Singleton, A. (2014). Unblock: A guide to the new continuous agile. Needham, MA: Assembla, Inc.
62 x Faster
U.S. DoD
IT Project
3,645 x Faster
U.S. DoD
IT Project

DevOps—Dot Com Case Study



36Ashman, D. (2014). Blackboard: Keep your head in the clouds. Proceedings of the 2014 Enterprise DevOps Summit, San Francisco, California, USA.
 Productivity STOPS due to excessive integration
 Implements DevOps & Microservices around 2010
 Waste elimination, productivity & innovation skyrocket
DevOps—Blackboard Case Study

DEVOPS &
MICROSERVICES
IMPLEMENTED

37Denayer, L. (2017). U.S. DHS citizenship and immigration services: USCIS agile development. Washington, DC. iSDLC Seminar.
 1st gen replete with large portfolios & governance
 2nd-3rd gen yield minor incremental improvements
 4th-5th gen enables big order-of-magnitude impacts
DevOps—U.S. DHS Case Study






    
Automated GovernanceManual Governance
 

 Tesla vehicle models are all electric automobiles
 Tesla autos have 100-200 million lines of code
 Tesla performs up to 130 deployments per day
38

DevOps—Tesla Software Releases
Choksi, N. (2016). How software lifecycle integration and devops are transforming car development. Goto Conference, Copenhagen, Denmark.
Vost, S., & Wagner, S. (2016). Towards continuous integration and continuous delivery in the automotive industry. Ithaca, NY: Cornell University.

39
 Detailed DevOps economics starting to emerge
 ROI ranges from $17M to $195M with minor costs
 Benefits from cost savings, revenue, and availability
Forsgren, N., Humble, J., & Kim, G. (2017). Forecasting the value of devops transformations: Measuring roi of devops. Portland, OR: DevOps Research.
Rico, D. F. (2017). Devops return on investment (ROI) calculator. Retrieved August 29, 2017, from http://davidfrico.com/devops-roi.xls

DevOps—Return on Investment

40
 DevOps adoption growing fast in-spite of slow start
 35% using, 27% thinking about it, & 38% are in-dark
 DevOps a global industry-wide extinction-level event
40
Brown, A. (2016). Devops and the need for speed, quality, and security: Do organizations really have to pick two out of three. Portland, OR: Puppet Labs.

DevOps—Adoption Statistics


41
 Having a DevOps rollout strategy is a key to success
 Phased, incremental, and situational implementation
 Includes build, testing, & IT operations, & practices
St-Cyr, J. (2015). Evolving devops: Advance alm and devops practices with cont. imp. Agile Dev, Better Software, & DevOps East Conference, Orlando, Florida, USA.

DevOps—Roadmap

42
 Industry leading DevOps assessments are emerging
 DORA Technology DevOps Assessment is popular
 Includes speed, deployments, reliability & morale
Kim, G., Forsgren, N., & Humble, J. (2017). The DORA technology performance assessment. Portland, OR: DevOps Research.

DevOps—Assessments

43Kim, G., Debois, P., Willis, J., & Humble, J. The devops handbook: How to create world-class agility, reliability, and security
in technology organizations. Portland, OR: IT Revolution Press.

 


 Everything begins with lean & agile principles
 Next step is smaller portfolio & simpler designs
 Final step is modular interfaces & E2E automation
DevOps—5 Keys to Success





 DevOps DOES NOT mean deliver it now and fix it later
 Lightweight, yet disciplined approach to development
 Reduced cost, risk, & waste while improving quality
44
Rico, D. F. (2012). What’s really happening in agile methods: Its principles revisited? Retrieved June 6, 2012, from http://davidfrico.com/agile-principles.pdf
Rico, D. F. (2012). The promises and pitfalls of agile methods. Retrieved February 6, 2013 from, http://davidfrico.com/agile-pros-cons.pdf
Rico, D. F. (2012). How do lean & agile intersect? Retrieved February 6, 2013, from http://davidfrico.com/agile-concept-model-3.pdf
What How Result
Flexibility Use lightweight, yet disciplined processes and artifacts Low work-in-process
Customer Involve customers early and often throughout development Early feedback
Prioritize Identify highest-priority, value-adding business needs Focus resources
Descope Descope complex programs by an order of magnitude Simplify problem
Decompose Divide the remaining scope into smaller batches Manageable pieces
Iterate Implement pieces one at a time over long periods of time Diffuse risk
Leanness Architect and design the system one iteration at a time JIT waste-free design
Swarm Implement each component in small cross-functional teams Knowledge transfer
Collaborate Use frequent informal communications as often as possible Efficient data transfer
Test Early Incrementally test each component as it is developed Early verification
Test Often Perform system-level regression testing every few minutes Early validation
Adapt Frequently identify optimal process and product solutions Improve performance

















DevOps—Summary

45
DevOps ensures enterprise success by delivering large
volumes of valuable, reliable, & secure IT products &
services to billions of users in fractions of a second ...
DevOps—Bottom Line?

Dave’s PROFESSIONAL CAPABILITIES
46
Software
Quality
Mgt.
Technical
Project
Mgt.
Software
Development
Methods
Leadership &
Org. Change
Cost Estimates
& Scheduling
Acquisition &
Contracting
Portfolio &
Program Mgt.
Strategy &
Roadmapping
Lean, Kanban,
& Six Sigma
Modeling &
Simulations
Big Data,
Cloud, NoSQL
Workflow
Automation
Metrics,
Models, & SPC
BPR, IDEF0,
& DoDAF
DoD 5000,
TRA, & SRA
PSP, TSP, &
Code Reviews
CMMI &
ISO 9001
Innovation
Management
Statistics, CFA,
EFA, & SEM
Evolutionary
Design
Systems
Engineering
Valuation — Cost-Benefit Analysis, B/CR, ROI, NPV, BEP, Real Options, etc.
Lean-Agile — Scrum, SAFe, Continuous Integration & Delivery, DevOpsSec, etc.
STRENGTHS – Lean & Agile Thinking • 360 Leadership Assessments • Executive & Agile Coaching • Enterprise Business Agility • Agile Acquisition
Contracts • Scaled Agile Framework (SAFe) • DevOps + Security (DevOpsSec) • Cloud Computing/Amazon Web Svcs. • Portfolio, Program, & Project Mgt. •
5x5x5 Innovation & Marketing Sprints • Strategic Planning & Technology Roadmaps • Program Increment & Big Room Planning • Emergent & Evolutionary
Microservices • Exploratory MVP, MVA, & MMF Experiments • Lean Startup Product-Focused Value Streams • Performance Metrics, Measures & Dashboards
● Data mining. Metrics, benchmarks, & performance.
● Simplification. Refactoring, refinement, & streamlining.
● Assessments. Audits, reviews, appraisals, & risk analysis.
● Coaching. Diagnosing, debugging, & restarting stalled projects.
● Business cases. Cost, benefit, & return-on-investment (ROI) analysis.
● Communications. Executive summaries, white papers, & lightning talks.
● Strategy & tactics. Program, project, task, & activity scoping, charters, & plans.
PMP, CSEP,
EBAS, BAF, FCP,
FCT, ACP, CSM,
DEVOPS, &
SAFE
36+ YEARS
IN IT
INDUSTRY

Kennedy, M. P., & Umphress, D. A. (2011). An agile systems engineering process: The missing link. Crosstalk, 24(3), 16-20.
 No. of software-intensive systems is growing
 80% of US DoD functions performed in software
 Major driver of cost, schedule, & tech. performance
48

Software—U.S. DoD Systems

Blackburn, M. R. (2014). Transforming systems engineering through a holistic approach to model centric engineering. Washington, DC: Stevens Institute of Technology.
 Software in U.S. DoD avionics growing exponentially
 10x growth from F-16 to F-22 (& another 10x to F-35)
 Productivity must grow by 10x for next gen systems
49

Software—U.S. DoD Fighter Jets

Rico, D. F. (2017). U.S. dod vs. amazon: 18 architectural principles to build fighter jets like amazon web services using devops. Retrieved November 7, 2017, from http://davidfrico.com
 F-35 software family of loosely-coupled ecosystems
 Each F-35 subsystem is ecosystem of microservices
 F-35 microservices iteratively developed with DevOps
50

Software—F-35 Avionics Software

51
Sheldon, F. T. et al. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20
Johnson, J. (2002). ROI: It's your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy.
 Requirements defects are #1 reason projects fail
 Traditional projects specify too many requirements
 More than 65% of requirements are never used at all
Other 7%
Requirements
47%
Design
28%
Implementation
18%
Defects
Always 7%
Often 13%
Sometimes
16%
Rarely
19%
Never
45%
Waste

Requirements—Defects & Waste

52
 Big projects result in poor quality and scope changes
 Productivity declines with long queues/wait times
 Large projects are unsuccessful or canceled
Jones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill.
Size vs. Quality
DEFECTS
0.00
3.20
6.40
9.60
12.80
16.00
0 2 6 25 100 400
SIZE
Size vs. Productivity
PRODUCTIVITY
0.00
1.00
2.00
3.00
4.00
5.00
0 2 6 25 100 400
SIZE
Size vs. Change
CHANGE
0%
8%
16%
24%
32%
40%
0 2 6 25 100 400
SIZE
Size vs. Success
SUCCESS
0%
12%
24%
36%
48%
60%
0 2 6 25 100 400
SIZE

Performance—Traditional Projects

53
Standish Group. (2015). Chaos summary 2015. Boston, MA: Author.
Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch.
 Challenged and failed projects hover at 67%
 Big projects fail more often, which is 5% to 10%
 Of $1.7T spent on IT projects, over $858B were lost
$0.0
$0.4
$0.7
$1.1
$1.4
$1.8
2002 2003 2004 2005 2006 2007 2008 2009 2010
Trillions(USDollars)
Expenditures Failed Investments

0% 20% 40% 60% 80% 100%
28%
34%
29%
35%
32%
33%
27%
28%
29%
49%
51%
53%
46%
44%
41%
56%
55%
52%
23%
15%
18%
19%
24%
26%
17%
17%
19%
2000
2002
2004
2006
2008
2010
2012
2014
2015
Year
Successful Challenged Failed
Failures—Global Traditional Projects

54
Capability #1
● Feature 1
● Feature 2
● Feature 3
● Feature 4
● Feature 5
● Feature 6
● Feature 7
Capability #2
● Feature 8
● Feature 9
● Feature 10
● Feature 11
● Feature 12
● Feature 13
● Feature 14
Capability #3
● Feature 15
● Feature 16
● Feature 17
● Feature 18
● Feature 19
● Feature 20
● Feature 21
Capability #4
● Feature 22
● Feature 23
● Feature 24
● Feature 25
● Feature 26
● Feature 27
● Feature 28
Capability #5
● Feature 29
● Feature 30
● Feature 31
● Feature 32
● Feature 33
● Feature 34
● Feature 35
Capability #6
● Feature 36
● Feature 37
● Feature 38
● Feature 39
● Feature 40
● Feature 41
● Feature 42
Capability #7
● Feature 43
● Feature 44
● Feature 45
● Feature 46
● Feature 47
● Feature 48
● Feature 49
1
2 3
4
5 6
7
8 9
10
11 12
13
14 15
16
17 18
19
20 21
Evolving “Unified/Integrated” Enterprise Data Model
“Disparate” LEGACY SYSTEM DATABASES (AND DATA MODELS)
ETL
A A
B C
D E F
G H I J K
A
B C
D E F
A
B C
D E
A
B C
D
A
B C
A
B
“Legacy” MS SQL Server Stovepipes “Inter-Departmental” Linux Blade/Oracle/Java/WebSphere Server
“Leased” DWA/HPC/Cloud Services
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6 Sprint 7
ETL ETL ETL ETL ETL ETL
Bente, S., Bombosch, U., & Langade, S. (2012). Collaborative enterprise architecture: Enriching EA with lean, agile, and enterprise 2.0 practices. Waltham, MA: Elsevier.
(for example, assume 25 user stories per feature, 175 user stories per capability, and 1,225 user stories total)
 Organize needs into capabilities, features, and stories
 Prioritize features, group releases, and initiate sprints
 Develop minimum set of features with highest value
Release
Release
Release
Release

MMF
- or -
MVP

Agile—Storymap, Roadmap, Arch.

Ladas, C. (2008). Scrumban: Essays on kanban systems for lean software development. Seattle, WA: Modus Cooperandi.
Reddy, A. (2016). The scrumban revolution: Getting the most out of agile, scrum, and lean-kanban. New York, NY: Addison-Wesley.

 Created by Corey Ladas of Modus Cooperandi (2008)
 Hybrid of Agile (Scrum) and Lean (Kanban) methods
 Scrum with one-piece-workflow vs sprints (batches)
55
ScrumBan—One-Piece Workflow

 Fewer integrations leave in higher bug counts
 Frequent, early integrations eliminate most defects
 Goal is to have as many early integrations as possible
56
Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392.
 
Number of
Integrations
Less Defects
•More Integrations
•Early IntegrationsMore Defects
•Few Integrations
•Late Integrations

Continuous Integration—Statistics

57Huett, B. (2017). The TMD toolbox. Retrieved May 5, 2017, from http://www.themoderndeveloper.com.
DevOps—TMD Toolbox

 Test-ing (tĕst′ĭng) An early, iterative, and automated
V&V of customer requirements; Incremental testing
 A testing approach embracing principles & values of lean
thinking, product development flow, & agile methods
 Early, collaborative, and automated form of incremental
development, integration, system, & operational testing
 Testing method that supports collaboration, teamwork,
iterative development, & responding to change
 Mult-tiered automated framework for TDD, Continuous
Integration, BDD, Continuous Delivery, & DevOps
 Maximizes BUSINESS VALUE of organizations, portfolios,
& projects by enabling buyers-suppliers to scale globally
58

Crispin, L., & Gregory, J. (2015). More agile testing: Learning journeys for the whole team. Boston, MA: Addison-Wesley.
Agile Testing—What is it?

Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.
Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.
 Most agile testing tools are “free” open source
 Build server costs no more than a commodity PC
 10x more efficient/effective than traditional testing
59

Agile Testing—Costs & Benefits

 Traditional testing is a late, manual process
 Agile testing is an early and automated process
 Goal to deliver early & often and V&V components
60
Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txt
Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.

AGILE TESTING
- Early Incremental Testing -
TRADITIONAL TESTING
- Late Big Bang Integration Testing -
Code is Frequently Checked In
Code Automatically Retrieved
Code Automatically Compiled
Tests Automatically Executed
Instant Feedback & Test Reports
Code Checked In Late in Project
Code Manually Submitted to Test
Code Manually Compiled & Built
Tests Manually Executed Late
Late Project Feedback & Reports
Code Automatically DeployedLate Defects Freeze Projects
Agile Testing—vs. Traditional
Test Criteria Accompany Stories
Automated Tests Written First
Units Coded-Tested One at Time
Test Criteria Written After Fact
Manual Tests Written Much Later
Units Coded Late All at One Time



 Agile teams don’t often use TDD, CI, CD & DevOps
 Implement independent test teams after Sprints done
 Sprint Waterfalling, Scrummerfalling, & Wagile result
61
Heusser, M. (2015). 12 years of agile testing: What do we know now. Proceedings of the Agile Gathering, Grand Rapids, Michigan, USA.



Incorrect
• Phased Testing
• Separate Teams
• Delayed Testing
Correct
• Integrated Testing
• Integrated Teams
• Continuous Testing
Agile Testing—Anti-Patterns


 Agile testing slows down with very large systems
 Slow testing slows integration and increases bugs
 Agile testing can speed back up with more attention
62
Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.
MICRO ADJUSTMENTS
- Focused Impact Tuning-
MACRO ADJUSTMENTS
- Wide Impact Tuning-
Reduce or Remove Test Timeouts
Select Different Tests
Refactor Code & Components
Tune Network & Software
Tune Database & Middleware
Remove Process Randomness
Use Faster Code & Test Tools
Incremental vs. Big Bang Tests
Parallelize Build & Install
Tune & Optimize Build Process
Agile Testing—Scaling Practices

Add More CPUs & Memory
Parallelize System Builds
Replace 3rd Party Test Libraries
In-Memory Compilation
Parallelize Test Runs
Pre-Install Test Libraries



 Industry very slow in adopting agile testing model
 Cost, difficulty, and territorialism are common issues
 Developers must take initiative for disciplined testing
63
Technical BarriersOrganizational Barriers
Developers don’t want to test
· Infrequently committing code
· Committing broken code
· Failing to immediately fix builds
· Not writing automated tests
· Not ensuring 100% of tests pass
· Not running private builds
· Resorting to traditional testing
Resistance to change
· Fear of investment costs
· Fear of learning new skills
· Test group territorialism
· Organizational policy conflicts
· Overhead of maintaining CI
· Complexity and scaling
· Not developing a quality culture
··
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.






Agile Testing—Common Barriers


 Eliminates big-bang integration in the 11th hour
 Creates a repeatable and reliable testing process
 Evaluates system-wide changes throughout project
64Maeda, M. K. (2009). Agile testing: Early, often, and smart. Arlington, MA: Cutter Consortium.
What’s the Bottom Line?
“Agile Testing Done Early & Often”
Agile TestingTraditional Testing
· High project visibility
· Greater confidence and morale
· Incremental business value
· 24x7 deployability to users
· Highly quality and reliability
· Lack of deployability
· Late big-bang integration
· Testing is a bottleneck
· Poor customer satisfaction
· Outright project failure


Late defect discovery
· Low quality software
· Poor project visibility
· Dramatically reduces risks
· Automates manual processes
· Instant verification & validation
·

 Agile test use is low in spite of its age, i.e., 15 years
 Many do not understand its utter simplicity and power
 Failure to use agile testing undermines project success
65
Kim, D. (2013). The state of scrum: Benchmarks and guidelines. Indianapolis, IN: Scrum Alliance.
Agile Practices
Retrospectives
Refactoring
Done Definition
Test Tools
Test Driven Dev.
CM Tools
Simplicity
Pair Programming
Technical Debt
Agile Testing 13%
Continuous Integrations
Weekly
Daily
2-3 Times
Per Day
Never
2-3
Times
Per
Iteration


Agile Testing—Adoption Statistics


 Microsoft created software security life cycle in 2002
 Waterfall approach tailored for Scrum sprints in 2009
 Uses security req, threat modeling & security testing
66
Microsoft. (2011). Security development lifecycle: SDL Process Guidance (Version 5.1). Redmond, WA: Author.
Microsoft. (2010). Security development lifecycle: Simplified implementation of the microsoft SDL. Redmond, WA: Author.
Microsoft. (2009). Security development lifecycle: Security development lifecycle for agile development (Version 1.0). Redmond, WA: Author.
Bidstrup, E., & Kowalczyk, E. C. (2005). Security development lifecycle. Changing the software development process to build in security from the start. Security Summit West.

SEE DETAILED - SECURITY LIFE CYCLE STEPS
http://davidfrico.com/agile-security-lifecycle.txt
Agile Testing—Security Life Cycle

 1st-generation systems used hardwired logic
 2nd-generation systems used PROMS & FPGAs
 3rd-generation systems use APP. SW & COTS HW
67
Pries, K. H., & Quigley, J. M. (2010). Scrum project management. Boca Raton, FL: CRC Press.
Pries, K. H., & Quigley, J. M. (2009). Project management of complex and embedded systems. Boca Raton, FL: Auerbach Publications.
Thomke, S. (2003). Experimentation matters: Unlocking the potential of new technologies for innovation. Boston, MA: Harvard Business School Press.

● Short Lead
● Least Cost
● Lowest Risk
● 90% Software
● COTS Hardware
● Early, Iterative Dev.
● Continuous V&V
● Moderate Lead
● Moderate Cost
● Moderate Risk
● 50% Hardware
● COTS Components
● Midpoint Testing
● “Some” Early V&V
● Long Lead
● Highest Cost
● Highest Risk
● 90% Hardware
● Custom Hardware
● Linear, Staged Dev.
● Late Big-Bang I&T
AGILE
“Software Model”
- MOST FLEXIBLE -
NEO-TRADITIONAL
“FPGA Model”
- MALLEABLE -
TRADITIONAL
“Hardwired Model”
- LEAST FLEXIBLE -
GOAL – SHIFT FROM LATE HARDWARE TO EARLIER SOFTWARE SOLUTION
RISK
Embedded
Systems
More HW
Than SW
STOP
Competing
With HW
START
Competing
With SW






Iterations,Integrations,&Validations
Agility—Hardware/Embedded Sys.

ROI & Business Value of CI, CD, DevOps, DevSecOps, & Microservices

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a ROI & Business Value of CI, CD, DevOps, DevSecOps, & Microservices

Similar a ROI & Business Value of CI, CD, DevOps, DevSecOps, & Microservices (20)

Más de David Rico

Más de David Rico (20)

Último

Último (20)

ROI & Business Value of CI, CD, DevOps, DevSecOps, & Microservices