This document summarizes MITRE's adoption of DevOps practices. It describes their old development and deployment process, how they defined success criteria with DevOps, and their implementation journey from 2011 to 2012. This included building automated builds, deployments, implementing continuous integration and delivery practices, and scaling up to support more applications. As a result, their average deployment cycle time decreased by 40% and throughput increased by 38% while freeing up staff. They continue working to change mindsets, incorporate more applications, and achieve full continuous delivery.

1. MITRE Bill & Aimee’s
Excellent DevOps Journey
How a Not-for-profit Corporation
Adopted DevOps
Aimee Bechtle
abechtle@mitre.org
Bill Donaldson
wbd@mitre.org
Approved for Public Release; Distribution Unlimited. Case Number: 14-3400 ©2014 The MITRE Corporation. ALL RIGHTS RESERVED.

2. What is MITRE? MITRE
MITRE is
a not-for-profit
organization that operates
Federally Funded Research
and Development Centers

3. Who Are We? MITRE
I’m Bill
I’m
Aimee
Dev
Ops
Road
Trip!

4. MITRE
The Starting Line
Production
1. Code, Build,
Unit Test
1...n
2. Commit
3. Run Build,
Development Test
1...n
4. Writes/Maintains 3a. Export Code
CM Form, Creates Change
Ticket
6. Rejects
CM Form
17. Install
Failed
7a. Export Code
14. Email Instructions
Integration
15. Execute
Task
Other
9. Execute
Task
18. Export Code
19. Install Failed
Rollback & Restore
Development
10. Notify
Complete
DBA
5. Receives & Reviews
CM Form
Instructions
7. Executes Instructions
(logs on, Exports code,
Runs Build, runs scripts
Copies files, cleans up)
8. Email
Instructions
16. Notify Complete
Deploy
Mgr
13. Notify
Complete
Sys
Admin
11. Email
Instructions
12. Execute
Task
DEV
20. Use
Application
User
Sandbox
SCM
OPERATIONS
(Deploys)
Paperwork
People
Processing
• JAVA, .Net
• ANT, Maven
• Subversion (SVN)
• Oracle, SqlServer,
MySQL
• RHEL, Windows 2008
• Avg Cycle Time = 15
days (30 mins – 486
days)
• # Deploys/yr = 2579

5. MITRE
Ops
In The Beginning – Aug 2009
The service
we provide
is great!
They are
slowing
me down!
Dev

6. Confronting the Chasm – May 2010 MITRE
24 hour SLA
is 23.5 hours
too long!
Huh?
Dev Ops

7. Bridging the Chasm – Aug 2010 MITRE
Let me
show you
how
this could
work.
OK!
Daev Ops

8. MITRE
Defining Success – Fall 2010
Let’s define
Acceptance
Criteria !
DevOps

9. Acceptance Criteria MITRE
I can live
with that.
 Single command to build app given an SVN revision #
 Single command to deploy to environment given an SVN revision #
 Build once. No development tools utilized on target environment.
 Deploy to test environment is self-service and can be done by any user without handoffs (no
SA, DBA) or additional passwords
 Deploy to PROD is done by release team without handoffs (no SA, DBA)
 Prior to deploying validates compliance with SONAR quality reports and code coverage
 Can determine what SVN revision is deployed in each environment, by who and when
 In PROD, on error or abort application is restored to prior state
 No additional instruction documentation is needed.
 In phase 1, utilize existing request system for approvals
 System validation is manual and outside of scope

10. MITRE
The Launch – May 2011
Build and Release
Engineer
Sys
Admin
DevOps
Java, .NET Developers DBA
DevOps
Everyone
on Board!

11. Implementation – (5/11 to 6/12) MITRE
Are we there
yet? Are we
there yet?
DevOps
DevOps

12. Course Correction – April 2012 MITRE
Got Training Brought in Expertise
Freed Up Resources A New Plan!

13. MITRE
Our DevOps Solution– June 2012
1 Developer develops commits code to repository
2 Commits automatically trigger builds
3 Depending on environment developer or
Release Engineer deploys build
Source Code Repository
2
Build Workflow
Checkout
Compile
Run Unit Tests
Run Code Analysis
Store Artifacts
Deploy Workflow
Cleanup Environment
Backup Environment
Deploy Binaries
Execute Rollback & Restore
Artifact Repository
Commit
Code &
Build
Files
Application &
Build Files
Release
Artifacts
Release
Artifacts
Run
Build
(revision #)
Run
Deploy
(build #)
Repeated for each target
Environment using the
same Build
Build/CI Server
1
3
* Developers can deploy to Non-Production Environments
CI
DEV
Test
Prod
DEV OPS

14. Scaling Up – June 2012 MITRE
Its working
Woo Hoooo!

15. Visible Tracking MITRE

16. Recipe For Onboarding MITRE

17. MITRE Guides and Samples

18. MITRE
Water cooler Talk
Snackfest
Formal briefings to
CIO and Senior Mgt
Visible Metrics
Marketing

19. MITRE The Results
• 68 applications in system
• Multiple Platforms
o ANT, Maven, Gradle
o SVN, GIT
o JAVA, .Net, NodeJS
o Oracle, MySQL, SQL Server,
MongoDB
• Avg Cycle Time = 9 days (-40%)
(range 2 mins – 483 days)
• ~75% of deploys are automated
o # Deploys/year = 3555
o Auto = 2665
o Manual = 890 (-65%)
o 38% Throughput Increase
• Freed up 2.5 deployment staff

20. MITRE
The Future: Continuous Delivery
Build & Release
Engineer
Sys
Admin
DevaOps
Java, .NET Developers DBA
DevaOps
Ops
Automation
Engineer
Test
Automation
Engineer
Let’s do it
again
Pleeease!

21. Bill & Aimee’s Excellent MITRE
DevOps Journey
Aug ’09:
Old
Process
July ’10:
Confront
May ’11:
Success
Defined,
Launch
Jun ’12:
Success!
• Anyone can do it
• Know where you want to go and why
• Patience - it takes time for change
• Education, education, education
• Use Agile (minimal product, iterations, product demonstrations)
Apr ’12:
Got Help
Today:
Results

22. MITRE
What We Need Help With
• Changing skillsets and mindsets to:
– Support infrastructure and test automation
– Break up silos and organize into cross-functional delivery teams
• Incorporating ERP and large COTS (or non-typical, non-web) applications into a
DevOps model

23. Thanks! MITRE
Aimee Bechtle
abechtle@mitre.org
Bill Donaldson
wbd@mitre.org
@wbdonaldson