1. Using Rundeck and Chef
to build DevOps
Toolchains
Anthony Shortland - anthony@dtosolutions.com
Moses Lei - mlei@dtosolutions.com

2. DTO Solutions

3. Process Improvement
Value Stream
Service Delivery Process
Program
Current state value stream map
Committee
Eric Switzer
Full Full
L/T Lead time
Projects / Projects /
Expedites Expedites P/T Process time
H/C Head count
EP
Business Technical S/R Scrap rate
Analysis Analysis
Luke Hanson Maya Highsmith
Estimates Estimates D Defects
and and EP Extra processes
requirements requirements
M Motion
PD Partially done
Project
Management TS Task switching
W Waiting
Eric Switzer
Business
Change
Design
request
Daily Scrum Change
Program Management
meeting Request QA Control Forms
Mx3 build Development
Dx3 Develop Technical
Specification Daily Scrum Deployment
web assets meeting QA
Meeting Business Systems
Jira issues Bugzilla
issues Support Release Engineering
L/T 8 weeks
Web Assets, Deployment
P/T 6 weeks Operations
flows and Jira issues Document Stan Walters
H/C 15 mockups
S/R 8%
Bob McNulty Wx2 PDx3 Dx2 RT tickets Change
TSx5 Dx2
Development Development W Release Control
Development Features and
fixes Build and deploy Integration testing Build and deploy
Set of
EP service L/T 10 weeks L/T 1 days L/T 2 day L/T 11 hours
Cut project branch branches P/T 6 weeks P/T 2 hours P/T 2 day P/T 8 hours
H/C 4 H/C 2 H/C 11 H/C 2 W W
D PD Mx7 Mx3
S/R 38% S/R 20% S/R 96% S/R 5% D PDx4 W D
L/T 20 minutes D
Joe Harbaugh Joe Harbaugh Joe Harbaugh Fernando Gomez W Deploy to EPx2
P/T 8 minutes Acceptance testing Feature testing Tested Merge project
Deploy to Staging Staging testing
H/C 2 packages Production branch
S/R 0%
L/T 5 days L/T 5 weeks L/T 7 days L/T 4 days L/T 10 hours L/T 1 hour
Chris Holmes
Development P/T 8 hours P/T 3.5 weeks P/T 110 minutes P/T 3.5 days P/T 7 hours P/T 30 minutes
Environments H/C 2 H/C 5 H/C 4 H/C 5 H/C 5 H/C 1
Documented S/R 10% S/R 20% S/R 70% S/R 66% S/R 50% S/R 50%
test plans Raj Lee Raj Lee Frances Middleton Frances Middleton Allen Cannata Fernando Gomez
and cases
EP PD Mx3
QA Staging Production
Test Requirements Build test plan Build test cases Environments Environment Environment
L/T 21 days L/T 8 days L/T 2 weeks
P/T 3 days P/T 4 days P/T 8 days
H/C 2 H/C 2 H/C 2
S/R 50% S/R 30% S/R 20%
Deploy to Performance
Luke Hanson Raj Lee Raj Lee
Performance testing
L/T 6 days L/T 1 week
P/T 90 minutes P/T 3 days
H/C 3 H/C 1
S/R 66% S/R 66%
Frances Middleton Allen Cannata
Performance
Environment
Process Time
5 weeks 5 weeks 5 hours 3 day 8 hours 6 hours 2 weeks 110 minutes = 59 days
4.5 days 9 hours
3 weeks 5 weeks 2.5 days 2 hours 5 days 2.5 weeks 6.75 days 2.5 days 2 hour Lead Time =
136 days
Product Milestone
RetailTrader - v133 "Add logic updates " 4 Days Remain 10d total
Metrics: 1 week, 1 week historical
Value-Creating Time (VCT)
65% +5%
Environment: Production
Stability
Delivery Cycle Time Delivery Support Team Utilization
7 Days 3h, -5h
5 10%
Support
85%, -5%
-5%
Scott, Erica, Reigner,…
Provision Lead Time
Deployments Work Tasks
2 Hours 15m
8 2 versions
4 environments
4/16 Open -30m
+2 environments +1 version 15 Closed Features
Provision Scrap Rate
Delivery Scrap Rate Build Success
4/5 Lead Time 5% 1/20
+1
20% 2/10
-3 80% -5% 34m avg
OK retailtr-v133.war

4. Process Automation
• End-to-end automation
• Supports business goals
• Optimize the system, not the parts

5. Introduction
Take a toolchain approach
E-commerce and software-as-a-service business models have matured quickly, but the
quality of the web operations that support these businesses has lagged behind.
Outages are all too common. High variability and defect rates are bemoaned but have
become an accepted reality. Key engineers spend all day (and sometimes all night)
mired in deployment issues and bottlenecks. And topping it all off, what tooling that
does exist are usually a custom one-offs that are brittle and expensive to maintain.
Today’s business of operating software over the Web as a revenue producing service is a
” Today’s business of dramatic departure from the days when software was primarily produced for delivery on
operating software over the physical mediums and IT Operations was considered a back-of-the-house support
Web as a revenue producing function. Shouldn't we be completely rethinking our tooling and operational capabilities
service is a dramatic to match these new innovations?
departure from the days
when software was primarily In short, we need to get out of Web Operations 1.0 -- mired in legacy tools, outdated
produced for delivery on approaches, and low expectations -- and into Web Operations 2.0 where tools and
physical mediums...” procedures are built from the ground up for highly efficient, reliable, and agile
Industrial Control Automation operations.
Runbook Automation
Control
Eventing, Alarm Mgmt
Charting, History, SPC
Measurement Instrumentation
System
+
There are multiple factors that go into achieving excellence in Web Operations, but the
linchpin that holds it all together is a fully automated provisioning system.
In this paper we will be:
1. Defining what we mean by "fully automated provisioning"
2. Explaining why virtualization and cloud computing efforts fail without fully
automated provisioning capabilities
3. Proposing a reference open source tool chain for fully automated provisioning
4. Describing a live implementation where a leading online retailer is actively rolling
out a fully automated provisioning system using all open source tools
Unix Tool Chain 2 Web Ops 2.0: Achieving Fully Automated Provisioning

6. Service Delivery Platform Elements

7. Design points
1. Everything is code
2. Everything is packaged
3. Separate code and configuration
4. Separate environment-dependent attributes
5. Balance distributed vs local orchestration
6. Resolve directed vs convergent orchestration
7. Integrate application and infrastructure provisioning
8. Design for flow not the organization

8. Example Service Delivery Platform

9. Integration Plugins
• Jenkins/GitHub - https://wiki.jenkins-ci.org/display/
JENKINS/Github+Plugin
• Jenkins/Rundeck - https://wiki.jenkins-ci.org/
display/JENKINS/RunDeck+Plugin - Vincent Behar
• Nexus/Rundeck - https://github.com/vbehar/
nexus-rundeck-plugin - Vincent Behar
• Nexus/Yum - http://code.google.com/p/nexus-
yum-plugin/
• Chef/Rundeck - https://github.com/opscode/chef-
rundeck - Adam Jacob

10. Example Service Delivery Platform
Layout
Deployment Application
Build Console Console Server Database
Build Artifact
Source Code Repository Server Client
Build Server Deploy Server Development Server

11. Develop

12. Build - Projects by artifact

13. Build - Trace by build number

14. Build - Template RPM SPEC
[ec2-user@centos-ami bank]$ cat SPECS/dukesbank-ear.spec.template
Summary: Duke's Bank ear file
Name: dukesbank-ear
Version: ${version}
Release: ${release}
License: LGPL
Group: Applications/System
Prefix: /opt/jboss-4.0.3SP1
BuildArch: noarch
Requires: jboss = 4.0.3SP1
.
.
.

15. Build - Register packages
.
.
.
<target name="rpmdeploy" depends="resolve,rpmbuild">
<ivy:makepom ivyfile="${basedir}/ivy.xml" pomfile="RPMS/noarch/dukesbank-ear-$
{version}-${release}.noarch.pom"/>
<ivy:publish resolver="nexus-deploy" pubrevision="${version}-${release}"
overwrite="true" publishivy="true">
<artifacts pattern="RPMS/noarch/[artifact]-[revision].noarch(-[classifier]).
[ext]"/>
<!--artifact name="dukesbank-ear" type="rpm" url="${build.dir}/rpm/RPMS/
noarch/dukesbank-1.0-2.noarch.rpm" /-->
</ivy:publish>
</target>
.
.
.
</project>

16. Repository - Build view

17. Repository - Deploy view

18. Deploy - Package version list
[root@centos-ami yum.repos.d]# cat /etc/yum.repos.d/dukesbank.repo
[dukesbank]
name=DukesBank demo repo
baseurl=http://ec2-75-101-135-205.compute-1.amazonaws.com:8081/nexus/content/
repositories/releases
enabled=1
gpgcheck=0
metadata_expire=1s
[root@centos-ami yum.repos.d]# repoquery --repoid=dukesbank --search dukesbank-
ear
dukesbank-ear-0:1.0-14.noarch
dukesbank-ear-0:1.0-15.noarch
dukesbank-ear-0:1.0-16.noarch
dukesbank-ear-0:1.0-18.noarch
dukesbank-ear-0:1.0-19.noarch
dukesbank-ear-0:1.0-20.noarch
dukesbank-ear-0:1.0-21.noarch
dukesbank-ear-0:1.0-22.noarch
dukesbank-ear-0:1.0-23.noarch
dukesbank-ear-0:1.0-24.noarch
dukesbank-ear-0:1.0-25.noarch
dukesbank-ear-0:1.0-26.noarch
[root@centos-ami yum.repos.d]# cat ~jenkins/jobs/dukesbank-ear.rpm/workspace/
versions.json
[ "1.0-14", "1.0-15", "1.0-16", "1.0-18", "1.0-19", "1.0-20", "1.0-21", "1.0-22",
"1.0-23", "1.0-24", "1.0-25", "1.0-26", "1.0-27", "1.0-28", "1.0-29", "1.0-30",
"1.0-31", "1.0-32", "1.0-33", "1.0-34" ]
[root@centos-ami yum.repos.d]#

19. Deploy - Nodes & roles

20. Deploy - Node provider
[ec2-user@ip-10-36-119-128 ~]$ curl http://
ec2-75-101-135-190.compute-1.amazonaws.com:9980/
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE project PUBLIC "-//DTO Labs Inc.//DTD Resources Document 1.0//EN"
"project.dtd">
<project>
<node name="ip-10-195-11-159.ec2.internal"
type="Node"
description="ip-10-195-11-159.ec2.internal"
osArch="x86_64"
osFamily="unix"
osName="centos"
osVersion="6.2"
tags="_default,base,chef_server"
username="ec2-user"
hostname="ip-10-195-11-159.ec2.internal"
editUrl="http://ip-10-195-11-159.ec2.internal:4040/nodes/
ip-10-195-11-159.ec2.internal/edit"/>
<node name="ip-10-36-119-128.ec2.internal"
type="Node"
description="ip-10-36-119-128.ec2.internal"
osArch="x86_64"
osFamily="unix"
osName="centos"
osVersion="6.2"
tags="dev,base,dukesbank_database_master,dukesbank"
username="ec2-user"
hostname="ip-10-36-119-128.ec2.internal"
editUrl="http://ip-10-195-11-159.ec2.internal:4040/nodes/
ip-10-36-119-128.ec2.internal/edit"/>
</project>

21. Deploy - Application attributes
[ec2-user@centos-ami chef-repo]$ cat data_bags/apps/dukesbank.json
{
"id": "dukesbank",
"server_roles": [
"dukesbank"
],
"type": {
"dukesbank": [
"jboss",
"jboss_webapp"
]
},
"database_master_role": [
"dukesbank_database_master"
],
"packages": {
"dev": {
"dukesbank-ear": "1.0-23"
}
},
"jboss_state": {
"dev": "start"
},
.
.
.

22. Links
• DTO Solutions - http://dtosolutions.com
• DTO Labs - http://dtolabs.com
• DTO Labs Github - https://github.com/dtolabs
• Rundeck - http://rundeck.org
• Devops-Toolchain -
http://groups.google.com/group/devops-
toolchain