Our pipeline that builds on individual configuration management modules to layer up to a full multi-node dev environment and on to production deploys

1. CI/CD Pipeline to
Deploy and Maintain an
OpenStack IaaS Cloud
Simon McCartney, Site Reliability Engineering
OpenStack Summit 2014 - Paris
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

2. Background
● This project predates HP Helion OpenStack
● We’re using Ubuntu 12.04, OpenStack Grizzly & SaltStack
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

3. Why?
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

4. Why Continuous Integration & Delivery?
• Repeatable, Reliable & Incrementally Improving
• Constant flow of changes to benefit users
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

5. Why Continuous Integration & Delivery?
● Infrastructure as Code > Infrastructure as Art
● Frequent Small Batches > Infrequent Big Batches
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

6. Why Continuous Integration & Delivery?
● Time Vampires
● Manual Builds
● Manual Testing
● Improved consistency & quality
● Faster Cycle Time
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

7. Why Continuous Integration & Delivery
● Unit Tests - salt formulae
● Integration Tests - do these salt formulae work together?
● End-to-End Tests - does the OpenStack Cloud we built work?
● Performance Test - did we break/de-tune something?
● Deployment Tests - do we have everything?
● Configuration Tests - can we test different environments before
hitting the real environment?
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

8. Our Pipeline
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

9. Our Tech Stack
● Vendor packages for OpenStack (Ubuntu)
● Configuration management (SaltStack)
● Software Engineering
● git, gerrit, gitshelf & jenkins
● test-kitchen (kitchen-salt, serverspec)
● Infrastructure Engineering
● Vagrant/VirtualBox & Public Cloud
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

10. Our Pipeline
1. Individual Salt Formula development
2. Personal multi-node, package based OpenStack environment on your
workstation for dev & validation
3. Go public – push to gerrit for review & automated testing
4. Pull in to the deploy-kit
5. deploy-kit tooling kicks in & builds deploy artefacts
6. Auto deploy to ephemeral public cloud test environment
7. Deploy to physical staging environment
8. Ready for production
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

11. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

12. Validating & Approving a change
• gerrit - defacto web based open source code review system
• submit changes (a patch set) for review by your peers..
• ..and for testing by Jenkins
• Jenkins & Gerrit Trigger
• Gerrit plugins allow certain jobs to be triggered on patch-set events (create,
merge etc)
• results of these jobs are then recorded in gerrit
• test-kitchen
• test harness to execute your configuration management code in isolation (i.e.
fresh VMs)
• providers (vagrant, LXC, openstack, gce, aws etc)
• provisioners (Chef, Salt, Puppet)
• testing frameworks (Bats, RSpec, serverspec etc)
• can be used interactively in your dev environment as well as in validation jobs
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

13. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

14. In-cloud validation
1. contractor creates instances &
networks
2. bootstrap.sh used prepare vanilla
image for use with salt
3. Jenkins uploads release artifacts and
deploys salt states
4. Performa validation tests
"roles": {
...
"compute": {
"image": "261844b3-479c-...",
"flavor": "101",
"keypair": "jenkins",
"instances": {
"ae1": [{
"az": "az3",
"number": 1,
"nics": [{
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
"network": "public-net",
"fixed_ip": "172.16.0.101",
"floating_ip": "15.126.241.109"
},
{
"network": "private-net",
"fixed_ip": "10.0.0.101"
}]

15. Deployment Automation
• Production Deploys
• Move slowly when required
• Confirm host is still “good” using existing monitoring framework
• Caution over nova-* service restarts
• Next
• rundeck
• chatops
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

16. Links
● http://devops.com/blogs/continuous-delivery-pipeline/
● http://martinfowler.com/bliki/DeploymentPipeline.html
● https://github.com/secure-pipeline/jenkins-example
● https://speakerdeck.com/garethr/continuous-integration-for-infrastructure
● https://github.com/gitshelf/gitshelf
● https://github.com/moniker-dns/contractor
● https://github.com/macgreagoir/gobstack
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

17. Questions?
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

18. Thank you!
Simon McCartney
@simonmcc
&
Mick Gregg
mickgregg.com
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.