It's no secret that python is fantastic when it comes to rapid prototyping and development. When it comes to deploying a web application, the road to glory isn't as well paved and navigating the array of techniques and tools can be daunting. This talk will address the advantages of continuous deployment, the success factors involved and the tools available, mainly focusing on experiences with Django web development.

1. Towards Continuous Deployment
with Django
Roger Barnes
@mindsocket
roger@mindsocket.com.au
PyCon Australia 2012
1

2. Beer
Adventure
Photography
Frisbee
Web development
Python/Django
Co-founder @ Arribaa
BTech ICS
1997 1999 2001 2003 2005 2007 2009 2011

3. Concepts from other talks
● The why
– Lean startups and customer discovery
– Do more with less
● The how
– EC2 / provisioning
– Architecture
– Developing for cloud deployment
– Monitoring live web applications
– IaaS vs PaaS

4. What is continuous delivery*
"Rapid, incremental, low-risk delivery of high quality, valuable
new functionality to users through automation of the build,
testing and deployment process" - Jez Humble
Deploying every good version of your software...
… or at least being able to
"Minimising MTTBID (Mean Time to Bad Idea Detection)" – Etsy
* delivery == deployment for the purposes of this talk

5. More Than Technology
● Technology - For automation
● People - Cross-functional team, organised around
product
● Processes – Conventions and some glue that
technology can't automate

6. Why
● Fail fast, win fast - Build, Measure, Learn
● Competitive advantage
● Avoid YAGNI - do less
● Less manual process, less risk
● Real-time control
– Deploy on demand
– Decoupled deployment and release
– Self service environments

7. Continuous Delivery in Practice
● Single path to production
● Optimise for resilience
● Get comfortable with being uncomfortable
● Automate as much as possible
● If it hurts, do it more often
● Becomes natural, return on investment is fast/high
● Lots of ways to do it, here's what I've done...

9. Develop Commit Build Stage Deploy Measure
Environments
● Make dev/test/prod as similar as possible
– Full stack, no "./manage.py runserver"
– Some exceptions make sense. eg for development:
● dummy email backend
● dummy message broker
● fewer threads/workers
● less memory
● less analytics instrumentation

10. Develop Commit Build Stage Deploy Measure
Environments
Dev/test virtualised using Vagrant
Create and configure lightweight, reproducible, and
portable development environments
$ vagrant up
$ vagrant ssh

11. Develop Commit Build Stage Deploy Measure
Environments
Vagrantfile configures base image, provisioning,
shared folders, forwarded ports
Vagrant::Config.run do |config|
config.vm.box = "precise64"
config.vm.box_url = "http://files.vagrantup.com/precise64.box"
config.vm.provision :puppet do |puppet|
puppet.manifests_path = "puppet/vagrant-manifests"
puppet.manifest_file = "dev.pp"
puppet.module_path = "puppet/modules"
end
config.vm.forward_port 80, 8000
config.vm.forward_port 3306, 3306
config.vm.share_folder "arribaa", "/opt/django-projects/arribaa", ".."
end

12. Develop Commit Build Stage Deploy Measure
Environments
● Repeatable, versioned configuration
– Snowflake bad, phoenix good
– Puppet (masterless) for provisioning OS and services
– Fabric for scripted tasks, eg:
● copy database from production
● update requirements
● migrate database
● commit and push to repository
● Anti-pattern
– Can't spin up new environment with one command

13. Develop Commit Build Stage Deploy Measure
Development top level puppet config
include uwsgi
include statsd
include solr
include memcached
include rabbitmq Test configuration is the same
include nginx
include testing Production adds:
include arribaa backup
include arribaa::db_node monitoring
include arribaa::nginx
include arribaa::celery

14. Develop Commit Build Stage Deploy Measure
Using Fabric with Vagrant
def vagrant():
# get vagrant ssh setup
vagrant_config = _get_vagrant_config()
env.key_filename = vagrant_config['IdentityFile']
env.hosts = ['%s:%s' % (vagrant_config['HostName'],
vagrant_config['Port'])]
env.user = vagrant_config['User']
def _get_vagrant_config():
with lcd('../vagrant'):
result = local('vagrant ssh-config', capture=True)
conf = {}
for line in iter(result.splitlines()):
parts = line.split()
conf[parts[0]] = ' '.join(parts[1:])
return conf
Based on https://gist.github.com/1099132

15. Develop Commit Build Stage Deploy Measure
Dependencies
● virtualenv – separate env for each app
● pip – install dependencies into virtualenv
– use requirements file
– use explicit versions
● for repository based dependencies, use commit id
● or fork on github

16. Develop Commit Build Stage Deploy Measure
Dependencies
● Using virtualenv and pip with fabric
env.site_dir = '/opt/django-projects/arribaa'
env.app_dir = '/opt/django-projects/arribaa/arribaa'
env.pip_file = 'requirements.txt'
def ve_run(command, func=run, base_dir=env.app_dir, *args, **kwargs):
with cd(base_dir):
with prefix("source /opt/virtualenvs/%s/bin/activate" % env.virtualenv):
return func(command, *args, **kwargs)
def update_reqs():
ve_run('pip install -r %s' % env.pip_file, base_dir=env.site_dir)

17. Develop Commit Build Stage Deploy Measure
Database migration
● South
– Schema changes
– Data migrations
● Deploy separate expand and contract operations, eg:
– Expand
● add new column (update model, manage.py schemamigration …)
● map from old column (manage.py datamigration …)
– Deploy
– Contract (optional)
● remove old column ( update model, manage.py schemamigration …)

18. Develop Commit Build Stage Deploy Measure
Database migration
● Using South with fabric
def syncdb():
ve_run("python manage.py syncdb --noinput --migrate")

19. Develop Commit Build Stage Deploy Measure
Tying it all together
● Update pip requirements on vagrant VM
$ fab vagrant update_reqs
● Run data migration on vagrant VM
$ fab vagrant syncdb

20. Develop Commit Build Stage Deploy Measure
Source control
● Pick one, learn it
● Use tags to keep track of build status
● Avoid long-lived branches
– or integrate them – more overhead
✔ application code
● Tracks all code/documentation ✔ deployment code
✔ provisioning code
✔ configuration code
✔ documentation
✗ not build artifacts

21. arribaa Application code
├── apps Django apps
│ └── ...
├── static Django static files
│ └── ...
├── templates Django templates
│ └── ...
├── fabfile.py Fabric scripts
├── requirements.txt Pip
└── ...
bin
└── jenkins.sh Jenkins job
docs
└── ...

22. vagrant
├── Vagrantfile Dev VM config
└── puppet
├── modules Puppet modules
│ ├── arribaa
│ ├── backup
│ ├── graphite
│ ├── ...
│ ├── uwsgi
│ └── wget
└── vagrant-manifests
├── dev.pp Puppet dev
├── prod.pp Puppet prod*
└── test.pp Puppet test
* Not used by vagrant, but convenient to store here

23. Develop Commit Build Stage Deploy Measure
Automated Testing
● Django’s test framework
● Continuous Integration and Testing
– Jenkins
● django-jenkins
– tests
– pylint
– coverage
– css/jslint
● factory_boy instead of fixtures

24. Develop Commit Build Stage Deploy Measure
Test separation
● Split up tests by type
● Can parallelise, keeping test run times low
– < 20 minutes from commit to production
● Unit tests first, faster feedback
● Run tests on different triggers, keep some out of the main
build
● Etsy - Divide and Concur
– http://codeascraft.etsy.com/2011/04/20/divide-and-concur/

25. Develop Commit Build Stage Deploy Measure
Test separation
● Unit
● Functional/UI
– Selenium et al
● Staging and production smoke tests
– Crawl URLs
– Load/performance testing
● Flaky tests, Slow tests
– exclude from regular build, run on a separate schedule

26. Develop Commit Build Stage Deploy Measure
Test separation
● How to do this with Django?
– Suites
– Custom test runner
– Other test systems (Nose has an attrib plugin)
● My current solution
– Annotating tests, and adapted django-jenkins command
class TestBooking(TestCase):
@pipelineTag ("unit")
def test_when_lapsed(self):
oldbooking = factories.BookingFactory.build(when=some_old_date)
self.assertTrue(oldbooking.when_lapsed())

27. Develop Commit Build Stage Deploy Measure
Test separation
# Used to ignore tests
ignore = lambda test_item: None
def pipelineTag(*args):
"""
Let a testcase run if any of the tags are in sys.argv and
none of the tags match not-* in sys.argv.
"""
# Abstain if we're not in a build pipeline (ie using django-jenkins)
# or if no tag(s) specified
if 'jenkins_pipeline' not in sys.argv or not any(['tagged-' in arg for arg in sys.argv]):
return _id
tags = args
assert set(tags) < set(['unit', 'functional', 'flaky', 'slow', 'staging'])
#Silently ignore if no matching tag
if not any(['tagged-'+tag in sys.argv for tag in tags]):
return ignore
# Skip if "not-blah" tagged
if any(['not-'+tag in sys.argv for tag in tags]):
return skip('tagged: ' + ','.join([tag for tag in tags if 'not-'+tag in sys.argv]))
# This test made it through
return _id

28. Develop Commit Build Stage Deploy Measure
Build Pipeline
● One pipeline
● All developers feed start of pipeline via VCS (master)
– Improve policy of "don't deploy broken code"...
– ...with system where it's harder to deploy broken code
● Jenkins Build Pipeline Plugin
– Series of dependant jobs, each runs different tests
– Feeds into pre-deploy staging job
– Successful staging job enables deploy job

29. Develop Commit Build Stage Deploy Measure
Build Pipeline
Currently simple linear flow

30. Develop Commit Build Stage Deploy Measure
Build Pipeline
● Jenkins job – unit test stage:
bash -ex ./bin/jenkins.sh tagged-unit not-flaky not-slow
● jenkins.sh
# Setup virtualenv, pip requirements, settings.py etc
...snip...
cd $WORKSPACE/arribaa
python manage.py clean_pyc
python manage.py jenkins_pipeline "$@"

31. Develop Commit Build Stage Deploy Measure
Staging/QA Job
● fab staging-deploy – very similar process to production deploy
● Test run of production deployment using Fabric
– Deploy code and config to staging environment (Vagrant VM)
– Apply dependency updates
● puppet config
● pip requirements
– Copy database from production and run data migration
● Should now have a deployed QA environment
● Browser smoke tests – currently selenium
– Not using Django 1.4 LiveTestCase here, we have a full stack to play with
● Tag successful build as deployable

32. Develop Commit Build Stage Deploy Measure
Deployment Strategies
● Optimistic
– Deploy, reload services – some appservers handle this well
● Blue/Green
– Parallel environments, 1 idle
– Idle environment gets new version
– Smoke tests and monitoring before switching over
● Canary
– Take subset of app servers out of pool
– Deploy to subset
– Smoke tests and monitoring before deploying rest of cluster
● Deploy whole new (app) server instance
– then rewire load-balancer

33. Develop Commit Build Stage Deploy Measure
Deployment with Jenkins & Fabric
Using Git tags
– Set in Jenkins after successful staging run
– Used by deploy in Fabric
def pull():
with cd(env.site_dir):
run('git fetch')
latest_tag = run('git describe --tags --match "ci-passed-staging-*" origin/master',
pty=False)
run('git reset --hard refs/tags/%s' % latest_tag, pty=False)

34. Develop Commit Build Stage Deploy Measure
Tags set by build pipeline

35. Develop Commit Build Stage Deploy Measure
Deployment with Jenkins & Fabric
Jenkins script:
virtualenv -q /opt/virtualenvs/arribaa-jenkins
source /opt/virtualenvs/arribaa-jenkins/bin/activate
pip install -r requirements.txt
pip install -r requirements-testing.txt
cd $WORKSPACE/arribaa
fab deploy

36. Develop Commit Build Stage Deploy Measure
Deployment with Jenkins & Fabric
def deploy():
dbbackup() # django-dbbackup – to dropbox
changes = pull() # git fetch, git describe, git log, git reset
apply_puppet() # puppet apply … prod.pp
update_requirements() # pip install -r requirements.txt
clean_pyc() # django-extensions command
syncdb() # and –migrate (South)
collectstatic()
reload_uwsgi() # reload app server
restart_celeryd() # bounce task server
run('git tag -f ci-deployed && git push –tags') # tag deployed version
send_email(changes) # send notification

37. Develop Commit Build Stage Deploy Measure
Decoupling deployment and release
● Feature flags vs feature branches
– Real time control with flags
– Cleanup/maint needed for flags
– Harder/slower to test different combinations in either case
● Good for testing ideas (on/off, A/B, power users, QA etc)
● Gargoyle
– includes optional admin interface
– selective criteria (%age of users, staff only, ...)
● Alternative: Django Waffle

38. Develop Commit Build Stage Deploy Measure
Measurement
● Eyes wide open
– Track system and user behaviour
● Post-deployment
– Error, performance and behaviour heuristics
– Trend monitoring
– Goal conversion rates

39. Develop Commit Build Stage Deploy Measure
Measurement

40. Develop Commit Build Stage Deploy Measure
Measurement - Munin

41. Develop Commit Build Stage Deploy Measure

42. Develop Commit Build Stage Deploy Measure
Measurement
● django-statsd, statsd and graphite
– page timing
– counters
http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/

43. Rollback options
● Level of automation
– Depends on risk aversion and complexity of environment
– 12+ years of deployment, average 5 deployments per week,
very very few actual rollbacks
● Engineer around it
– lots of small changes → more likely to "roll-forward"
– staged deployment → minimise harm
● Engineer for it
– avoid backwards incompatible data migrations
– tag deployments, keep previous versions handy

44. Rollback options
"My deployment rollback strategy is
like a car with no reverse gear.
You can still go backwards,
but you have to get out and push.
Knowing that makes you drive carefully."
- Me

45. Where to from here
Have an existing app with some gaps?
Think about what's slowing you down the most
● Suggested priorities
– Automated tests (Django built-in)
– Continuous integration (jenkins, django-jenkins)
– Measure all the things (sentry, statsd + graphite, new relic)
– Scripted database (South)
– Scripted deployment (fabric)
– Configuration management (puppet)
– Virtualised development (Vagrant)
– Test separation (test decorator, customise test runner)
– Feature flags (gargoyle)

46. Looking Forward
● Better version pinning
● Fully automated provisioning
● Better test separation and parallelisation
● Combined coverage results
● Reduce dependence on external services
● Make deployment more atomic
● Staged rollouts to cluster – canary deployment
● Make the "big red deploy button" big and red

47. Resources
● Slides – http://slideshare.net/mindsocket/
● Continuous delivery/deployment
– Continuous Delivery Primer - http://www.informit.com/articles/article.aspx?p=1641923
– Anatomy of the Deployment Pipeline - http://www.informit.com/articles/printerfriendly.aspx?p=1621865
– Code as Craft – Etsy http://codeascraft.etsy.com/
– Safe deploying on the cutting edge – Urban Airship http://lanyrd.com/2011/djangocon-us/shbqz/
● Vagrant – http://vagrantup.com
● Vagrant with Fabric - https://gist.github.com/1099132
● Jenkins Build Pipeline Plugin -
http://www.morethanseven.net/2011/03/20/A-continuous-deployment-example-setup.html
● Git/fabric based rollback options
– http://dan.bravender.us/2012/5/11/git-based_fabric_deploys_are_awesome.html
– http://www.askthepony.com/blog/2011/07/setup-a-complete-django-server-deploy-rollback-%E2%80%93-all-in-one-
powerful-script/
– http://lethain.com/deploying-django-with-fabric/

48. Thank You!
http://slideshare.net/mindsocket
Questions?