1.
Best of both worlds
Deploying Django projects
with Capistrano and Supervisord
Maciej Pasternacki
SetJam, Inc.
maciej@pasternacki.net
Wednesday, May 26, 2010 1

2.
Best of both worlds
Should we only use Python tools with Django,
or rather pick and mix best tools available,
regardless of the ecosystem they come from?
http://www.setjam.com/
Wednesday, May 26, 2010 2

3.
Background
SetJam.com: an online TV guide
Started June 2009,
public beta December 2009,
incorporated February 2010
Agile & Open Source-based
http://www.setjam.com/
Wednesday, May 26, 2010 3

4.
What’s this talk about?
Release management
Automated deployment
Upstream libraries
Process management
Scaling in cloud
Centralized logging
http://www.setjam.com/
Wednesday, May 26, 2010 4

5.
Release management
Weekly iterations
Devel → Staging → Production
All in git branches
http://www.setjam.com/
Wednesday, May 26, 2010 5

6.
Capistrano
automating the deployment
Separate, independent releases
Mature & complete
Full workflow
Transactions and rollbacks
http://www.setjam.com/
Wednesday, May 26, 2010 6

7.
Capistrano
directory layout
releases YYYYMMDDHHMMSS
20100512210109
20100514101512
20100515085101
20100515101756
current log
var/ solr_data
shared
processed_images
local_settings.py
…
…
http://www.setjam.com/
Wednesday, May 26, 2010 7

8.
deploy:setup Prepares one or more servers for
deployment.
deploy:default Deploys your project.
deploy:update Copies your project and updates
the symlink.
deploy:update_code Copies your project to the remote servers.
strategy.deploy!
deploy:finalize_update [internal] Touches up the released code.
deploy:symlink Updates the symlink to the most recently
deployed release
deploy:restart Restarts your application.
http:/
Transaction
/www.setjam.com/
http://www.capify.org/index.php/Default_Deployment_Path
Wednesday, May 26, 2010 8

9.
Capistrano-offroad
an open source pack of Capistrano utils
Reset Capistrano’s railsy defaults
Handful of utility functions
Django and Supervisord recipes
Want to improve!
http://www.setjam.com/
Wednesday, May 26, 2010 9

10.
Capistrano
configuration samples: settings
# Capistrano-offroad modules
load 'lib/capistrano-offroad/offroad.rb'
offroad_modules 'defaults', 'django', 'supervisord'
# Server config
server "staging.setjam.com", :web, :db, :primary => true
role :app, "molly.setjam.com"
group :staging_fetch, :backend
set :deploy_to, "/srv/staging"
set :deploy_user, "staging"
set :deploy_group, "staging"
set :python, "/opt/python26/bin/python2.6"
set :django_project_subdirectory, "setjam"
set :django_use_south, true
http://www.setjam.com/
Wednesday, May 26, 2010 10

11.
Capistrano
configuration samples: dependencies
depend :remote, :command, "make"
depend :remote, :command, "s3cmd"
[…]
depend :remote, :python_module, "PIL"
depend :remote, :python_module, "flup"
depend :remote, :python_module, "lxml"
depend :remote, :python_module, "MySQLdb"
http://www.setjam.com/
Wednesday, May 26, 2010 11

12.
Capistrano
configuration samples: tasks
desc "[internal] finalize code update"
task :finalize_update do
run "ln -sfv #{shared_path}/local_settings*.py #{latest_release}/setjam/"
run "make -C #{latest_release}/lib/ wait-for-cached download-cached"
run "mv #{latest_release}/var #{latest_release}/var.orig"
run "ln -sv #{shared_path}/var #{latest_release}/var"
django_manage "synccompress", :roles => [:app, :web]
_role_supervisord_conf :backend
_role_supervisord_conf :app
end
desc "[internal] Wait until processes are reachable"
task :wait, :roles => :app do
run "#{current_path}/scripts/waitfcgi #{fcgi_port}"
run "#{current_path}/scripts/waitfcgi #{fcgi_api_port}"
end
http://www.setjam.com/
Wednesday, May 26, 2010 12

13.
Capistrano
configuration samples: hooks
after "deploy:setup", "setjam:setup"
before "deploy:finalize_update", "setjam:finalize_update"
before "deploy:symlink", "smoke:offline"
after "deploy:start", "smoke"
after "deploy:restart", "smoke"
before "smoke", "setjam:wait"
before "deploy:cleanup" do set :use_sudo, true end
after "deploy:cleanup" do set :use_sudo, false end
http://www.setjam.com/
Wednesday, May 26, 2010 13

14.
Upstream libraries
how to get them?
System-wide?
Project-wide (shared?)
Release-wide!
http://www.setjam.com/
Wednesday, May 26, 2010 14

15.
Upstream libraries
standard approach: virtualenv+pip
pulling from upstream VC?
add local patches?
cache library set?
non-python dependencies?
http://www.setjam.com/
Wednesday, May 26, 2010 15

16.
Upstream libraries
current approach: Makefile
Make tasks to get and patch libs
Timestamp files for dependencies
Cache versioned tarballs in S3
Symlinks to use downloaded stuff
http://www.setjam.com/
Wednesday, May 26, 2010 16

17.
Upstream libraries
current approach: Makefile
all: Django.stamp _submodules.stamp
beautifulsoup.stamp
boto.stamp
concurrentloghandler.stamp
oauth.stamp
piston.stamp
httplib2.stamp
pyflakes.stamp
python-memcached.stamp
south.stamp
test_coverage.stamp
urllib3.stamp
pyyaml.stamp
feedparser.stamp
unittest2.stamp
chmod -R a+rX .
south.stamp: south.release south-return_status.diff
rm -rf south
hg clone -r `head -1 south.release` http://bitbucket.org/andrewgodwin/south/
cd south && patch -p1 < ../south-return_status.diff
date > $@
http://www.setjam.com/
Wednesday, May 26, 2010 17

18.
Upstream libraries
current approach: Makefile
.PHONY: _submodules.stamp
_submodules.stamp:
cd .. && git submodule update --init
date > $@
DJANGO_VERSION=1.1.1
DJANGO_BASENAME=Django-$(DJANGO_VERSION)
$(DJANGO_BASENAME).tar.gz:
wget -q -O $@ http://www.djangoproject.com/download/$(DJANGO_VERSION)/tarball/
touch $@ # needed - wget doesn't update timestamp properly
Django.stamp: $(DJANGO_BASENAME).tar.gz Django-test_suite.3.diff Django-unittest2.diff
rm -rf $(DJANGO_BASENAME) Django
tar -xzf $<
cd $(DJANGO_BASENAME) && patch -p0 < ../Django-test_suite.3.diff
cd $(DJANGO_BASENAME) && patch -p0 < ../Django-test-client-cookie-fix.diff
cd $(DJANGO_BASENAME) && patch -p1 < ../Django-unittest2.diff
ln -s $(DJANGO_BASENAME) Django
date > $@
http://www.setjam.com/
Wednesday, May 26, 2010 18

19.
Supervisord
process management
Manages foreground processes
Easy to use locally
Easy to configure — .ini file
Easy to extend
Robust and reliable
http://www.setjam.com/
Wednesday, May 26, 2010 19

20.
Supervisord
configuration sample
[program:fcgi]
command = python2.6 manage.py runfcgi daemonize=false …
stopsignal = HUP
directory = %(here)s/../setjam/
[program:api_fcgi]
command = python2.6 manage.py runfcgi daemonize=false …
stopsignal = HUP
directory = %(here)s/../setjam/
environment = DJANGO_SETTINGS_MODULE=settings_api
[program:classifier]
command = python2.6 manage.py runreceiver classification
directory = %(here)s/../setjam/
autostart = false
process_name = classify-%(process_num)s
numprocs = 3
http://www.setjam.com/
Wednesday, May 26, 2010 20

21.
Scaling in cloud
Few central servers
Symmetrical processes
Queuing to distribute work: django-sqs
EC2 security groups to mark roles:
capistrano-ec2group
http://www.setjam.com/
Wednesday, May 26, 2010 21

22.
Scaling in cloud
deploying a new server
Start new instance
cap HOSTFILTER=… deploy:setup
Copy local_settings.py
cap HOSTFILTER=… deploy
Maybe update central server config
http://www.setjam.com/
Wednesday, May 26, 2010 22

23.
Logging
logging.handlers.SysLogHandler
Central rsyslogd writes to MySQL
phplogcon for log browsing
Customized log reports from MySQL
http://www.setjam.com/
Wednesday, May 26, 2010 23

24.
http://www.setjam.com/
Wednesday, May 26, 2010 24

25.
http://www.setjam.com/
Wednesday, May 26, 2010 25

26.
Future work
Monitoring and status - Nagios?
Automate machine administration - Chef?
Polish and gemify capistrano-offroad
Rake instead of make for upstream libs
Capistrano rendering configs from templates
http://www.setjam.com/
Wednesday, May 26, 2010 26

27.
Questions?
More info: www.setjam.com/blog/
Code: github.com/mpasternacki/
More code: gist.github.com/mpasternacki/
E-mail: maciej@pasternacki.net
Twitter: @mpasternacki
http://www.setjam.com/
Wednesday, May 26, 2010 27