Best Of Both Worlds

Best of both worlds
Deploying Django projects
with Capistrano and Supervisord

Maciej Pasternacki
SetJam, Inc.
maciej@pasternacki.net

Wednesday, May 26, 2010 1

Best of both worlds

Should we only use Python tools with Django,
or rather pick and mix best tools available,
regardless of the ecosystem they come from?

http://www.setjam.com/

Background

SetJam.com: an online TV guide
Started June 2009,
public beta December 2009,
incorporated February 2010
Agile & Open Source-based


What’s this talk about?
Release management

Automated deployment

Upstream libraries

Process management

Scaling in cloud

Centralized logging


Release management

Weekly iterations
Devel → Staging → Production
All in git branches


Capistrano
automating the deployment

Separate, independent releases
Mature & complete
Full workﬂow
Transactions and rollbacks


Capistrano
directory layout
releases YYYYMMDDHHMMSS

20100512210109

20100514101512

20100515085101

20100515101756

current log

var/ solr_data
shared
processed_images
local_settings.py
…

…

deploy:setup Prepares one or more servers for
deployment.

deploy:default Deploys your project.

deploy:update Copies your project and updates
the symlink.

deploy:update_code Copies your project to the remote servers.

strategy.deploy!

deploy:ﬁnalize_update [internal] Touches up the released code.

deploy:symlink Updates the symlink to the most recently
deployed release

deploy:restart Restarts your application.

http:/
Transaction
/www.setjam.com/
http://www.capify.org/index.php/Default_Deployment_Path

Capistrano-offroad
an open source pack of Capistrano utils

Reset Capistrano’s railsy defaults
Handful of utility functions
Django and Supervisord recipes
Want to improve!


Capistrano
conﬁguration samples: settings
# Capistrano-offroad modules
load 'lib/capistrano-offroad/offroad.rb'
offroad_modules 'defaults', 'django', 'supervisord'

# Server config
server "staging.setjam.com", :web, :db, :primary => true
role :app, "molly.setjam.com"
group :staging_fetch, :backend

set :deploy_to, "/srv/staging"
set :deploy_user, "staging"
set :deploy_group, "staging"

set :python, "/opt/python26/bin/python2.6"
set :django_project_subdirectory, "setjam"
set :django_use_south, true


Capistrano
conﬁguration samples: dependencies

depend :remote, :command, "make"
depend :remote, :command, "s3cmd"
[…]
depend :remote, :python_module, "PIL"
depend :remote, :python_module, "flup"
depend :remote, :python_module, "lxml"
depend :remote, :python_module, "MySQLdb"


Capistrano
conﬁguration samples: tasks

desc "[internal] finalize code update"
task :finalize_update do
run "ln -sfv #{shared_path}/local_settings*.py #{latest_release}/setjam/"
run "make -C #{latest_release}/lib/ wait-for-cached download-cached"
run "mv #{latest_release}/var #{latest_release}/var.orig"
run "ln -sv #{shared_path}/var #{latest_release}/var"
django_manage "synccompress", :roles => [:app, :web]
_role_supervisord_conf :backend
_role_supervisord_conf :app
end

desc "[internal] Wait until processes are reachable"
task :wait, :roles => :app do
run "#{current_path}/scripts/waitfcgi #{fcgi_port}"
run "#{current_path}/scripts/waitfcgi #{fcgi_api_port}"
end


Capistrano
conﬁguration samples: hooks

after "deploy:setup", "setjam:setup"
before "deploy:finalize_update", "setjam:finalize_update"

before "deploy:symlink", "smoke:offline"
after "deploy:start", "smoke"
after "deploy:restart", "smoke"
before "smoke", "setjam:wait"

before "deploy:cleanup" do set :use_sudo, true end
after "deploy:cleanup" do set :use_sudo, false end


Upstream libraries
how to get them?

System-wide?
Project-wide (shared?)
Release-wide!


Upstream libraries
standard approach: virtualenv+pip

pulling from upstream VC?
add local patches?
cache library set?
non-python dependencies?


Upstream libraries
current approach: Makeﬁle

Make tasks to get and patch libs
Timestamp ﬁles for dependencies
Cache versioned tarballs in S3
Symlinks to use downloaded stuff


Upstream libraries
all: Django.stamp _submodules.stamp
beautifulsoup.stamp
boto.stamp
concurrentloghandler.stamp
oauth.stamp
piston.stamp
httplib2.stamp
pyflakes.stamp
python-memcached.stamp
south.stamp
test_coverage.stamp
urllib3.stamp
pyyaml.stamp
feedparser.stamp
unittest2.stamp
chmod -R a+rX .

south.stamp: south.release south-return_status.diff
rm -rf south
hg clone -r `head -1 south.release` http://bitbucket.org/andrewgodwin/south/
cd south && patch -p1 < ../south-return_status.diff
date > $@

Upstream libraries
.PHONY: _submodules.stamp
_submodules.stamp:
cd .. && git submodule update --init
date > $@
DJANGO_VERSION=1.1.1
DJANGO_BASENAME=Django-$(DJANGO_VERSION)
$(DJANGO_BASENAME).tar.gz:
wget -q -O $@ http://www.djangoproject.com/download/$(DJANGO_VERSION)/tarball/
touch $@ # needed - wget doesn't update timestamp properly

Django.stamp: $(DJANGO_BASENAME).tar.gz Django-test_suite.3.diff Django-unittest2.diff
rm -rf $(DJANGO_BASENAME) Django
tar -xzf $<
cd $(DJANGO_BASENAME) && patch -p0 < ../Django-test_suite.3.diff
cd $(DJANGO_BASENAME) && patch -p0 < ../Django-test-client-cookie-fix.diff
cd $(DJANGO_BASENAME) && patch -p1 < ../Django-unittest2.diff
ln -s $(DJANGO_BASENAME) Django
date > $@


Supervisord
process management

Manages foreground processes
Easy to use locally
Easy to conﬁgure — .ini ﬁle
Easy to extend
Robust and reliable


Supervisord
conﬁguration sample
[program:fcgi]
command = python2.6 manage.py runfcgi daemonize=false …
stopsignal = HUP
directory = %(here)s/../setjam/

[program:api_fcgi]
command = python2.6 manage.py runfcgi daemonize=false …
stopsignal = HUP
environment = DJANGO_SETTINGS_MODULE=settings_api

[program:classifier]
command = python2.6 manage.py runreceiver classification
autostart = false
process_name = classify-%(process_num)s
numprocs = 3


Scaling in cloud

Few central servers
Symmetrical processes
Queuing to distribute work: django-sqs
EC2 security groups to mark roles:
capistrano-ec2group


Scaling in cloud
deploying a new server

Start new instance
cap HOSTFILTER=… deploy:setup
Copy local_settings.py

cap HOSTFILTER=… deploy
Maybe update central server conﬁg


Logging

logging.handlers.SysLogHandler
Central rsyslogd writes to MySQL
phplogcon for log browsing
Customized log reports from MySQL


Future work

Monitoring and status - Nagios?

Automate machine administration - Chef?

Polish and gemify capistrano-offroad

Rake instead of make for upstream libs

Capistrano rendering conﬁgs from templates


Questions?
More info: www.setjam.com/blog/
Code: github.com/mpasternacki/
More code: gist.github.com/mpasternacki/
E-mail: maciej@pasternacki.net
Twitter: @mpasternacki


Best Of Both Worlds

Recomendados

Recomendados

Más contenido relacionado

Último

Último (20)

Destacado

Destacado (20)

Best Of Both Worlds