Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Best Of Both Worlds

2.882 visualizaciones

Publicado el

While Django is a great Web framework, there are places where Python ecosystem falls short of its competitors. However, nothing prevents us from picking the best parts to create a mixed environment that is productive and nice to work with. I will present my approach to deployment of Django webapps controlled by supervisord, using Capistrano - an automated deployment tool belonging to the Ruby on Rails ecosystem - and my capistrano-offroad package that adapts Capistrano for non-Rails applications. I will also talk about release management process, based on the Git version control system, upon which the deployment setup is built.

Publicado en: Tecnología
  • Inicia sesión para ver los comentarios

Best Of Both Worlds

  1. 1. Best of both worlds Deploying Django projects with Capistrano and Supervisord Maciej Pasternacki SetJam, Inc. Wednesday, May 26, 2010 1
  2. 2. Best of both worlds Should we only use Python tools with Django, or rather pick and mix best tools available, regardless of the ecosystem they come from? Wednesday, May 26, 2010 2
  3. 3. Background an online TV guide Started June 2009, public beta December 2009, incorporated February 2010 Agile & Open Source-based Wednesday, May 26, 2010 3
  4. 4. What’s this talk about? Release management Automated deployment Upstream libraries Process management Scaling in cloud Centralized logging Wednesday, May 26, 2010 4
  5. 5. Release management Weekly iterations Devel → Staging → Production All in git branches Wednesday, May 26, 2010 5
  6. 6. Capistrano automating the deployment Separate, independent releases Mature & complete Full workflow Transactions and rollbacks Wednesday, May 26, 2010 6
  7. 7. Capistrano directory layout releases YYYYMMDDHHMMSS 20100512210109 20100514101512 20100515085101 20100515101756 current log var/ solr_data shared processed_images … … Wednesday, May 26, 2010 7
  8. 8. deploy:setup Prepares one or more servers for deployment. deploy:default Deploys your project. deploy:update Copies your project and updates the symlink. deploy:update_code Copies your project to the remote servers. strategy.deploy! deploy:finalize_update [internal] Touches up the released code. deploy:symlink Updates the symlink to the most recently deployed release deploy:restart Restarts your application. http:/ Transaction / Wednesday, May 26, 2010 8
  9. 9. Capistrano-offroad an open source pack of Capistrano utils Reset Capistrano’s railsy defaults Handful of utility functions Django and Supervisord recipes Want to improve! Wednesday, May 26, 2010 9
  10. 10. Capistrano configuration samples: settings # Capistrano-offroad modules load 'lib/capistrano-offroad/offroad.rb' offroad_modules 'defaults', 'django', 'supervisord' # Server config server "", :web, :db, :primary => true role :app, "" group :staging_fetch, :backend set :deploy_to, "/srv/staging" set :deploy_user, "staging" set :deploy_group, "staging" set :python, "/opt/python26/bin/python2.6" set :django_project_subdirectory, "setjam" set :django_use_south, true Wednesday, May 26, 2010 10
  11. 11. Capistrano configuration samples: dependencies depend :remote, :command, "make" depend :remote, :command, "s3cmd" […] depend :remote, :python_module, "PIL" depend :remote, :python_module, "flup" depend :remote, :python_module, "lxml" depend :remote, :python_module, "MySQLdb" Wednesday, May 26, 2010 11
  12. 12. Capistrano configuration samples: tasks desc "[internal] finalize code update" task :finalize_update do run "ln -sfv #{shared_path}/local_settings*.py #{latest_release}/setjam/" run "make -C #{latest_release}/lib/ wait-for-cached download-cached" run "mv #{latest_release}/var #{latest_release}/var.orig" run "ln -sv #{shared_path}/var #{latest_release}/var" django_manage "synccompress", :roles => [:app, :web] _role_supervisord_conf :backend _role_supervisord_conf :app end desc "[internal] Wait until processes are reachable" task :wait, :roles => :app do run "#{current_path}/scripts/waitfcgi #{fcgi_port}" run "#{current_path}/scripts/waitfcgi #{fcgi_api_port}" end Wednesday, May 26, 2010 12
  13. 13. Capistrano configuration samples: hooks after "deploy:setup", "setjam:setup" before "deploy:finalize_update", "setjam:finalize_update" before "deploy:symlink", "smoke:offline" after "deploy:start", "smoke" after "deploy:restart", "smoke" before "smoke", "setjam:wait" before "deploy:cleanup" do set :use_sudo, true end after "deploy:cleanup" do set :use_sudo, false end Wednesday, May 26, 2010 13
  14. 14. Upstream libraries how to get them? System-wide? Project-wide (shared?) Release-wide! Wednesday, May 26, 2010 14
  15. 15. Upstream libraries standard approach: virtualenv+pip pulling from upstream VC? add local patches? cache library set? non-python dependencies? Wednesday, May 26, 2010 15
  16. 16. Upstream libraries current approach: Makefile Make tasks to get and patch libs Timestamp files for dependencies Cache versioned tarballs in S3 Symlinks to use downloaded stuff Wednesday, May 26, 2010 16
  17. 17. Upstream libraries current approach: Makefile all: Django.stamp _submodules.stamp beautifulsoup.stamp boto.stamp concurrentloghandler.stamp oauth.stamp piston.stamp httplib2.stamp pyflakes.stamp python-memcached.stamp south.stamp test_coverage.stamp urllib3.stamp pyyaml.stamp feedparser.stamp unittest2.stamp chmod -R a+rX . south.stamp: south.release south-return_status.diff rm -rf south hg clone -r `head -1 south.release` cd south && patch -p1 < ../south-return_status.diff date > $@ Wednesday, May 26, 2010 17
  18. 18. Upstream libraries current approach: Makefile .PHONY: _submodules.stamp _submodules.stamp: cd .. && git submodule update --init date > $@ DJANGO_VERSION=1.1.1 DJANGO_BASENAME=Django-$(DJANGO_VERSION) $(DJANGO_BASENAME).tar.gz: wget -q -O $@$(DJANGO_VERSION)/tarball/ touch $@ # needed - wget doesn't update timestamp properly Django.stamp: $(DJANGO_BASENAME).tar.gz Django-test_suite.3.diff Django-unittest2.diff rm -rf $(DJANGO_BASENAME) Django tar -xzf $< cd $(DJANGO_BASENAME) && patch -p0 < ../Django-test_suite.3.diff cd $(DJANGO_BASENAME) && patch -p0 < ../Django-test-client-cookie-fix.diff cd $(DJANGO_BASENAME) && patch -p1 < ../Django-unittest2.diff ln -s $(DJANGO_BASENAME) Django date > $@ Wednesday, May 26, 2010 18
  19. 19. Supervisord process management Manages foreground processes Easy to use locally Easy to configure — .ini file Easy to extend Robust and reliable Wednesday, May 26, 2010 19
  20. 20. Supervisord configuration sample [program:fcgi] command = python2.6 runfcgi daemonize=false … stopsignal = HUP directory = %(here)s/../setjam/ [program:api_fcgi] command = python2.6 runfcgi daemonize=false … stopsignal = HUP directory = %(here)s/../setjam/ environment = DJANGO_SETTINGS_MODULE=settings_api [program:classifier] command = python2.6 runreceiver classification directory = %(here)s/../setjam/ autostart = false process_name = classify-%(process_num)s numprocs = 3 Wednesday, May 26, 2010 20
  21. 21. Scaling in cloud Few central servers Symmetrical processes Queuing to distribute work: django-sqs EC2 security groups to mark roles: capistrano-ec2group Wednesday, May 26, 2010 21
  22. 22. Scaling in cloud deploying a new server Start new instance cap HOSTFILTER=… deploy:setup Copy cap HOSTFILTER=… deploy Maybe update central server config Wednesday, May 26, 2010 22
  23. 23. Logging logging.handlers.SysLogHandler Central rsyslogd writes to MySQL phplogcon for log browsing Customized log reports from MySQL Wednesday, May 26, 2010 23
  24. 24. Wednesday, May 26, 2010 24
  25. 25. Wednesday, May 26, 2010 25
  26. 26. Future work Monitoring and status - Nagios? Automate machine administration - Chef? Polish and gemify capistrano-offroad Rake instead of make for upstream libs Capistrano rendering configs from templates Wednesday, May 26, 2010 26
  27. 27. Questions? More info: Code: More code: E-mail: Twitter: @mpasternacki Wednesday, May 26, 2010 27