Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Best Of Both Worlds

2.882 visualizaciones

Publicado el

While Django is a great Web framework, there are places where Python ecosystem falls short of its competitors. However, nothing prevents us from picking the best parts to create a mixed environment that is productive and nice to work with. I will present my approach to deployment of Django webapps controlled by supervisord, using Capistrano - an automated deployment tool belonging to the Ruby on Rails ecosystem - and my capistrano-offroad package that adapts Capistrano for non-Rails applications. I will also talk about release management process, based on the Git version control system, upon which the deployment setup is built.

Publicado en: Tecnología
  • Inicia sesión para ver los comentarios

Best Of Both Worlds

  1. 1. Best of both worlds Deploying Django projects with Capistrano and Supervisord Maciej Pasternacki SetJam, Inc. maciej@pasternacki.net Wednesday, May 26, 2010 1
  2. 2. Best of both worlds Should we only use Python tools with Django, or rather pick and mix best tools available, regardless of the ecosystem they come from? http://www.setjam.com/ Wednesday, May 26, 2010 2
  3. 3. Background SetJam.com: an online TV guide Started June 2009, public beta December 2009, incorporated February 2010 Agile & Open Source-based http://www.setjam.com/ Wednesday, May 26, 2010 3
  4. 4. What’s this talk about? Release management Automated deployment Upstream libraries Process management Scaling in cloud Centralized logging http://www.setjam.com/ Wednesday, May 26, 2010 4
  5. 5. Release management Weekly iterations Devel → Staging → Production All in git branches http://www.setjam.com/ Wednesday, May 26, 2010 5
  6. 6. Capistrano automating the deployment Separate, independent releases Mature & complete Full workflow Transactions and rollbacks http://www.setjam.com/ Wednesday, May 26, 2010 6
  7. 7. Capistrano directory layout releases YYYYMMDDHHMMSS 20100512210109 20100514101512 20100515085101 20100515101756 current log var/ solr_data shared processed_images local_settings.py … … http://www.setjam.com/ Wednesday, May 26, 2010 7
  8. 8. deploy:setup Prepares one or more servers for deployment. deploy:default Deploys your project. deploy:update Copies your project and updates the symlink. deploy:update_code Copies your project to the remote servers. strategy.deploy! deploy:finalize_update [internal] Touches up the released code. deploy:symlink Updates the symlink to the most recently deployed release deploy:restart Restarts your application. http:/ Transaction /www.setjam.com/ http://www.capify.org/index.php/Default_Deployment_Path Wednesday, May 26, 2010 8
  9. 9. Capistrano-offroad an open source pack of Capistrano utils Reset Capistrano’s railsy defaults Handful of utility functions Django and Supervisord recipes Want to improve! http://www.setjam.com/ Wednesday, May 26, 2010 9
  10. 10. Capistrano configuration samples: settings # Capistrano-offroad modules load 'lib/capistrano-offroad/offroad.rb' offroad_modules 'defaults', 'django', 'supervisord' # Server config server "staging.setjam.com", :web, :db, :primary => true role :app, "molly.setjam.com" group :staging_fetch, :backend set :deploy_to, "/srv/staging" set :deploy_user, "staging" set :deploy_group, "staging" set :python, "/opt/python26/bin/python2.6" set :django_project_subdirectory, "setjam" set :django_use_south, true http://www.setjam.com/ Wednesday, May 26, 2010 10
  11. 11. Capistrano configuration samples: dependencies depend :remote, :command, "make" depend :remote, :command, "s3cmd" […] depend :remote, :python_module, "PIL" depend :remote, :python_module, "flup" depend :remote, :python_module, "lxml" depend :remote, :python_module, "MySQLdb" http://www.setjam.com/ Wednesday, May 26, 2010 11
  12. 12. Capistrano configuration samples: tasks desc "[internal] finalize code update" task :finalize_update do run "ln -sfv #{shared_path}/local_settings*.py #{latest_release}/setjam/" run "make -C #{latest_release}/lib/ wait-for-cached download-cached" run "mv #{latest_release}/var #{latest_release}/var.orig" run "ln -sv #{shared_path}/var #{latest_release}/var" django_manage "synccompress", :roles => [:app, :web] _role_supervisord_conf :backend _role_supervisord_conf :app end desc "[internal] Wait until processes are reachable" task :wait, :roles => :app do run "#{current_path}/scripts/waitfcgi #{fcgi_port}" run "#{current_path}/scripts/waitfcgi #{fcgi_api_port}" end http://www.setjam.com/ Wednesday, May 26, 2010 12
  13. 13. Capistrano configuration samples: hooks after "deploy:setup", "setjam:setup" before "deploy:finalize_update", "setjam:finalize_update" before "deploy:symlink", "smoke:offline" after "deploy:start", "smoke" after "deploy:restart", "smoke" before "smoke", "setjam:wait" before "deploy:cleanup" do set :use_sudo, true end after "deploy:cleanup" do set :use_sudo, false end http://www.setjam.com/ Wednesday, May 26, 2010 13
  14. 14. Upstream libraries how to get them? System-wide? Project-wide (shared?) Release-wide! http://www.setjam.com/ Wednesday, May 26, 2010 14
  15. 15. Upstream libraries standard approach: virtualenv+pip pulling from upstream VC? add local patches? cache library set? non-python dependencies? http://www.setjam.com/ Wednesday, May 26, 2010 15
  16. 16. Upstream libraries current approach: Makefile Make tasks to get and patch libs Timestamp files for dependencies Cache versioned tarballs in S3 Symlinks to use downloaded stuff http://www.setjam.com/ Wednesday, May 26, 2010 16
  17. 17. Upstream libraries current approach: Makefile all: Django.stamp _submodules.stamp beautifulsoup.stamp boto.stamp concurrentloghandler.stamp oauth.stamp piston.stamp httplib2.stamp pyflakes.stamp python-memcached.stamp south.stamp test_coverage.stamp urllib3.stamp pyyaml.stamp feedparser.stamp unittest2.stamp chmod -R a+rX . south.stamp: south.release south-return_status.diff rm -rf south hg clone -r `head -1 south.release` http://bitbucket.org/andrewgodwin/south/ cd south && patch -p1 < ../south-return_status.diff date > $@ http://www.setjam.com/ Wednesday, May 26, 2010 17
  18. 18. Upstream libraries current approach: Makefile .PHONY: _submodules.stamp _submodules.stamp: cd .. && git submodule update --init date > $@ DJANGO_VERSION=1.1.1 DJANGO_BASENAME=Django-$(DJANGO_VERSION) $(DJANGO_BASENAME).tar.gz: wget -q -O $@ http://www.djangoproject.com/download/$(DJANGO_VERSION)/tarball/ touch $@ # needed - wget doesn't update timestamp properly Django.stamp: $(DJANGO_BASENAME).tar.gz Django-test_suite.3.diff Django-unittest2.diff rm -rf $(DJANGO_BASENAME) Django tar -xzf $< cd $(DJANGO_BASENAME) && patch -p0 < ../Django-test_suite.3.diff cd $(DJANGO_BASENAME) && patch -p0 < ../Django-test-client-cookie-fix.diff cd $(DJANGO_BASENAME) && patch -p1 < ../Django-unittest2.diff ln -s $(DJANGO_BASENAME) Django date > $@ http://www.setjam.com/ Wednesday, May 26, 2010 18
  19. 19. Supervisord process management Manages foreground processes Easy to use locally Easy to configure — .ini file Easy to extend Robust and reliable http://www.setjam.com/ Wednesday, May 26, 2010 19
  20. 20. Supervisord configuration sample [program:fcgi] command = python2.6 manage.py runfcgi daemonize=false … stopsignal = HUP directory = %(here)s/../setjam/ [program:api_fcgi] command = python2.6 manage.py runfcgi daemonize=false … stopsignal = HUP directory = %(here)s/../setjam/ environment = DJANGO_SETTINGS_MODULE=settings_api [program:classifier] command = python2.6 manage.py runreceiver classification directory = %(here)s/../setjam/ autostart = false process_name = classify-%(process_num)s numprocs = 3 http://www.setjam.com/ Wednesday, May 26, 2010 20
  21. 21. Scaling in cloud Few central servers Symmetrical processes Queuing to distribute work: django-sqs EC2 security groups to mark roles: capistrano-ec2group http://www.setjam.com/ Wednesday, May 26, 2010 21
  22. 22. Scaling in cloud deploying a new server Start new instance cap HOSTFILTER=… deploy:setup Copy local_settings.py cap HOSTFILTER=… deploy Maybe update central server config http://www.setjam.com/ Wednesday, May 26, 2010 22
  23. 23. Logging logging.handlers.SysLogHandler Central rsyslogd writes to MySQL phplogcon for log browsing Customized log reports from MySQL http://www.setjam.com/ Wednesday, May 26, 2010 23
  24. 24. http://www.setjam.com/ Wednesday, May 26, 2010 24
  25. 25. http://www.setjam.com/ Wednesday, May 26, 2010 25
  26. 26. Future work Monitoring and status - Nagios? Automate machine administration - Chef? Polish and gemify capistrano-offroad Rake instead of make for upstream libs Capistrano rendering configs from templates http://www.setjam.com/ Wednesday, May 26, 2010 26
  27. 27. Questions? More info: www.setjam.com/blog/ Code: github.com/mpasternacki/ More code: gist.github.com/mpasternacki/ E-mail: maciej@pasternacki.net Twitter: @mpasternacki http://www.setjam.com/ Wednesday, May 26, 2010 27

×