Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Python & Devops: Your own heroku

3.758 visualizaciones

Publicado el

Using python and uWSGI for agile deployment, effortless :)

Publicado en: Tecnología
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    ¿Estás seguro?    No
    Tu mensaje aparecerá aquí

Python & Devops: Your own heroku

  1. 1. Python & DevOps Your own heroku jib.li
  2. 2. Chakib Benziane @sp4ke● Freelance Full Stack Developer / DevOps● CTO & CoFounder at Jib.li● Python Enthusiast
  3. 3. Summary1. Environment & Stack2. Agile deployment with uWSGI
  4. 4. 1. Environment & Stack Our needs: ● Web Application ● Deep linking with social networks ● Agile development ● Community and available packages
  5. 5. 1. Environment & Stack Our needs: ● Web Application ● Deep linking with social networks ● Agile development ● Community and available packages = Django + MongoDB + Github
  6. 6. 1. Environment & Stack Our needs: ● Web Application ● Deep linking with social networks ● Agile development ● Community and available packages = Django + MongoDB + Github gevent-socketio, zmq, celery, AWS boto ...
  7. 7. 1. Environment & Stack Local environment : The essentials ● Virtualenv ● PIP
  8. 8. 1. Environment & Stack Local environment : The essentials ● Virtualenv ● PIP ● But they have RVM !
  9. 9. 1. Environment & Stack Local environment : The essentials ● Virtualenv ● PIP ● But they have RVM ! ● Pythonbrew: utahta/pythonbrew.git
  10. 10. 1. Environment & Stack● Pythonbrew: utahta/pythonbrew.git ○ Compile system independent pythons $ pythonbrew install 2.7.3 $ pythonbrew use 2.7.3 $ pythonbrew list && which python # pythonbrew pythons Python-2.7.3 (*) /home/spike/.pythonbrew/pythons/Python-2.7.3/bin/python
  11. 11. 1. Environment & Stack● Pythonbrew: utahta/pythonbrew.git ○ Easy management of virtualenvs $ pythonbrew venv create jibli $ pythonbrew venv use jibli && which python && which pip # Using `jibli` environment # To leave an environment, simply run `deactivate` /home/spike/.pythonbrew/venvs/Python-2.7.3/jibli/bin/python /home/spike/.pythonbrew/venvs/Python-2.7.3/jibli/bin/pip
  12. 12. 1. Environment & Stack Environment bootstrap: git clone jibli/project && cd project pythonbrew create venv jibli && pythonbrew activate jibli pip install -r dependencies.txt
  13. 13. 1. Environment & Stack Environment bootstrap: git clone jibli/project && cd project pythonbrew create venv jibli && pythonbrew activate jibli pip install -r dependencies.txt --download-cache=CACHE
  14. 14. 1. Environment & Stack pip & dependencies.txt: package-foo dont package-bar==4.2 do
  15. 15. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo
  16. 16. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo git+https://github.com/user/repo#egg=mon-package
  17. 17. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo git+https://github.com/user/repo#egg=mon-package git+https://github.com/user/repo@branch
  18. 18. 1. Environment & Stack pip & dependencies.txt: package-foo package-bar==4.2 git+https://github.com/user/repo git+https://github.com/user/repo#egg=mon-package git+https://github.com/user/repo@branch ● Quick dependencies update pip freeze > dependencies.txt
  19. 19. 1. Environment & Stack MongoDB ● NoSQL, Schemaless, Document Oriented ● BSON data format ● Advantage: Python Dict -> JSON ● Good Python API pymongo
  20. 20. 1. Environment & Stack MongoDB● MongoDB Javascript Console $ mongo jibli MongoDB shell version: 2.0.6 connecting to: jibli > db.users.find( {profil.age: 10} );
  21. 21. 1. Environment & Stack MongoDB● MongoDB Javascript Console $ mongo jibli MongoDB shell version: 2.0.6 connecting to: jibli > db.users.find( {profil.age: 10} );● PyMongo equivalent u = pymongo.Connection(host=localhost, port=27017)[jibli][users] u.find( {profil.age: 10} )
  22. 22. 2. Agile deployment with uWSGI Local development● Git branch feature● Unit test● Implement● Test on local server (./manage.py runserver)● Commit and merge on master branch
  23. 23. 2. Agile deployment with uWSGI Agile deployment ● Many features require a production like environment: ○ OAuth Authentication and Social Networks ○ Async tasks, Celery (notifications, crons ... ) ○ Push Notifications ○ Hard to clone a production environment in local
  24. 24. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature
  25. 25. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push
  26. 26. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push ○ My branch is UP on feature.dev.com
  27. 27. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push ○ My branch is UP on feature.dev.com ○ Remote access to the deployed app from local shell with a production environment (ie. restart, upgrade, ipython, mongo shell ...)
  28. 28. 2. Agile deployment with uWSGI ● Development Scenario ○ Start a new feature $ git checkout -b feature ○ Implement and push on dev server $ fab push ○ My branch is UP on feature.dev.com ○ Remote access to the deployed app from local shell with a production environment (ie. restart, upgrade, ipython, mongo shell ...) ○ Once satisfied merge on master and push on prod
  29. 29. 2. Agile deployment with uWSGI Solution● Nginx● Github● Fabric+● uWSGI
  30. 30. 2. Agile deployment with uWSGI Solution● Nginx● Github● Fabric+● uWSGI = your heroku like solution
  31. 31. 2. Agile deployment with uWSGI uWSGI● Create developement stacks● Host application clusters
  32. 32. 2. Agile deployment with uWSGI uWSGI ● how ? Web Server Django (nginx) uWSGI App WSGI Interface also: FastCGI,CGI, PHP, Rack, ...
  33. 33. 2. Agile deployment with uWSGI uWSGI ● how ? Django App Web Server Django (nginx) uWSGI App Load balancing Django App
  34. 34. 2. Agile deployment with uWSGI uWSGI ● how ? Django App1 Web Server Django (nginx) uWSGI App2 Router app1.dev.com app2.dev.com app3.dev.com Proxy Flask App3
  35. 35. 2. Agile deployment with uWSGI Django uWSGI App1 ● how ? Django App2 Web Server Flask (nginx) uWSGI App3 Emperor app1.dev.com app2.dev.com Massive app app3.dev.com *.dev.com deployment App n
  36. 36. 2. Agile deployment with uWSGI uWSGI Emperor ● Event based dynamic handling of applications (Vassals)Default: ● Scan for config files in directories (.ini, .xml, .yml, .json ... ) ● dir:// & glob:// for conf files monitoring ● Much more plugins available (mongodb, ampq, ldap ... )
  37. 37. 2. Agile deployment with uWSGI glob:// plugin uwsgi --emperor /opt/apps/*/*.ini
  38. 38. 2. Agile deployment with uWSGI glob:// plugin uwsgi --emperor "/opt/apps/*/*.ini" One does not simply use glob patterns !
  39. 39. 2. Agile deployment with uWSGI glob:// plugin uwsgi --emperor "/opt/apps/*/*.ini" Example: ○ New file "/opt/apps/appn/uwsgi.ini" ● Spawn vassal ○ File modified ● Restart vassal ○ File removed ● Kill vassal ○ Emperor dies ● All vassals die with him
  40. 40. 2. Agile deployment with uWSGI ● Create a conf file for each deployed app ? /opt/apps/app1/uwsgi.ini /opt/apps/app2/uwsgi.ini /opt/apps/appn/uwsgi.ini
  41. 41. 2. Agile deployment with uWSGI ● Create a conf file for each deployed app ? /opt/apps/app1/uwsgi.ini /opt/apps/app2/uwsgi.ini /opt/apps/appn/uwsgi.ini ● ln -s ○ Use template conf files /opt/apps/template ln -s /opt/apps/template /opt/apps/app1/app1.ini
  42. 42. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings chdir = %(djangoproject) module = uwsgi_app socket = 127.0.0.1:0 master = true processes = 1 idle = 300 subscribe-to = 127.0.0.1:9999:%n.dev.com logto = %d/log/uwsgi.log
  43. 43. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ ● Use variables like here djangoproject ● Magic variables : ○ %d - Absolute path to configuration file ○ %n - Name of configuration file without extension
  44. 44. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings ● Define your apps virtualenv
  45. 45. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings ● Define your apps virtualenv ● Python search paths (You can repeat this one)
  46. 46. 2. Agile deployment with uWSGI Template conf file [uwsgi] djangoproject = %d/app/ home = %d/virt pythonpath = %d/ env = DJANGO_SETTINGS_MODULE=app.settings ● Define your apps virtualenv ● Python search paths (You can repeat this one) ● Custom environment variables
  47. 47. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] chdir = %(djangoproject) module = uwsgi_app ● Which module to run when starting application ○ django.core.handlers.wsgi:WSGIHandler() ● Best spot to run your custom scripts and setup environment before launching application ○ ie. compile static, zMQ sockets, syncdb ...
  48. 48. 2. Agile deployment with uWSGI Up until now we can: ○ Git push origin feature ○ Clone feature in remote /opt/apps/feature ○ Prepare dirs structure ○ Create venv & install dependencies ○ Symllink to the uWSGI template file ○ uWSGI emperor launches feature app We still need to: ○ access our feature using subdomains ● feature.dev.com
  49. 49. 2. Agile deployment with uWSGI FastRouter ○ Proxy/Load Balancing/Router ○ Speaks uWSGI protocol ○ Unlimited setup possibilities ○ Key/Value store
  50. 50. 2. Agile deployment with uWSGI FastRouter ○ Proxy/Load Balancing/Router ○ Speaks uWSGI protocol ○ Unlimited setup possibilities ○ Key/Value store Example: uwsgi --fastrouter /tmp/fastrouter.socket -- fastrouter-subscription-server 127.0.0.1:9999
  51. 51. 2. Agile deployment with uWSGI Always use unix sockets instead of localhost tcp
  52. 52. 2. Agile deployment with uWSGI Nginx server { listen 80; server_name dev.com *.dev.com; location / { include /etc/nginx/uwsgi_params; uwsgi_param UWSGI_FASTROUTER_KEY $host; uwsgi_pass unix: /tmp/fastrouter.socket; } }
  53. 53. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] ... socket = 127.0.0.1:0 subscribe-to = 127.0.0.1:9999:%n.dev.com
  54. 54. 2. Agile deployment with uWSGI Template conf file (Django App) [uwsgi] ... socket = 127.0.0.1:0 subscribe-to = 127.0.0.1:9999:%n.dev.com /opt/apps/feature1/feature1.ini
  55. 55. 2. Agile deployment with uWSGI Request on feature1.dev.com nginx Fastrouteur + Subscription server { feature1.dev.com : 127.0.0.1:n1, feautre2.dev.com : 127.0.0.1:n2, ... } uWSGI emperor feature1 feature2
  56. 56. 2. Agile deployment with uWSGI HTOP Deploying feature1 ├─ supervisord │ ├─ /uwsgi --fastrouter ... --emperor /opt/apps/*/*.ini │ │ ├─ /usr/local/bin/uwsgi #fastrouter │ │ └─ /usr/local/bin/uwsgi #master │ │ ├─uwsgi --ini /opt/apps/feature1/feature1.ini
  57. 57. 2. Agile deployment with uWSGI HTOP Deploying feature2 ├─ supervisord │ ├─ /uwsgi --fastrouter ... --emperor /opt/apps/*/*.ini │ │ ├─ /usr/local/bin/uwsgi #fastrouter │ │ └─ /usr/local/bin/uwsgi #master │ │ ├─uwsgi --ini /opt/apps/feature1/feature1.ini │ │ └─uwsgi --ini /opt/apps/feature2/frature2.ini
  58. 58. End twitter: @sp4ke email: spike@jib.lislides: sp4ke.com/pythondevops

×