The document discusses the author's approach to setting up a development environment for Django projects. It describes establishing a project layout with separate folders for source code, virtual environments, requirements files, and more. It also covers tools and practices for tasks like dependency management, testing, debugging, deployment, and overall software development philosophy.

1. Development environment
for Django project - my way
!
Robert Lujo, 2014

2. about me
• software
• professionally 17 y.
• python since 2.0, django since 0.96
• freelancer
• more info -> linkedin

3. “Software Development
Environment” is …?
I found no unique / unified answer …

4. Software Development
Environment (1)
refers to a server tier designated to a specific stage in a
release process.
List of tiers: Local, Development, Integration, Test/QA, UAT,
Stage/Pre-production, Production/Live
Hm…!
https://en.wikipedia.org/wiki/Development_environment_
%28software_development_process%29

5. Software Development
Environment (2)
a collection of procedures and tools for developing, testing
and debugging an application or program.
Getting closer …!
http://www.techopedia.com/definition/16376/development-environment

6. Integrated development
environment - IDE
!
software application that provides comprehensive facilities to
computer programmers for software development.
… source code editor, build automation tools and a debugger...
Intelligent code completion features ... compiler, interpreter, ...
version control system and various tools are integrated to simplify
the construction of a GUI ... class browser, an object browser,
and a class hierarchy diagram.
Closer and closer …!
https://en.wikipedia.org/wiki/Integrated_development_environment

7. Software Development
Environment (3)
contains everything required by a team to build and
deploy software-intensive systems (where software is an
essential and indispensable element)
I approve!!
http://www.ibm.com/developerworks/rational/library/define-scope-development-
environment/index.html

8. What is “everything”?
Software development lifecycle (SDLC)
• Planning
• Designing
• Implementation, testing and documenting
• Deployment and maintenance
https://en.wikipedia.org/wiki/Software_development
!

9. What about “a team”?
Interacts with:
• Software project management - organisation,
coordination, covers: risk, change, software configuration
and release management
https://en.wikipedia.org/wiki/Software_project_management
• Software development process a.k.a. software
development methodology - e.g. waterfall, agile …
https://en.wikipedia.org/wiki/Software_development_process

10. My priorities
1. implement at least 90% of the most important
requested features
2. shortest possible time needed for “from request
to deploy” cycle
3. low-cost scaling - human/machines - initial
setup of DevEnv or running system
4. ensure high quality deliverables

11. !
now, the real stuff
!
btw. where has Django disappeared?

12. Initial setup - proj layout
~/env
├── proj-1
├── cli-1
├── …
├── biz
├── …
└── proj-N

13. Initial setup - proj layout
~/env/biz
├── archive
├── biz-web (GIT)
│ ├── ... (to be continued)
├── biz-data(GIT)
├── biz-ie (GIT)
├── doc -> ln -s ~/Dropbox/…/biz/doc
└── temp

14. Initial setup - python proj layout
~/env/biz/biz-web (GIT)
├── bin
├── doc (sphinx,rst: readme,todo,
│ install, ...)
├── pybiz (venv python installation)
├── reqs (base.txt, dev.txt)
├── src
│ ├── (to be continued)
│ ...
└── var
├── biz.sqlite
├── biz-app.log
├── biz-jobs.log
└── bulk-data

15. Initial setup - django proj layout
~/env/biz/biz-web/src
├── bizproj (global project folder - formats, settings)
├── bizjobs
│ ├── ...
├── bizapp
│ ├── management / commands
│ ├── migrations
│ ├── scripts
│ ├── templates
│ │ ├── biz
│ │ │ ├── components
│ │ │ ├── backoffice
│ │ │ ├── protected
│ │ │ ├── public
│ │ │ └── trial
│ │ ├── admin
│ │ ├── rosetta
│ └── templatetags
├── locale (de/en)
├── media
└── static (js, css, images, libs, fonts) # to bizapp?

16. Initial setup - django settings
~/env/biz/biz-web/src/bizproj/settings
!
base.py
from .secret import *
!
dev.py
dev_demo.py
!
test.py
!
secret.py # not in git repo
EMAIL_HOST_USER = "..."
DATABASES = {
!
secrete.py.template # is in git repo
!
dbrouter.py
testrunner.py

17. Initial setup - .gitignore
!
~/env/biz/biz-web/.gitignore
…
pybiz
var
src/media/*
secret.py
fabfile_settings.py
*.pyc
*.swp
*.pickle
*.sqlite
*.bak
…

18. File and directory naming
• very careful in naming !
• naming convention - inherit existing +
adaptations!
• very careful in choosing folder/structure!
• bigger files better than many files!
• less files/folders is better than more!

19. Project dependencies
• as small as possible - more components -> more complex -> more integration/
upgrade/other issues
• choose very carefully
• proven, stable, I/team is familiar with
• easily installable/upgradeable
• adaptable (when possible)
• when choosing new
• read documentation/read reviews/choose several/experiment/make
decision
• bigger community -> better

20. Project dependencies
• solutions
1. make a script to setup everything from the
scratch
2. install manually/reuse existing images and
then freeze state in VM, Vagrant, Docker
image
3. combine both

21. Project dependencies
• make a shell script
• two types of dependencies
• pip installable
pip install -r reqs/dev.txt
• pip non-installable
• set of shell commands to install everything that is missing
• Problem: different OS-es, e.g. OSX, Linux CentOS?
if [ $(uname) == 'Linux' ] …

22. Project deps - pip requirements
!
~/env/biz/biz-web/reqs
!
base.txt :
django>=1.5,<1.6
pillow
reportlab
johnny-cache
markdown
elasticsearch
beautifulsoup4
lxml
requests
celery[redis]
…
!
dev.txt :
-r base.txt
south
django_extensions
django-debug-toolbar
django-rosetta # for translation
mock
ipdb
…

23. Project deps - shell script
!
non-PIP installables
!
~/env/biz/biz-web/bin/biz # chmod u+x
!
case "$1" in
env)
if [ ! -d $BIZ_HOME/pybiz ]; then
cd $BIZ_HOME
sudo apt-get install python-devel
sudo apt-get install python-pip
sudo apt-get install python-xml
sudo apt-get install elasticsearch
...
pip install virtualenv
virtualenv pybiz
fi
!
(to be continued …)

24. Project deps - shell script
!
PIP installables
!
(continued …)
!
if [ -f $BIZ_HOME/pybiz/bin/activate ]; then
source $BIZ_HOME/pybiz/bin/activate
echo "pip=`which pip`, python=`which python`”
!
pip install -q --requirement=$BIZ_HOME/reqs/dev.txt
!
cd $BIZ_HOME/src/bizapp
export PYTHONPATH=$PYTHONPATH:$BIZ_HOME/src
echo "== env 'BIZ' initialized, checking base packages"
python -c"import django;
print(' Django - ok v%s' % django.get_version())"
python -c"import redis ;print(' redis - ok')"
python -c"import celery ;print(' celery - ok')"
python -c"import bizappp ;print(' biz app - ok’)"
…
fi
!
(to be continued …)

25. Project deps - shell script
!
check services
!
echo "== Is redis running?"
ps ax | grep -v grep | grep redis
!!
echo "== Is elasticsearch running?"
ps ax | grep -v grep | grep elasticsearch | awk "{print $1, $2,
$3, $4, $5, "..."}"
!
…

26. Project management shell script
• why?
• to have one central place where to do project management
• what? everything that needs automation
• setup initial environment, update environment
• open editor/IDE
• run tests
• deployment, remote management

27. Project management shell script
!
# biz
!
Usage:
!
*env - will initialize dev environment
!
redis - start redis in background
elas - start elasticsearch in background
cel - start celery worker in background
!
doc-update/du - will update doc/build/html wiki doc
doc - will open doc/build/html/index.html - home of wiki
!
log2 - will tail app log (if located in local var directory)
!
run/n - django-admin runserver
django/d - django-admin
sp - django-admin shell_plus
!
test-ping/tp - ping production
test/t - django-admin test --settings=test_settings
trans-make|tm - TRANSLATION - calls makemessages
!
r/remote - will call fabric script for deployment and remote operations
!
migrate|mig - do south migration - only dev system
mig-auto - create new south auto-migration - only dev system
!
browse|b - browse to localhost:8000
issues - browse to opened issues on bitbucket
!
* - should be called with source/.

28. Editor: ViM
• although steep learning curve, is:
• very adaptable
• unique features philosophy
• not in the way, very effective
• personal preference:
• I don’t feel comfortable in IDE
• I like to be close to “bare metal”
• to use editor designed for source code editing

29. • personal preference:
Editor: ViM
• I don’t like plugins too much, only .vimrc
• when working on the server to have the same possibilities
• configuration
• server
scp ~/.vimrc user@server:/home/user/
• local development machine
ln -s ~/Dropbox/env/.vimrc ~
• flavors
• gVim / MacVim - GUI, ViM - on every server

30. Development on the server
• with project shell script one can do directly on
the server
• devenv setup/upgrade
• webapp setup/upgrade
• vim & shell-script -> do “live” development
• run dev server

31. File browser
• personal preference:
• I like side by side file browsers
• usual file operations: create folders, check files,
search for files, zip etc.
• easy integration with editor (F4)
• not found “one-to-rule-them-all”
• TotalCommander, FreeCommander, DoubleCmd …

32. Idea evaluation
• evaluate a new idea or to test concept
• ipython / django shell_plus
• use reload:
from biz import models
models.test_something()
# change models and save
reload(models)
models.test_something()
• next step is - create a module/function

33. • I use a lot
Debugging
• plain-old pdb or sometimes ipdb
• conditional breakpoints:
if i==500:
import pdb; pdb.set_trace()
• conditional breakpoints (2):
try:
bad_bad_code()
except:
import pdb; pdb.set_trace()
• server side - logging, send emails on error
logger.error(“this will be send by email)
logger.error(“in try/except - will be send by email with traceback”)

34. • Unit tests
• integration tests
Testing
• “click-testing” - local development server
• automating http requests to staging/production
servers (from curl, nagios to dedicated services)
• “click-testing” - staging/production servers

35. Deployment
• must be simple, automated, fast and reliable
• usually:
• setup environment
• git push+pull / collectstatic / restart server
• analyse logs
• run integration tests on server instance
• example:
git ci -am”very important fix”
biz r test upgrade prod upgrade

36. Philosophy
• “Unix philosophy by Mike Gancarz” filtered:
Make each program do one thing well.
Use software leverage to your advantage.
Use shell scripts to increase leverage and
portability.
(btw. Store data in flat text files)
https://en.wikipedia.org/wiki/Unix_philosophy

37. Thanks for the patience!
Questions?
!
robert.lujo@gmail.com
@trebor74hr