This presentation describes different strategies for installing Python software packages. This includes standard techniques like easy_install and pip, as well as newer techniques like virtualenv that are well-suited for users who do not have administrative privileges.

1. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,
for the United States Department of Energy’s National Nuclear Security Administration
under contract DE-AC04-94AL85000.
Installing Python Software Packages:
The Good, The Bad and the Ugly
William E. Hart
Sandia National Laboratories
wehart@sandia.gov

2. Overview
This talks focuses on installing on Python packages
– Installing Python itself is usually a no-brainer
The Good
1. Installing on Windows with an installer executable
2. Installing with Linux application utility
3. Installing a Python package from the PyPI repository
4. Installing a Python package from source
The Bad
5. Using a virtual environment to isolate package installations
6. Using an installer executable on Windows with a virtual environment
The Ugly
7. Installing a Python extension package from source
8. PyCoinInstall – Managing builds for Python extension packages
Slide 2

3. About Python
Recent releases:
– 2.5 Very stable
– 2.6 Very stable
– 2.7 Recently released. Supports transition to Python 3.x
– 3.0 Includes changes that are not backwards compatible.
– 3.1 Stable 3.x release. Resolves 3.0 performance issues.
Package management for 3.x is still a bit snarky. We’ll focus on examples for
2.x
Slide 3

4. Python Packages
Package distributions
– Include one or more *.py files
– May include compiled source files (e.g. C++)
– Include a top-level setup.py file to coordinate installation
Three categories:
– Pure-Python packages
• Only include Python *.py files
– Independent extension packages
• Include compiled source files only on standard system libraries
– Dependent extension packages
• Include compiled source files that rely on third-party libraries
Slide 4

5. The Good
Assumption: you have administrative privileges
– Windows administrator group
– Linux super-user
Impact: you can install Python in system directories!
Idea:
– Install packages in the system Python directories
• Specifically, the site-packages directory is used
– Can easily install pure-Python packages or pre-compiled extension
packages
Slide 5

6. Example 1
Installing on Windows with an installer executable
Steps:
1.Download installer executable from package website
2.Execute installer
Notes:
– This uses the registry to find the Python installation
– The installer can contain pre-compiled extension packages
• These may differ for 32- and 64-bit installations
– The installer will likely be specific to the Python version
– This may work OK for dependent extension packages if the dependent
libraries are stored in a standard location
Slide 6

7. Example 2
Installing with Linux application utility
Idea: use a standard Linux package manager
– Ubuntu (and other Debian-based Linux variants)
• apt-get install packagename
– Fedora
• yum install packagename
Notes:
– The package manager can install pre-compiled extension packages
– The package manager handles package dependencies automatically
– Not all Python packages are available within these managers
– It’s unclear whether these packages are updated frequently
Slide 7

8. Example 3
Installing a Python package from the PyPI repository
PyPI:
– A web-based service for hosting Python packages
– Includes some pre-compiled extension packages
– Several Python packages are used to install from PyPI
• setuptools, distribute, pip
Installing with pip:
– pip install packagename
Notes:
– The package manager handles package dependencies automatically
– Most Python packages are available from PyPI, including most releases
– PyPI does not include many pre-compiled extension packages
– The PyPI service has become a bit overloaded in the past year
– You need to specify the HTTP_PROXY environment to talk to proxy servers
Slide 8

9. Example 3 (cont)
Problem: pip is not a standard Python package!
Installing pip:
– wget http://peak.telecommunity.com/dist/ez_setup.py
– python ez_setup.py
– easy_install pip
Notes:
– pip is now generally recommended over setuptools and distribute
– The wget command is not a standard MS Windows utility
• Download from http://users.ugent.be/~bpuype/wget/
Slide 9

10. Example 4
Installing a Python package from source
Steps:
– Download package source from PyPI or project website
– Extract package from tarball or zipfile
– python setup.py install
Notes:
– This process avoids the need to install pip
– This is useful for packages that are not available on PyPI
– This installation process can build and install some extension packages
• Configuration of the compiler may be a problem
• Configuration of extension package dependencies is a problem
– This installation does not handle package dependencies
• Package dependencies may cause an install to fail
Slide 10

11. The Bad
Problem: users do not have administrative privileges
– This is not uncommon in industry
Problem: different users on a computer need different package versions
Idea:
– A user can create a virtual Python environment
– The user installs packages in this virtual Python directories
Impact:
– Users have complete control over their Python environment
• Can even have multiple environments (e.g. for developers)
– Can install both pure-Python packages or pre-compiled extension
packages
Slide 11

12. Virtual Python
Idea: create an isolated Python environment
virtualenv:
– Creates a directory structure that mimics the system Python installation
– Works on MS Windows, Linux and Mac OS
Installing virtualenv (with administrative privileges)
– Debian Linux
• apt-get install python-virtualenv
– Other
• wget http://peak.telecommunity.com/dist/ez_setup.py
• python ez_setup.py
• easy_install virtualenv
Slide 12

13. Virtual Python (cont)
Creating a virtual Python environment:
– virtualenv directory
Virtual Python executable:
– MS Windows: directory/Scripts/python.exe
– Linux: directory/bin/python
Notes:
– On MS Windows the executables are installed in the Scripts directory
• For simplicity, all subsequent examples will use the bin directory
– The setuptools package is installed in a virtual environment
• Can use the easy_install command to install packages
Example:
– directory/bin/easy_install pip
Slide 13

14. Virtual Python (cont)
Problem: setup a virtual environment without administrative privileges
vpy_install:
– Bootstrap a virtualenv installation from a script
Steps:
– wget http://doiop.com/pyutilib/vpy_install
– python vpy_install directory
Notes:
– Standard tools like setuptools and pip are installed by vpy_install
– The specified directory is a virtualenv Python environment
– On Linux, you can use a simpler command-line:
• vpy_install directory
– The –-site-packages option can be used to expose packages that are
installed in the system Python installation
• This is particularly useful when using precompiled extension packages,
which are installed in system Python directories
Slide 14

15. Example 5
Using a virtual environment to isolate package installations
Idea:
– Create a virtual python installation with vpy_install
– Install packages with pip
Example (Linux):
– wget http://doiop.com/pyutilib/vpy_install
– vpy_install mypy
– mypy/bin/pip install packagename
Notes:
– Within the virtual environment, pip is installed as an executable
• You do not need to execute it with the Python executable
Slide 15

16. Example 5 (cont)
Using a virtual environment to isolate package installations
Note: source package installations using a virtual Python executable are
installed in the virtual environment
Example (Linux):
– wget http://doiop.com/pyutilib/vpy_install
– vpy_install mypy
– cd packagename
– ../mypy/bin/python setup.py install
Slide 16

17. Example 6
Using an installer executable on Windows with a virtual environment
Problem:
– MS Windows installers use the system Python installation
– If a user does not have administrative privileges, then an installer cannot
be used directly!
Idea:
– “Install” the executable with the easy_install command
Example:
– easy_install.exe matplotlib-1.0.0.win32-py2.6.exe
Note: this only works if the executable was created by the Python distutils
package!
Slide 17

18. Example 6 (cont)
Using an installer executable on Windows with a virtual environment
Idea:
– Modify the MS Windows registry to register the virtual python executable
Steps:
1.Download the the_python.py script that is described at
http://doiop.com/nedbatchelder/register_python.html
2.Register the virtual python executable (as administrator)
• mypyScriptspython.exe the_python.py
1.Execute the MS Windows installer
2.Reregister the original python executable (as administrator)
Slide 18

19. The Ugly
Problem:
– you need to use an extension package that has dependencies
– prebuilt packages are not available
Observation:
– There is very little consistency between installation setup for different
packages
– Few packages use configuration management tools like autotools or
cmake
– Build instructions can be very different for different platforms
There are few options beyond diving deep into the package build process!
Slide 19

20. Example 7
Installing a Python extension package from source
Note: there is no standard way for doing this
Example: SciPy documentation
– General build instructions
– Separate instructions for Linux, Mac OS, Windows, and Solaris
Recommendations:
– Make sure that this package is going to be worth your time
– Don’t hesitate to ask the developer team for help
– Have patience ...
Slide 20

21. Example 8
PyCoinInstall – Managing builds for Python extension packages
Note: this script is a relatively new project within CoinBazaar
– https://projects.coin-or.org/CoinBazaar/wiki/Projects/PyCoinInstall
Idea:
– Create a virtual environment with virtualenv
– Define package installation with a simple plugin system
• Packages define download, build and/or install operations
– A user creates a simple configuration file to specify the packages that will be
installed
– The pci command is used to manage the installation
Notes:
– Current development is focused on COIN-OR related packages
– Current plugins are tailored for building under Linux
Slide 21

22. Final Thoughts
Installing and distributing pure-Python packages is quite easy.
Installing extension packages is much more difficult.
– Try to leverage pre-compiled extension packages!
Virtual Python environments offer a flexible paradigm for managing different
Python configurations
– These are worth using even if you have administrative privileges!
Slide 22