Using Virtual Environments - Python I

In a previous post I gave virtual environments as a solution to the problem of dependency and version collision for software built using languages like Python, Ruby, Node.js, and Go.

In this post I give the details and examples of using the Python virtual environment. First I walk through everything using OS X–i.e., using a Mac–and then, briefly, I describe the same procedures using Windows.

• Python installation (OS X)
• Creating a python virtual environment
• Activating and deactiving the virtual environment
• Virtual environment creation in Python 3.3 and later
• Adding packages to the virtual environment
• Key Features of virtual environments
• Next steps
• Python virtual environments on Windows
  • (Windows) Installing Python
• Next steps (reiterated)

Python installation (OS X)

Python is, typically, installed side by side on a system at the granularity of major and minor releases. For example, on my OS X system I use the package manager¹ MacPorts to install new versions of Python. MacPorts installs all its software under the top level directory /opt. Here are all the Python versions currently installed on my OS X system via MacPorts:

~$ ls /opt/local/Library/Frameworks/Python.framework/Versions/
2.7 3.3 3.4 3.5

OS X comes with Python already installed. I’m still running Mavericks, a.k.a. OS X 10.9.5, and that ships with Python 2.7.5. Usually Apple does not update the system version of Python, and so my system will remain at 2.7.5 (circa 2013) as long as I keep it running Mavericks (also circa 2013).

As you can tell from the directly listing above, I’ve used MacPorts to independently install Python versions 2.7, 3.3, 3.4, and 3.5. Another way to see that is by asking MacPorts to list the Python packages:

~$ port installed python*
The following ports are currently installed:
  python2_select @0.0_1 (active)
  python3_select @0.0_1 (active)
  python27 @2.7.11_0 (active)
  python33 @3.3.6_5 (active)
  python34 @3.4.4_0 (active)
  python35 @3.5.1_0 (active)
  python_select @0.3_6 (active)

I installed each of those using a command like:

~$ sudo port install python27

or

~$ sudo port install python35

The various python_select packages come along with any MacPorts Python package. They provide a command you can use to change the default python package to one selected from the various versions you’ve installed via MacPorts. I don’t use python_select because I find using virtual environments to be a better mechanism.

Creating a python virtual environment

The Python port maintainers at MacPorts are pretty good about upgrades, and so the Python 2.7 installed in the /opt tree is likely to be recent. To check that, let’s create a virtual environment for Python 2.7 from MacPorts.

1. Open a terminal window and create an empty directory somewhere–I called mine “using-python-venv”–and cd into it.

2. Check the path to the Python binary and the Python version. This will be whatever is the default on your system.

   ~/using-python-venv$ which python
   /usr/bin/python
   ~/using-python-venv$ python --version
   Python 2.7.5

3. Save a record of all the current environment variables (so we can see what changes when the virtual environment is activated) using this command:

   ~/using-python-venv$ env > orig.env.txt

4. Create the virtual environment deployment directory using this command:[^venvdir]

   ~/using-python-venv$ /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenv venv27

   Note: the path will be different with Fink or Homebrew. If you are using either of those package managers, you’ll need to locate the Python 2.7 installation directory to be used in the command.

Activating and deactiving the virtual environment

1. Activate the virtual environment using this “source” command:

   ~/using-python-venv$ . venv27/bin/activate
   (venv27)~/using-python-venv$

   Notice that the command prompt now has a parenthetical prefix that shows the active virtual environment deployment directory.

2. Save a record of the new environment variables:

   (venv27)~/using-python-venv$ env > venv27.env.txt

3. Check the python path and version version

   (venv27)~/using-python-venv$ which python
   /Users/neo/using-python-venv/venv27/bin/python
   (venv27)~/using-python-venv$ python --version
   Python 2.7.11

4. Deactivate the virtual environment using the deactivate command:

   (venv27)~/using-python-venv$ deactivate
   ~/using-python-venv$

   Notice that the command prompt changed to no longer shows the virtual environment deployment directory. And if you try the deactivate command again, you’ll get a command not found error.

5. Diff the original (default) environment against the new environment to see what changed when the Python virtual environment was activated:

   ~/using-python-venv$ diff orig.env.txt venv27.env.txt

   Here’s the diff output looked like on my system; I used the command diff orig.env.txt venv27.env.txt:

   13a14
   > VIRTUAL_ENV=/Users/neo/using-python-venv/venv27
   15c16
   < PATH=/Users/neo/.rbenv/shims:/Users/neo/bin:/Users/neo/local/bin:/xpt/local/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/texbin:/Applications/git-annex.app/Contents/MacOS
   ---
   > PATH=/Users/neo/using-python-venv/venv27/bin:/Users/neo/.rbenv/shims:/Users/neo/bin:/Users/neo/local/bin:/xpt/local/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/texbin:/Applications/git-annex.app/Contents/MacOS
   21c22
   < PS1=\[\033[1;36m\]$(parse_git_branch)\[\033[1;32m\]\w$\[\033[0m\]\n
   ---
   > PS1=(venv27)\[\033[1;36m\]$(parse_git_branch)\[\033[1;32m\]\w$\[\033[0m\]\n

   As you can see, activating the virtual environment has an effect. A new environment variable, VIRTUAL_ENV, is created and identifies the location of the Python deployment directory. The $PATH is changed to have the virtual Python environment searched first. And the command line prompt is changed to indicate which Python virtual environment deployment directory is active.

Virtual environment creation in Python 3.3 and later

The virtual environment mechanism is built in to the Python Library in Python version 3.3. and later, in the venv module. The command to create the virtual environment using the venv module would be:

~/using-python-venv$ /opt/local/Library/Python.framework/Versions/3.3/bin/python3 -m venv venv33

However, on my Mac using the MacPorts distribution of Python 3.3, the virtual environment that is created has no packages whatsoever. Specifically, it does not have setuptools or pip, and both those tools are immensely useful.²

So I continue to use the virtualenv command, even with Python 3.3 and later, when on my Mac.³ The command to create a 3.3 virtual environment that way is:

~/using-python-venv$ /opt/local/Library/Frameworks/Python.framework/Versions/3.3/bin/virtualenv venv33

Activating the 3.3 virtual environment created with virtualenv, and using the pip list command confirms the presence of setuptools and pip (albeit slightly outdated).

Adding packages to the virtual environment

When the virtual environment is active you can install packages an not be concerned about polluting the global Python environment or breaking dependencies for any other project or any application installed on the system.

The first thing I usually do after creating a new virtual environment is to activate it and update setuptools and pip.

~/using-python-venv$ . venv33/bin/activate
(venv33)~/using-python-venv$ pip list
pip (1.4.1)
setuptools (0.9.8)
(venv33)~/using-python-venv$ pip install --upgrade setuptools
(venv33)~/using-python-venv$ pip install --upgrade pip
venv33)~/using-python-venv$ pip list
pip (8.0.2)
setuptools (20.0)
(venv33)~/using-python-venv$

And now you can install any other packages you need.

Key Features of virtual environments

With these basic steps I have demonstrated the key characteristics of a virtual environment for a project or application. These characteristics are common to all virtual environments, not just for Python.

Virtual environments have:

• A local, mutable, deployment directory for the execution environment that is independent of the global (system) environment.

  With Python, this is accomplished with the virtualenv utility or the venv module to create and provision a local deployment directory.

• An activation mechanism for standing up the virtual environment—for switching an execution context to use the local deployment directory preferentially ahead of the global environment, or masking the global environment.

  With Python virtual environments, this is accomplished using the activate script⁴ to change the search path to look in the local deployment directory before looking in other locations. The acivate script also generates the deactivate script, changes the CLI prompt to signal that the virtual environment is activate, and adds a special VIRTUAL_ENV environment variable to aid virtual environment support in Python itself and in project or application scripting.

• A deactivation mechanism for standing down the virtual environment—for switching the execution context back to using the global environment and ignoring the deployment directory.

  As mentioned above, Python virtual environments generate a deactivate script to accomplish this. The script self-deletes when executed.

Next steps

In the next post I’ll discuss how to do upgrades of dependencies to pick up patches without picking up major or minor upgrades, how to test for the correct virtual environment and dependency versions in script (e.g. test scripts, build scripts), and how to provision the virtual environment from a list of known dependencies.

Python virtual environments on Windows

Using Python on Windows is only slightly different, but worth describing. In this post I’ll use the standard command window. In the next post I’ll switch to Powershell.

(Windows) Installing Python

Python install kits are available here: https://www.python.org/downloads/windows/. I’ll be using only the 64-bit installers in these examples.

Python 2 and older Python 3 installers use a root directory as the default install location (e.g. C:\Python27 or C:\Python33). Starting with Python 3.5, the installer places Python in the \Program Files directory. I like that; it’s in keeping with traditional Windows software installation practices. So when I use the older installers, I change the install directory to match the new convention.

Here’s how to install Python 2.7, create a virtual environment, activate it, update it, and deactivate it from a Command window.

1. Install Python 2.7.11 from the python-2.7.11.amd64.msi installer.

2. When prompted for the install path, set it to C:\Program Files\Python 2.7.

3. After installing, open a command window using “Run as administrator …” and enter these commands:

   C:\>"C:\Program Files\Python 2.7\python.exe" -m pip install --upgrade pip
   C:\>"C:\Program Files\Python 2.7\python.exe" -m pip install --upgrade setuptools
   C:\>"C:\Program Files\Python 2.7\python.exe" -m pip install virtualenv

4. Open a new command window (without using “Run as administrator …”) and create a virtual environment inside a new directory using:

   C:\>mkdir C:\test
   C:\>cd C:\test
   C:\test>mkdir using-python-venv
   C:\test>cd Using-python-venv
   C:\test\using-python-venv>"C:\Program Files\Python 2.7\Scripts\virtualenv.exe" venv27

5. Activate the virtual environment using:

   C:\test\using-python-venv>venv27\Scripts\activate

6. Check the version and upgrade setuptools and pip:

   (venv27) C:\test\xx-yy-zz>python --version
   Python 2.7.11
   (venv27) C:\test\xx-yy-zz>pip list
   pip (8.0.2)
   setuptools (20.0)
   wheel (0.29.0)
   (venv27) C:\test\xx-yy-zz>pip install --upgrade setuptools
   (venv27) C:\test\xx-yy-zz>pip install --upgrade pip
   (venv27) C:\test\xx-yy-zz>pip list
   pip (8.0.2)
   setuptools (20.0)

7. Deactivate the virtual environment using:

   (venv27) C:\test\xx-yy-zz>deactivate
   C:\test\xx-yy-zz>

Installing Python 3.3 and later is similar, except that the virtual environment is created using the venv module.

1. Install Python 3.4.4 from the python-3.4.4.amd64.msi installer.

2. When prompted for the install path, set it to C:\Program Files\Python 3.4.

3. After installing, open a command window using “Run as administrator …” and enter these commands:

   C:\>"C:\Program Files\Python 3.4\python.exe" -m pip install --upgrade pip
   C:\>"C:\Program Files\Python 3.4\python.exe" -m pip install --upgrade setuptools

4. Open a new command window (without using “Run as administrator …”) and create a virtual environment inside a new directory using:

   C:\>cd C:\test\using-python-venv
   C:\test\using-python-venv>"C:\Program Files\Python 3.3\Scripts\python.exe" -m venv venv34

5. Activate the virtual environment using:

   C:\test\using-python-venv>venv34\Scripts\activate.bat

6. Check the version and upgrade setuptools and pip:

   (venv34) C:\test\xx-yy-zz>python --version
   Python 3.4.4
   (venv34) C:\test\xx-yy-zz>pip list
   pip (7.1.2)
   setuptools (18.2)
   You are using pip version 7.1.2, however version 8.0.2 is available.
   You should consider upgrading via the 'python -m pip install --upgrade pip' command.
   (venv34) C:\test\xx-yy-zz>python -m pip install --upgrade setuptools
   (venv34) C:\test\xx-yy-zz>python -m pip install --upgrade pip
   (venv34) C:\test\xx-yy-zz>python -m pip list
   pip (8.0.2)
   setuptools (20.0)

7. Deactivate the virtual environment using:

   (venv34) C:\test\xx-yy-zz>deactivate
   C:\test\xx-yy-zz>

Next steps (reiterated)

In the next post I’ll discuss how to do upgrades of dependencies to pick up patches without picking up major or minor upgrades, how to test for the correct virtual environment and dependency versions in script (e.g. test scripts, build scripts), and how to provision the virtual environment from a list of known dependencies.