Using Virtual Environments - Python II

In the last post I explained how to install multiple versions of Python, how to create Python virtual environments, and how to activate and deactivate those virtual environments.

I want to dive a little deeper now, and start building up some support for automation in a Python virtual environment.¹

• About the example
• Package management automation
• … with pip
• … without pip
• Automating other actions
• The example using Python 3.5
• Package management automation on Windows

About the example

I’m going to use an example development project that builds a website using Flask and Redis. I imagine a historical version of the project that used Python 2.7 with the packages Flask 0.8 and redis 2.6.² I imagine the current version of the project that uses Python 3.5 with the packages Flask 0.10.1 and redis 2.10. The scenario I imagine is that the developers still support the historical version, but active development is on the current version.

I’ve made the example with all the scripts in this post available in a GitHub repository called using-virtual-environments. Use git checkout example-venv27 for the Python 2.7 example files, and git checkout example-venv35 for the Python 3.5 example files. There are some scripts in that repo that I won’t discuss, like the scripts to create and initialize the 2.7 and 3.5 virtual environments; the full set of scripts may be worth investigating on your own.

Package management automation

As discussed in previous posts, a major benefit of virtual environments is the isolation from changes in package installation and upgrades. Package installation changes made globally on the system, for other projects and applications, won’t impact the local virtual environment.

Another benefit is the ability to control changes to packages installed in the local virtual environment. For most development projects it is important to control which versions of packages are installed and to control when and whether to accept patches or upgrades to those packages.

What is needed are some scripts that:

• install the required versions of the necessary packages and their dependent packages
• apply patches to the necessary packages and their dependencies while maintaining the version constraints
• validate the virtual environment, checking that:
  • a virtual environment is active
  • the virtual environment is using the correct version of Python
  • the virtual environment holds the necessary packages and their dependencies at the required versions

… with pip

Although some Python packages cannot be installed using pip, most are. And pip has a nice feature called requirements files that can be used to direct which packages are installed and constrain the versions of those packages.

You can create a requirements file by installing some packages and then issuing the command: pip freeze > requirements.txt. If I activate a Python 2.7 virtual environment³ and install the Flask and redis packages, and then issue that pip freeze command, the requirements file will look something like this:

Flask==0.10.1
itsdangerous==0.24
Jinja2==2.8
MarkupSafe==0.23
redis==2.16.1
Werkzeug==0.11.3

pip will generally install the latest version of each package, and the list above does indeed show the latest versions (as of this writing). In the example I’m using, however, the older version of the project that is built using Python 2.7 is using older versions of Flask and redis and their dependent components, circa 2012-2013 or so. To specify those versions⁴—and to constrain the packages to those versions—I create a requirements file, requirements-venv.txt, that looks like this:

# redis and its dependencies
#
redis>=2.6,<2.7

# Flask and its dependencies
#
MarkupSafe==0.23
Werkzeug>=0.6,<0.7
Jinja2>=2.4,<2.5
Flask>=0.8,<0.9

The greater than and less than constraints⁵ will hold the packages to a particular major and minor version, but will also allow upgrades to the latest patch version (e.g. Flask 0.8.1 upgraded to 0.8.2).

We can now make two scripts; one to provision the virtual environment with the correct packages, and another to upgrade the packages to pick up available patches (without violating version constraints). The provisioning script would be used when setting up the development environment, and the upgrade script would be used on occasion as patches become available or simply periodically to stay current with patches.

The provisioning script, provision-venv.sh, looks like this:

#! /bin/bash
#
# These scripts assumes that Python 2.7 is installed via Mac Ports.
# If Python 2.7 is installed some other way, then the scripts will
# need some adjustment.
#

SCRIPTNAME_="$0"

# Get the directory holding this script
# (method from: http://stackoverflow.com/a/12694189/1392864)
#
SCRIPTDIR_="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi

# Pick up the verification functions
#
. "${SCRIPTDIR_}/check-functions.src"

# Check that we are running in a Python 2.7 virtual environment
#
. "${SCRIPTDIR_}/check-active-venv.src"

# Install required packages
#
pip install -q -r requirements-venv.txt

# Check that we have the needed packages
#
. "${SCRIPTDIR_}/check-dependencies.src"

# Done!
#
echo ""
echo -n "OK. Provisioned required packages for project in "
echo "Python 2.7 virtual environment."
echo ""

The core of the script is, of course, the pip install command. But before running that command, the script checks that the expected virtual environment exists and is active. After running the command, the script verifies that the expected versions of the required packages are in place.

That kind of checking is key to successful automation. You want to declare and validate as many assumptions as possible to avoid stupid mistakes and undetected problems. The worst kind of error is one that misleads you into relying on an invalid assumption; that can lead to costly surprised down the road.

You might want to pull the example repo from GitHub, checkout the example-venv27 branch, and have a look at the three sourced scripts used to validate assumptions and results:

• python/check-functions.src
• python/check-active-venv.src
• python/check-dependencies.src

The upgrade script is similar; it uses the --upgrade option on the pip install command. The script, upgrade-venv.sh, looks like this:

#! /bin/bash
#
# These scripts assumes that Python 2.7 is installed via Mac Ports.
# If Python 2.7 is installed some other way, then the scripts will
# need some adjustment.
#

SCRIPTNAME_="$0"

# Get the directory holding this script
# (method from: http://stackoverflow.com/a/12694189/1392864)
#
SCRIPTDIR_="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi

# Pick up the verification functions
#
. "${SCRIPTDIR_}/check-functions.src"

# Check that we are running in a Python 2.7 virtual environment
#
. "${SCRIPTDIR_}/check-active-venv.src"

# Check that we have the needed packages
#
. "${SCRIPTDIR_}/check-dependencies.src"

# Upgrade the required packages
#
pip install --upgrade -q -r requirements-venv.txt

# Check that we still have the needed packages
#
. "${SCRIPTDIR_}/check-dependencies.src"

# Done!
#
echo ""
echo -n "OK. May have upgraded required packages for project in "
echo "Python 2.7 virtual environment."
echo ""

… without pip

Some dependencies might not be installed via pip. NumPy is an example; it must be built from source. The Redis server is another example. Normally packages that are not installed via pip are added to the virtual environment in this way:

1. build from source
2. install to project subdirectory, perhaps the virtual environment deployment directory
3. create a custom activation script that adds the installed dependencies to the PATH and activates the Python virtual environment
4. create a custom deactivation script that deactivates the Python virtual environment and resets the PATH

Sometimes multiple versions of these non-pip dependencies can be installed side-by-side. If that’s the case then you can use the normal installers, or system package managers like Mac Ports, to install multiple versions of those applications or libraries and point the current context to the appropriate version in the custom activation script.

You should also add validation checks in the check-dependencies.src script to ensure the non-pip dependencies were present, correct, and available. That, again, is key to avoiding errors or mistakes that will, sooner or later, cause trouble that could have been easily avoided from the start.

I’m not going to give a detailed example here. As this series on automating virtual environments proceeds, there will be other opportunities to give examples of custom activation scripts.

Automating other actions

In a development project there are other actions that could be automated, such as build and test scripts. They will need similar validation of the virtual environment. You can use the same validation approach in those scripts as was done for the provision-venv.sh script.

The example using Python 3.5

There is little difference between the provisioning and upgrade scripts for the historic release (on Python 2.7) and the current release (on Python 3.5). The scripts are essentially the same except for the versions of Python and the packages, and the mix of packages has changed slightly, and a slightly different command for creating the Python 3.5 virtual environment.

Here is what the requirements-venv.txt file looks like for the project’s current version:

# redis and its dependencies
#
redis>=2.10,<2.11

# Flask and its dependencies
#
Werkzeug>=0.11,<0.12
MarkupSafe==0.23
Jinja2>=2.8,<2.9
itsdangerous==0.24
Flask>=0.10,<0.11

Note that Flask has a new dependency, itsdangerous. Otherwise, only the version numbers changed.

To access and review the example project’s current version automation scripts, pull the example repo from GitHub, and checkout the example-venv35 branch.

Package management automation on Windows

Finally, I present Windows examples of the same commands and scripting used to automate creation, provisioning, and upgrading of the virtual environments. I use Powershell to do so because it provides the capabilities I need for validation of the virtual environment.

Restating what I wrote near the beginning of this post: I’ve made the example with all the scripts in this post available in a GitHub repository called using-virtual-environments. Use git checkout example-venv27 for the Python 2.7 example files, and git checkout example-venv35 for the Python 3.5 example files. There are some scripts in that repo that I won’t discuss, like the scripts to create and initialize the 2.7 and 3.5 virtual environments; the full set of scripts may be worth investigating on your own.

The provisioning script, provision-venv.ps1, looks like this:

# This script assumes that Python 2.7 is installed in
# C:\Program Files\Python 2.7
#
# If Python 2.7 is installed some other way, then the script will
# need some adjustment.
#
Set-Strictmode -version Latest

# Capture the file name of this powershell script
#
$scriptName_ = $MyInvocation.InvocationName

$scriptDir_ = $PSScriptRoot

# Set error code
#
$errcode_ = 0

# Pick up the verification functions
#
. "$scriptDir_\check-functions.ps1"

# Check that we are running in a Python 2.7 virtual environment
#
. "$scriptDir_\check-active-venv.ps1"
If (0 -ne $errcode_) {
  Exit
}

# Install required packages
#
$rcmd_ = "pip"
$rargs_ = "install -q -r requirements-venv.txt" -split " "
Invoke-Expression "$rcmd_ $rargs_"

# Check that we have the needed packages
#
. "$scriptDir_\check-dependencies.ps1"
If (0 -ne $errcode_) {
  Exit
}

# Done!
#
Write-Output ""
Write-Output ("OK. Provisioned required packages for project in " `
  + "Python 2.7 virtual environment.")
Write-Output ""

And the upgrade script, upgrade-venv.ps1, looks like this:

# This script assumes that Python 2.7 is installed in
# C:\Program Files\Python 2.7
#
# If Python 2.7 is installed some other way, then the script will
# need some adjustment.
#
Set-Strictmode -version Latest

# Capture the file name of this powershell script
#
$scriptName_ = $MyInvocation.InvocationName

$scriptDir_ = $PSScriptRoot

# Set error code
#
$errcode_ = 0

# Pick up the verification functions
#
. "$scriptDir_\check-functions.ps1"

# Check that we are running in a Python 2.7 virtual environment
#
. "$scriptDir_\check-active-venv.ps1"
If (0 -ne $errcode_) {
  Exit
}

# Check that we have the needed packages
#
. "$scriptDir_\check-dependencies.ps1"
If (0 -ne $errcode_) {
  Exit
}

# Install required packages
#
$rcmd_ = "pip"
$rargs_ = "install --upgrade -q -r requirements-venv.txt" -split " "
Invoke-Expression "$rcmd_ $rargs_"

# Check that we still have the needed packages
#
. "$scriptDir_\check-dependencies.ps1"
If (0 -ne $errcode_) {
  Exit
}

# Done!
#
Write-Output ""
Write-Output ("OK. May have upgraded required packages for project in " `
  + "Python 2.7 virtual environment.")
Write-Output ""