Using Virtual Environments

In the last post I explained how to install multiple versions of Ruby, how to create Python virtual environments, and how to activate and deactivate those virtual environments. I also demonstrated how to provision a Ruby virtual environment with the necessary gems.

In this post I show how to fully automate the creation of Ruby virtual environments, to automate provisioning, activation and deactivation. I also cover updating the gems within version constraints.¹

About the example
Automating virtual environment management
Script source code
Automating the creation of the virtual environment
Automating virtual environment activation and deactivation
Automating virtual environment provisioning
- Gemfile
Automating gem upgrades
Automating other actions
The example using Ruby 2.2.3
Automation of a Ruby virtual environment on Windows

About the example

In the previous post I described² a website example. To author the CSS used on that website I want to use Sass and it’s companion Compass—both written in Ruby and available as gems.

I’m going to show automation scripts that use Ruby 2.0.0p598 and gems constrained to versions circa June 2011. Then I’ll show how to upgrade the gems to versions circa March 2012 or so. And finally, I’ll show automation scripts that use Ruby 2.2.3p173 (the latest Ruby as of this writing) and very recent gem versions.

Automating virtual environment management

When working on a project I don’t want to fumble around trying to figure out how to create a virtual environment with the correct version of Ruby and provision it with the needed gems at the required versions.

What I want is to be able to pull a fresh source tree from source control (GitHub, Atlassian, a private server) and:

run a script that automatically creates the correct virtual environment,
run another script to activate the virtual environment,
run another script to provision the virtual environment with the gems required by the project.

I also want my command prompt to indicate that a virtual environment is active, and I want:

a script to deactivate the virtual environment, and
a script to upgrade gems to pick up compatible patches and updates

These scripts should validate that the correct and complete environment expected is in place before taking any actions to modify or manage that environment.

Those capabilities should be considered standard features of virtual environments. For a clear statement of the principle requirements of a virtual environment see “Package management automation”.

Script source code

I’ve made the example with all the scripts in this post available in a GitHub repository called using-virtual-environments. Use git checkout example-ruby200 for the Ruby 2.0.0p598 example files, and git checkout example-ruby223 for the Ruby 2.2.3p173 example files. There are some scripts in that repo that I won’t discuss, like the scripts that contain helper functions (like functions to parse and compare version strings); the full set of scripts may be worth investigating on your own.

Automating the creation of the virtual environment

On OS X, the script to make the virtual environment is called make-venv.sh. It performs the following steps:

Check whether rbenv is installed
Check the rbenv version
Check the rbenv plugins
If necessary, reset the local ruby version maintained by rbenv
Select the required Ruby version
Check whether rbenv has it available
Prompt for install if that version is not already in the rbenv cache
Set the local Ruby version
Set the gemset directory
Install the bundler gem

Here is what the script to automate the creation of a Ruby 2.0.0p598 virtual environment looks like (and latest version is available in the Git repo):

#! /bin/bash
#
# This script assumes that rbenv 0.4.0 is installed.
#

SCRIPTNAME_="$0"

# Get the directory holding this script
# (method from: http://stackoverflow.com/a/12694189/1392864)
#
SCRIPTDIR_="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi

# Pick up the verification functions
#
. "${SCRIPTDIR_}/check-functions.src"

# Check whether rbenv is installed
#
`which -s rbenv`
RBENV_IN_PATH_=$?
if (( 0 != RBENV_IN_PATH_ )); then
  echo "ERROR: rbenv is not installed."
  echo "Install using 'sudo port install rbenv'."
  exit 2
fi

# Check the rbenv version
#
RBENV_VSN_EXP_="1.0.0"
RBENV_VSN_MAX_="1.1.0"
RBENV_VSN_=`rbenv -v`
parse_version "${RBENV_VSN_}"
if (( 0 != $? )); then
  echo "ERROR: failed to parse RBENV_VSN_"
  exit 2
fi
RBENV_VSN_PARTS_=( ${parse_version[@]} )
RBENV_VSN_="${RBENV_VSN_PARTS_[0]}"
RBENV_VSN_="${RBENV_VSN_}.${RBENV_VSN_PARTS_[1]}"
RBENV_VSN_="${RBENV_VSN_}.${RBENV_VSN_PARTS_[2]}"
version_is_at_least "${RBENV_VSN_EXP_}" "${RBENV_VSN_}"
RBENV_VSN_MIN_OK_=${version_is_at_least}
version_is_less_than "${RBENV_VSN_MAX_}" "${RBENV_VSN_}"
RBENV_VSN_MAX_OK_=${version_is_less_than}
if (( 0 == ${RBENV_VSN_MIN_OK_} )) \
  || (( 0 == ${RBENV_VSN_MAX_OK_} )); then
  echo
  echo -n "ERROR: Expecting rbenv ${RBENV_VSN_EXP_} or later, "
  echo "up to ${RBENV_VSN_MAX_}."
  echo "Found ${RBENV_VSN_}"
  echo
  exit 10
fi

# Check the plugins
#
RBENV_REQUIRED_PLUGINS_=( "rbenv-gemset" "ruby-build" )
RBENV_PLUGINS_=( $HOME/.rbenv/plugins/* )
for (( i = 0; i < ${#RBENV_REQUIRED_PLUGINS_[@]}; i++ ))
do
  FOUND_=0
  for (( k = 0; k < ${#RBENV_PLUGINS_[@]}; k++ ))
  do
    PLUGIN_=`basename ${RBENV_PLUGINS_[k]}`
    if [[ "${RBENV_REQUIRED_PLUGINS_[i]}" == "${PLUGIN_}" ]]; then
      FOUND_=1
      break
    fi
  done
  if (( 0 == ${FOUND_} )); then
    echo
    echo "ERROR: Missing rbenv plugin ${RBENV_REQUIRED_PLUGINS_[i]}."
    exit 8
  fi
done

# Unset the local rbenv ruby
#
if [[ -f "${SCRIPTDIR_}/.rbenv" ]]; then
  rbenv local --unset
fi

# Determine the version of Ruby we want
#
RUBY_VERSION_="2.0.0p598"
parse_version "${RUBY_VERSION_}"
if (( 0 != $? )); then
  echo "ERROR: failed to parse RUBY_VERSION_"
  exit 2
fi
RUBY_VERSION_PARTS_=( ${parse_version[@]} )
RBENV_RUBY_VERSION_="${RUBY_VERSION_PARTS_[0]}"
RBENV_RUBY_VERSION_="${RBENV_RUBY_VERSION_}.${RUBY_VERSION_PARTS_[1]}"
RBENV_RUBY_VERSION_="${RBENV_RUBY_VERSION_}.${RUBY_VERSION_PARTS_[2]}"
if [[ ! -z "${RUBY_VERSION_PARTS_[3]}" ]]; then
  RBENV_RUBY_VERSION_="${RBENV_RUBY_VERSION_}-${RUBY_VERSION_PARTS_[3]}"
fi

# Check if the Ruby version we want is available
#
is_this_ruby_available "${RBENV_RUBY_VERSION_}"
if (( 0 != $? )); then
  echo "ERROR: failed to parse RBENV_RUBY_VERSION_"
  exit 2
fi
FOUND_=$is_this_ruby_available

# If not found, can it be installed? should it be installed?
#
if (( 0 == $FOUND_ )); then
  FOUND_=0
  AVAILABLE_VERSIONS_=( `rbenv install --list` )
  for (( i - 0 ; i < ${#AVAILABLE_VERSIONS_[@]} ; i++ ))
  do
    VSN_="${AVAILABLE_VERSIONS_[$i]}"
    parse_version "${VSN_}"
    if (( 0 != $? )); then
      continue;
    fi
    VSN_PARTS_=( ${parse_version[@]} )
    if [[ "${RUBY_VERSION_PARTS_[@]}" == "${VSN_PARTS_[@]}" ]]; then
      FOUND_=1
      break
    fi
  done
  if (( 0 == $FOUND_ )); then
    echo "ERROR: Could not find Ruby ${RUBY_VERSION_} in the versions"
    echo "       available to be installed from 'rbenv'."
    exit 2
  fi
  echo "WARNING: Ruby ${RUBY_VERSION_} is not available to be set from rbenv."
  read -p "Do you want to make it available to 'rbenv'? [y/N]" -n 1 -r
  echo
  if [[ ! $REPLY =~ ^[Yy]$ ]]
  then
    echo "No action taken."
    exit 6
  fi
  rbenv install ${RBENV_RUBY_VERSION_}
  is_this_ruby_available "${RBENV_RUBY_VERSION_}"
  if (( 0 != $? )); then
    echo "ERROR: failed to parse RBENV_RUBY_VERSION_"
    exit 2
  fi
  FOUND_=$is_this_ruby_available
  if (( 0 == $FOUND_ )); then
    echo "ERROR: failed to install Ruby ${RBENV_RUBY_VERSION_}"
    exit 10
  fi
fi

# Set the local Ruby version
#
rbenv local ${RBENV_RUBY_VERSION_}

# Check the local Ruby version
#
is_this_ruby_active "${RBENV_RUBY_VERSION_}"
if (( 0 != $? )); then
  echo "ERROR: failed to determine active Ruby version"
  exit 2
fi
if (( 0 == $is_this_ruby_available )); then
  echo "ERROR: failed to activate Ruby ${RBENV_RUBY_VERSION_}"
  exit 4
fi

# Set the gemset directory
#
rbenv local --unset
if [[ ! -f ".rbenv-gemsets" ]]; then
  echo ".gems" > .rbenv-gemsets
  echo "" >> .rbenv-gemsets
fi

# Install bundler
#
rbenv local ${RBENV_RUBY_VERSION_}
BUNDLER_P_=`gem list --no-versions | grep -E ^bundler$`
if [[ -z "${BUNDLER_P_}" ]]; then
  gem install bundler
fi
BUNDLER_P_=`gem list --no-versions | grep -E ^bundler$`
rbenv local --unset
if [[ -z "${BUNDLER_P_}" ]]; then
  echo "ERROR: failed to install the gem 'bundler'."
  exit 6
fi

# Done!
#
echo ""
echo "OK. Virtual environment for Ruby ${RUBY_VERSION_} is created."
echo ""
echo -n "To activate, use the source command "
echo "'. ${SCRIPTDIR_}/activate_project.src'."
echo ""
echo "Alternatively, use the command 'rbenv local ${RBENV_RUBY_VERSION_}' "
echo "to activate, and the command 'rbenv local --unset' to deactivate."
echo ""

Running the script produces output like this:

./ruby/make-venv.sh
Successfully installed bundler-1.11.2
Parsing documentation for bundler-1.11.2
Installing ri documentation for bundler-1.11.2
1 gem installed

OK. Virtual environment for Ruby 2.0.0p598 is created.

To activate, use the source command '. ./ruby/activate_project.src'.

Alternatively, use the command 'rbenv local 2.0.0-p598'
to activate, and the command 'rbenv local --unset' to deactivate.

Automating virtual environment activation and deactivation

The activation script should:

activate the virtual environment,
indicate a Ruby virtual environment is active by adding “(rb)” to the start of the command prompt,
create the deactivation script

My activation script looks like this … yours may need some tweaking to get the command prompt changed correctly.

# Switch to project-specific environments for ruby
#
# Want to put '(rb)' after the git branch in the command prompt.
#
# Note: My PS1 looks like this:
#
#  PS1=\[\033[1;36m\]$(parse_git_branch)\[\033[1;32m\]\w$\[\033[0m\]\n
#

# Save PS1
#
#\[\033[1;36m\]$(parse_git_branch)\[\033[1;32m\]\w$\[\033[0m\]\n
#
RE_='s/^(.*)(\\\[\\033\[1;32m\\\]\\w\$.+)$/\1/'
export PRE_PS1_=`echo -n "$PS1" | sed -E ${RE_}`
RE_='s/^(.*)(\\\[\\033\[1;32m\\\]\\w\$.+)$/\2/'
export POST_PS1_=`echo -n "$PS1" | sed -E ${RE_}`
CUR_PS1_=""

# Switch to ruby 2.0.0p598
#
rbenv local 2.0.0-p598
CUR_PS1_="(rb)$CUR_PS1_"
export PS1="${PRE_PS1_}${CUR_PS1_}${POST_PS1_}"

# A bit of cleanup
#
unset CUR_PS1_
unset RE_

# Prepare the script to restore default environments
#
PROJECT_ROOT_=`pwd`
# echo "deactivate" > deactivate_project.src
echo "rbenv local --unset" >> deactivate_project.src
echo "export PS1=\"\${PRE_PS1_}\${POST_PS1_}\"" >> deactivate_project.src
echo "unset PRE_PS1_" >> deactivate_project.src
echo "unset POST_PS1_" >> deactivate_project.src
echo "rm deactivate_project.src" >> deactivate_project.src
echo "unalias deactivate_project" >> deactivate_project.src

# Setup the alias to deactivate the local environments
#
alias deactivate_project='. '$PROJECT_ROOT_'/deactivate_project.src'
unset PROJECT_ROOT_

# Inform user of how to get default environment back
#
echo "Run 'deactivate_project' to restore default Ruby environment"

Running the script (using a source command) creates a deactivation script and an alias deactivate_project that sources the deactivation script.

The generated deactivate script looks like this:

rbenv local --unset
export PS1="${PRE_PS1_}${POST_PS1_}"
unset PRE_PS1_
unset POST_PS1_
rm deactivate_project.src
unalias deactivate_project

It deactivates the local Ruby, restores the command prompt, and removes the deactivation script and alias.

Note that activation script could be improved to do some validation checks to ensure that an appropriate Ruby virtual environment is available to be activated.

Automating virtual environment provisioning

An automation script that provisions the virtual environment should:

check that the appropriate virtual environment is activated
install the gems as specified by the Gemfile
check that the required dependencies are now in place

My implementation of the provisiong script is called provision-venv.sh and looks like this:

#! /bin/bash
#

SCRIPTNAME_="$0"

# Get the directory holding this script
# (method from: http://stackoverflow.com/a/12694189/1392864)
#
SCRIPTDIR_="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi

# Ruby and rbenv local Ruby versions
#
RUBY_VSN_="2.0.0p598"
RUBY_VSN_EXP_="2.0.0p598"
RUBY_VSN_MAX_="2.0.1"
RBENV_RUBY_VSN_EXP_="2.0.0-p598"
RBENV_RUBY_VSN_MAX_="2.0.1"

# Pick up the verification functions
#
. "${SCRIPTDIR_}/check-functions.src"

# Check that we are running in a Ruby virtual environment
#
. "${SCRIPTDIR_}/check-active-venv.src"

# Install required packages
#
bundle install --quiet

# Check that we have the needed packages
#
. "${SCRIPTDIR_}/check-dependencies.src"

# Done!
#
echo ""
echo -n "OK. Provisioned required packages for project in "
echo "Ruby ${RUBY_VSN_} virtual environment."
echo ""

You can find the included scripts check-functions.src, check-active-venv.src, and check-dependencies.src, as well as the latest version of provision-venv.sh and all the other scripts, in the GitHub repo.

Gemfile

The Gemfile used is the same as from the previous post:

source "https://rubygems.org"

gem 'sass', '3.1.3'
gem 'chunky_png', '>= 1.2.0', '< 1.2.1'
gem 'fssm', '0.2.7'
gem 'compass', '0.11.3'

Automating gem upgrades

Sometimes you may want to pick up patches or updates to gems used by your project. If you are working on an older project you probably won’t want the most recent gems (because they may require a later Ruby or bring along a whole mess of other gems you really didn’t want to mess with). So you handle that by updating the Gemfile to constrain the version range of the gems and, importantly, by using bundler to update the gems – not by using the gem update command.

For this example, I want to pick up some more recent gems for my Ruby 2.0.0 project. I was using gems circa mid-2011, and now I want to pick up gems circa early 2012. To do that I update the Gemfile to:

source "https://rubygems.org"

# Circa March 2012

gem 'sass', '>= 3.1.3', '<= 3.1.17'
gem 'chunky_png', '>= 1.2.0', '<= 1.2.5'
gem 'fssm', '>= 0.2.7', '<= 0.2.9'
gem 'compass', '>= 0.11.3', '<= 0.12.1'

Then I run the upgrade script, upgrade-venv.sh. That script looks like:

#! /bin/bash
#
# These scripts assumes that rbenv is installed.
#

SCRIPTNAME_="$0"

# Get the directory holding this script
# (method from: http://stackoverflow.com/a/12694189/1392864)
#
SCRIPTDIR_="${BASH_SOURCE%/*}"
if [[ ! -d "$DIR" ]]; then DIR="$PWD"; fi

# Ruby and rbenv local Ruby versions
#
RUBY_VSN_="2.0.0p598"
RUBY_VSN_EXP_="2.0.0p598"
RUBY_VSN_MAX_="2.0.1"
RBENV_RUBY_VSN_EXP_="2.0.0-p598"
RBENV_RUBY_VSN_MAX_="2.0.1"

# Pick up the verification functions
#
. "${SCRIPTDIR_}/check-functions.src"

# Check that we are running in a Python 3.5 virtual environment
#
. "${SCRIPTDIR_}/check-active-venv.src"

# Check that we have the needed packages
#
. "${SCRIPTDIR_}/check-dependencies.src"

# Upgrade the required packages
#
bundle update
gem cleanup

# Check that we still have the needed packages
#
. "${SCRIPTDIR_}/check-dependencies.src"

# Done!
#
echo ""
echo -n "OK. May have upgraded required packages for project in "
echo "Ruby ${RUBY_VSN_} virtual environment."
echo ""

Before running the script, the gems were:

bigdecimal (1.2.0)
bundler (1.11.2)
chunky_png (1.2.0)
compass (0.11.3)
fssm (0.2.7)
io-console (0.4.2)
json (1.7.7)
minitest (4.3.2)
psych (2.0.0)
rake (0.9.6)
rdoc (4.0.0)
sass (3.1.3)
test-unit (2.0.0.0)

After updating the Gemfile and running the upgrade script, the gems were:

bigdecimal (1.2.0)
bundler (1.11.2)
chunky_png (1.2.5)
compass (0.12.1)
fssm (0.2.9)
io-console (0.4.2)
json (1.7.7)
minitest (4.3.2)
psych (2.0.0)
rake (0.9.6)
rdoc (4.0.0)
sass (3.1.17)
test-unit (2.0.0.0)

Note that the check of dependencies will fail because the check-dependencies.src script uses hard-coded version numbers for the gems it is checking. So you’ll need to update that script after upgrading.³

If I’m happy with those gems, then I can lock in those versions (preventing them from getting changed by future upgrade-venv.sh invocations) by changing the Gemfile to be very restrictive, like this:

source "https://rubygems.org"

# Circa March 2012

gem 'sass', '3.1.17'
gem 'chunky_png', '1.2.5'
gem 'fssm', '0.2.9'
gem 'compass', '0.12.1'

Automating other actions

In a development project there are other actions that could be automated, such as build and test scripts. They will need similar validation of the virtual environment. You can use the same validation approach in those scripts as was done for the provision-venv.sh script.

The example using Ruby 2.2.3

In the GitHub repo I also provide automation scripts that specifiy a recent version of Ruby, version 2.2.3, and very recent versions of the Compass and Sass gems.

Other than changing version numbers, there is really no significant difference in the scripts, so I’m not going to detail them here. Have a look in the GitHub repo to get those example scripts.

Automation of a Ruby virtual environment on Windows

The same set of scripts is needed – a virtual environment creation script, an activation script, a provisioning script, and an upgrade script. However, creating a virtual environment on Windows is a bit clumsier (because no rbenv on Windows). And, of course, we use Powershell to do industrial strength scripting on Windows, not Bash.

All the scripts are in the Ruby example branches in the GitHub repo. I’ve done Powershell scripts for both the Ruby 2.0.0 and the Ruby 2.2.3 examples.

Automating the creation of the virtual environment (Windows)

On Windows, the script to make the virtual environment is called make-venv.ps1. It performs the following steps:

Check whether the required version of Ruby is installed on the system
Check whether Ruby is in the local deployment directory .rblcl
- If it is, check the version
- If it is not, copy the installed Ruby into the local deployment directory
Add the local Ruby to the $Path
Check the local Ruby version
Install the gem bundler
Remove the local Ruby from the $Path

This script does nothing tricky, so I won’t reproduce it here. Have a look at it in the GitHub repo.

However, the script does depend on the expected versions of Ruby being installed in known locations. See “Installing Ruby (Windows)” for installation guidance.

Automating the virtual environment activation and deactivation (Windows)

Rather than sourcing the activation script, on Windows you just execute it directly. The script is called activate-project.ps1. This script:

Adds the local Ruby to the $Path
Prepends “(rb)” to the command prompt, to indicate an active Ruby virtual environment
Creates the deactivate script

The deactivation script just removes the local Ruby from the $Path and removes “(rb)” from the command prompt.

The only tricky bit is how the command prompt is managed. It is worth having a look at the example script.

Provision the virtual environment (Windows)

The provisioning script, provision-venv.ps1, is very similar to the OS X version, at least in the steps it takes. It uses bundler and a Gemfile to control the gem installation.

Upgrade the virtual environment (Windows)

Similarly, the upgrade script, upgrade-venv.ps1, is very similar to the OS X version as regards the steps it takes.

Summary (Windows)

If you want to use Ruby on Windows, I encourage you to pull the examples from the GitHub repo and try them out. For me, they’ve made using Ruby on Windows quite pleasant – mainly because they’ve made establishing and using Ruby virtual environments a straightforward excercise.

And, as in the last post, I give examples initially for OS X and, at the end of the post, provide an abbreviated section with Windows examples. ↩
The example is descibed in the bundler section. ↩
I hope to modify the dependency check to use the Gemfile itself; that should mitigate this issue with the upgrade script. ↩