In a previous post I postulated virtual environments as a solution to the problem of dependency and version collision for software built using languages like Python, Ruby, Node.js, and Go.

In this post I give the details and some examples of using a Go virtual environment. First I walk through everything using OS X—i.e., using a Mac—and then I describe the same procedures using Windows.

As with the Node.js virtual environments post, instead of breaking the work into two posts, I’ll try to work it all into a single post (made possible by skipping direct inspection of the automation scripts).

Caveat: As I write this I am very new to Go. I’ve tried to do a good job representing the common workflow and tools of Go developers, but may have missed important things. I recommend supplementing this post with StackOverflow search, Go docs, and reading postings from other Go developers.

About Go

The homepage of Go, https://golang.org/, says:

Go is an open source programming language that makes it easy to build simple, reliable, and efficient software.

According to Wikipedia, the designers of Go:

Go originated as an experiment by Google engineers Robert Griesemer, Rob Pike, and Ken Thompson to design a new programming language that would resolve common criticisms of other languages while maintaining their positive characteristics.[20] The new language was to:

  • be statically typed, scalable to large systems (as Java and C++);
  • be productive and readable, without too many mandatory keywords and repetition (“light on the page” like dynamic languages);
  • not require tooling, but support it well,
  • support networking and multiprocessing.

But that really doesn’t explain why a developer would want to learn or us Go. The fact that it was developed by Google, might make it of interest. The fact that Ken Thompson and Rob Pike had a hand in the design and creation of Go, might be compelling. But for me, the most compelling reason to learn Go is that many tools having to do with containers and virtualization and networking have been written using Go; Docker is written in Go; Packer is written in Go; CoreOS is written in Go; oh, and Syncthing is written in Go.

Go installation (OS X)

If you go to the Go downloads page you’ll see that there are source archives (from which you can build the Go compiler) and installers and plain binary archives. If you use the installer, then you’ll only have one version of Go on your system—whichever version you last installed—and that’s the antithesis of a virtual environment. Thankfully, the plain binary trees are available, and that’s what should be used to install a particular version of Go as a virtual environment.

I follow the same process as I do for Node.js virtual environments—I put all the binary archives (*-amd64.tar.gz) of Go under the directory ~/Library/Frameworks in a sub-directory named go.framework.

For example, currently on my system the ~/Library/Frameworks/go.framework contains:

  • go1.4.3.darwin-amd64.tar.gz
  • go1.6.3.darwin-amd64.tar.gz
  • go1.7.1.darwin-amd64.tar.gz

Go workspaces

According to the official documentation, the organization of a Go project is a workspace directory containing three sub-directories: bin, pkg, and src. Within src each component is one or more source repository, stored with directory namespace that maps to the primary repository location.

There is also emerging the idea that dependencies should reside in a fourth sub-directory, vendor. It was an earlier practice to place those dependencies within the src folder, but the downsides of doing that outweigh the conveniences. N.B., we are in a transition period; many projects have not moved to the vendor practice.

Go environment variables

Go uses a handful of environment variables to control behavior and specify default locations. You can see all the important environment variables using the command go env.

For the purposes of virtual environment support, only three environment variables are important:

  • GOROOT is the location of the Go installation. go itself should be found at $GOROOT/bin/go, and $GOROOT/bin should be in the command shell’s search path.
  • GOPATH is the location of the Go project workspace.
  • GOBIN is the location of the Go project binaries; usually this is $GOPATH/bin. For convenience, it is common to put $GOBIN in the path.

Go virtual environments

The example

I wanted to pick a real world example. Unlike the previous virtual environment examples, for Go I’ll picked a project having to do with distributed computing—Hashicorp’s Go implemetnation of the Raft protocol, which is used for consensus in a distributed system.

Essentially the example will create a virtual environment in which this Go implementation of Raft can be built and modified. I would do this if I wanted to pickup the latest fix, or if I wanted to make a change or fix a bug and generate a pull request. This is just an example with just enough complexity to illustrate some key virtual environment concepts. The main thing is to pay attention to the techniques used to create and manage the virtual environment because they’ll be used for any Go project, regardless of size or complexity.

We’ll get the code for Hashicorp’s Raft implementation from the GitHub repo.

Automating virtual environment management

As I’ve said before, when working on a project I don’t want to fumble around trying to figure out how to create a virtual environment with the correct version of Node.js and provision it with the needed npm packages at the required versions.

What I want is to be able to pull a fresh source tree from source control (GitHub, Atlassian, a private server) and:

  • run a script that automatically creates the correct virtual environment,
  • run another script to activate the virtual environment,
  • run another script to provision the virtual environment with the packages required by the project.

I also want my command prompt to indicate that a virtual environment is active, and I want:

  • a script to deactivate the virtual environment, and
  • a script to upgrade packages to pick up compatible patches and updates

These scripts should validate that the correct and complete environment expected is in place before taking any actions to modify or manage that environment.

Except for the provisioning and upgrade scripts, those capabilities should be considered standard features of virtual environments. For a clear statement of the principle requirements of a virtual environment see “Package management automation”.

The problem with Go Get

A Go project typically has some dependencies (just like Ruby projects and Python projects and Node.js projects). Go dependencies are resolved by adding the source of the dependent packages to the local workspace of the Go project. This is done with go get, and you can list dependencies with go list.

go get always gets the latest version of main branch of any dependency it retrieves. That behavior is not compatible with the concept of providing version specific support for a product or project, and does not protect a project from breaking changes in dependencies. Put simply, each time you pull a fresh copy of your project and run go get then you will get the most recent version of each dependency, and what you get today will almost certainly be different than what you or your colleagues got last week. Not good.

For more on the topic, there is an interesting StackOverflow discussion

A solution using glide

I’m new to Go programming, but I believe that for a long time the dependency control problem was solved using godep. That tool will allow you to specify which commit to pull from the repository of each dependency.

More recently, I believe, the community has started using the Glide tool. Beside locking a dependency to a specific commit, Glide also allows specification by semantic version. Glide will parse godep JSON files, and provides a nice wizard for creating the initial YAML config files from an existing dependency tree. Glide is like Ruby’s gem and bundler rolled into one tool.

Glide will work with Go 1.5 and later; it requires use of the vendor feature for package management, and that feature was introduced in Go 1.5.

If you have to work with Go 1.4 and earlier, use godep; otherwise, use Glide. I’ll be using Glide in the example.

The two files of importance to Glide are glide.yaml and glide.lock. The first, glide.yaml is generated by running glide init from $GOPATH; the project source code must be in place within the workspace src directory. Glide will create a glide.yaml file that holds a shallow dependency tree. You can edit that file to specify particular versions or version ranges for each dependency.

The second file, glide.lock, is produced by running glide update from $GOPATH; this will add each dependency to the vendor subdirectory, and will store the commit id for each dependency in the glide.lock. It will also do a deeper dependency search than was done to produce glide.yaml and will map a full dependency tree into glide.lock.

Once glide.yaml and glide.lock are established, they can be used to reproduce the exact dependencies required by a project, such that the dependencies will be identical across each instance of the project, whenever and wherever the project is provisioned.

Script source code

I’ve made the example with all the scripts in this post available in a GitHub repository called using-virtual-environments. Use git checkout example-go163 for the Go v1.6.3 example files. I thought about doing a Go v1.7.1 example, but there is no difference sufficient to warrant the extra work. There are some scripts in that repo that I won’t discuss, like the scripts that contain helper functions (like functions to parse and compare version strings); the full set of scripts may be worth investigating on your own.

Automating the creation of the virtual environment

On OS X, the script to make the virtual environment is called make-venv.sh. It is quite straightforward. It performs the following steps:

  • Check whether a Go virtual environment exists
  • If not, check whether a tar.gz distribution exists; install it to a local directory in the project directory tree
  • Check whether the Go virtual environment contains the expected version of Go

Take a look at golang/make-venv.sh from the example-go163 branch of the Git repo). Notice that the local directory is .golcl.1

Automating the activation of the virtual environment

The script to activate the virtual environment performs these steps:

  • sets the GOROOT environment variable to the local Go installation
  • sets GOPATH and GOBIN environment variables to the local Go workspace
  • prepends GOROOT/bin to the $PATH environment variable
  • prepends GOBIN to the $PATH environment variable
  • adds (go) to the command prompt to signal that a Go virtual environment is active
  • creates the deactivation script
  • creates an alias deactivate-venv to invoke the deactivation script

This script must be executed using the source command:

$ . golang/activate-venv.src

Take a look at golang/activate-venv.sh from the example-go163 branch of the Git repo).

Automating the provisioning of the virtual environment

The script to provision the virtual environment performs these steps:

  • Check that the Go virtual environment is active and running the expected version of Go
  • Ensure that the workspaces has the three primary subfolders, bin, pkg, and src
  • Simple check that source code exists.
  • Check that glide is installed and has the correct version.
  • Use glide to ensure the correct versions of the package dependencies are installed. For any not installed, install them.

Take a look at golang/provision-venv.sh from the example-go163 branch of the Git repo).

Go on Windows

Using Go on Windows systems is just as easy as it is on OS X. Especially so if you don’t install Go using an install kit because, similar to the situation on OS X, there is no side-by-side installation of different Go versions.

The main issue you might run into is the nasty path length problem. Windows path lengths cannot exceed 260 characters. Actually they can, and will (thanks to go get and glide), but many native CLI and GUI programs will break if the absolute path of a file or directory exceeds 260 characters. I don’t have a great solution to this, but there are some workarounds out there; caveat emptor.

The post on Node.js virtual environments discusses the path length issue in more detail, including a discussion of workarounds.2

Installing Go (Windows)

The goal of this installation is to make a source image of a particular version of Go available for virtual environment creation. This is not intended to install Go for general or global use.

Note: this method works around the “lack of side-by-side installation” issue.

If you go to the Go downloads page you’ll see that there are source archives (from which you can build the Go compiler) and installers and plain binary archives. If you use the installer, then you’ll only have one version of Go on your system—whichever version you last installed—and that’s the antithesis of a virtual environment. Thankfully, the plain binary trees are available, and that’s what should be used to install a particular version of Go as a virtual environment.

I put all the binary archives (*.windows-amd64.zip) of Go under the directory $Env:USERPROFILE\Frameworks\go.framework. (Note: the example automation scripts assume that location.)

For example, currently on my system the ~\Frameworks\go.framework directory contains:

  • go1.6.3.windows-amd64.zip
  • go1.7.1.windows-amd64.zip

7-zip (Windows)

To unpack the Go binary archive from a PowerShell script I decided, in my example automation scripts, to rely on 7-zip. 7-zip has a nice CLI executable that is very compatible with scripting.

To use the example automation scripts, then, you’ll need to download and install 7-zip.

Automating the Go virtual environment (Windows)

As mentioned above, I’ve made the example with all the scripts in this post—including the Powershell scripts for Windows—available in a GitHub repository called using-virtual-environments.

The make-venv.ps1 script will install the virtual environment for a specific version of Go by copying the pristine installation image into a local .golcl folder for the project—first checking whether .golcl already exists and contains a valid virtual environment.

The activate-venv.ps1 script will check for a valid virtual environment local directory (i.e., a valid .golcl directory) and will then create the GOROOT, GOPATH, and GOBIN environment variables and add GOROOT\bin and GOBIN to the path. Then it will prefix the command prompt with “(go)”. Finally it will create a deactivation script deactivate-venv.ps1 that should be used to deactivate the virtual environment (remove the virtual environment from the PATH and remove the command prompt prefix).

The provision-venv.ps1 script will create the bin and pkg directories, install glide, and install all the package dependencies into the vendor directory.

So, when you download a fresh source tree from the source control repository, you only need run:

  • make-venv.ps1
  • activate-venv.ps1
  • provision-venv.ps1

And you’ll have a fully provisioned and active Node.js virtual environment ready for your project to use.

Note: I did not create an example task script as I did for the Node.js virtual environment example (see example-task.ps1 in the Node example scripts for an example of a project task script, such as a build or test script, that validates the virtual environment before performing any work), but you can—and should—do similar validation in your Go task scripts.

Removing the Go virtual environment (Windows)

If you run into path length issues trying to remove the Go virtual environment, see the Node.js virtual environment post for a discussion, with examples, of how to work around that issue.

In sum (Windows)

Using Go virtual environments on Windows is straightforward and recommended. It is easier to do that Node.js virtual environments. I think most Go projects could take advantage of a virtual environment on Windows systems.

  1. This name avoids conflict with .rblcl, which is what I used in the virtual environment mechanism on Windows. Eventually I want to have simultaneous virtual environments for a project: concurrently active virtual environments for Python, Ruby, Node.js, and Go. ↩

  2. If not, there’s always Bash on Windows. ↩