Using Docker Machine from an Azure VM

I’m scheduled to take a training class that requires a Windows laptop running Visual Studio and Docker Toolbox. But I don’t have a Windows laptop.

• The problem
• The solution
• Spinning up a Windows 10 Visual Studio system in Azure
• No Hyper-V and no Virtual Box
• Creating a machine using the Azure driver
• Using the new Docker VM
• Details on provisioning the Windows 10 VM

The problem

I am gainfully employed as a developer, supporting and building software that runs on Windows servers and desktops. Part of my responsibilities include understanding how to leverage cloud platforms. To help improve my understanding, I’m scheduled to participate in a training class from Microsoft called “Containers for Developers and IT Pros”. For the class I need an active Azure subscription and a Windows machine running Visual Studio and Docker Toolbox.

I use a desktop system for development (along with a host of Windows servers). I don’t have a Windows laptop. What to do?

The solution

I do have a Mac laptop that I use for travel and meetings and remote access—just not for slinging code for my day job. And I do have an Azure subscription because my company provides an MSDN subscription for each developer, and that comes with an Azure subscription with a $150 monthly credit.¹

It seems possible, as a colleague brilliantly suggested, that I could just run a Windows 10 VM in Azure and RDP to that VM in the class.

Great idea, in theory—and also, it turns out, in practice. Though, as with almost everything in computing, the implementation of that solution was a bit more complex than immediately apparent.

The rest of this post describes what I did, with enough detail to help you do something similar.

Spinning up a Windows 10 Visual Studio system in Azure

Keep in mind that up to this point I’d had very little hand’s on with Azure, and all of that experience was several years old – before the new Azure Resource Manager existed. Things had changed significantly.

THe first thing I did was clean out all my old Azure resources, all the storage and VMs that I had created in past investigations with Azure. I couldn’t remember anything about them and there was no point in keeping them around. So my starting point is a clean subscription with no extant resources.

The first thing I did was spin up a Windows 10 VM running Visual Studio 2015. Microsoft makes a variety of VM images available with various Microsoft software products pre-installed. There are a number of different Visual Studio VMs with various combinations of Windows operating systems and Visual Studio types and releases.

I selected an image from the Azure Marketplace by searching for the string “windows 10 visual studio 2015”. This resulted in four potential images, from which I selected the one named “Visual Studio Enterprise 2015 Update 3 with Universal Windows Tools and Azure SDK 2.9 on Windows 10 Enterprise N (x64)”.

When I created the VM I was prompted for a resource group name to supply the resource group that had to be created to organize the resources that would be created to create and host the new VM. I chose “dev”. I was also prompted for a userid and password, which I supplied.

After the VM was created, I connected to it using the “connect” link in the Azure Portal. That downloaded an RDP file which I opened using the excellent Jump Desktop RDP application on the Mac. It worked just fine to connect to the Windows 10 VM.

After the VM was started, I created and attached a second disk, this time an HDD, on which I intend to store all my Visual Studio projects and all my Docker related files (including Docker Machine storage).

Note: the details of provisioning and configuring the VM are more complex and nuanced than described above. You can find the complete details at the end of this post.

No Hyper-V and no Virtual Box

I then installed Docker Toolbox for Windows using the default settings. I was already very familiar with Docker Toolbox from using it on my Mac, and so I immediately tried a “docker-machine create --driver virtualbox ...”. It failed with an error saying that VirtualBox is incompatible with Hyper-V.

So I followed my own guidance about configuring a Hyper-V host system to work with Docker Toolbox, and tried a “docker-machine.exe create --driver hyperv ...”. It failed with an error that I ultimately understood to be generated because Azure VMs do not support nested virtualization. In other words, you can’t host a Hyper-V hypervisor on an VM; it’s not turtles all the way down, at least not in Azure.²

Just to confirm that, I removed the Hyper-v feature from the Windows 10 VM and restored the network adapter configuration needed by VirtualBox. Then I tried to create a machine on VirtualBox and that failed, as expected, with a message saying the BIOS hadn’t enabled virtualization.³

Creating a machine using the Azure driver

Docker Machine now ships with an Azure driver, so it seemed possible that I could use Azure to create and manage new Docker VMs using Docker Machine. Experiments bore that out.

NOTE: My MSDN subscription is tied to a Microsoft Live account, not an Azure Active Directory account. The old Live accounts are not very compatible with the various authentication and authorization mechanisms of Azure, and the Docker Machine documentation about the Azure driver has this warning: “There is a known issue with Azure Active Directory causing stored credentials to expire within hours rather than 14 days when the user logs in with personal Microsoft Account (formerly Live ID) instead of an Active Directory account. Currently, there is no ETA for resolution, however in the meanwhile you can create an AAD account and login with that as a workaround.” The guide in that link is a bit out of date, but I follow the intent of it and was able to create a new AAD account tied to my MSDN subscription and give it admin rights to that it can do everything my primary userid can do (except manage billing). I used that new userid for all my Azure authentication to Docker Machine.

I created a Powershell script with this command:⁴

docker-machine.exe `
  --storage-path "E:\.docker\machine" `
  create `
  --driver azure `
  --azure-subscription-id "bc7e016e-00e1-4cfe-8358-2129313f0507" `
  --azure-location southcentralus `
  --azure-resource-group dev `
  --azure-vnet dev-vnet `
  --azure-subnet default `
  --azure-subnet-prefix "10.0.0.0/24" `
  --azure-use-private-ip `
  --azure-no-public-ip `
  mtc-dkr

I ran it in an administrative Powershell window and got this output:

PS E:\docker> .\create-docker-vm-azure.ps1
Creating CA: E:\.docker\machine\certs\ca.pem
Creating client certificate: E:\.docker\machine\certs\cert.pem
Running pre-create checks...
(mtc-dkr) Microsoft Azure: To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code F7G34VTMC to authenticate.
(mtc-dkr) Completed machine pre-create checks.
Creating machine...
(mtc-dkr) Querying existing resource group.  name="dev"
(mtc-dkr) Resource group "dev" already exists.
(mtc-dkr) Configuring availability set.  name="docker-machine"
(mtc-dkr) Configuring network security group.  name="mtc-dkr-firewall" location="southcentralus"
(mtc-dkr) Querying if virtual network already exists.  name="dev-vnet" location="southcentralus"
(mtc-dkr) Virtual network already exists.  location="southcentralus" name="dev-vnet"
(mtc-dkr) Configuring subnet.  vnet="dev-vnet" cidr="10.0.0.0/24" name="default"
(mtc-dkr) Not creating a public IP address.
(mtc-dkr) Creating network interface.  name="mtc-dkr-nic"
(mtc-dkr) Creating storage account.  name="vhdshsra2b15kq332q9jbp7p" location="southcentralus"
(mtc-dkr) Creating virtual machine.  location="southcentralus" size="Standard_A2" username="docker-user" osImage="canonical:UbuntuServer:16.04.0-LTS:latest" name="mtc-dkr"
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with ubuntu(systemd)...
Installing Docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: C:\Program Files\Docker Toolbox\docker-machine.exe env mtc-dkr
PS E:\docker>

In the middle of that I got prompted to authenticate with a message like this, which required me to open a browser and authenticate:

(mtc-dkr) Microsoft Azure: To sign in, use a web browser to open the page https://aka.ms/devicelogin and enter the code F7G34VTMC to authenticate.

According to the Azure driver documentation, that authentication will need to be renewed after a couple weeks.

This command shows the machine is there and running.

docker-machine.exe --storage-path "E:\.docker\machine" ls

NAME      ACTIVE   DRIVER   STATE     URL                   SWARM   DOCKER    ERRORS
mtc-dkr   *        azure    Running   tcp://10.0.0.5:2376           v1.13.0

Using the new Docker VM

I can set an environment variable so that I don’t have to keep supplying the storage path to docker-machine. I do that with: $Env:MACHINE_STORAGE_PATH = "E:\.docker\machine".

Now I can point my local docker client to that machine and pull down an image and run my first container:

& docker-machine env mtc-dkr | Invoke-Expression
docker pull alpine:3.4
docker run --rm -i -t alpine:3.4 /bin/ash

And I can get a terminal session on the new Docker VM with the command docker-machine ssh mtc-dkr.

It all seems to work fine!

Just for my own interest, I checked the IP address of the various systems:

• container (Alpine:3.4) – 172.17.0.2
• mtc-dkr (Ubuntu16.04) – 172.17.0.1, 10.0.0.5
• heind001-az (Windows 10) – 10.0.0.4

and of course Azure maps a public IP to heind001-az.

Important: To avoid unexpected costs and charges, you want to deallocate the VMs when you are not using them. You’ll want to stop both the Windows 10 VM and the Docker VM from the Azure Portal so that the machines are both shutdown and deallocated. Just using docker-machine stop mtc-dkr would only shutdown the machine, but it would remain allocated and accumulating charges.

Details on provisioning the Windows 10 VM

Here I’m attempting to provide all the nitty gritty details of what I did so that you get a better chance at reproducing the example. Unless you are really interested, you can probably skip this section.

The details of provisioning are this:

1. Browse to Azure Portal.
2. In the Portal nav bar on the left, select “Virtual machines”.
3. Click the “Add” button.
4. Search for “Windows 10 Visual Studio 2015 Update 3”.
5. Select the correct image, then make sure the “Resource Manager” deployment model is selected, and click “Create”.
6. In the Basic Configuration step, pick a name. I picked “heind001-az”.
7. Pick a disk type. SSD is fine.
8. Pick a username and password.
9. Make sure correct subscription is selected. Mine was “Visual Studio Enterprise”.
10. Supply a resource group name. I picked “dev”.
11. Pick a location. I picked South Central US.
12. Click OK.
13. In the Size configuration step, select the machine sizing. I just picked the cheapest of the recommendations, which was DS2_V2.
14. In the Settings configuration step, accept all the defaults.

After the machine is provisioned it will be launched. I logged in and did the following:

1. Enable ClearType.
2. Configure File Explorer to show extensions and hidden files.
3. Install x64 Notepad++ using all defaults.
4. Install Chrome and configure with uBlock Origin and to not save passwords.
5. Install Firefox, set as default, and configure with uBlock Origin and to not save passwords.
6. Configure Edge to not save passwords and to use Google as the initial page for new windows and tabs.
7. Install Docker Toolbox (using all defaults).
8. Clear out most of the panes in the Start Menu and add Powershell, Notepad++, and the browsers to the Start menu and Taskbar.

Next I added a second disk, a 128GB hard disk (because it’s cheap and performance isn’t critical).

1. In the VM details, click on Disks and then click Add.
2. Keep the generated name, change to HDD instead of SSD, and select 128GB size.
3. Navigate from “Location” and add a new storage account. I named mine “devdisks5491” and used Standard storage and LRS redundancy.
4. Add a container to the new storage account. I called mine “data-disks” and made it private.
5. Select that new container.
6. Click OK to create the new disk.
7. Azure will provision the disk and attach it to the VM.
8. In the RDP session, use Disk Management in the VM to format the new disk. I selected a GUID Partition Table (GPT) and quick formatted the New Simple Volume with NTFS, naming the volume “DATA”. I changed the drive letter to “E:”.