As part of my efforts to learn python I decided to create a docker-machine CLI wrapper that can be used from within Python applications to create and manage remote/cloud VM services using docker-machine. The wrapper also has support for upping and monitoring services, using docker-compose.
Key features of Python I had to learn:
- calling subprocesses using
Popen
- multi-threading
- queues
- file streams
NOTE: checkout https://aduvenhage.github.io/raytracer/ for an example of where I use this API to run raytracing in the cloud.
Docker-machine works well to provision remote VMs since it hides much of the boilerplate effort required to create a remote machine, manage keys and create a basic image with Docker installed. All docker-engine settings are managed via environment variables, and it even makes it easy to secure copy and ssh on the remote VM (with the keys stored in the machine environment). Docker-machine has built-in drivers for a whole range of cloud providers, including Amazon Web Services, Digital Ocean and Google Compute Engine.
The install on my macbook was straightforward:
- Docker: download docker desktop from https://www.docker.com/products/docker-desktop
- docker-machine: brew install docker-machine, doctl
- create API token: https://cloud.digitalocean.com/account/api/tokens
- login on API:
doctl auth init -t $TOKEN
- list droplet sizes:
doctl compute size ls
for example:Slug Memory VCPUs Disk Price Monthly Price Hourly c-4 8192 4 50 80.00 0.119050 c-32 65536 32 400 640.00 0.952380
- list available regions:
doctl compute region ls
for example:nyc1 New York 1 true sfo1 San Francisco 1 false nyc2 New York 2 false ams2 Amsterdam 2 false sgp1 Singapore 1 true lon1 London 1 true nyc3 New York 3 true ams3 Amsterdam 3 true fra1 Frankfurt 1 true tor1 Toronto 1 true sfo2 San Francisco 2 true blr1 Bangalore 1 true sfo3 San Francisco 3 true
- create VM (ubuntu 18.04 LTS -- Digital Ocean):
docker-machine create --driver digitalocean --digitalocean-image ubuntu-18-04-x64 --digitalocean-access-token=... do01
- activate docker env:
eval $(docker-machine env do01) .
- deactivate docker env:
eval $(docker-machine env -u)
- ssh into remote machine:
docker-machine ssh do01
- list machines:
docker-machine ls
- remove machines:
docker-machine rm do01
- provision a system: docker-machine (create --> eval ... --> docker-compose up)
From source root python setup.py sdist
pip install docker-machine-api-x.x.x.tar.gz
on built package file, orpip install git+https://github.com/aduvenhage/docker-machine-api
to install directly from github.
from docker_machine_api.cl_api import DockerMachine
# create machine API
dm = DockerMachine(name='raytracer',
cwd='../',
config={
'driver': 'digitalocean',
'digitalocean-image': 'ubuntu-18-04-x64',
'digitalocean-access-token': '....',
'engine-install-url': 'https://releases.rancher.com/install-docker/19.03.9.sh'
},
user_env={...})
# watch machine output
idle = False
while dm.busy() or not idle:
idle = True
try:
text = dm._stdout_queue.get(block=False)
logger.info(text)
idle = False
except Exception:
pass
try:
text = dm._stderr_queue.get(block=False)
logger.error(text)
idle = False
except Exception:
pass
if idle:
time.sleep(0.2)
- Docker machine task manager
- Tasks to provision, start/stop, get status, etc.
- sub-process stdout/stderr stream available to user
- Config options for AWS, GCP and DO