Storj-base

Glossary

This repo provides the following:

• Devops CLI tooling + Build images/compositions + Development enviroment setup + Test automation / CI
• Docker images + storjlabs/storj:base + storjlabs/storj:thor

TODO:

• File/dir naming conventions
• Devops CLI generators/templates + Makefiles + Docker composition files + Dockerfiles

Devops CLI tooling

Ideally, all storj repos use the same tooling to provide a consistent devops interface across projects.

Goals

1. Flexible and extensible
2. Minimal dependencies

Constraints

1. Compatible with docker
2. Compatible with kubenetes

Dependencies

• gnu make
• docker
• docker-compose

Methodology

Storj-base uses make for the main point-of-entry. make is an extremely common build tool and should be most widely available. Make is used as a wrapper around more powerful tools that run inside a docker container, where they have all the dependencies they need without the hassle of installation onto the host system.

Control-flow:

1. User executes make <some goal> (see make and goal in the glossary)
2. Make recipe runs a docker container or composition/service, executing the corresponding thor task

Thor is used to perform most tasks as it's much better suited for defining and performing devops tasks than a javascript task runner. To eliminate the need for a ruby interpreter the docker dependency is reused in conjunction with the storjlabs/storj:thor to provide an environment where the thorfiles are available and executable.

Additionally, when working in the storj-base repo, a docker composition (thor.yml) is used to provide access to the thor container as well as mounting the repo directory as a volume at /storj-base.

Usage

To see what commands are availble run make [command [sub-command [...]]] help.

Building Images

To build the images that this repo provides run:

make build

By default this will only build the storjlabs/storj:thor image. If you want to also build storjlabs/storj:base you have to untag/remove it and then build:

docker rmi storjlabs/storj:base
make build

Development environment Setup

From the storj-base project root:

make setup <repo name> <your github username>

This will use the git submodules associated with this repo and init and update them (i.e. clone and checkout).

(If the repo you're working on isn't available as a git submodule, you can add it)

After your repo is inited and updateed, a dependency tree is crawled starting from the repo's package.json's devopsBase.npmLinkDeps object. This tree is used to run a series of npm link and npm link <module name> commands to ensure that all storj dependencies are linked locally and that there is only 1 instant of any of these modules.

Docker Images

storjlabs/storj:base

Serves as a base image for all storj containers; it includes ruby 2 and node 4 interpreters.

storjlabs/storj:thor

Provides the thorfiles, allowing the use of our devops tooling within images that build FROM it.

Tool Docs / Glossary

Make (docs):

goal:

What make refers to as goals are analogous to targets or tasks in other task runner tools (e.g. grunt or thor).

recipe:

The bash code to execute for a given goal; the "body" if you will.

Docker (docs):

image:

The output of a docker build ... or docker-compose build ...; can be listed with docker images. These images are used to create containers

container:

An ephemeral virtual machine (linux container)

composition:

Refers to a .yml file used by docker-compose or the collective services' containers which make it up.

service:

Refers to a single container or container configuration in a docker-compose composition. See docker's docs

volume: see docker-compose docs

Git (docs)

submodule: see git submodule docs

Thor (docs)

Guides

Adding A Repo Submodule

You can add a git submodule to this repo so that it will be available when running make setup ... like so:

git submodule add <https github repo url>

(NOTE: you must use the https url instead of the ssh url because the git commands are being executed in a docker container that doesn't have access to your ssh keys. Git remotes urls will be automatically rewritten to the ssh version when using make setup)

Once you've added the submodule, you'll notice a file in your git diff corresponding to the submodule's directory on the filesystem e.g. for the bridge-gui submodule:

diff --git i/bridge-gui w/bridge-gui
index 45fbe3e..8ac466b 160000
--- i/bridge-gui
+++ w/bridge-gui
@@ -1 +1 @@
+Subproject commit 8ac466b79304abe7e24545fee72c6b651d03cf84

This file tells git where in version control history should be checked out when doing a git submodule update for that submodule and as such should be kept up to date in this repo (i.e. your fork / feature branch of storj-base)

Finally, you should ensure that you're new submodule's repo's package.json conatains a devopsBase object with an npmLinkDeps property whose value is an object that maps the npm module name to the git submodule name, optionally including a git refspec indicating where in version control history to check out. If a refspec is not provided, make setup defaults to highest compatible npm-published version tag.

TODO: add documentation for devopsBase and npmLinkDeps TODO: rename devopsBase to storjBase or something