Docker Registry Implementation

Maintainer: @lwander

Note - this is mean to serve as a technical guide to the Docker Registry implementation. A more general walkthrough can (soon) be found on spinnaker.io.

The Docker Registry is not for deploying Docker containers, instead, it serves to cache and expose all the tags associated with a set of Docker image repositories. Furthermore, it assumes that the provided Registry is a compliant with the v2 API.

Authentication

The provider specification is as follows:

dockerRegistry:
  enabled:          # boolean indicating whether or not to use docker registries as a provider
  accounts:         # list of docker registry accounts
    - name:         # required unique name for this account
      address:      # required address of the registry. e.g. https://index.docker.io
      username:     # optional username for authenticating with the registry
      password:     # optional password for authenticating with the registry
      email:        # optional email for authenticating with the registry
      repositories: # optional list of registries. if none configured, registry must support `/_catalog` endpoint
      cacheThreads: # optional (default is 1) number of threads to cache registry contents across
      clientTimeoutMillis: # optional (default is 1 minute) time before the registry connection times out
      paginateSize: # optional (default is 100) number of entries to request from /_catalog at a time

Authentication is handled by the Clouddriver microservice, and was introduced by in clouddriver/pull#268.

Clouddriver authenticates itself following the official v2 Docker registry specification. The motivation for storing the credentials here, rather than loading them from ~/.docker/config, was to allow the user to segregate accounts with access to the same Registry, but with different sets of permissions.

Caching

For each listed repository the full list of tags is fetched from the /v2/<name>/tags/list endpoint, and stored using the Netflix CATS API. Each tag's manifest is then pulled from the /v2/<name>/manifests/<reference>/ endpoint, and it's digest (Docker-Content-Digest) is cached as well. The images can then be fetched at clouddriver endpoint: /dockerRegistry/images/find?q=<image:tag>&account=<account> endpoint. If either parameter q or account is left blank, it is assumed to mean * (everything). This was implemented in clouddriver/pull#282.

If the user omits the list of repositories they want to cache, the provider will attempt to retrieve every repository in the registry with the _/catalog endpoint at every cache cycle. If this endpoint is not supported, the provider will fail to start. When there are a very large number of repositories, it is recommended to set the cacheThreads count to something larger than 1. The caching will be split among all Clouddriver instances, so scale the cacheThreads accordingly.

Triggers

Since Clouddriver caches every image digest, we can track when individual tags are created or updated. The Igor microservice was extended to poll for incoming docker changes for every account configured in Clouddriver in igor/pull#64. Once it sees a tag was created, or a tag's digest has changed, it sends an event to the Echo microservice, which acts as an event bus. If the updated repository or tag matches one of the exiting pipeline triggers, Echo will start that pipeline. This was added in echo/pull#76. The Deck microservice had the ability to expose adding Docker triggers to pipelines added in deck/pull#2059.

If a tag is omitted from a Docker trigger, the trigger will fire on any change to the selected repository. The the tag that caused the pipeline to trigger will be forwarded to the pipeline to be used during a deploy. (This last step is currently missing, and should be added this month).