azure-pipelines-templates

Opinionated templates for azure pipelines builds.

This repo is to be used as central repository for an organizations azure pipelines templates, so that all libs in an organization can reuse the same core build template, without having to update each lib independently when changes to build process are made.

Supported Build Types

1. Python package builds - single python version
2. Golang module builds
3. Rust crate builds

Getting started

1. Fork this repo into your organization. Azure devops requires that remote templates referenced by a pipeline must be part of the same organization.

2. Setup an Azure Devops account and Azure Pipelines project.

3. Connect a pipeline to your github repo.

Repo Requirements - General

1. Accessible via ssh: This template uses ssh to handle git commands. Directions for enabling ssh access

2. Dev Branch: Builds are triggered from the dev branch. Changes are merged into master at the end of a successful build.

Related Repos

There are a few other repos related to these templates.

• azure-pipelines-scripts: Helper scripts for some pipeline steps. Can be forked or left as-is if no changes are desired. If forked, the BUILD_SCRIPTS_REPO variable in variables.yml should be changed to the fork.

• occlib-py: Python package template with all tooling configs preset.

• occlib-go: Golang module template with all tooling configs preset.

• occlib-rs: Rust module template with all tooling configs preset.

Repo Yaml Templates

The /repo_yaml_templates folder contains template pipeline definitions to be placed in your repo's and references these core templates.

General Build Pipeline

Builds are performed in a single Linux job, in order to reduce build time. All pipelines follow the same general order

1. Lint: Run linters to check code style.

2. Test: Run tests.

3. Check test Coverage: By default, 85% code coverage is required.

4. Version up: The pipeline automatically chooses the next available patch version to be used for the target major, minor version in setup.cfg

5. Build Docs: static html docs are built to ./docs/. There are options for publishing to an S3 bucket, github pages, or both. This build is added to the merge for master.

6. Build software: Creates target build type for the software (go binary / python package / docker container) if needed.

7. Upload build: (to pypi / dockerhub, etc.)

8. Git tag version: Version tag added to git, (ex: v1.1.13)

9. Force merge master: The source code is tagged with the version and force merged into master, favoring the dev branch in code conflict.

10. Master and tags pushed to git: New master branch and build tags are pushed to git.

Pull Request Checks.

The pipeline is run on any PR request made to dev by default. Build steps (stages 5-11 above) are not executed during PR validation.

Documentation Publication Options.

Publish to Github Pages: Since docs are always copied and committed to /docs/ on the master branch, its easy to configure documentation publication through github pages. NOTE: GITHUB PAGES WILL ALWAYS BE PUBLIC, EVEN ON A PRIVATE REPO

Azure Pipelines Variable Groups.

These templates rely on access to logins or credentials for other services, and accesses them via [pipeline variables]. The following variables are required for these templates (depending on the type of build.) They are broken into suggested groups.

• GIT SETTINGS: Used for adding tags and updating master.

  • GIT_USERNAME: User to use with git commits.
  • GIT_EMAIL: Email to use with git commits.

• TWINE CREDENTIALS: Used for uploading public packages to pypi.org

  • PYPIORG_USER: username for pypi.org
  • PYPIORG_PASSWORD: password for pypi.org.

• Crates.io Credentials Used for uploading Rust creates to crates.io

  • CRATES_IO_TOKEN: the API token for your organization's crates.io account

• CONTAINER REGISTRY CREDENTIALS: Used for uploading service images.

  • CONTAINER_REGISTRY_URL: URL for pushing / pulling from container registry.
  • CONTAINER_REGISTRY_ID: ID to sign into container registry.
  • CONTAINER_REGISTRY_PASSWORD: Password to sign into container registry

NOTE: These groups will need to be added to every build pipeline individually. The configuration pane to add groups to a pipeline is currently a little hard to find.

In the overview page for a pipeline go to: Edit -> Hamburger Menu -> Triggers. This will bring you to a general settings page. At the top choose Varaibles -> Variable Groups -> Link variable group to give a pipeline access to a variable group.

To make a new group, look at the left hand pane and go to Pipelines -> Library -> Variable groups.

Python Packages Builds

The template library for using this build pipelines can be found here. The following tools are used during python package builds:

• Dependency Installation: Handled via [pip].

• Linters: Linting is done via

  • Flake8,
  • Black
  • mypy for static-type analysis.

• Tests: Handled via pytest.

• Docs: Handled via sphinx.

• Builds: Handled via setuptols.

• Package Uploads: Handled via twine.

Go Module Builds

The template library for using this build pipelines can be found here. The following tools are used during python package builds:

• Dependency Installation: Handled via go get. Git is configured to use ssh instead of http to enable private package fetching.

• Linters: Linting is done via:

  • Revive
  • Go Vet
  • Go Lint
  • Misspell

• Tests: Handled via go test.

• Docs: Handled via sphinx for quickstarts and guides + a cli tool to generate API Documentation via godoc.

• Builds: Go modules do not need to be built.

• Package Uploads: Handled via merge into master and version tag.

Rust Crate Builds

The template library for using this build pipelines can be found here. The following tools are used during python package builds:

• Dependency Installation: Handled via cargo.

• Linters: Linting is done via

  • rustfmt,
  • clippy
  • Misspell

• Tests: Handled via cargo test.

• Docs: Handled via cargo doc.

• Builds: Handled via cargo.

• Package Uploads: Handled via cargo publish.

Docker Image Builds:

• Root Template: docker_image_main.yml

• Image Builds: Handled via docker commandline + dockerfile.

• Image Uploads: Handled via docker commandline.