Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

378 Convert source tree to python-copier-template #667

Merged
merged 16 commits into from
Jul 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .copier-answers.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Changes here will be overwritten by Copier
_commit: 2.1.0
_src_path: gh:DiamondLightSource/python-copier-template
author_email: [email protected]
author_name: Dominic Oram
component_owner: group:data-acquisition
description: Ophyd devices and other utils that could be used across DLS beamlines
distribution_name: dls-dodal
docker: false
docs_type: sphinx
git_platform: github.com
github_org: DiamondLightSource
package_name: dodal
pypi: true
repo_name: dodal
type_checker: pyright
52 changes: 22 additions & 30 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,52 +3,44 @@
"name": "Python 3 Developer Container",
"build": {
"dockerfile": "../Dockerfile",
"target": "build",
// Only upgrade pip, we will install the project below
"args": {
"PIP_OPTIONS": "--upgrade pip"
}
"target": "developer"
},
"remoteEnv": {
// Allow X11 apps to run inside the container
"DISPLAY": "${localEnv:DISPLAY}"
},
// Add the URLs of features you want added when the container is built.
"features": {
"ghcr.io/devcontainers/features/common-utils:1": {
"username": "none",
"upgradePackages": false
}
},
// Set *default* container specific settings.json values on container create.
"settings": {
"python.defaultInterpreterPath": "/venv/bin/python"
},
"customizations": {
"vscode": {
// Set *default* container specific settings.json values on container create.
"settings": {
"python.defaultInterpreterPath": "/venv/bin/python"
},
// Add the IDs of extensions you want installed when the container is created.
"extensions": [
"ms-python.python",
"github.vscode-github-actions",
"tamasfe.even-better-toml",
"redhat.vscode-yaml",
"ryanluker.vscode-coverage-gutters"
"ryanluker.vscode-coverage-gutters",
"charliermarsh.ruff",
"ms-azuretools.vscode-docker"
]
}
},
// Make sure the files we are mapping into the container exist on the host
"initializeCommand": "bash -c 'for i in $HOME/.inputrc; do [ -f $i ] || touch $i; done'",
"features": {
// Some default things like git config
"ghcr.io/devcontainers/features/common-utils:2": {
"upgradePackages": false
}
},
"runArgs": [
// Allow the container to access the host X11 display and EPICS CA
"--net=host",
"--security-opt=label=type:container_runtime_t"
],
"mounts": [
"source=${localEnv:HOME}/.ssh,target=/root/.ssh,type=bind",
"source=${localEnv:HOME}/.inputrc,target=/root/.inputrc,type=bind",
// map in home directory - not strictly necessary but useful
"source=${localEnv:HOME},target=${localEnv:HOME},type=bind,consistency=cached"
// Make sure SELinux does not disable with access to host filesystems like tmp
"--security-opt=label=disable"
],
// make the workspace folder the same inside and outside of the container
"workspaceMount": "source=${localWorkspaceFolder},target=${localWorkspaceFolder},type=bind",
"workspaceFolder": "${localWorkspaceFolder}",
// Mount the parent as /workspaces so we can pip install peers as editable
"workspaceMount": "source=${localWorkspaceFolder}/..,target=/workspaces,type=bind",
// After the container is created, install the python project in editable form
"postCreateCommand": "pip install -e .[dev]"
"postCreateCommand": "pip install $([ -f dev-requirements.txt ] && echo '-c dev-requirements.txt') -e '.[dev]' && pre-commit install"
}
27 changes: 27 additions & 0 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Contribute to the project

Contributions and issues are most welcome! All issues and pull requests are
handled through [GitHub](https://github.com/DiamondLightSource/dodal/issues). Also, please check for any existing issues before
filing a new one. If you have a great idea but it involves big changes, please
file a ticket before making a pull request! We want to make sure you don't spend
your time coding something that might not fit the scope of the project.

## Issue or Discussion?

Github also offers [discussions](https://github.com/DiamondLightSource/dodal/discussions) as a place to ask questions and share ideas. If
your issue is open ended and it is not obvious when it can be "closed", please
raise it as a discussion instead.

## Code Coverage

While 100% code coverage does not make a library bug-free, it significantly
reduces the number of easily caught bugs! Please make sure coverage remains the
same or is improved by a pull request!

## Developer Information

It is recommended that developers use a [vscode devcontainer](https://code.visualstudio.com/docs/devcontainers/containers). This repository contains configuration to set up a containerized development environment that suits its own needs.

This project was created using the [Diamond Light Source Copier Template](https://github.com/DiamondLightSource/python-copier-template) for Python projects.

For more information on common tasks like setting up a developer environment, running the tests, and setting a pre-commit hook, see the template's [How-to guides](https://diamondlightsource.github.io/python-copier-template/2.1.0/how-to.html).
35 changes: 0 additions & 35 deletions .github/CONTRIBUTING.rst

This file was deleted.

66 changes: 20 additions & 46 deletions .github/actions/install_requirements/action.yml
Original file line number Diff line number Diff line change
@@ -1,60 +1,34 @@
name: Install requirements
description: Run pip install with requirements and upload resulting requirements
description: Install a version of python then call pip install and report what was installed
inputs:
requirements_file:
description: Name of requirements file to use and upload
required: true
install_options:
python-version:
description: Python version to install, default is from Dockerfile
default: "dev"
pip-install:
description: Parameters to pass to pip install
required: true
artifact_name:
description: A user friendly name to give the produced artifacts
required: true
python_version:
description: Python version to install
default: "3.x"
default: "$([ -f dev-requirements.txt ] && echo '-c dev-requirements.txt') -e .[dev]"

runs:
using: composite

steps:
- name: Get version of python
run: |
PYTHON_VERSION="${{ inputs.python-version }}"
if [ $PYTHON_VERSION == "dev" ]; then
PYTHON_VERSION=$(sed -n "s/ARG PYTHON_VERSION=//p" Dockerfile)
fi
echo "PYTHON_VERSION=$PYTHON_VERSION" >> "$GITHUB_ENV"
shell: bash

- name: Setup python
uses: actions/setup-python@v5
with:
python-version: ${{ inputs.python_version }}
python-version: ${{ env.PYTHON_VERSION }}

- name: Pip install
run: |
touch ${{ inputs.requirements_file }}
# -c uses requirements.txt as constraints, see 'Validate requirements file'
pip install -c ${{ inputs.requirements_file }} ${{ inputs.install_options }}
- name: Install packages
run: pip install ${{ inputs.pip-install }}
shell: bash

- name: Create lockfile
run: |
mkdir -p lockfiles
pip freeze --exclude-editable > lockfiles/${{ inputs.requirements_file }}
# delete the self referencing line and make sure it isn't blank
sed -i'' -e '/file:/d' lockfiles/${{ inputs.requirements_file }}
shell: bash

- name: Upload lockfiles
uses: actions/[email protected]
with:
name: lockfiles-${{ inputs.python_version }}-${{ inputs.artifact_name }}-${{ github.sha }}
path: lockfiles

# This eliminates the class of problems where the requirements being given no
# longer match what the packages themselves dictate. E.g. In the rare instance
# where I install some-package which used to depend on vulnerable-dependency
# but now uses good-dependency (despite being nominally the same version)
# pip will install both if given a requirements file with -r
- name: If requirements file exists, check it matches pip installed packages
run: |
if [ -s ${{ inputs.requirements_file }} ]; then
if ! diff -u ${{ inputs.requirements_file }} lockfiles/${{ inputs.requirements_file }}; then
echo "Error: ${{ inputs.requirements_file }} need the above changes to be exhaustive"
exit 1
fi
fi
- name: Report what was installed
run: pip freeze
shell: bash
8 changes: 6 additions & 2 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,15 @@ updates:
schedule:
interval: "weekly"
groups:
github-artifacts:
actions:
patterns:
- actions/*-artifact
- "*"

- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "weekly"
groups:
dev-dependencies:
patterns:
- "*"
2 changes: 1 addition & 1 deletion .github/pages/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@
<link rel="canonical" href="main/index.html">
</head>

</html>
</html>
29 changes: 12 additions & 17 deletions .github/pages/make_switcher.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,28 +3,27 @@
from argparse import ArgumentParser
from pathlib import Path
from subprocess import CalledProcessError, check_output
from typing import List


def report_output(stdout: bytes, label: str) -> List[str]:
def report_output(stdout: bytes, label: str) -> list[str]:
ret = stdout.decode().strip().split("\n")
print(f"{label}: {ret}")
return ret


def get_branch_contents(ref: str) -> List[str]:
def get_branch_contents(ref: str) -> list[str]:
"""Get the list of directories in a branch."""
stdout = check_output(["git", "ls-tree", "-d", "--name-only", ref])
return report_output(stdout, "Branch contents")


def get_sorted_tags_list() -> List[str]:
def get_sorted_tags_list() -> list[str]:
"""Get a list of sorted tags in descending order from the repository."""
stdout = check_output(["git", "tag", "-l", "--sort=-v:refname"])
return report_output(stdout, "Tags list")


def get_versions(ref: str, add: str | None, remove: str | None) -> List[str]:
def get_versions(ref: str, add: str | None) -> list[str]:
"""Generate the file containing the list of all GitHub Pages builds."""
# Get the directories (i.e. builds) from the GitHub Pages branch
try:
Expand All @@ -36,15 +35,12 @@ def get_versions(ref: str, add: str | None, remove: str | None) -> List[str]:
# Add and remove from the list of builds
if add:
builds.add(add)
if remove:
assert remove in builds, f"Build '{remove}' not in {sorted(builds)}"
builds.remove(remove)

# Get a sorted list of tags
tags = get_sorted_tags_list()

# Make the sorted versions list from main branches and tags
versions: List[str] = []
versions: list[str] = []
for version in ["master", "main"] + tags:
if version in builds:
versions.append(version)
Expand All @@ -58,9 +54,12 @@ def get_versions(ref: str, add: str | None, remove: str | None) -> List[str]:

def write_json(path: Path, repository: str, versions: str):
org, repo_name = repository.split("/")
pages_url = f"https://{org}.github.io"
if repo_name != f"{org}.github.io":
# Only add the repo name if it isn't the source for the org pages site
pages_url += f"/{repo_name}"
struct = [
{"version": version, "url": f"https://{org}.github.io/{repo_name}/{version}/"}
for version in versions
{"version": version, "url": f"{pages_url}/{version}/"} for version in versions
]
text = json.dumps(struct, indent=2)
print(f"JSON switcher:\n{text}")
Expand All @@ -69,16 +68,12 @@ def write_json(path: Path, repository: str, versions: str):

def main(args=None):
parser = ArgumentParser(
description="Make a versions.txt file from gh-pages directories"
description="Make a versions.json file from gh-pages directories"
)
parser.add_argument(
"--add",
help="Add this directory to the list of existing directories",
)
parser.add_argument(
"--remove",
help="Remove this directory from the list of existing directories",
)
parser.add_argument(
"repository",
help="The GitHub org and repository name: ORG/REPO",
Expand All @@ -91,7 +86,7 @@ def main(args=None):
args = parser.parse_args(args)

# Write the versions file
versions = get_versions("origin/gh-pages", args.add, args.remove)
versions = get_versions("origin/gh-pages", args.add)
write_json(args.output, args.repository, versions)


Expand Down
27 changes: 27 additions & 0 deletions .github/workflows/_check.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
on:
workflow_call:
outputs:
branch-pr:
description: The PR number if the branch is in one
value: ${{ jobs.pr.outputs.branch-pr }}

jobs:
pr:
runs-on: "ubuntu-latest"
outputs:
branch-pr: ${{ steps.script.outputs.result }}
steps:
- uses: actions/github-script@v7
id: script
if: github.event_name == 'push'
with:
script: |
const prs = await github.rest.pulls.list({
owner: context.repo.owner,
repo: context.repo.repo,
head: context.repo.owner + ':${{ github.ref_name }}'
})
if (prs.data.length) {
console.log(`::notice ::Skipping CI on branch push as it is already run in PR #${prs.data[0]["number"]}`)
return prs.data[0]["number"]
}
Loading
Loading