Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conda lock files #5827

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions .github/workflows/wave.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
name: Generate conda-lock
# When environment.yml is changed
on:
pull_request:
paths:
- "**/environment.yml"

jobs:
gen-matrix:
name: generate-matrix
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4

- name: Calculate file differences
id: diff
uses: tj-actions/changed-files@v44
with:
json: true
quotepath: false
files: |
modules/**/environment.yml
- name: Debug
run: echo ${{ steps.diff.outputs.all_changed_files }}
- id: set-matrix
run: echo "matrix={\"files\":${{ steps.diff.outputs.all_changed_files }} }" >> "$GITHUB_OUTPUT"

outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}

lock:
if: "${{ fromJson(needs.gen-matrix.outputs.matrix) }}"
needs: gen-matrix
name: Generate conda-lock file
permissions:
contents: write
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix: "${{ fromJson(needs.gen-matrix.outputs.matrix) }}"
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4
with:
ref: ${{ github.event.pull_request.head.ref }}

- uses: mamba-org/setup-micromamba@v1
with:
environment-name: conda-lock-env
create-args: >-
python=3.11
conda-lock
mamba

- name: Get the lock file path
uses: actions/github-script@v7
id: lockfile-name
with:
result-encoding: string
script: |
return '${{ matrix.files }}'.replace('/environment.yml', '/conda-lock.yml');

- name: Run conda-lock
run: |
rm --force "${{steps.lockfile-name.outputs.result}}"
conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"
Copy link
Member

@pinin4fjords pinin4fjords Aug 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I tested why the lock file didn't work, and I think it's because of the default unified lock files. To get a lock file compatible with Nextflow (which just does like this), you have to render it to a platform specific one:

conda-lock render -p osx-64  
Rendering lockfile(s) for osx-64...
 - Install lock using : conda create --name YOURENV --file conda-osx-64.lock

We can also just do:

conda-lock --kind explicit -f environment.yml

... which automatically renders to all the specified platforms.

In any case, for nextflow as-is, we will need platform-specific lockfiles.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! I didn't really want to have 4+ lock files floating around. I didn't know this was a possibility, great find!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid we're stuck with those 4 lock files, unless we do that rendering dynamically somehow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"
conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind explicit --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"
mv conda-linux-64.lock conda-lock-linux-64.yml

Nextflow needs it to be a .yml, but use of --kind explicit means we get .lock files, so we'll need to do some file renaming.

Platforms should probably be set in the environment.ymls actually.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just do...

Suggested change
conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"
conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}".yml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't look like that works 😞

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The explicit thing is important. It makes you a lock file for each platform which is consumable by conda env create, and therefore by Nextflow.

But by doing explicit you lose control over the naming of the multiple outputs, and Nextflow needs .yml, hence the rename hack I suggested.

env:
MAMBA: "--mamba"
shell: bash -l {0}

- name: Commit & push changes
id: commit-and-push
run: |
git config user.email "[email protected]"
git config user.name "nf-core-bot"
git config push.default upstream
git add .
git status
git commit -m "[automated] autogenerated conda-lock file"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
git commit -m "[automated] autogenerated conda-lock file"
git commit -m "[automated] autogenerated conda-lock files"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should just be a single commit per lock file I think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One lock file per platform

git push

- name: If no conda-lock update
if: steps.commit-and-push.outcome == 'failure'
run: |
echo "No conda-lock update"

# TODO Build containers with Wave
1 change: 1 addition & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ __pycache__
*.pyo
*.pyc
.github/renovate.json5
**/conda-lock.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**/conda-lock.yml
**/conda-lock*.yml

Loading
Loading