Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Conda lock files #5827

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Conda lock files #5827

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
dbad13e
ci(#2193): Add condalock file generation
edmundmiller Jun 18, 2024
6ea762d
ci: Add write permissions
edmundmiller Jun 18, 2024
b740f85
ci: Add gh pr checkout
edmundmiller Jun 18, 2024
95ba6cc
ci: Try a pre-canned action
edmundmiller Jun 18, 2024
07a250e
ci: Cut out conda-lock-refresh
edmundmiller Jun 18, 2024
b3ef764
ci: Use plain GITHUB_TOKEN secret
edmundmiller Jun 18, 2024
98cbce3
ci: Skip committing
edmundmiller Jun 18, 2024
253a398
ci: Use Commit step from prettier linting
edmundmiller Aug 8, 2024
f085cc1
ci: Try actions/checkout#719
edmundmiller Aug 8, 2024
fe828fa
ci: Clean up commit message
edmundmiller Aug 8, 2024
8433c87
ci: Output lock file to the directory
edmundmiller Aug 8, 2024
a9f4d26
chore: Update an environment file for testing
edmundmiller Aug 8, 2024
22c4985
[automated] autogenerated conda-lock file
nf-core-bot Aug 8, 2024
3afecb6
ci: Clean up the matrix to avoid a race condition
edmundmiller Aug 8, 2024
c6b7ae6
style: Ignore conda-lock for prettier
edmundmiller Aug 8, 2024
251b8ca
ci: Allow no conda-lock updates
edmundmiller Aug 8, 2024
de2fdf7
build: Try a conda-lock file
edmundmiller Aug 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions .github/workflows/wave.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
name: Generate conda-lock
# When environment.yml is changed
on:
pull_request:
paths:
- "**/environment.yml"

jobs:
gen-matrix:
name: generate-matrix
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4

- name: Calculate file differences
id: diff
uses: tj-actions/changed-files@v44
with:
json: true
quotepath: false
files: |
modules/**/environment.yml
- name: Debug
run: echo ${{ steps.diff.outputs.all_changed_files }}
- id: set-matrix
run: echo "matrix={\"files\":${{ steps.diff.outputs.all_changed_files }} }" >> "$GITHUB_OUTPUT"

outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}

lock:
if: "${{ fromJson(needs.gen-matrix.outputs.matrix) }}"
needs: gen-matrix
name: Generate conda-lock file
permissions:
contents: write
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix: "${{ fromJson(needs.gen-matrix.outputs.matrix) }}"
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4
with:
ref: ${{ github.event.pull_request.head.ref }}

- uses: mamba-org/setup-micromamba@v1
with:
environment-name: conda-lock-env
create-args: >-
python=3.11
conda-lock
mamba

- name: Get the lock file path
uses: actions/github-script@v7
id: lockfile-name
with:
result-encoding: string
script: |
return '${{ matrix.files }}'.replace('/environment.yml', '/conda-lock.yml');

- name: Run conda-lock
run: |
rm --force "${{steps.lockfile-name.outputs.result}}"
conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"
Copy link
Member

@pinin4fjords pinin4fjords Aug 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I tested why the lock file didn't work, and I think it's because of the default unified lock files. To get a lock file compatible with Nextflow (which just does like this), you have to render it to a platform specific one:

conda-lock render -p osx-64  
Rendering lockfile(s) for osx-64...
 - Install lock using : conda create --name YOURENV --file conda-osx-64.lock

We can also just do:

conda-lock --kind explicit -f environment.yml

... which automatically renders to all the specified platforms.

In any case, for nextflow as-is, we will need platform-specific lockfiles.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! I didn't really want to have 4+ lock files floating around. I didn't know this was a possibility, great find!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid we're stuck with those 4 lock files, unless we do that rendering dynamically somehow.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"
conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind explicit --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"
mv conda-linux-64.lock conda-lock-linux-64.yml

Nextflow needs it to be a .yml, but use of --kind explicit means we get .lock files, so we'll need to do some file renaming.

Platforms should probably be set in the environment.ymls actually.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just do...

Suggested change
conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}"
conda-lock lock $MAMBA --file "${{ matrix.files }}" --kind lock --platform linux-64 --lockfile "${{steps.lockfile-name.outputs.result}}".yml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't look like that works 😞

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The explicit thing is important. It makes you a lock file for each platform which is consumable by conda env create, and therefore by Nextflow.

But by doing explicit you lose control over the naming of the multiple outputs, and Nextflow needs .yml, hence the rename hack I suggested.

env:
MAMBA: "--mamba"
shell: bash -l {0}

- name: Commit & push changes
id: commit-and-push
run: |
git config user.email "[email protected]"
git config user.name "nf-core-bot"
git config push.default upstream
git add .
git status
git commit -m "[automated] autogenerated conda-lock file"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
git commit -m "[automated] autogenerated conda-lock file"
git commit -m "[automated] autogenerated conda-lock files"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should just be a single commit per lock file I think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One lock file per platform

git push

- name: If no conda-lock update
if: steps.commit-and-push.outcome == 'failure'
run: |
echo "No conda-lock update"

# TODO Build containers with Wave
1 change: 1 addition & 0 deletions .prettierignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ __pycache__
*.pyo
*.pyc
.github/renovate.json5
**/conda-lock.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**/conda-lock.yml
**/conda-lock*.yml

Loading
Loading