Extract conda-lock workflow into standalone GitHub Action?

[weiji14]

Continuing discussion on whether to move some of the conda-lock related automation into a standalone GitHub Actions Continuous Integration workflow that can be shared with https://github.com/pangeo-data/pangeo-docker-images or other people.

@weiji14 how do you feel about extracting the conda-lock stuff from here into an action by itself? Then we can use that action in pangeo-docker-images, thus removing duplicate work...

Originally posted by @yuvipanda in #13 (comment)

To be clear, do we mean:

• The /condalock command added in Add slash command /condalock to autogenerate conda-lock files #5, which I think pangeo-docker-images has a version of already maintained by @scottyhq?
• The 'installing from conda-lock.yml' script being worked on at Using unified conda-lock file in postBuild for reproducible conda environment #14? A quick search led to conda-incubator/setup-miniconda fails when using conda-lock.yml conda/conda-lock#245 (comment) and apparently there might be something re-usable already at https://github.com/mamba-org/provision-with-micromamba#example-with-customization?

[scottyhq]

hi @weiji14 i like the 2nd option of always installing from a lockfile. the /condalock command to be honest was a somewhat experimental thing that we tried early in learning github actions! the goal was to allow anyone to add packages (even from forks), but run the locking behind the scenes.

[weiji14]

i like the 2nd option of always installing from a lockfile

Yeah, I agree that lockfiles are good for reproducible environments. I know pangeo-docker-images is currently installing from explicit lockfiles (conda-linux-64.lock) at https://github.com/pangeo-data/pangeo-docker-images/blob/2e6d92f01e959ad2a5cc224e5090023dab81d828/base-image/Dockerfile#L155-L156, but the unified lockfile (conda-lock.yml) is something I'm keen to push on since it handles conda/pip dependencies, and is cross-platform Linux/macOS/Windows.

What are your thoughts @scottyhq from a maintenance perspective? This issue is really to discuss where best to put the 'install-from-lockfiles' piece of code in the context of building reproducible docker images. Should this be added to repo2docker? I.e. this issue at jupyterhub/repo2docker#1157? Or somewhere else?

And yes, the /condalock slash command is great 😄

[scottyhq]

Good questions!

The goal is to have a reproducible environment installed either in CI or locally and right now you either have to 1. render a bunch of files with platform-specific locked packages or 2. have people install conda-lock and install the environment from there. (or 3. use Docker everywhere)

Short term, I'm supportive of moving to embracing a single conda-lock.yml and consequently micromamba or conda-lock in the CI workflow.

I don't always want to use repo2docker because it can be helpful to have more control over the Dockerfile. But ultimately, a way to contribute upstream and have the widest impact might be to help standardize the conda-lock.yml format. I don't know how much the people working on conda, mamba, and conda-lock are cross-communicating currently, but ideally the conda-lock.yml would be a standard recognized by different tools. Most people use conda or mamba locally and unfortunately you can't mamba create -f conda-lock.yml mamba-org/mamba#1884...

[weiji14]

Short term, I'm supportive of moving to embracing a single conda-lock.yml and consequently micromamba or conda-lock in the CI workflow.

Yes, and for GitHub Actions CI, using the unified conda-lock.yml lockfile might be as easy as changing from conda-incubator/setup-miniconda to mamba-org/provision-with-micromamba according to conda/conda-lock#245 (comment) (though I haven't tested this).

ideally the conda-lock.yml would be a standard recognized by different tools. Most people use conda or mamba locally and unfortunately you can't mamba create -f conda-lock.yml mamba-org/mamba#1884...

Agree, having the conda-lock.yml format supported upstream by mamba/conda would be the most seamless! This could then trickle downstream to repo2docker that could list conda-lock.yml as another supported config file, and advanced users using Dockerfile would be able to use conda-lock.yml too by just having mamba/conda installed.

While we wait for those ☝️ to be implemented by smart people, is a /condalock GitHub Action or GitHub App still something worthwhile to code up? I guess that means packaging what's in CondaLock.yml and conda-lock-command.yml (plus the ChatOps stuff) into something reusable across different repositories.

[yuvipanda]

I'd love for the CondaLock.yml thingy to be packaged up and moved into a default, especially in a form that can also handle pip dependencies. Additionally, I think it should also autorun and not require a chatops command. If we can make that work, I'd love to incorporate that into https://github.com/2i2c-org/hub-user-image-template/ :)

[yuvipanda]

Coming back after the holidays, @weiji14 do you think you would be able to work on this? :)

[weiji14]

Coming back after the holidays, @weiji14 do you think you would be able to work on this? :)

I'm currently back in Malaysia visiting family, and won't have much capacity until mid Feb at least. You can start working on some pieces first if you're interested!

[yuvipanda]

@weiji14 oooh have fun! I don't think I'll have much time either, so this can wait :)

[weiji14]

But ultimately, a way to contribute upstream and have the widest impact might be to help standardize the conda-lock.yml format. I don't know how much the people working on conda, mamba, and conda-lock are cross-communicating currently, but ideally the conda-lock.yml would be a standard recognized by different tools.

Just noticed that conda-lock graduated from conda-incubator to the main conda organisation at https://github.com/conda/conda-lock! Hopefully this means conda-lock.yml uptake will increase this year 😄

My thought is that we should wait for a conda-lock 1.5.0 release (see conda/conda-lock#336) for them to iron out any issues with the organization migration, and then someone could start working on the /condalock GitHub Action/App. There's also a big issue with conda-lock and conda=22.11.1 at conda/conda-lock#292 which I hit into recently, and I kinda want to see that resolved before we get working on anything serious.

Since this /condalock GitHub Action/App isn't cryo-specific, what organization do you think we should host it under? I see @scottyhq opened a parallel issue at pangeo-data/pangeo-docker-images#414, so should it go under the pangeo-data org, or somewhere else?

[weiji14]

I've got a conda-lock composite GitHub Action over at https://github.com/weiji14/conda-lock-refresh, and a working demo over at weiji14/conda-lock-refresh-demo#6 that seems to be working for a simple user repo.

As a next step, it would be good to test things out a bit on a organization repo + forked repos to make sure things like permissions work ok, and would like to use this CryoCloud hub-image repo as a test if people are ok with it. It would involve replacing the slash-command-dispatch.yml and conda-lock-command.yml workflows with a single workflow, similar to this example here.

I'll open up a few PRs in the next few days to get things implemented, and if it works well, I'll look at transferring the GitHub Action repo over to a proper organization, cc @yuvipanda.