Using unified conda-lock file in postBuild for reproducible conda environment #14

weiji14 · 2022-11-29T18:44:02Z

Towards better reproducibility of the conda environment. Done by having a custom conda-lock command in postBuild, rather than letting repo2docker build directly from an enviroment.yml file. Suggestion from #11 (comment)

TODO:

Rename environment.yml file to environment-cryocloud.yml (so that Binder doesn't use it. Install from lockfile happens in postBuild)
Change conda-lock command from generating explicit lockfiles to unified lockfiles
Implement postBuild bash script
Test to ensure things work (Test locking unified condalock attempt 5 weiji14/hub-image#8)

Addresses #11. Closes #15.

So that repo2docker doesn't use it to build the docker image.

github-actions · 2022-11-29T18:44:15Z

👈 Test this PR on Binder

Minor version updates to fix `mamba` solver.

weiji14 · 2022-11-29T20:33:47Z

/condalock

Handle both conda and pip dependencies by changing from the old explicit conda-lock (<1.0) format to the newer unified format. Deleting conda-linux-64.lock and replacing it with a placeholder conda-lock.yml for now.

So that the /condalock slash command works.

weiji14 · 2022-11-29T20:45:38Z

/condalock

…ent.yml" This reverts commit 95d8ec8.

https://repo2docker.readthedocs.io/en/latest/config_files.html#postbuild-run-code-after-installing-the-environment

Need a proper placeholder conda-lock.yml to prevent `AttributeError: 'NoneType' object has no attribute 'pop'` when running `conda-lock lock`.

The `--file` option doesn't actually work, even though it's documented like so.

See #15.

Update instructions on how to add or update dependencies in the cryocloud docker image. Specifically, the part where `conda-lock` is used for a fully reproducible conda environment.

Needed to avoid `Warning: found pip deps, but conda-lock was installed without pypi support. pip dependencies will not be included in the lock file`. See https://conda-incubator.github.io/conda-lock/getting_started.

No more explicit conda-linux-64.lock, just the unified conda-lock.yml.

Follow the default environment name on the JupyterHub rather than the conda environment's name.

weiji14 · 2022-11-29T22:14:26Z

.github/workflows/conda-lock-command.yml

@@ -42,12 +42,12 @@ jobs:

      # Install conda-lock library
      - name: Install conda-lock
-        run: pip install conda-lock
+        run: pip install conda-lock[pip_support]==1.0.5


Using conda-lock 1.0.5 instead of the latest 1.2.1 to keep the relative file paths intact. See #15.

Ok, might need to upgrade to conda-lock 1.2.1 actually, because it contains important bugfixes related to pip dependency hashes. E.g. see https://github.com/CryoInTheCloud/hub-image/actions/runs/3578432352/jobs/6018581680#step:5:4747

ERROR:root:ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them. ERROR:root: fastjsonschema@ https://files.pythonhosted.org/packages/e4/be/cf1b876348070a23cb0c3ebfee7a452ad3a91b07b456dade3bd514656009/fastjsonschema-2.16.2-py3-none-any.whl#md5=None from https://files.pythonhosted.org/packages/e4/be/cf1b876348070a23cb0c3ebfee7a452ad3a91b07b456dade3bd514656009/fastjsonschema-2.16.2-py3-none-any.whl#md5=None (from -r /tmp/tmpn_q_dp50 (line 1)): ERROR:root: Expected md5 None ERROR:root: Got fff77553c8612b49c8782a5233541199 ERROR:root: ERROR:root: ERROR:root:ERROR conda.cli.main_run:execute(49): `conda run pip install --no-deps -r /tmp/tmpn_q_dp50` failed. (See above for error) Traceback (most recent call last): File "/srv/conda/envs/notebook/bin/conda-lock", line 8, in <module> File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1130, in __call__ File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1055, in main File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1657, in invoke File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1404, in invoke File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 760, in invoke File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func File "/srv/conda/envs/notebook/lib/python3.7/site-packages/conda_lock/conda_lock.py", line 1268, in install File "/srv/conda/envs/notebook/lib/python3.7/site-packages/conda_lock/conda_lock.py", line 219, in do_conda_install File "/srv/conda/envs/notebook/lib/python3.7/site-packages/conda_lock/invoke_conda.py", line 131, in _invoke_conda poetry.utils._compat.CalledProcessError: Command '['/srv/conda/bin/mamba', 'run', '--name', 'notebook', 'pip', 'install', '--no-deps', '-r', '/tmp/tmpn_q_dp50']' returned non-zero exit status 1. Removing intermediate container 4d439bce016f The command '/bin/sh -c ./postBuild' returned a non-zero code: 1

weiji14 · 2022-11-29T22:15:46Z

postBuild

+#!/bin/bash
+set -eux
+pip install conda-lock[pip_support]==1.0.5
+conda-lock install --mamba --name notebook conda-lock.yml


Not sure if it matters, but I've set the conda environment name as notebook here instead of cryocloud, because that's the default conda environment name on the CryoCloud JupyterHub.

Also drop the pip_support extras since poetry has been vendored in conda-lock 1.2.0.

Using GitHub Actions hosted runner filepath.

postBuild

yuvipanda

couple comments, otherwise this looks great!

Co-Authored-By: Yuvi Panda <[email protected]>

weiji14 · 2022-11-29T23:32:54Z

Hitting into this bug with pywinpty, see conda/conda-lock#236. Error from https://github.com/CryoInTheCloud/hub-image/actions/runs/3578701513/jobs/6019136464#step:5:4686

  INFO:root:Collecting pywinpty@ https://files.pythonhosted.org/packages/22/77/59645ee028f410ef1d582fc2923d2cb61016ed38e6ae5022f405227ca2ad/pywinpty-2.0.9.tar.gz#sha256=01b6400dd79212f50a2f01af1c65b781290ff39610853db99bf03962eb9a615f
  INFO:root:  Downloading pywinpty-2.0.9.tar.gz (26 kB)
  INFO:root:  Installing build dependencies: started
  INFO:root:  Installing build dependencies: finished with status 'done'
  INFO:root:  Getting requirements to build wheel: started
  INFO:root:  Getting requirements to build wheel: finished with status 'done'
  INFO:root:  Preparing metadata (pyproject.toml): started
  INFO:root:  Preparing metadata (pyproject.toml): finished with status 'error'
  INFO:root:
  ERROR:root:  error: subprocess-exited-with-error
  ERROR:root:
  ERROR:root:  × Preparing metadata (pyproject.toml) did not run successfully.
  ERROR:root:  │ exit code: 1
  ERROR:root:  ╰─> [6 lines of output]
  ERROR:root:
  ERROR:root:      Cargo, the Rust package manager, is not installed or is not on PATH.
  ERROR:root:      This package requires Rust and Cargo to compile extensions. Install it through
  ERROR:root:      the system's package manager or via https://rustup.rs/
  ERROR:root:
  ERROR:root:      Checking for Rust toolchain....
  ERROR:root:      [end of output]
  ERROR:root:
  ERROR:root:  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR:root:error: metadata-generation-failed
  ERROR:root:
  ERROR:root:× Encountered error while generating package metadata.
  ERROR:root:╰─> See above for output.
  ERROR:root:
  ERROR:root:note: This is an issue with the package mentioned above, not pip.
  ERROR:root:hint: See above for details.
  ERROR:root:
  ERROR:root:ERROR conda.cli.main_run:execute(49): `conda run pip install --no-deps -r /tmp/tmpp9s90hbc` failed. (See above for error)
  Traceback (most recent call last):
    File "/srv/conda/envs/notebook/bin/conda-lock", line 8, in <module>
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1055, in main
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/conda_lock/conda_lock.py", line 1288, in install
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/conda_lock/conda_lock.py", line 220, in do_conda_install
    File "/srv/conda/envs/notebook/lib/python3.7/site-packages/conda_lock/invoke_conda.py", line 131, in _invoke_conda
  conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['/srv/conda/bin/mamba', 'run', '--name', 'notebook', 'pip', 'install', '--no-deps', '-r', '/tmp/tmpp9s90hbc']' returned non-zero exit status 1.
  Removing intermediate container f13e567c749f
  The command '/bin/sh -c ./postBuild' returned a non-zero code: 1

This might take a while to resolve on conda-lock, and I'm not sure if there's a good workaround. I think it'll be best if I cherry-pick 90fe9f6 into a separate PR so that #13 can proceed.

Add back in the 'start'` file to `.github/workflows/build.yaml`.

Replaced by conda-linux.yml.

weiji14 · 2022-12-04T04:02:25Z

Marking as draft again as the bug at #14 (comment) isn't fixed yet, waiting on conda-forge/staged-recipes#21369 🤞 (which is itself waiting on a few other things).

yuvipanda · 2022-12-04T22:57:19Z

conda-forge/staged-recipes#21369 might take a while I think :D Does this mean @weiji14 this repo can only install from conda and can't install from pip?

weiji14 · 2022-12-05T01:13:13Z

conda-forge/staged-recipes#21369 might take a while I think :D Does this mean @weiji14 this repo can only install from conda and can't install from pip?

Yes, only via mamba/conda. Using conda-lock.yml won't work unless we get jupyter-desktop-server on conda-forge, or the issue with winpty gets resolved in conda-lock.

yuvipanda · 2022-12-05T01:39:33Z

Hmm, that seems a major limitation going forward honestly :| PyPI will always have things that won't be on conda-forge I think. Do you know if conda-lock might resolve this, or is it a non-goal?

weiji14 · 2022-12-05T01:44:07Z

Hmm, that seems a major limitation going forward honestly :| PyPI will always have things that won't be on conda-forge I think. Do you know if conda-lock might resolve this, or is it a non-goal?

The issue is tracked in conda/conda-lock#236, and actually, jupyter-desktop-server is a special case because it indirectly depends on winpty which is Windows only. Most other PyPI packages won't have this issue, so I still think it's ok.

yuvipanda · 2022-12-05T01:44:41Z

AAAAAAH, apologies @weiji14 for not following links - and thank you so much for clarifying this is desktop-server specific!

weiji14 · 2023-05-28T22:32:55Z

Ready for review @yuvipanda! I've updated to conda-lock=2.0.0 which fixes the Windows-only pywinpty issue (thanks to @scottyhq's patch in conda/conda-lock#295), so we don't need to wait for a jupyter-desktop-server conda-forge package anymore 🎉 See weiji14@ec088de for an example of /condalock working successfully with the new update.

Once this is merged, and tested a few times, we should be able to move ahead with the /condalock GitHub Action work as mentioned in #18.

.github/workflows/build.yaml

.github/workflows/test.yaml

Co-authored-by: Matt Fisher <[email protected]>

Conflicts: conda-linux-64.lock

Bumps [conda-lock](https://github.com/conda/conda-lock) from 2.0.0 to 2.1.0. - [Release notes](https://github.com/conda/conda-lock/releases) - [Commits](conda/conda-lock@v2.0.0...v2.1.0)

weiji14 · 2024-08-15T01:09:59Z

Note to self. A 'conda-lock.yml' file was added in #122 already, though the builds which rely on repo2docker are technically still using 'environment.yml' so not 100% reproducible yet

hub-image/.github/workflows/build.yaml

Lines 35 to 36 in e0ad226

    
           - name: Build and push the image to quay.io 
        
             uses: jupyterhub/repo2docker-action@master

Maybe wait for repo2docker to support conda-lock natively (jupyterhub/repo2docker#1312), and then the 'postBuild' hack here can go away.

Rename environment.yml file to environment-cryocloud.yml

5857db0

So that repo2docker doesn't use it to build the docker image.

weiji14 self-assigned this Nov 29, 2022

Bump to cartopy 0.21.0, rasterio 1.3.3, hdf5 1.12.2, qgis 3.22.11

90fe9f6

Minor version updates to fix `mamba` solver.

weiji14 added 2 commits November 29, 2022 15:43

Replace explicit conda-linux-64.lock file with unified conda-lock.yml

1d23d26

Handle both conda and pip dependencies by changing from the old explicit conda-lock (<1.0) format to the newer unified format. Deleting conda-linux-64.lock and replacing it with a placeholder conda-lock.yml for now.

Temporarily rename environment-cryocloud.yml back to environment.yml

95d8ec8

So that the /condalock slash command works.

weiji14 and others added 10 commits November 29, 2022 15:50

Revert "Temporarily rename environment-cryocloud.yml back to environm…

09ecd55

…ent.yml" This reverts commit 95d8ec8.

Add postBuild script to install dependencies from unified conda-lock.yml

00cc297

https://repo2docker.readthedocs.io/en/latest/config_files.html#postbuild-run-code-after-installing-the-environment

Minimal conda-lock.yml with only Python 3.10

d07eafb

Need a proper placeholder conda-lock.yml to prevent `AttributeError: 'NoneType' object has no attribute 'pop'` when running `conda-lock lock`.

Fix conda-lock install command in postBuild

ef9cc29

The `--file` option doesn't actually work, even though it's documented like so.

Pin to conda-lock 1.0.5 to keep relative paths

e609d9c

See #15.

Document how to update conda-lock.yml file in the main README.md

4ea512b

Update instructions on how to add or update dependencies in the cryocloud docker image. Specifically, the part where `conda-lock` is used for a fully reproducible conda environment.

Install conda-lock with pip_support extras

abb8710

Needed to avoid `Warning: found pip deps, but conda-lock was installed without pypi support. pip dependencies will not be included in the lock file`. See https://conda-incubator.github.io/conda-lock/getting_started.

Run docker build and test workflow when conda-linux.yml is modified

33436bc

No more explicit conda-linux-64.lock, just the unified conda-lock.yml.

Pin minimum jupyterlab to 3.5

3e08468

[condalock-command] autogenerated conda-lock files

85b01cb

weiji14 marked this pull request as ready for review November 29, 2022 22:09

Set notebook as conda environment name in docker image

f82355e

Follow the default environment name on the JupyterHub rather than the conda environment's name.

weiji14 commented Nov 29, 2022

View reviewed changes

weiji14 and others added 3 commits November 29, 2022 17:37

Bump conda-lock from 1.0.5 to 1.2.1

f0b86fe

Also drop the pip_support extras since poetry has been vendored in conda-lock 1.2.0.

Update conda-lock's dependency file source

20ec34b

Using GitHub Actions hosted runner filepath.

[condalock-command] autogenerated conda-lock files

3c6a826

yuvipanda reviewed Nov 29, 2022

View reviewed changes

postBuild Outdated Show resolved Hide resolved

yuvipanda reviewed Nov 29, 2022

View reviewed changes

postBuild Outdated Show resolved Hide resolved

yuvipanda reviewed Nov 29, 2022

View reviewed changes

Improvements to postBuild script

70fb920

Co-Authored-By: Yuvi Panda <[email protected]>

weiji14 mentioned this pull request Nov 29, 2022

Bump to cartopy 0.21.0, rasterio 1.3.3, hdf5 1.12.2, qgis 3.22.11 #16

Merged

tsnow03 marked this pull request as ready for review November 30, 2022 22:16

weiji14 added 3 commits December 3, 2022 22:49

Merge branch 'main' into unified-lockfile

9f36ec5

Fix incorrect merge conflict omission

7f9db0c

Add back in the 'start'` file to `.github/workflows/build.yaml`.

Delete conda-linux-64.lock

27f3a8b

Replaced by conda-linux.yml.

weiji14 marked this pull request as draft December 4, 2022 04:01

weiji14 mentioned this pull request Mar 3, 2023

Can we use compatible release spec in environment.yml? #46

Closed

weiji14 added 3 commits May 29, 2023 09:34

Merge branch 'main' into unified-lockfile

5486608

Bump conda-lock to 2.0.0

18b1b25

Use conda-lock 2.0.0 in postBuild

8353e4d

weiji14 mentioned this pull request May 28, 2023

update conda-lock version after updates #15

Closed

Update conda-lock.yml to v1 schema

4a21b81

weiji14 marked this pull request as ready for review May 28, 2023 22:32

mfisher87 reviewed Jun 9, 2023

View reviewed changes

.github/workflows/build.yaml Outdated Show resolved Hide resolved

mfisher87 reviewed Jun 9, 2023

View reviewed changes

.github/workflows/test.yaml Outdated Show resolved Hide resolved

weiji14 and others added 2 commits June 10, 2023 14:01

Fix typo conda-linux.yml to conda-lock.yml

71a2f4e

Co-authored-by: Matt Fisher <[email protected]>

Merge branch 'main' into unified-lockfile

4cff62b

Conflicts: conda-linux-64.lock

weiji14 mentioned this pull request Jun 10, 2023

Simplify condalock workflow to work without chatops pangeo-data/pangeo-docker-images#414

Open

weiji14 added 2 commits June 17, 2023 13:04

Merge branch 'main' into unified-lockfile

bccefa2

Bump conda-lock from 2.0.0 to 2.1.0

9c8e894

Bumps [conda-lock](https://github.com/conda/conda-lock) from 2.0.0 to 2.1.0. - [Release notes](https://github.com/conda/conda-lock/releases) - [Commits](conda/conda-lock@v2.0.0...v2.1.0)

weiji14 marked this pull request as draft August 29, 2023 01:45

weiji14 mentioned this pull request May 17, 2024

Update xarray and friends #113

Merged

weiji14 mentioned this pull request Aug 6, 2024

Commit conda-lock.yml to repository #122

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using unified conda-lock file in postBuild for reproducible conda environment #14

Using unified conda-lock file in postBuild for reproducible conda environment #14

weiji14 commented Nov 29, 2022 •

edited

Loading

github-actions bot commented Nov 29, 2022

weiji14 commented Nov 29, 2022

weiji14 commented Nov 29, 2022

weiji14 Nov 29, 2022

weiji14 Nov 29, 2022

weiji14 Nov 29, 2022

yuvipanda left a comment

weiji14 commented Nov 29, 2022 •

edited

Loading

weiji14 commented Dec 4, 2022

yuvipanda commented Dec 4, 2022

weiji14 commented Dec 5, 2022

yuvipanda commented Dec 5, 2022

weiji14 commented Dec 5, 2022

yuvipanda commented Dec 5, 2022

weiji14 commented May 28, 2023

weiji14 commented Aug 15, 2024

Using unified conda-lock file in postBuild for reproducible conda environment #14

Are you sure you want to change the base?

Using unified conda-lock file in postBuild for reproducible conda environment #14

Conversation

weiji14 commented Nov 29, 2022 • edited Loading

github-actions bot commented Nov 29, 2022

weiji14 commented Nov 29, 2022

weiji14 commented Nov 29, 2022

weiji14 Nov 29, 2022

Choose a reason for hiding this comment

weiji14 Nov 29, 2022

Choose a reason for hiding this comment

weiji14 Nov 29, 2022

Choose a reason for hiding this comment

yuvipanda left a comment

Choose a reason for hiding this comment

weiji14 commented Nov 29, 2022 • edited Loading

weiji14 commented Dec 4, 2022

yuvipanda commented Dec 4, 2022

weiji14 commented Dec 5, 2022

yuvipanda commented Dec 5, 2022

weiji14 commented Dec 5, 2022

yuvipanda commented Dec 5, 2022

weiji14 commented May 28, 2023

weiji14 commented Aug 15, 2024

weiji14 commented Nov 29, 2022 •

edited

Loading

weiji14 commented Nov 29, 2022 •

edited

Loading