Problem installing Python packages that need compiling

[alexgleith]

Describe the bug
I'm trying to install a Python package from GitHub that needs some rust components compiled. I've added

To Reproduce
Steps to reproduce the behavior:

1. Add these to the environment.yaml:

  # Compilers
  - gcc_linux-64
  - gxx_linux-64
  - sysroot_linux-64

  # Rust
  - rust

2. Add this to Python requirements.txt

git+https://github.com/opendatacube/odc-algo.git@add-rust-geomedian-impl

4. That's probably enough for it to fail.

Expected behavior

The Python package builds and installs in the Conda environment.

Docker Image Version (e.g. quay.io/pangeo/ml-notebook:2023.02.27):

FROM pangeo/base-image:2024.03.22

Infrastructure (Where you are running this image):

• Builds on GitHub: https://github.com/digitalearthpacific/dep-analytics-lab-image/pull/5/files

Error output is this:

× Building wheel for odc-algo (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [41 lines of output]
    Running `maturin pep517 build-wheel -i /srv/conda/envs/notebook/bin/python3.11 --compatibility off`
    📦 Including license file "/tmp/pip-req-build-kq0ba07m/LICENSE"
    🍹 Building a mixed python/rust project
    🔗 Found pyo3 bindings
    🐍 Found CPython 3.11 at /srv/conda/envs/notebook/bin/python3.11
    📡 Using build options profile from pyproject.toml
       Compiling autocfg v1.1.0
       Compiling target-lexicon v0.12.11
       Compiling cfg-if v1.0.0
       Compiling libc v0.2.147
       Compiling scopeguard v1.2.0
       Compiling crossbeam-utils v0.8.16
    error: linking with `cc` failed: exit status: 1
      |
      = note: LC_ALL="C" PATH="/srv/conda/envs/notebook/lib/rustlib/x86_64-unknown-linux-gnu/bin:/tmp/pip-build-env-qfi6t3dr/overlay/bin:/tmp/pip-build-env-qfi6t3dr/normal/bin:/srv/conda/envs/notebook/bin:/srv/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcYzB7w0/symbols.o" "/tmp/pip-req-build-kq0ba07m/target/release/build/libc-d36222c216519f5a/build_script_build-d36222c216519f5a.build_script_build.283f1c8ce0e68003-cgu.0.rcgu.o" "/tmp/pip-req-build-kq0ba07m/target/release/build/libc-d36222c216519f5a/build_script_build-d36222c216519f5a.build_script_build.283f1c8ce0e68003-cgu.1.rcgu.o" "/tmp/pip-req-build-kq0ba07m/target/release/build/libc-d36222c216519f5a/build_script_build-d36222c216519f5a.build_script_build.283f1c8ce0e68003-cgu.2.rcgu.o" "/tmp/pip-req-build-kq0ba07m/target/release/build/libc-d36222c216519f5a/build_script_build-d36222c216519f5a.2nt4p8xasu24jgsz.rcgu.o" "-Wl,--as-needed" "-L" "/tmp/pip-req-build-kq0ba07m/target/release/d
      = note: /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc: No such file or directory
              collect2: error: ld returned 1 exit status

[weiji14]

Hi @alexgleith, nice to see those Rust geomedian algorithms coming along!

I'm trying to see what the best approach here would be. We probably don't want to install all those Linux compilers explicitly in base-image, given that we're trying to move towards cross-platform builds (e.g. @scottyhq's comment on having osx-arm64 builds at #396 (comment)).

We could just depend on rust itself, and let conda/mamba pull in the necessary compilers for each platform. But looking at https://anaconda.org/conda-forge/rust/files?version=1.77.2, the Rust package itself is 100-200MB in size, and there will be more MBs added for the compilers, so not sure if this is a good idea...

That said, we could consider making a base-rust image perhaps? The packaging situation for Rust is a little different since most people just use cargo to install crates rather than conda/pip, so we'd need to think this through a bit.

[alexgleith]

Hey @weiji14

This is for my own Docker Image, so I don't really mind how big it is, at least for now. I'd just like a solution that works! I don't really want to have an external build process for this Python library, though admittedly, it does make sense to remove the compiler and rust dependencies...

Anyhow, this should be possible to do, right? I can't understand why I can't install GCC and Rust and compile the package!

[weiji14]

Ah ok, I thought you were asking for a solution on pangeo-docker-images to make this work out of the box 😅

Could you try some of the suggestions at conda-forge/rust-feedstock#71? I see something about setting a RUSTFLAGS environment variable, though that should be taken care of if you have activated the conda environment correctly.

[alexgleith]

That looks like a good option! Testing now...

[alexgleith]

Doesn't seem to have helped... I'll do some testing locally in a built Docker image, rather than in CI.

[alexgleith]

Testing in a running Docker image I get the below:

env | grep RUST

RUSTFLAGS=-C link-arg=-Wl,-rpath-link,/srv/conda/envs/notebook/lib -C link-arg=-Wl,-rpath,/srv/conda/envs/notebook/lib -L /srv/conda/envs/notebook/lib

python -m pip install git+https://github.com/opendatacube/odc-algo.git@add-rust-geomedian-impl

Results in a huge list of errors including:

note: /srv/conda/envs/notebook/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find -lgcc: No such file or directory

[scottyhq]

To narrow down that possible issues a bit, if you just use an environment.yml with the micromamba docker image https://micromamba-docker.readthedocs.io/en/latest/quick_start.html does it work?

[maxrjones]

It wasn't trivial, so here's how I got rustup working on a JupyterHub compatible docker image to compile Python bindings with a rust crate in case others find the approach helpful - https://github.com/developmentseed/veda-optimized-data-delivery-image/pull/12/files