Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Conda build error #187

Closed
paskino opened this issue Mar 18, 2024 · 10 comments · Fixed by #189
Closed

Conda build error #187

paskino opened this issue Mar 18, 2024 · 10 comments · Fixed by #189
Assignees
@casperdcl @gfardell

Comments

@paskino
Copy link
Contributor

paskino commented Mar 18, 2024 •
edited
Loading

After #183, on Jenkins the conda build fails with the following, seemingly not having made the package correctly?

The error seems to originate from CMake finding the base python of my mamba installation rather than the one used in the build environment

-- Found Python: /home/ofn77899/mambaforge/bin/python3.10 (found version "3.10.10") found components: Interpreter

======================================================================
ERROR: test_CPU_regularisers (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_CPU_regularisers
Traceback (most recent call last):
  File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/test_tmp/test/test_CPU_regularisers.py", line 6, in <module>
    from ccpi.filters.regularisers import FGP_TV, SB_TV, TGV, LLT_ROF, FGP_dTV, NDF, Diff4th, ROF_TV, PD_TV
ModuleNotFoundError: No module named 'ccpi'


======================================================================
ERROR: test_run_test (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_run_test
Traceback (most recent call last):
  File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/jenkins/conda-bld/ccpi-regulariser_1710763552122/test_tmp/test/test_run_test.py", line 5, in <module>
    from ccpi.filters.regularisers import ROF_TV, FGP_TV, PD_TV, SB_TV, TGV, LLT_ROF, FGP_dTV, NDF, Diff4th
ModuleNotFoundError: No module named 'ccpi'
@paskino paskino changed the title Conda build error Conda build error on Jenkins Mar 19, 2024
@paskino
Copy link
Contributor Author

paskino commented Mar 19, 2024

I tested on my machine and the conda build seems to work ???

@paskino
Copy link
Contributor Author

paskino commented Mar 20, 2024 •
edited
Loading

I forced the latest conda on Jenkins and now the error is

AssertionError: Can't merge/copy source into subdirectory of itself.  Please create separate spaces for these things.
      src: /home/jenkins/workspace/CILsingle/CCPi-Regularisation-Toolkit
      dst: /home/jenkins/workspace/CILsingle/CCPi-Regularisation-Toolkit/conda-bld/ccpi-regulariser_1710924290226/work

@paskino
Copy link
Contributor Author

paskino commented Mar 22, 2024 •
edited
Loading

I ran the conda build locally and it failed on the following combination

    _libgcc_mutex:    0.1-conda_forge            conda-forge
    _openmp_mutex:    4.5-2_gnu                  conda-forge
    bzip2:            1.0.8-hd590300_5           conda-forge
    ca-certificates:  2024.2.2-hbcca054_0        conda-forge
    ccpi-regulariser: 3.0-np123py38_25           local
    ld_impl_linux-64: 2.40-h41732ed_0            conda-forge
    libblas:          3.9.0-21_linux64_openblas  conda-forge
    libcblas:         3.9.0-21_linux64_openblas  conda-forge
    libffi:           3.4.2-h7f98852_5           conda-forge
    libgcc-ng:        13.2.0-h807b86a_5          conda-forge
    libgfortran-ng:   13.2.0-h69a702a_5          conda-forge
    libgfortran5:     13.2.0-ha4646dd_5          conda-forge
    libgomp:          13.2.0-h807b86a_5          conda-forge
    liblapack:        3.9.0-21_linux64_openblas  conda-forge
    libnsl:           2.0.1-hd590300_0           conda-forge
    libopenblas:      0.3.26-pthreads_h413a1c8_0 conda-forge
    libsqlite:        3.45.2-h2797004_0          conda-forge
    libstdcxx-ng:     13.2.0-h7e041cc_5          conda-forge
    libuuid:          2.38.1-h0b41bf4_0          conda-forge
    libxcrypt:        4.4.36-hd590300_1          conda-forge
    libzlib:          1.2.13-hd590300_5          conda-forge
    ncurses:          6.4.20240210-h59595ed_0    conda-forge
    numpy:            1.23.5-py38h7042d01_0      conda-forge
    openssl:          3.2.1-hd590300_1           conda-forge
    pip:              24.0-pyhd8ed1ab_0          conda-forge
    python:           3.8.19-hd12c33a_0_cpython  conda-forge
    python_abi:       3.8-4_cp38                 conda-forge
    readline:         8.2-h8228510_1             conda-forge
    setuptools:       69.2.0-pyhd8ed1ab_0        conda-forge
    tk:               8.6.13-noxft_h4845f30_101  conda-forge
    wheel:            0.43.0-pyhd8ed1ab_0        conda-forge
    xz:               5.2.6-h166bdaf_0           conda-forge

@paskino paskino changed the title Conda build error on Jenkins Conda build error Mar 23, 2024
@paskino
Copy link
Contributor Author

paskino commented Mar 25, 2024 •
edited
Loading

@casperdcl on further investigation, conda build:

  • python 3.8 FAIL
  • python 3.9 FAIL
  • python 3.10 OK

This is what the tar.bz2 produced by conda contains for python 3.9: that is it doesn't contain the python module

>>> tar tjf /home/ofn77899/mambaforge/conda-bld/broken/ccpi-regulariser-3.0-np124py39_26.tar.bz2
info/hash_input.json
info/files
info/test/run_test.sh
info/test/test/.vscode/settings.json
info/recipe/build.sh
info/index.json
info/paths.json
info/recipe/bld.bat
info/recipe/conda_build_config.yaml
info/recipe/meta.yaml.template
info/test/test/testroutines.py
info/recipe/meta.yaml
info/test/test/debug.py
info/about.json
info/test/test/test_CPU_regularisers.py
info/test/test/test_run_test.py
info/git
info/test/test/test_imageLena.bin
libcilreg.so
libcilregcuda.so

By comparison the same command produces the following for Python 3.10

tar tjf /home/ofn77899/mambaforge/conda-bld/linux-64/ccpi-regulariser-3.0-np124py310_25.tar.bz2
info/hash_input.json
info/test/run_test.sh
info/test/test/.vscode/settings.json
info/recipe/build.sh
info/index.json
info/recipe/bld.bat
info/recipe/conda_build_config.yaml
info/recipe/meta.yaml.template
info/test/test/testroutines.py
info/files
info/recipe/meta.yaml
info/test/test/debug.py
info/about.json
info/test/test/test_CPU_regularisers.py
info/paths.json
info/test/test/test_run_test.py
info/git
info/test/test/test_imageLena.bin
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/INSTALLER
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/METADATA
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/WHEEL
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/RECORD
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/REQUESTED
libcilreg.so
lib/python3.10/site-packages/ccpi/filters/libcilreg.so
libcilregcuda.so
lib/python3.10/site-packages/ccpi/filters/libcilregcuda.so
lib/python3.10/site-packages/ccpi/cuda_kernels/__init__.py
lib/python3.10/site-packages/ccpi/filters/utils.py
lib/python3.10/site-packages/ccpi/filters/regularisers.py
lib/python3.10/site-packages/ccpi/supp/qualitymetrics.py
lib/python3.10/site-packages/ccpi/filters/diffusion.py
lib/python3.10/site-packages/ccpi/filters/regularisers_old.py
lib/python3.10/site-packages/ccpi/filters/regularisersCuPy.py
lib/python3.10/site-packages/ccpi/filters/TV.py
lib/python3.10/site-packages/ccpi/supp/__init__.py
lib/python3.10/site-packages/ccpi/filters/__init__.py
lib/python3.10/site-packages/ccpi/__init__.py
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/top_level.txt
lib/python3.10/site-packages/ccpi_regularisers-24.0.0.dist-info/direct_url.json
lib/python3.10/site-packages/ccpi/__pycache__/__init__.cpython-310.pyc
lib/python3.10/site-packages/ccpi/supp/__pycache__/__init__.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/__init__.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/utils.cpython-310.pyc
lib/python3.10/site-packages/ccpi/cuda_kernels/__pycache__/__init__.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/regularisers.cpython-310.pyc
lib/python3.10/site-packages/ccpi/supp/__pycache__/qualitymetrics.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/diffusion.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/regularisers_old.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/regularisersCuPy.cpython-310.pyc
lib/python3.10/site-packages/ccpi/filters/__pycache__/TV.cpython-310.pyc

@casperdcl
Copy link
Member

about this, what was the working (3.10) package list?

@paskino
Copy link
Contributor Author

paskino commented Mar 25, 2024 •
edited
Loading

I think that was the python

about this, what was the working (3.10) package list?

That was the python 3.10 that I showed above. Does this answer you?

I checked that my system created the following conda packages before failing on the 3.9

ccpi-regulariser-3.0-np121py310_25.tar.bz2  
ccpi-regulariser-3.0-np121py310_35.tar.bz2  
ccpi-regulariser-3.0-np124py310_25.tar.bz2
ccpi-regulariser-3.0-np122py310_25.tar.bz2  
ccpi-regulariser-3.0-np123py310_25.tar.bz2
ccpi-regulariser-3.0-np124py311_25.tar.bz2
ccpi-regulariser-3.0-np123py311_25.tar.bz2

@casperdcl
Copy link
Member

casperdcl commented Mar 25, 2024 •
edited
Loading

No I mean this (conda env spec) not this (dist artefacts)

@paskino
Copy link
Contributor Author

paskino commented Mar 26, 2024

I don't understand what you are asking.

I ran conda build recipe -c conda-forge --python=3.8, that produced a conda package which didn't contain the python code:

>>> tar tjf /home/ofn77899/mambaforge/conda-bld/broken/ccpi-regulariser-3.0-np123py38_25.tar.bz2
info/hash_input.json
info/files
info/test/run_test.sh
info/test/test/.vscode/settings.json
info/recipe/build.sh
info/index.json
info/paths.json
info/recipe/bld.bat
info/recipe/conda_build_config.yaml
info/recipe/meta.yaml.template
info/test/test/testroutines.py
info/recipe/meta.yaml
info/test/test/debug.py
info/about.json
info/test/test/test_CPU_regularisers.py
info/test/test/test_run_test.py
info/git
info/test/test/test_imageLena.bin
libcilreg.so
libcilregcuda.so

The same happens with python 3.8 and 3.9, not on 3.10 and 3.11.

What package could have changed between 3.9 and 3.10 that produces this error? We are using pyproject.toml, which is new to me.

@paskino
Copy link
Contributor Author

paskino commented Mar 26, 2024 •
edited
Loading

Ah, I understand now! This is the environment that was created with py3.10 is the following

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
  bzip2              conda-forge/linux-64::bzip2-1.0.8-hd590300_5
  ca-certificates    conda-forge/linux-64::ca-certificates-2024.2.2-hbcca054_0
  ccpi-regulariser   home/ofn77899/mambaforge/conda-bld/linux-64::ccpi-regulariser-3.0-np121py310_35
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.40-h41732ed_0
  libblas            conda-forge/linux-64::libblas-3.9.0-21_linux64_openblas
  libcblas           conda-forge/linux-64::libcblas-3.9.0-21_linux64_openblas
  libffi             conda-forge/linux-64::libffi-3.4.2-h7f98852_5
  libgcc-ng          conda-forge/linux-64::libgcc-ng-13.2.0-h807b86a_5
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-13.2.0-h69a702a_5
  libgfortran5       conda-forge/linux-64::libgfortran5-13.2.0-ha4646dd_5
  libgomp            conda-forge/linux-64::libgomp-13.2.0-h807b86a_5
  liblapack          conda-forge/linux-64::liblapack-3.9.0-21_linux64_openblas
  libnsl             conda-forge/linux-64::libnsl-2.0.1-hd590300_0
  libopenblas        conda-forge/linux-64::libopenblas-0.3.26-pthreads_h413a1c8_0
  libsqlite          conda-forge/linux-64::libsqlite-3.45.2-h2797004_0
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-13.2.0-h7e041cc_5
  libuuid            conda-forge/linux-64::libuuid-2.38.1-h0b41bf4_0
  libxcrypt          conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1
  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5
  ncurses            conda-forge/linux-64::ncurses-6.4.20240210-h59595ed_0
  numpy              conda-forge/linux-64::numpy-1.21.6-py310h45f3432_0
  openssl            conda-forge/linux-64::openssl-3.2.1-hd590300_1
  pip                conda-forge/noarch::pip-24.0-pyhd8ed1ab_0
  python             conda-forge/linux-64::python-3.10.14-hd12c33a_0_cpython
  python_abi         conda-forge/linux-64::python_abi-3.10-4_cp310
  readline           conda-forge/linux-64::readline-8.2-h8228510_1
  setuptools         conda-forge/noarch::setuptools-69.2.0-pyhd8ed1ab_0
  tk                 conda-forge/linux-64::tk-8.6.13-noxft_h4845f30_101
  tzdata             conda-forge/noarch::tzdata-2024a-h0c530f3_0
  wheel              conda-forge/noarch::wheel-0.43.0-pyhd8ed1ab_0
  xz                 conda-forge/linux-64::xz-5.2.6-h166bdaf_0

The unittest ran correctly.

@casperdcl
Copy link
Member

Oh sorry, you already had 3.8 here - that's what I wanted

casperdcl added a commit that referenced this issue Mar 26, 2024
@casperdcl
fix python path for conda builds
b41850a 
- fixes #187
@casperdcl casperdcl mentioned this issue Mar 26, 2024
Merged
casperdcl added a commit that referenced this issue Mar 26, 2024
@casperdcl
fix python path for conda builds
770ee3f 
- fixes #187
casperdcl added a commit that referenced this issue Mar 26, 2024
@casperdcl
fix python path for conda builds
235d84c 
- fixes #187
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@casperdcl casperdcl

@gfardell gfardell

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix conda build TomographicImaging/CCPi-Regularisation-Toolkit
3 participants
@casperdcl @paskino @gfardell