Cannot install import hook when monorepo #15

jeertmans · 2024-12-16T09:06:09Z

Bug Description

Hello!

I currently develop tools that is split into two Python modules:

differt-core: lower-level Python module built on Rust code (using PyO3 and maturin);
differt: the main Python module, that imports the lower-level module and provides all high-level API.

The project is a bit like pydantic-core and pydantic, except that I use a monorepo architecture (thanks to uv's workspace feature).

The (simplified) project layout is as follows:

.
├── Cargo.lock
├── Cargo.toml
├── differt
│   ├── pyproject.toml
│   └── src
├── differt-core
│   ├── Cargo.toml
│   ├── pyproject.toml
│   ├── python
│   └── src
├── pyproject.toml
└── uv.lock

When trying to install the import hook, it complains that the top-level pyproject.toml does not contain required information (i.e., build section). However, it should actually look at differt-core/pyproject.toml instead.

I did try to cd differt-core && python -m maturin_import_hook site install, but it stills uses the top-level pyproject.toml file. Is there any way to make it use the correct project file?

Steps to Reproduce

Checkout my repository and checkout import_hook branch;
cd differt-core && uv run python -m maturin_import_hook site install --detect-uv

Repository: https://github.com/jeertmans/DiffeRT/tree/import-hook

Please provide the output of `python -m maturin_import_hook version` (or provide manually)

failed to resolve project "/home/jeertmans/repositories/DiffeRT": pyproject.toml is invalid (does not have required fields)
OS: Linux-6.8.0-49-generic-x86_64-with-glibc2.39
Python: CPython 3.11.8
maturin-import-hook: 0.2.0
maturin: maturin 1.7.8
rustc: rustc 1.83.0 (90b35a623 2024-11-26)
pip: ?

Does `maturin develop` work when run manually for your project?

Yes/No (leave blank if not applicable)

The text was updated successfully, but these errors were encountered:

mbway · 2024-12-16T20:58:38Z

you say

it stills uses the top-level pyproject.toml

Are you saying that the import hook is not rebuilding when it should for you? When I follow the steps it works despite the warnings:

failed to resolve project "/workspace/DiffeRT": pyproject.toml is invalid (does not have required fields)
...
>>> import differt
failed to resolve project ".": pyproject.toml is invalid (does not have required fields)
building "differt_core"
rebuilt and loaded package "differt_core" in 12.251s

so the hook is functional but I agree that the messages ideally shouldn't be there.

The problem is that the top level workspace looks like it could be a maturin project (has pyproject.toml and Cargo.toml) but running maturin build at the top level will fail with

💥 maturin failed
  Caused by: pyproject.toml at /workspace/DiffeRT/pyproject.toml is invalid
  Caused by: TOML parse error at line 1, column 1
  |
1 | [project]
  | ^
missing field `build-system`

which is what the import hook is also complaining about. It's printing the error because it was assumed that a user working with a project that looks like a maturin project that fails to build is working with a broken project.

adding the following to the pyproject.toml makes the warnings go away:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

for some reason there is a build-system section for the example uv workspaces in the documentation https://docs.astral.sh/uv/concepts/projects/workspaces/
It seems that the pyproject.toml spec doesn't require the [build-system] section, it just says that it is 'strongly recommended' (but for workspaces I don't think it makes much sense) https://packaging.python.org/en/latest/guides/writing-pyproject-toml/

however, more generally I think it would be an improvement to the ProjectResolver to not fully resolve packages unless required (which would resolve this class of issue). Here, if there was ProjectResolver.resolve_package_name(), that would be sufficient to determine if a particular package should be rebuilt. So long as differt_dev (the workspace name) is not imported then the root never has to be fully resolved so no error would be printed. I will look into this.

jeertmans · 2024-12-17T16:22:50Z

Thanks for your reply @mbway! What command did you exactly run?

Because it doesn't work for me.

cd differt-core && uv run python -m maturin_import_hook site install --detect-uv && cd -

Importing differt fails.

differt-devjeertmans@jeertmans:~/repositories/DiffeRT$ uv run python
Uninstalled 1 package in 0.35ms
Installed 1 package in 0.51ms
failed to resolve project "/home/jeertmans/repositories/DiffeRT": pyproject.toml is invalid (does not have required fields)
Python 3.11.8 (main, Feb 25 2024, 04:18:18) [Clang 17.0.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import differt
failed to resolve project ".": pyproject.toml is invalid (does not have required fields)
building "differt_core"
maturin_import_hook [ERROR] command "/home/jeertmans/.local/bin/maturin develop --manifest-path /home/jeertmans/repositories/DiffeRT/differt-core/Cargo.toml --color always --uv" returned non-zero exit status: 1
maturin_import_hook [ERROR] maturin output:
📦 Including license file "/home/jeertmans/repositories/DiffeRT/differt-core/LICENSE.md"
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.9
🐍 Not using a specific python interpreter
📡 Using build options features, bindings from pyproject.toml
Audited 1 package in 2ms
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.05s
📦 Built wheel for abi3 Python ≥ 3.9 to /tmp/.tmpvwB1H2/differt_core-0.0.30-cp39-abi3-linux_x86_64.whl
✏️  Setting installed package as editable
💥 maturin failed
  Caused by: failed to list package files

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jeertmans/repositories/DiffeRT/differt/src/differt/__init__.py", line 3, in <module>
    from ._version import VERSION
  File "/home/jeertmans/repositories/DiffeRT/differt/src/differt/_version.py", line 1, in <module>
    from differt_core import __version__  # Re-export version from code module
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jeertmans/repositories/DiffeRT/.venv/lib/python3.11/site-packages/maturin_import_hook/project_importer.py", line 151, in find_spec
    spec, rebuilt = self._rebuild_project(package_name, project_dir)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jeertmans/repositories/DiffeRT/.venv/lib/python3.11/site-packages/maturin_import_hook/project_importer.py", line 261, in _rebuild_project
    maturin_output = develop_build_project(self.find_maturin(), resolved.cargo_manifest_path, settings)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jeertmans/repositories/DiffeRT/.venv/lib/python3.11/site-packages/maturin_import_hook/_building.py", line 181, in develop_build_project
    raise MaturinError(msg)
maturin_import_hook.error.MaturinError: Failed to build package with maturin
>>> differt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'differt' is not defined

Pytest also fails:

differt-devjeertmans@jeertmans:~/repositories/DiffeRT$ uv run pytest -n0
Uninstalled 1 package in 0.36ms
Installed 1 package in 1ms
failed to resolve project "/home/jeertmans/repositories/DiffeRT": pyproject.toml is invalid (does not have required fields)
============================================================== test session starts ==============================================================
platform linux -- Python 3.11.8, pytest-8.3.3, pluggy-1.5.0
codspeed: 3.0.0 (disabled, mode: walltime, timer_resolution: 1.0ns)
benchmark: 5.1.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=True warmup_iterations=100000)
rootdir: /home/jeertmans/repositories/DiffeRT
configfile: pyproject.toml
plugins: codspeed-3.0.0, xdist-3.6.1, anyio-4.6.2.post1, dash-2.18.2, typeguard-2.13.3, cov-6.0.0, missing-modules-0.2.1, benchmark-5.1.0, jaxtyping-0.2.34, env-1.1.5
collecting ... building "differt_core"
maturin_import_hook [ERROR] command "/home/jeertmans/.local/bin/maturin develop --manifest-path /home/jeertmans/repositories/DiffeRT/differt-core/Cargo.toml --color always --uv" returned non-zero exit status: 1
maturin_import_hook [ERROR] maturin output:
📦 Including license file "/home/jeertmans/repositories/DiffeRT/differt-core/LICENSE.md"
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.9
🐍 Not using a specific python interpreter
📡 Using build options features, bindings from pyproject.toml
Audited 1 package in 3ms
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.04s
📦 Built wheel for abi3 Python ≥ 3.9 to /tmp/.tmpNa8FbP/differt_core-0.0.30-cp39-abi3-linux_x86_64.whl
✏️  Setting installed package as editable
💥 maturin failed
  Caused by: failed to list package files

collecting 0 items / 1 error                                                                                                                    building "differt_core"
maturin_import_hook [ERROR] command "/home/jeertmans/.local/bin/maturin develop --manifest-path /home/jeertmans/repositories/DiffeRT/differt-core/Cargo.toml --color always --uv" returned non-zero exit status: 1
maturin_import_hook [ERROR] maturin output:
📦 Including license file "/home/jeertmans/repositories/DiffeRT/differt-core/LICENSE.md"
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.9
🐍 Not using a specific python interpreter
📡 Using build options features, bindings from pyproject.toml
Audited 1 package in 2ms
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.04s
📦 Built wheel for abi3 Python ≥ 3.9 to /tmp/.tmpeL3YYb/differt_core-0.0.30-cp39-abi3-linux_x86_64.whl
✏️  Setting installed package as editable
💥 maturin failed
  Caused by: failed to list package files

collecting 7 items / 2 errors / 2 skipped                                                                                                       building "differt_core"
maturin_import_hook [ERROR] command "/home/jeertmans/.local/bin/maturin develop --manifest-path /home/jeertmans/repositories/DiffeRT/differt-core/Cargo.toml --color always --uv" returned non-zero exit status: 1
maturin_import_hook [ERROR] maturin output:
📦 Including license file "/home/jeertmans/repositories/DiffeRT/differt-core/LICENSE.md"
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.9
🐍 Not using a specific python interpreter
📡 Using build options features, bindings from pyproject.toml
Audited 1 package in 2ms
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.04s
📦 Built wheel for abi3 Python ≥ 3.9 to /tmp/.tmp2FOVwG/differt_core-0.0.30-cp39-abi3-linux_x86_64.whl
✏️  Setting installed package as editable
💥 maturin failed
  Caused by: failed to list package files

collecting 102 items / 3 errors / 5 skipped                                                                                                     building "differt_core"
maturin_import_hook [ERROR] command "/home/jeertmans/.local/bin/maturin develop --manifest-path /home/jeertmans/repositories/DiffeRT/differt-core/Cargo.toml --color always --uv" returned non-zero exit status: 1
maturin_import_hook [ERROR] maturin output:
📦 Including license file "/home/jeertmans/repositories/DiffeRT/differt-core/LICENSE.md"
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.9
🐍 Not using a specific python interpreter
📡 Using build options features, bindings from pyproject.toml
Audited 1 package in 2ms
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.04s
📦 Built wheel for abi3 Python ≥ 3.9 to /tmp/.tmp0HRBUI/differt_core-0.0.30-cp39-abi3-linux_x86_64.whl
✏️  Setting installed package as editable
💥 maturin failed
  Caused by: failed to list package files

collected 135 items / 4 errors / 7 skipped                                                                                                      

==================================================================== ERRORS =====================================================================
_____________________________________________________ ERROR collecting differt/src/differt ______________________________________________________
<frozen importlib._bootstrap_external>:940: in exec_module
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
differt/src/differt/__init__.py:3: in <module>
    from ._version import VERSION
differt/src/differt/_version.py:1: in <module>
    from differt_core import __version__  # Re-export version from code module
.venv/lib/python3.11/site-packages/maturin_import_hook/project_importer.py:151: in find_spec
    spec, rebuilt = self._rebuild_project(package_name, project_dir)
.venv/lib/python3.11/site-packages/maturin_import_hook/project_importer.py:261: in _rebuild_project
    maturin_output = develop_build_project(self.find_maturin(), resolved.cargo_manifest_path, settings)
.venv/lib/python3.11/site-packages/maturin_import_hook/_building.py:181: in develop_build_project
    raise MaturinError(msg)
E   maturin_import_hook.error.MaturinError: Failed to build package with maturin
___________________________________________________ ERROR collecting differt/tests/benchmarks ___________________________________________________
differt/tests/benchmarks/conftest.py:8: in <module>
    from ..scene.fixtures import simple_street_canyon_scene, sionna_folder
differt/tests/scene/fixtures.py:7: in <module>
    from differt.geometry._triangle_mesh import TriangleMesh
differt/src/differt/geometry/__init__.py:26: in <module>
    from ._triangle_mesh import (
differt/src/differt/geometry/_triangle_mesh.py:13: in <module>
    import differt_core.geometry
.venv/lib/python3.11/site-packages/maturin_import_hook/project_importer.py:162: in find_spec
    spec, rebuilt = self._rebuild_project(package_name, project_dir)
.venv/lib/python3.11/site-packages/maturin_import_hook/project_importer.py:261: in _rebuild_project
    maturin_output = develop_build_project(self.find_maturin(), resolved.cargo_manifest_path, settings)
.venv/lib/python3.11/site-packages/maturin_import_hook/_building.py:181: in develop_build_project
    raise MaturinError(msg)
E   maturin_import_hook.error.MaturinError: Failed to build package with maturin
____________________________________________________ ERROR collecting differt/tests/geometry ____________________________________________________
differt/tests/geometry/conftest.py:10: in <module>
    from .fixtures import (
differt/tests/geometry/fixtures.py:6: in <module>
    from differt.geometry._triangle_mesh import TriangleMesh
differt/src/differt/geometry/__init__.py:26: in <module>
    from ._triangle_mesh import (
differt/src/differt/geometry/_triangle_mesh.py:13: in <module>
    import differt_core.geometry
.venv/lib/python3.11/site-packages/maturin_import_hook/project_importer.py:162: in find_spec
    spec, rebuilt = self._rebuild_project(package_name, project_dir)
.venv/lib/python3.11/site-packages/maturin_import_hook/project_importer.py:261: in _rebuild_project
    maturin_output = develop_build_project(self.find_maturin(), resolved.cargo_manifest_path, settings)
.venv/lib/python3.11/site-packages/maturin_import_hook/_building.py:181: in develop_build_project
    raise MaturinError(msg)
E   maturin_import_hook.error.MaturinError: Failed to build package with maturin
_____________________________________________________ ERROR collecting differt/tests/scene ______________________________________________________
differt/tests/scene/conftest.py:10: in <module>
    from ..geometry.fixtures import (
differt/tests/geometry/fixtures.py:6: in <module>
    from differt.geometry._triangle_mesh import TriangleMesh
differt/src/differt/geometry/__init__.py:26: in <module>
    from ._triangle_mesh import (
differt/src/differt/geometry/_triangle_mesh.py:13: in <module>
    import differt_core.geometry
.venv/lib/python3.11/site-packages/maturin_import_hook/project_importer.py:162: in find_spec
    spec, rebuilt = self._rebuild_project(package_name, project_dir)
.venv/lib/python3.11/site-packages/maturin_import_hook/project_importer.py:261: in _rebuild_project
    maturin_output = develop_build_project(self.find_maturin(), resolved.cargo_manifest_path, settings)
.venv/lib/python3.11/site-packages/maturin_import_hook/_building.py:181: in develop_build_project
    raise MaturinError(msg)
E   maturin_import_hook.error.MaturinError: Failed to build package with maturin

---------- coverage: platform linux, python 3.11.8-final-0 -----------
Coverage XML written to file coverage.xml

============================================================ short test summary info ============================================================
ERROR differt/src/differt - maturin_import_hook.error.MaturinError: Failed to build package with maturin
ERROR differt/tests/benchmarks - maturin_import_hook.error.MaturinError: Failed to build package with maturin
ERROR differt/tests/geometry - maturin_import_hook.error.MaturinError: Failed to build package with maturin
ERROR differt/tests/scene - maturin_import_hook.error.MaturinError: Failed to build package with maturin
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================= 7 skipped, 4 errors in 17.67s =========================================================

mbway · 2024-12-17T16:31:28Z

This seems to be this maturin bug
PyO3/maturin#2388

I assume running maturin develop --uv manually (in the appropriate directory) also doesn't work?
Try upgrading UV and try again. I also have on my TODO list to include the UV version in the python -m maturin_import_hook version cli option as I can't tell what version you're using currently

jeertmans · 2024-12-17T16:39:57Z

It worked after upgrading uv! Thanks :D

…ages (#2389) since version 0.4.25, `uv pip show --files` is supported and can be used to set the package installed by `maturin develop` as editable without having to fallback to using `pip`. However several users are using old versions of `uv` and it would be better if the error message was clearer if an outdated uv is used. This has caused confusion in these issues: - #2388 - PyO3/maturin-import-hook#15 before: ``` ✏️ Setting installed package as editable 💥 maturin failed Caused by: failed to list package files ``` after: ``` ✏️ Setting installed package as editable ⚠️ Warning: failed to set package as editable: uv >= 0.4.25 is required for `show --files`. Version 0.4.23 was found. 🛠 Installed $PACKAGE ``` (note that installation now proceeds if setting as editable fails)

jeertmans added the bug Something isn't working label Dec 16, 2024

mbway mentioned this issue Dec 17, 2024

check the install backend version to give more informative error messages PyO3/maturin#2389

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot install import hook when monorepo #15

Cannot install import hook when monorepo #15

jeertmans commented Dec 16, 2024

mbway commented Dec 16, 2024 •

edited

Loading

jeertmans commented Dec 17, 2024

mbway commented Dec 17, 2024

jeertmans commented Dec 17, 2024

Cannot install import hook when monorepo #15

Cannot install import hook when monorepo #15

Comments

jeertmans commented Dec 16, 2024

Bug Description

Steps to Reproduce

Please provide the output of python -m maturin_import_hook version (or provide manually)

Does maturin develop work when run manually for your project?

mbway commented Dec 16, 2024 • edited Loading

jeertmans commented Dec 17, 2024

mbway commented Dec 17, 2024

jeertmans commented Dec 17, 2024

Please provide the output of `python -m maturin_import_hook version` (or provide manually)

Does `maturin develop` work when run manually for your project?

mbway commented Dec 16, 2024 •

edited

Loading