Editable Mode install rebuilds don't update *.so in site-packages

[RisinT96]

Hi,

Unfortunately, I can't share my project, however here's the main details:

Configured cmake to install my built .so into ${SKBUILD_PLATLIB_DIR}/project/lib:

install(
    TARGETS
        bindings
    DESTINATION
        ${SKBUILD_PLATLIB_DIR}/project/lib/
)

When installing a package using pip install -v --no-build-isolation -e . I can see that cmake is installing the .so into the correct path:

  -- Installing: /tmp/tmpl60cdmsu/wheel/platlib/project/lib/bindings.cpython-312-x86_64-linux-gnu.so

Inspecting site-packages the .so is present:

> ls -lh .venv/lib/python3.12/site-packages/project/lib
-rw-r--r-- 1 user user  77K Aug 20 08:52 bindings.cpython-312-x86_64-linux-gnu.so

When importing the package, the .so is rebuilt:

> python -c "import project"
Running cmake --build & --install in /workspace/project/build
[0/2] Re-checking globbed directories...
ninja: no work to do.
-- Install configuration: "Release"
-- Installing: /tmp/tmpl60cdmsu/wheel/platlib/project/lib/bindings.cpython-312-x86_64-linux-gnu.so

(also tested where code changes were made an rebuild was actually necessary, got same behavior)

This only updates the .so in the temp dir, but doesn't update it in site-packages (see timestamp):

> ls -lh /tmp/tmpl60cdmsu/wheel/platlib/project/lib/
-rw-r--r-- 1 user user  77K Aug 20 09:02 bindings.cpython-312-x86_64-linux-gnu.so

> ls -lh .venv/lib/python3.12/site-packages/project/lib
-rw-r--r-- 1 user user  77K Aug 20 08:52 bindings.cpython-312-x86_64-linux-gnu.so

The running code is reusing the old shared object so any changes don't take effect.

I also tried installing locally (in addition to platlib):

install(
    TARGETS
        bindings
    DESTINATION
        ${CMAKE_CURRENT_SOURCE_DIR}/project/lib/
)

This will place the .so both in site-packages and in the local lib directory.

The local .so is updated on rebuild, however depending on environment and runpath, sometimes python picks up the local .so, and sometimes from site-packages.

This is unstable and so not very usable.

I also tried detecting editable installs by checking SKBUILD_STATE, and then installing only locally, but this prevents rebuilds on import.

Is this behavior expected, or am I missing something?

Thanks!

[LecrisUT]

Did you set editable.rebuild = true? Otherwise the only other thing could be to try drop the ${SKBUILD_PLATLIB_DIR}/ part (though that would be a bug if it was the issue). Increasing the logging.level would also help some, though I don't think there are logging messages in that part of the code.

You can also edit the _{name}_editable.py file and see if there is something amiss there.

[RisinT96]

Yes, I set the rebuild flag, and can see the rebuild running when importing the package.

I updated the install path in the cmake like so:

install(
    TARGETS
        bindings
    DESTINATION
        project/lib/
)

This seems to have actually $${\color{lightgreen}fixed \space the \space issue}$$:

> pip install -v --no-build-isolation -e .
...
 *** Installing project into wheel...
  -- Install configuration: "Release"
  -- Installing: /tmp/tmpx0s1g0_f/wheel/platlib/project/lib/bindings.cpython-312-x86_64-linux-gnu.so
...

> python -c "import project"
Running cmake --build & --install in /workspaces/project/build
[0/2] Re-checking globbed directories...
ninja: no work to do.
-- Install configuration: "Release"
-- Installing: .venv/lib/python3.12/site-packages/project/lib/bindings.cpython-312-x86_64-linux-gnu.so

# Change code

> python -c "import project"
Running cmake --build & --install in /workspaces/project/build
[0/2] Re-checking globbed directories...
[5/5] Generating /workspaces/project/project/lib/bindings.pyi
-- Install configuration: "Release"
-- Installing: .venv/lib/python3.12/site-packages/project/lib/bindings.cpython-312-x86_64-linux-gnu.so

and the .so is now updated in the site-packages.

So, it looks like there's some $${\color{salmon}possible \space bug}$$ that installs the targets into the wrong place if I use ${SKBUILD_PLATLIB_DIR}/project/lib/ vs project/lib/.

I thought I had to use SKBUILD_PLATLIB_DIR as it's not very clear from the documentation.

Thanks for your help :)

[LecrisUT]

Yes I don't think there would be any special handling for ${SKBUILD_PLATLIB_DIR} 🤔. Will need to check what's going on in there

[RisinT96]

I think the main issue here is that SKBUILD_PLATLIB_DIR is set in the cmake cache, pointing to the temp dir, and when the import-rebuild runs, it uses the same cmake cache, causing it to install into the temp dir.

The initial build passes the tmp dir as the install prefix.

The auto rebuild runs cmake --install . --prefix <path to site-packages>, which doesn't affect the installer as it's hardcoded to SKBUILD_PLATLIB_DIR and doesn't care about the cmake prefix path.

[LecrisUT]

Ah, I did not notice that. Indeed the install paths are actually different. Not sure why we don't set the CMAKE_INSTALL_PREFIX and such to the venv path directly 🤔.