libpython missing

[CowKeyMan]

libpython should be left in the environment. This is useful for us developing c libraries with python bindings. And in fact python expects this as well (look at the not found line below):

(myenv) username@device:/opt/pyenvs$ which python
/opt/pyenvs/myenv/bin/python
(myenv) username@device:/opt/pyenvs$ ldd /opt/pyenvs/myenv/bin/python
        linux-vdso.so.1 (0x00007ffc04b9d000)
        /opt/pyenvs/myenv/bin/../lib/libpython3.11.so.1.0 => not found
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbf79aec000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fbf79ae7000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fbf79ae2000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbf799fb000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fbf799f4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbf797cb000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fbf79b0e000)

Right now, it is instead being put in: ${HOME}.local/share/uv/python/cpython-3.11.9-linux-x86_64-gnu/lib, which is great for and speed, but super awful for portability. Maybe there can be a flag for this? I thought the uv venv --relocatable flag would help, but it does not. Maybe a --portable flag would be good for this?

Thanks for the awesome tool otherwise!

[frederikschubert]

We are also encountering the same issue when trying to run pytriton within a uv environment.

[smheidrich]

I guess this is only an issue for the python-build-standalone binaries uv uses by default.

If you let uv create the venv with a Python binary from another source, e.g. your system Python installation or one installed via pyenv, it works just fine because they don't have a shared library dependency on libpython*.so:

$ uv venv --python /usr/bin/python3 system-py-venv
...
$ ldd system-py-venv/bin/python
	linux-vdso.so.1 (0x00007ffc870c9000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1233c94000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1233c75000)
	libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f1233c4a000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1233a69000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f1233d9f000)

Judging by the sizes of these binaries (python-build-standalone: ~2 MB, pyenv/system: ~30 MB), I guess the latter just link libpython statically.

So maybe uv should somehow find out whether the binary in question has a relative libpython*.so dependency or not and only copy/symlink libpython*.so if so?

But this issue also messes with venvs created with python-build-standalone binaries independently of uv, so here's an issue on their side to maybe do something about it:

• Creating venvs via standard library's venv.create(..., with_pip=True) fails for binaries linked against libpython dynamically python-build-standalone#381

[charliermarsh]

Is this still an issue on latest uv, if you reinstall Python?

[frederikschubert]

I can confirm that the latest version solves this issue for us! 👍

Repro for testing which failed previously but now succeeds:

Running this script with uv run --with torch --with nvidia-pytriton --isolated server.py

[nazdridoy]

no it's not solved for me

❯ source .venv/bin/activate
❯ ldd `which python`
	linux-vdso.so.1 (0x00007f030312d000)
	/home/uv/UV/Workspace/TEMP/miscProjects/uv/.venv/bin/../lib/libpython3.12.so.1.0 => not found
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f03030dd000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f03030d8000)
	libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f03030d3000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f0302fe4000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007f0302fdd000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f0302dec000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f030312f000)

❯ uv --version
uv 0.5.16 (333f03f11 2025-01-08)

[zanieb]

❯ uv self update
info: Checking for updates...
success: Upgraded uv from v0.5.17 to v0.5.18! https://github.com/astral-sh/uv/releases/tag/0.5.18
❯ uv python install 3.13 --reinstall
Installed Python 3.13.1 in 2.37s
 ~ cpython-3.13.1-linux-x86_64-gnu
❯ uv python find 3.13
/home/zb/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/bin/python3.13
❯ ldd /home/zb/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/bin/python3.13
	linux-vdso.so.1 (0x00007b41f3d00000)
	/home/zb/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/bin/../lib/libpython3.13.so.1.0 (0x00007b41f2400000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007b41f3ccf000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007b41f3cca000)
	libutil.so.1 => /usr/lib/libutil.so.1 (0x00007b41f3cc5000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007b41f3bd6000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007b41f3bcf000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007b41f220f000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007b41f3d02000)
❯ uv venv -p 3.13
Using CPython 3.13.1
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate
❯ ldd .venv/bin/python
	linux-vdso.so.1 (0x000076c3c32b0000)
	/fast/workspace/uv/.venv/bin/../lib/libpython3.13.so.1.0 => not found
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x000076c3c327f000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x000076c3c327a000)
	libutil.so.1 => /usr/lib/libutil.so.1 (0x000076c3c3275000)
	libm.so.6 => /usr/lib/libm.so.6 (0x000076c3c3186000)
	librt.so.1 => /usr/lib/librt.so.1 (0x000076c3c3181000)
	libc.so.6 => /usr/lib/libc.so.6 (0x000076c3c2f8e000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x000076c3c32b2000)

[zanieb]

This seems to be "as-designed" upstream https://bugs.python.org/issue43334#msg388720

[charliermarsh]

Is there anything to resolve here then? Or na?

[zanieb]

Unless someone can point to comments contrary to what I linked, yeah I think our current behavior is correct. I guess the build system should handle resolving to the upstream binary?

[geofft]

One small note here on something that surprised me: not found in ldd output usually indicates that the binary isn't going to run, but clearly it does run. What's going on is that the bin/python is a symlink to the real location, and the ${ORIGIN}-relative path works when computed based on the target of the symlink.

$ ls -l .venv/bin/python
lrwxrwxrwx 1 nixos users 82 Jan 27 19:35 .venv/bin/python -> /home/nixos/.local/share/uv/python/cpython-3.13.1-linux-aarch64-gnu/bin/python3.13
$ ldd .venv/bin/python | grep libpython
        /home/nixos/Downloads/.venv/bin/../lib/libpython3.13.so.1.0 => not found
$ ldd `readlink -f .venv/bin/python` | grep libpython
        /home/nixos/.local/share/uv/python/cpython-3.13.1-linux-aarch64-gnu/bin/../lib/libpython3.13.so.1.0 (0x0000ffff88610000)

This appears to be a bug in ldd: https://sourceware.org/bugzilla/show_bug.cgi?id=25263

Notes on why the bug happens:

ldd works by running the dynamic linker (ld.so) in a mode where it dumps out paths and exits instead of actually running the program. In the past, ldd used to just set an environment variable LD_TRACE_LOADED_OBJECTS=1 and run the argument. If you do it that way, the output is correct:

$ LD_TRACE_LOADED_OBJECTS=1 .venv/bin/python | grep libpython
        /home/nixos/.local/share/uv/python/cpython-3.13.1-linux-aarch64-gnu/bin/../lib/libpython3.13.so.1.0 (0x0000ffffad130000)

The problem with this implementation of ldd is that it has unexpected effects if the program you're running is not in fact dynamically linked and doesn't change its behavior with that variable—you're actually running the program. So, instead, current versions of ldd run by invoking the dynamic linker explicitly.

$ LD_TRACE_LOADED_OBJECTS=1 /lib/ld-linux-aarch64.so.1 .venv/bin/python | grep libpython
        /home/nixos/Downloads/.venv/bin/../lib/libpython3.13.so.1.0 => not found

(On x86-64 the dynamic linker is at /lib64/ld-linux-x86-64.so.2.) Sure enough, if you leave out the environment variable but explicitly invoke the dynamic linker, the program does not start:

$ /lib/ld-linux-aarch64.so.1 .venv/bin/python 
.venv/bin/python: error while loading shared libraries: /home/nixos/Downloads/.venv/bin/../lib/libpython3.13.so.1.0: cannot open shared object file: No such file or directory

The difference here is that when you invoke ld.so with a path, it gets the path name and uses that for $ORIGIN. However, when you run a program directly, it gets its path by reading the /proc/self/exe symlink (see sysdeps/unix/sysv/linux/dl-origin.c in the glibc source). And /proc/self/exe is computed by looking at the open struct file, which I believe is after symlink resolution (see function proc_exe_link in fs/proc/base.c in the Linux kernel source).

I wonder if fixing the bug in ldd would be enough to fix whatever build systems are unhappy about this. @CowKeyMan / @nazdridoy, can you comment on what is going wrong at a higher level, e.g., with a pointer to what specific build system is failing?

[CowKeyMan]

It seems that this issue was closed as not planned, which is fair enough. The only provider that offers this is conda. I have since moved on to using a more standard application environment manager - asdf, instead of uv. It also does not offer this functionality, but my workaround now is to find wherever these libraries are stored and link to there, or then copy them over to the python path. Unfortunate workaround :/

To your reply of which system is failing, I had a C++ project which tries to link to the python library. CMake build fails because it does not find the python path. It would not do this when the library is as stated in the post above, but also maybe I'm using the build system wrong. I'll update this post when I have time to get around to tinkering with that again.

[geofft]

OK, I'd be curious to take a look when you get a chance. The standard CMake FindPython doesn't seem to have issues:

$ cat CMakeLists.txt 
cmake_minimum_required(VERSION 3.15)
project(cm VERSION 1.0 LANGUAGES C)
find_package(Python COMPONENTS Interpreter Development)
add_executable(cm cm.c)
target_include_directories(cm PRIVATE ${Python_INCLUDE_DIRS})
target_link_libraries(cm PRIVATE Python::Python)
$ cat cm.c
#include <Python.h>

int main(void) {
        Py_Initialize();
        PyRun_SimpleString("from __future__ import braces");
}
$ cmake -S . -B build 
-- Could NOT find Python (missing: Python_EXECUTABLE Python_INCLUDE_DIRS Python_LIBRARIES Interpreter Development Development.Module Development.Embed)
[and a bunch of other errors]
$ uv venv -p 3.13
Using CPython 3.13.1
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate
$ ldd .venv/bin/python | grep libpython
        /home/nixos/cm/.venv/bin/../lib/libpython3.13.so.1.0 => not found
$ source .venv/bin/activate
(cm) $ cmake -S . -B build 
-- Found Python: /home/nixos/cm/.venv/bin/python3.13 (found version "3.13.1") found components: Interpreter Development Development.Module Development.Embed
-- Configuring done (0.2s)
-- Generating done (0.0s)
-- Build files have been written to: /home/nixos/cm/build
(cm) $ cmake --build build 
[ 50%] Building C object CMakeFiles/cm.dir/cm.c.o
[100%] Linking C executable cm
[100%] Built target cm
(cm) $ build/cm
  File "<string>", line 1
SyntaxError: not a chance

(This is on NixOS 24.05 stable inside nix-shell -p cmake, i.e., I have no python command on my PATH.)

Conda by its nature has its own environment implementation. I don't think uv should diverge from how upstream venv works, but it definitely makes sense to make sure that build tools know how to work right with upstream venv and locate things properly, either by changing those build tools (which is I think what the suggestion in the Python bug report was) or changing upstream venv.