Namespacing shared libraries for python to avoid name clashes #243

Is your feature request related to a problem? Please describe.

eckitlib (python) bundles certain libraries like libldap in its eckitlib.libs folder. This is great under the assumption that no other libraries are doing the same.

Unfortunately, psycopg2-binary (driver for postgres databases) is doing the same thing, with what seems to be a considerable overlap in library names.

Here's a small MRE that causes a segfault:

docker run --rm -it python:3.13 sh -c 'pip install cfgrib psycopg2-binary; python3 -c "import gribapi;im port psycopg2"'

There's a uv script approach as an alternative MRE:

#!/usr/bin/env -S uv run

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "psycopg2-binary==2.9.11",
#   "cfgrib==0.9.15.1",
# ]
# ///

import gribapi
import psycopg2

print("gribapi module:", getattr(gribapi, "__version__", "unknown"))
print("psycopg2 version:", psycopg2.__version__)
print("about to exit normally, but watch for invalid pointer or segfault messages")

This script seems to give free(): invalid pointer when run with my system python but always segfault in docker.

Describe the solution you'd like

Some ideas (ai-assisted):

• build vendored libs with private SONAMEs (e.g. libldap_eckit.so), and link own extension modules against that SONAME.
• Or, build them as static libraries and link them into a single private .so that doesn’t leak generic symbols.
  • Use symbol visibility controls / version scripts so that: nothing like ldap_initialize, SSL_CTX_new, etc. is exported globally from the wheel’s private copy.

I know this is probably a very significant effort. However, it seems a real issue at the moment that different python "stacks" are growing that are fundamentally incompatible due to "claiming" the same shared library names in an incompatible manner.

Describe alternatives you've considered

No response

Additional context

I've posted this on psycopg2 issue tracker as well. Possibly there will be some insight there: psycopg/psycopg2#1824

Organisation

No response