libadalang test runner on Windows with Python3.8+ cannot find dlls

[LordAro]

Seems that Python changed how it handles dll dependencies in 3.8, and now extra paths need to be added explicitly:
https://docs.python.org/3/library/os.html#os.add_dll_directory

Currently it fails with a message something along the lines of:

Traceback (most recent call last):
  File "C:\libadalang\tmp\name_resolution__S709-005\test.py", line 1, in <module>
    import libadalang as lal
  File "C:\libadalang\build\python\libadalang\__init__.py", line 94, in <module>
    _c_lib = ctypes.cdll.LoadLibrary(_c_lib_path)
  File "C:\Python310\lib\ctypes\__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "C:\Python310\lib\ctypes\__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'libadalang.dll' (or one of its dependencies). Try using the full path with constructor syntax.

As a cheap and nasty hack, the following in module_py.mako just above the LoadLibrary call seems to work, but could probably be improved:

for path in os.environ['PATH'].split(';'):
    if os.path.exists(path):
        os.add_dll_directory(path)

langkit/langkit/templates/python_api/module_py.mako

Line 79 in d79dfd3

_c_lib = ctypes.cdll.LoadLibrary(_c_lib_path)

[pmderodat]

It seems that we will indeed need to do something similar. We’ll take care of it, thank you for bringing this to our attention!

[LordAro]

Additional find - mypy 0.800 used by the langkit tests is too old for os.add_dll_directory. Looks like 0.900 should be sufficient has a new enough version of os.pyi, though it didn't seem to fix the error for me, and I got other errors (with 0.950)

langkit\utils\memoization.py:21: error: Incompatible types in assignment (expression has type "Optional[Any]", variable has type "List[Dict[Any, Any]]")
contrib\python\build\python\libpythonlang\__init__.py:72: error: Module has no attribute "add_dll_directory"
contrib\lkt\build\python\liblktlang\__init__.py:72: error: Module has no attribute "add_dll_directory"
langkit\auto_properties_dsl.py:6: error: Library stubs not installed for "docutils.parsers.rst" (or incompatible with Python 3.7)
langkit\auto_properties_dsl.py:6: note: Hint: "python3 -m pip install types-docutils"
langkit\auto_properties_dsl.py:6: note: (or run "mypy --install-types" to install all missing stub packages)
langkit\auto_properties_dsl.py:6: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
langkit\auto_properties_dsl.py:6: error: Library stubs not installed for "docutils" (or incompatible with Python 3.7)
langkit\auto_properties_dsl.py:6: error: Library stubs not installed for "docutils.parsers" (or incompatible with Python 3.7)
langkit\auto_properties_dsl.py:7: error: Library stubs not installed for "docutils.statemachine" (or incompatible with Python 3.7)
langkit\libmanage.py:690: error: Incompatible types in assignment (expression has type "Callable[[Type[BaseException], BaseException, TracebackType], Any]", variable has type "Callable[[Type[BaseException], BaseException, Optional[TracebackType]], Any]")

[pmderodat]

Hello! We are unsure about the best course of action for this issue.

For the moment we think that making the Python bindings call os.add_dll_directory themselves feels to go against the design of this new mechanism. Our understanding of this situation is that it would be up to the “application packager” to do this, so whatever uses the Langkit-generated library.

With this in mind, our conclusion so far would be that there is nothing to do in Langkit.

[pmderodat]

Hello,

Some time ago, we introduced a --generate-auto-dll-dirs to manage.py generate|make commands (62b8245) so that the generated code for Python bindings do the hack that you mentionned in your first message. We kept this behavior optional and disabled by default because we feel this defeats the purpose of Python’s 3.8 DLL lookup mechanism, so we likely won’t be able to do much more…