Skip to content

python_bindings.md

Latest commit

 

History

History
118 lines (82 loc) · 5.64 KB

python_bindings.md

File metadata and controls

118 lines (82 loc) · 5.64 KB
Raw

Python Bindings and Packaging

Python Interface

The wrapper interface uses an object-based interface instead of explicitly passing device and object on every call.

anariFooBar(device, object, ...) turns into object.fooBar(...).

Object lifetime is automatically handled and doesn't require explicit calls to retain/release.

See examples/python/anari_tutorial_device.py for an example.

Frontend Python Bindings

The SDK python bindings consist of two layers. The raw cffi bindings generated by code_gen/cffi_gen.py and a python wrapper generated from code_gen/__init__.py.in. They are built via the neo_bindings CMake target.

Building

A python package can be built/installed via the pyproject.toml in the SDK's root directory.

Either directly from github: pip3 install git+https://github.com/KhronosGroup/ANARI-SDK

or from an already checked out version of the repo via either pip3 install ANARI_SDK/ or python3 -m build ANARI-SDK/

Device Bindings

Devices are exposed through their own python package containing the device library (eg. libanari_library_helide.so) and a __init__.py that is generated from code_gen/device_trampoline.py.in. The trampoline file simply imports the anari frontend module and loads the device specific library.

This mechanism is chosen to allow all modules to be self contained. Python packages don't have an intrinsic mechanism to determine each others install locations making their contents not easily accessible to the frontend module loadLibrary function. Instead by importing for example import helide the helide module provides its own location and can inject the library location into a newDevice wrapper without making assumptions about install locations.

The plain anari module can still be used the same way as the C API. Using anari.loadLibrary('helide') will find a system installed helide device but not one installed as a python package.

Readying a Device for Packaging

CMake additions

To setup a device for packaging its CMakeLists.txt needs to configure the trampoline file:

set(DEVICE_LIBRARY_NAME helide)
configure_file(${ANARI_CODE_GEN_ROOT}/device_trampoline.py.in ${CMAKE_CURRENT_BINARY_DIR}/__init__.py)

and add install targets for the configured file and the library in a component specific to the python bindings (COMPONENT helide_module here). These install targets will determine what gets put into the package.

install(
  FILES ${CMAKE_CURRENT_BINARY_DIR}/__init__.py
  EXCLUDE_FROM_ALL
  COMPONENT helide_module
  DESTINATION helide/)
install(TARGETS anari_library_helide
  EXCLUDE_FROM_ALL
  COMPONENT helide_module
  LIBRARY DESTINATION helide
  RUNTIME DESTINATION helide)

pyproject.toml

Write a pyproject.toml file using the py-build-cmake build backend by configuring the build in the [tool.py-build-cmake.*] sections.

install_components should match the component name from the previous step.

[project] # Project metadata
name = "helide"
requires-python = ">=3.7"
readme = "README.md"
license = "APACHE"
license-files = ["LICENSE"]
dependencies = ["cffi","anari"]
version = "1.0"

[build-system] # How pip and other frontends should build this project
requires = ["py-build-cmake~=0.5.0"]
build-backend = "py_build_cmake.build"

[tool.py-build-cmake.sdist]
include = ['*']

[tool.py-build-cmake.module]
generated = "package"

[tool.py-build-cmake.cmake] # How to build the CMake project
build_type = "RelWithDebInfo"
source_path = "."
build_args = ["-j"]
install_components = ["helide_module"]

[tool.cibuildwheel.linux]
before-all = """
git clone -b next_release https://github.com/KhronosGroup/ANARI-SDK.git
mkdir ANARI-SDK/build
cd ANARI-SDK/build
cmake ../ -DBUILD_EXAMPLES=OFF -DBUILD_HELIDE_DEVICE=OFF -DBUILD_TESTING=OFF -DINSTALL_VIEWER_LIBRARY=OFF
make && make install
"""

Building "portable" Packages

The pyproject.toml example above contains a [tool.cibuildwheel.linux] section. cibuildwheel is a tool to create portable packages for upload to package repositories. It builds the package in a variety of standardized environments (containers) that allow the packages to be binary compatible with various target systems.

For simple packages like the anari frontend this works without much effort since it has no dependencies other than "usual" system libraries (standard library, libdl etc.).

Device libraries however are likely to have non-trivial dependencies like graphics APIs, CUDA etc.. For the build in the container to work these dependencies likely need to be installed since the containers are minimal. This can be done via the before-all step (or fetch content in the cmake build etc.).

However since those dependencies are not part of the standardized base environment cibuildwheel will detect what other .so the packaged device library depends on and include them in the package. This bloats the package and more importantly probably leads to the package "shipping" third party binaries/code.

If the device for example relies on OpenGL one could apt install the required packages (mesa etc.) in the before-all step and just naively add OpenGL::OpenGL to the target_link_libraries. This would then likely lead to libgl.so and other parts of the mesa infrastructure to get copied into the package which is likely undesirable.

To avoid this issue the device should be kept clean of dynamic link dependencies. In the above opengl example this could be accomplished by using a gl loader library like glad which will fetch the entrypoints dynamically (dlsym etc.) instead of linking against it at compile time triggering the package inclusion.