feat(build): Python wheels workflow and build backend (#4428)

### Summary
This PR is the spiritual successor to #4011. It implements a
`scikit-build-core`-based python build-backend, making it possible to
use `pip install .` to build from source; and it adds a Github workflow
for building with `cibuildwheel` and publishing to pypi.org binary
distributions (bdists) of the Python module / extensions / CLI tools for
cpython 3.8-3.13, across major operating systems and architectures.

When you `pip install OpenImageIO`, pip attempts to retrieve an
OpenImageIO bdist from pypi.org for the host's platform / architecture /
Python interpreter. If it can't find something appropriate, pip will
attempt to build locally from the OpenImageIO source distribution
(sdist), downloading and temporarily installing cmake and ninja if
necessary.

### PEP-Compliant Packaging: `pyproject.toml`

The `pyproject.toml` file is organized in three parts:
1. **Package metadata**: standard attributes identifying and describing
the Python package, its run-time and build-time requirements,
entry-points to executable scripts, and so forth.
2. **scikit-build-core options**: governs how `pip install ...`
interacts with `cmake`.
3. **cibuildwheel options**: additional steps and considerations for
building, packaging, and testing relocatable wheel build artifacts in
isolated environments.

### Additions to `__ init __.py` 

Previously, we were using a custom OpenImageIO/__ init__.py file to help
Python-3.8+ on Windows load the shared libraries linked by the Python
module (i.e., the .dll files that live alongside oiiotool.exe under
$PATH).

This PR adds an additional method for loading the DLL path, necessitated
by differences between pip-based and traditional CMake-based installs.

It also adds a mechanism for invoking binary executables found in the
.../site-packages/OpenImageIO/bin directory. This provides a means for
exposing Python script "shims" for each CLI tool, installed to
platform-specific locations under $PATH, while keeping the actual
binaries in a static location relative to the dynamic libraries. Upshot
is, in `pyproject.toml`,
each item under `[project.scripts]` is turned into a Python script upon
installation that behaves indistinguishably to the end user to the CLI
binary executable of the same name.

### Relocatable Binary Distributions with `cibuildwheel` + `repairwheel`

[cibuildwheel](https://github.com/pypa/cibuildwheel) is a widely-used
tool for drastically streamlining the process of building, repairing,
and testing Python wheels across platforms, architectures, interpreters,
and interpreter versions.

Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to
"MinSizeRel" to optimize for size (instead of speed) -- this seems to
shave ~1.5MB off each .whl's size, compared to "Release"

### "Wheels" Github workflow

I straight-up copied `.github/workflows/wheel.yml` from OpenColorIO and
made a few OIIO-specific modifications. When pushing a commit tagged
v3*, the workflow will invoke a platform-agnostic "build sdist" (source
distribution) task, followed by a series of tasks for building OIIO
wheels for cpython-3.8-3.13 on Windows, Linux (x86_64 + aarch64, new
libstdc++), and MacOS (x86_64 + arm64) and persisting build artifacts;
followed finally by a task for publishing the build artifacts to
pypi.org

Note: For the sake of simplicity and troubleshooting, I've made as few
changes to OpenColorIO's wheel.yml as I could get away with; but in the
future, we can also build wheels for the PyPy interpreter, and possibly
pyodide.

Note: A "trusted publisher" must be set up on pypi.org. See
https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/

### Other Changes

I made some minor adjustments to `pythonutils.cmake` and
`fancy_add_executable.cmake` that only affect scikit-build-core-based
installs:
- -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property
to point to the relative path to the dynamic libraries, for the Python
module and CLI tools, respectively. This helps ensure that pip-based
builds and installs from source (as opposed to installs from repaired,
pre-built wheels) yield relocatable, importable packages, without
needing to mess with $LD_LIBRARY_PATH etc.


## Tests

`cibuildwheel` tests if `oiiotool --buildinfo` runs. If that command
elicits code zero, it means the "oiiotool" Python script installed by
the wheel is able to `import OpenImageIO`; that the actual binary
executable `oiiotool` is properly packaged and exists in the expected
location (e.g., at `.../site-packages/OpenImageIO/bin`); and that all
runtime dependencies are found.

## Inspiration, Credit, Prior Art
- @aclark4life's and @JeanChristopheMorinPerso's efforts + direction +
discussion + advice. See
[#3249](#3249),
and
[#4011](#4011),
as well as JCM's
[python_wheels](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels)
and
[python_wheels_windows](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels_windows)
branches. This PR is an attempt to leverage OIIO-2.6+
self-building-dependency features with # 4011's minimalist and modern
approach to packaging.
- OpenColorIO -- I tried to copy as much as I could from @remia et al's
fantastic work with all things wheels-related. The __init __.py
modifications, the way we're wrapping the CLI tools, and the github
Wheels workflow are lifted almost-verbatim from OCIO. Insert pun about
reinventing the wheel here.
  - @joaovbs96's help and patience with testing stuff on Windows.

---------

Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Larry Gritz <[email protected]>
Signed-off-by: Anton Dukhovnikov <[email protected]>
Signed-off-by: Basile Fraboni <[email protected]>
Signed-off-by: Vlad (Kuzmin) Erium <[email protected]>
Signed-off-by: Joseph Goldstone <[email protected]>
Signed-off-by: Darby Johnston <[email protected]>
Signed-off-by: Jeremy Retailleau <[email protected]>
Signed-off-by: zachlewis <[email protected]>
Co-authored-by: Larry Gritz <[email protected]>
Co-authored-by: Anton Dukhovnikov <[email protected]>
Co-authored-by: Basile Fraboni <[email protected]>
Co-authored-by: Vlad (Kuzmin) Erium <[email protected]>
Co-authored-by: Joseph Goldstone <[email protected]>
Co-authored-by: Darby Johnston <[email protected]>
Co-authored-by: Jeremy Retailleau <[email protected]>
Co-authored-by: Jean-Christophe Morin <[email protected]>

Author: 9 people

.github/workflows/wheel.yml

@@ -4,13 +4,17 @@ branches/
 build/
 dist/
 ext/
+wheelhouse/
+_coverage/
+.python-version
 .idea
 .vscode
 .cproject
 .project
 .DS_Store
 *.pyc
-_coverage/
+/*.lock
+gastest.o
 
 # Exclude test files I tend to leave around at the top level. The leading
 # slash ensures that files with these extensions within subdirectories are not
@@ -21,3 +25,5 @@ _coverage/
 /*.jxl
 /*.tx
 /*.log
+
+

.gitignore

@@ -181,6 +181,38 @@ include (compiler)
 # Dependency finding utilities and all dependency-related options
 include (dependency_utils)
 
+option (IGNORE_HOMEBREWED_DEPS "If ON, will ignore homebrew-installed dependencies" OFF)
+if (IGNORE_HOMEBREWED_DEPS)
+    # Define the list of prefixes to ignore
+    set (HOMEBREW_PREFIXES
+         /opt/homebrew
+         /usr/local
+         /usr/X11
+         /usr/X11R6
+         /opt/X11
+    )
+    message (STATUS "Ignoring Homebrew dependencies and adjusted environment and CMake variables accordingly.")
+    foreach (_cmake_var
+             CMAKE_SYSTEM_INCLUDE_PATH
+             CMAKE_SYSTEM_LIBRARY_PATH
+             CMAKE_PREFIX_PATH)
+        remove_prefixes_from_variable (CMAKE ${_cmake_var} "${HOMEBREW_PREFIXES}")
+    endforeach ()
+
+    # Adjust CMAKE_IGNORE_PATH
+    foreach (_prefix IN LISTS HOMEBREW_PREFIXES)
+        list (APPEND CMAKE_IGNORE_PATH
+              "${_prefix}"
+              "${_prefix}/lib"
+              "${_prefix}/bin"
+              "${_prefix}/include"
+             )
+    endforeach ()
+
+    message (STATUS "CMAKE_IGNORE_PATH: ${CMAKE_IGNORE_PATH}")
+endif ()
+
+
 # Utilities and options related to finding python and making python bindings
 include (pythonutils)
 
@@ -238,6 +270,17 @@ if (NOT BUILD_OIIOUTIL_ONLY)
     add_subdirectory (src/libOpenImageIO)
 endif ()
 
+# Disable building of certain tools when building Python wheels
+if (SKBUILD)
+    set (ENABLE_iconvert OFF)
+    set (ENABLE_idiff OFF)
+    set (ENABLE_igrep OFF)
+    set (ENABLE_iinfo OFF)
+    set (ENABLE_testtex OFF)
+    set (ENABLE_iv OFF)
+endif ()
+
+
 if (OIIO_BUILD_TOOLS AND NOT BUILD_OIIOUTIL_ONLY)
     add_subdirectory (src/iconvert)
     add_subdirectory (src/idiff)
@@ -258,10 +301,16 @@ if (NOT EMBEDPLUGINS AND NOT BUILD_OIIOUTIL_ONLY)
     endforeach ()
 endif ()
 
-if (USE_PYTHON AND Python3_Development_FOUND AND NOT BUILD_OIIOUTIL_ONLY)
+
+if (WIN32)
+    set (_py_dev_found Python3_Development_FOUND)    
+else ()
+    set (_py_dev_found Python3_Development.Module_FOUND)
+endif ()
+if (USE_PYTHON AND ${_py_dev_found} AND NOT BUILD_OIIOUTIL_ONLY)
     add_subdirectory (src/python)
 else ()
-    message (STATUS "Not building Python bindings: USE_PYTHON=${USE_PYTHON}, Python3_Development_FOUND=${Python3_Development_FOUND}")
+    message (STATUS "Not building Python bindings: USE_PYTHON=${USE_PYTHON}, Python3_Development.Module_FOUND=${Python3_Development.Module_FOUND}")
 endif ()
 
 add_subdirectory (src/include)

CMakeLists.txt

@@ -230,19 +230,20 @@ Make targets you should know about:
 
 Additionally, a few helpful modifiers alter some build-time options:
 
-|  Target                   |  Command                                       |
-| ------------------------- | ---------------------------------------------- |
-| make VERBOSE=1 ...        |  Show all compilation commands                 |
-| make STOP_ON_WARNING=0    |  Do not stop building if compiler warns        |
-| make EMBEDPLUGINS=0 ...   |  Don't compile the plugins into libOpenImageIO |
-| make USE_OPENGL=0 ...     |  Skip anything that needs OpenGL               |
-| make USE_QT=0 ...         |  Skip anything that needs Qt                   |
-| make MYCC=xx MYCXX=yy ... |  Use custom compilers                          |
-| make USE_PYTHON=0 ...     |  Don't build the Python binding                |
-| make BUILD_SHARED_LIBS=0  |  Build static library instead of shared        |
-| make LINKSTATIC=1 ...     |  Link with static external libraries when possible |
-| make SOVERSION=nn ...     |  Include the specified major version number in the shared object metadata |
-| make NAMESPACE=name       |   Wrap everything in another namespace         |
+|  Target                       |  Command                                                                  |
+| ----------------------------- | ------------------------------------------------------------------------- |
+| make VERBOSE=1 ...            |  Show all compilation commands                                            |
+| make STOP_ON_WARNING=0        |  Do not stop building if compiler warns                                   |
+| make EMBEDPLUGINS=0 ...       |  Don't compile the plugins into libOpenImageIO                            |
+| make USE_OPENGL=0 ...         |  Skip anything that needs OpenGL                                          |
+| make USE_QT=0 ...             |  Skip anything that needs Qt                                              |
+| make MYCC=xx MYCXX=yy ...     |  Use custom compilers                                                     |
+| make USE_PYTHON=0 ...         |  Don't build the Python binding                                           |
+| make BUILD_SHARED_LIBS=0      |  Build static library instead of shared                                   |
+| make IGNORE_HOMEBREWED_DEPS=1 |  Ignore homebrew-managed dependencies                                     |
+| make LINKSTATIC=1 ...         |  Link with static external libraries when possible                        |
+| make SOVERSION=nn ...         |  Include the specified major version number in the shared object metadata |
+| make NAMESPACE=name           |  Wrap everything in another namespace                                     |
 
 The command 'make help' will list all possible options.
 
@@ -366,6 +367,29 @@ If you've built OIIO from source and ``import OpenImageIO`` is throwing a Module
 ``OPENIMAGEIO_PYTHON_LOAD_DLLS_FROM_PATH=1``. 
 
 
+
+Python-based Builds and Installs
+--------------------------------
+
+**Installing from prebuilt binary distributions**
+
+If you're only interested in the Python module and the CLI tools, you can install with `pip` or `uv`:
+
+> ```pip install OpenImageIO```
+
+**Building and installing from source**
+
+If you have a C++ compiler installed, you can also use the Python build-backend to compile and install
+from source by navigating to the root of the repository and running: ```pip install .```
+
+This will download and install CMake and Ninja if necessary, and invoke the CMake build system; which,
+in turn, will build missing dependencies, compile OIIO, and install the Python module, the libraries, 
+the headers, and the CLI tools to a platform-specific, Python-specific location. 
+
+See the [scikit-build-core docs](https://scikit-build-core.readthedocs.io/en/latest/configuration.html#configuring-cmake-arguments-and-defines)
+for more information on customizing and overriding build-tool options and CMake arguments.
+
+
 Test Images
 -----------
 

INSTALL.md

@@ -378,6 +378,7 @@ help:
 	@echo "      USE_NUKE=0               Don't build Nuke plugins"
 	@echo "      Nuke_ROOT=path           Custom Nuke installation"
 	@echo "      NUKE_VERSION=ver         Custom Nuke version"
+	@echo "      IGNORE_HOMEBREWED_DEPS=1 Don't use dependencies installed by Homebrew"
 	@echo "  OIIO build-time options:"
 	@echo "      INSTALL_PREFIX=path      Set installation prefix (default: ./${INSTALL_PREFIX})"
 	@echo "      NAMESPACE=name           Override namespace base name (default: OpenImageIO)"

Makefile

@@ -0,0 +1,122 @@
+[project]
+name = "OpenImageIO"
+# The build backend ascertains the version from the CMakeLists.txt file.
+dynamic = ["version"]
+description = "Reading, writing, and processing images in a wide variety of file formats, using a format-agnostic API, aimed at VFX applications."
+authors = [
+    {name = "Larry Gritz", email = "[email protected]"},
+    {name = "OpenImageIO Contributors", email = "[email protected]"}
+]
+maintainers = [
+    {name = "OpenImageIO Contributors", email="[email protected]"},
+]
+readme = "README.md"
+license = {text = "Apache-2.0"}
+classifiers = [
+    "Natural Language :: English",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "License :: OSI Approved :: Apache Software License",
+    "Topic :: Multimedia :: Graphics",
+    "Topic :: Multimedia :: Video",
+    "Topic :: Multimedia :: Video :: Display",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+requires-python = ">= 3.8"
+
+[project.urls]
+Homepage = "https://openimageio.org/"
+Source = "https://github.com/AcademySoftwareFoundation/OpenImageIO"
+Documentation = "https://docs.openimageio.org"
+Issues = "https://github.com/AcademySoftwareFoundation/OpenImageIO/issues"
+
+
+[project.scripts]
+# Use the convention below to expose CLI tools as Python scripts.
+maketx = "OpenImageIO:_command_line"
+oiiotool = "OpenImageIO:_command_line"
+
+[build-system]
+build-backend = "scikit_build_core.build"
+requires = [
+    "scikit-build-core>=0.10.6,<1",
+    "pybind11",
+]
+
+[tool.scikit-build]
+build.verbose = true
+# Exclude unnecessary directories from the source distribution.
+sdist.exclude = [".github", "testsuite", "ASWF", "docs"]
+# Pin minimum scikit-build-core to that specified in build-system.requires.
+minimum-version = "build-system.requires"
+# Pin minimum CMake version to that specified in CMakeLists.txt.
+cmake.version = "CMakeLists.txt"
+wheel.license-files = ["LICENSE.md", "THIRD-PARTY.md"]
+# Make sure the package is structured as expected.
+wheel.install-dir = "OpenImageIO"
+# Only install the user and fonts components.
+install.components = ["user", "fonts"]
+
+[tool.scikit-build.cmake.define]
+# Build missing dependencies. See src/cmake for details.
+OpenImageIO_BUILD_MISSING_DEPS = "all"
+# Don't build tests. Dramatically improves build time.
+OIIO_BUILD_TESTS = "0"
+# Prefer linking static dependencies when possible.
+LINKSTATIC = "1"
+# Standardize the install directory for libraries, as expected by
+# other parts of the wheels build process.
+CMAKE_INSTALL_LIBDIR = "lib"
+
+# Dynamically set the package version metadata by pasrsing CMakeLists.txt.
+[tool.scikit-build.metadata.version]
+provider = "scikit_build_core.metadata.regex"
+input = "CMakeLists.txt" 
+regex = 'set \(OpenImageIO_VERSION "(?P<value>[0-9a-z.]+)"\)'
+
+# On macOS, ensure dependencies are only built for the target architecture.
+[[tool.scikit-build.overrides]]
+if.platform-system = "darwin"
+if.platform-machine  = "arm64"
+inherit.cmake.define = "append"
+cmake.define.CMAKE_OSX_ARCHITECTURES = "arm64"
+
+[[tool.scikit-build.overrides]]
+if.platform-system = "darwin"
+if.platform-machine  = "x86_64"
+inherit.cmake.define = "append"
+cmake.define.CMAKE_OSX_ARCHITECTURES = "x86_64"
+
+[tool.cibuildwheel]
+build-verbosity = 1
+skip = [
+    # Skip 32-bit builds
+    "*-win32",
+    "*-manylinux_i686",
+    # Building with musl seems to work, but the repair-wheel step seems to fail...
+    # This may be a bug in repairwheel (or auditwheel)?
+    "*musllinux*",  
+]
+test-command = "oiiotool --buildinfo"
+
+[tool.cibuildwheel.macos.environment]
+SKBUILD_CMAKE_ARGS = "-DLINKSTATIC=1; -DIGNORE_HOMEBREWED_DEPS=1"
+# C++17 - std::filesystem is only available in macOS 10.15 and later; ARM compatibility introduced in 11.
+MACOSX_DEPLOYMENT_TARGET = "11"
+# Optimize for size (not speed).
+SKBUILD_CMAKE_BUILD_TYPE = "MinSizeRel"
+
+[tool.cibuildwheel.linux.environment]
+SKBUILD_CMAKE_ARGS = "-DLINKSTATIC=1"
+# Suppress warnings that cause linux cibuildwheel build to fail
+CXXFLAGS = "-Wno-error=stringop-overflow -Wno-pragmas"
+SKBUILD_CMAKE_BUILD_TYPE = "MinSizeRel"
+
+[tool.cibuildwheel.windows.environment]
+SKBUILD_CMAKE_BUILD_TYPE = "MinSizeRel"

pyproject.toml

@@ -17,9 +17,9 @@ set_cache (Freetype_BUILD_SHARED_LIBS  OFF
 
 string (MAKE_C_IDENTIFIER ${Freetype_BUILD_VERSION} Freetype_VERSION_IDENT)
 
-# Conditionally disable support for PNG-compressed OpenType embedded bitmaps on Apple Silicon
-# https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4423#issuecomment-2455034286
-if ( APPLE AND ( CMAKE_SYSTEM_PROCESSOR MATCHES "arm64" OR CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64" ) )
+# Conditionally disable support for PNG-compressed OpenType embedded bitmaps on MacOS
+# https://github.com/AcademySoftwareFoundation/OpenImageIO/pull/4423#issuecomment-2455217897
+if ( APPLE )
     set (_freetype_EXTRA_CMAKE_ARGS -DFT_DISABLE_PNG=ON )
 endif ()
 

src/cmake/build_Freetype.cmake

@@ -6,7 +6,7 @@
 # OpenColorIO by hand!
 ######################################################################
 
-set_cache (OpenColorIO_BUILD_VERSION 2.4.0 "OpenColorIO version for local builds")
+set_cache (OpenColorIO_BUILD_VERSION 2.4.1 "OpenColorIO version for local builds")
 set (OpenColorIO_GIT_REPOSITORY "https://github.com/AcademySoftwareFoundation/OpenColorIO")
 set (OpenColorIO_GIT_TAG "v${OpenColorIO_BUILD_VERSION}")
 set_cache (OpenColorIO_BUILD_SHARED_LIBS  OFF #ON

src/cmake/build_OpenColorIO.cmake

@@ -10,7 +10,7 @@ set_cache (PNG_BUILD_VERSION 1.6.44 "PNG version for local builds")
 set (PNG_GIT_REPOSITORY "https://github.com/glennrp/libpng")
 set (PNG_GIT_TAG "v${PNG_BUILD_VERSION}")
 
-set_cache (PNG_BUILD_SHARED_LIBS ${LOCAL_BUILD_SHARED_LIBS_DEFAULT}
+set_cache (PNG_BUILD_SHARED_LIBS OFF
            DOC "Should execute a local PNG build, if necessary, build shared libraries" ADVANCED)
 
 string (MAKE_C_IDENTIFIER ${PNG_BUILD_VERSION} PNG_VERSION_IDENT)
@@ -28,7 +28,7 @@ build_dependency_with_cmake (PNG
     GIT_REPOSITORY  ${PNG_GIT_REPOSITORY}
     GIT_TAG         ${PNG_GIT_TAG}
     CMAKE_ARGS
-        -D PNG_SHARED=OFF
+        -D PNG_SHARED=${PNG_BUILD_SHARED_LIBS}
         -D PNG_STATIC=ON
         -D PNG_EXECUTABLES=OFF
         -D PNG_TESTS=OFF

src/cmake/build_PNG.cmake

@@ -9,7 +9,7 @@
 set_cache (libdeflate_BUILD_VERSION 1.20 "libdeflate version for local builds")
 set (libdeflate_GIT_REPOSITORY "https://github.com/ebiggers/libdeflate")
 set (libdeflate_GIT_TAG "v${libdeflate_BUILD_VERSION}")
-set_cache (libdeflate_BUILD_SHARED_LIBS  ${LOCAL_BUILD_SHARED_LIBS_DEFAULT}
+set_cache (libdeflate_BUILD_SHARED_LIBS OFF # ${LOCAL_BUILD_SHARED_LIBS_DEFAULT}
            DOC "Should a local libdeflate build, if necessary, build shared libraries" ADVANCED)
 
 string (MAKE_C_IDENTIFIER ${libdeflate_BUILD_VERSION} libdeflate_VERSION_IDENT)
@@ -20,6 +20,7 @@ build_dependency_with_cmake(libdeflate
     GIT_TAG         ${libdeflate_GIT_TAG}
     CMAKE_ARGS
         -D BUILD_SHARED_LIBS=${libdeflate_BUILD_SHARED_LIBS}
+        -D LIBDEFLATE_BUILD_SHARED_LIB=${libdeflate_BUILD_SHARED_LIBS}
         -D CMAKE_POSITION_INDEPENDENT_CODE=ON
         -D CMAKE_INSTALL_LIBDIR=lib
         -D LIBDEFLATE_BUILD_GZIP=OFF
@@ -32,6 +33,7 @@ set (libdeflate_ROOT ${libdeflate_LOCAL_INSTALL_DIR})
 # Signal to caller that we need to find again at the installed location
 set (libdeflate_REFIND TRUE)
 set (libdeflate_REFIND_ARGS CONFIG)
+set (libdeflate_REFIND_VERSION ${libdeflate_BUILD_VERSION})
 
 if (libdeflate_BUILD_SHARED_LIBS)
     install_local_dependency_libs (libdeflate libdeflate)