Failure to find python Interpreter and Development.Module running under yocto/oe

[sv662]

I am attempting to use scikit-build-core 0.10.7 to build python3-pyzmq bindings at 26.2.0 in a yocto environment.
There was no existing recipe within meta-openembedded yet, so I have attempted to create my own.
It is starting to build, but there appears to be a failure in the execution of cmake where it cannot locate python.

CMake Error at /src/platform/build/tmp/work/cortexa9t2hf-neon-xilinx-linux-gnueabi/python3-pyzmq/26.2.0/recipe-sysroot-native/usr/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:230 (message): Could NOT find Python (missing: Interpreter Development.Module)

The environment for this recipe appears to be being setup correctly by bitbake. The local recipe native sysroot does have python3 in it and the PYTHON environment variables are being set to point at those locations, so I'm wondering what it is trying to look for that it is not finding. Python is this context is version 3.12.

The PYTHON env variable is set:
export PYTHON="/src/platform/build/tmp/work/cortexa9t2hf-neon-xilinx-linux-gnueabi/python3-pyzmq/26.2.0/recipe-sysroot-native/usr/bin/python3-native/python3"

... and that location does indeed have the python3 executable:
$ ls -l ./work/cortexa9t2hf-neon-xilinx-linux-gnueabi/python3-pyzmq/26.2.0/recipe-sysroot-native/usr/bin/python3-native/python3*
lrwxrwxrwx 1 10 Nov 26 16:24 ./work/cortexa9t2hf-neon-xilinx-linux-gnueabi/python3-pyzmq/26.2.0/recipe-sysroot-native/usr/bin/python3-native/python3 -> python3.12
-rwxr-xr-x 7 14360 Nov 20 15:59 ./work/cortexa9t2hf-neon-xilinx-linux-gnueabi/python3-pyzmq/26.2.0/recipe-sysroot-native/usr/bin/python3-native/python3.12

I'm attaching the run.do_compile and log.do_compile files from the build so you can see the environment and calls into the build for context.

Any assistance you could give would be gratefully received.

Does scikit-build-core fully pass the environment through to cmake or is there some additional setup that needs to be done?
Would the following command line args to python when running the build within pyzmq have any adverse effect on scikit-build-core?
-m build --no-isolation --wheel --outdir

log.do_compile.txt
run.do_compile.txt

[henryiii]

I'll try to look at this soon, probably after Monday.

[LecrisUT]

The thing I'm worried about with openembeded is getting the cross-compilation right and there are no contamination of packages. With FindPython you have the additional complication that you need to make sure the python of the bitbake executor is not used because afaiu you want to pick up the python.h and library of the runtime system.

I tried to look into it, but the hardest part is setting up the initial docker environment with openembeded, and a test environment. Could you help share a dockerfile with a minimal setup?

[sv662]

pyzmq_test.zip

Hi @LecrisUT

I don't know if this would be the sort of thing you were expecting, but I've put together a failing example. Uses a dockerfile and docker compose setup to create a build container based on ubuntu. The docker compose service will run the build.sh script which pulls all the meta layers using repo and kick off a bitbake using a new layer with the recipes I was trying to scikit-build-core and pyzmq. You can always override the command and just go in on a bash prompt and interact with it directly on the terminal after that inital build fails.

Removing the REQUIRED_VERSION line from local.conf would put it back to the version released in scarthgap.

This might not be the normal way to use docker, it's just how I know to set it up to build a project with source on the host machine mounted as a volume.

I think the only environment dependency would be setting env variables DOCKER_UID and DOCKER_GID to match your users id and groups like this:
export DOCKER_GID=$(id -g)
export DOCKER_UID=$(id -u)

To run this do the following:
unzip pyzmq_test.zip
cd pyzmq_test
docker compose build yocto_xlnx
docker compose run --rm yocto_xlnx

If that's not what you were expecting or you have any trouble with it, do let me know.

[coolo]

@sv662 I'm in the same boat as you and what helped kicking this further down the road was this patch:

@@ -0,0 +1,22 @@
+From 42573aad35732223bb1467cbfdd35a5081e80d33 Mon Sep 17 00:00:00 2001
+From: Stephan Kulow <[email protected]>
+Date: Thu, 19 Dec 2024 12:56:28 +0100
+Subject: [PATCH] fix: check for PYTHON_INCLUDE_DIR
+
+---
+ src/scikit_build_core/builder/builder.py | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/src/scikit_build_core/builder/builder.py b/src/scikit_build_core/builder/builder.py
+index 437265f..18d55b9 100644
+--- a/src/scikit_build_core/builder/builder.py
++++ b/src/scikit_build_core/builder/builder.py
+@@ -203,7 +203,7 @@ class Builder:
+         python_sabi_library = (
+             get_python_library(self.config.env, abi3=True) if limited_api else None
+         )
+-        python_include_dir = get_python_include_dir()
++        python_include_dir = os.getenv("PYTHON_INCLUDE_DIR") or get_python_include_dir()
+         numpy_include_dir = get_numpy_include_dir()
+ 
+         # Classic Find Python

I hope it helps to find a proper solution. At least I've got myself a pyzmq-26.2 package now :)