Free-threaded build config fixes #4719

ngoldbaum · 2024-11-18T22:10:26Z

Trying to get a library name or build pyo3-ffi against a free-threaded ABI for Python earlier than 3.13 now panics.

Also adds tests for the panics and for the free-threaded library names as followup for #4690 where I ran out of steam to add new tests.

While working on this I noticed that there was code to work around python/cpython#101614 which was fixed in CPython for 3.10 and newer, so I modified the workaround code path to only apply for Python 3.9 and older.

ngoldbaum · 2024-11-19T15:19:51Z

pyo3-build-config/src/impl_.rs

+        major: 3,
+        minor: 13,
+    };
+    #[allow(dead_code)]


this is leftover from a mistaken refactoring that removed the only internal use of this outside tests and should be deleted

ngoldbaum · 2024-11-19T15:21:05Z

pyo3-build-config/src/impl_.rs

@@ -1642,24 +1645,33 @@ fn default_lib_name_windows(
    debug: bool,
    gil_disabled: bool,
 ) -> String {
-    if debug {
+    if debug && version.minor < 10 {


maybe define a PY310 constant?

pyo3-build-config/src/impl_.rs

davidhewitt · 2024-11-19T20:40:39Z

👍 thanks, do you want me to review before the above comments are resolved?

ngoldbaum · 2024-11-19T20:45:20Z

Go ahead.

Tiny bit of context: I often look at PRs on my phone since it helps me catch things if I look over code in a different context from where I wrote it.

davidhewitt

Thanks, this seems reasonable to me 👍

davidhewitt · 2024-11-21T22:41:21Z

pyo3-ffi/build.rs

+                    if version < PythonVersion::PY313 {
+                        panic!(


This should use ensure! rather than panic!, it leads to a friendlier presentation of the error to users.

davidhewitt · 2024-11-22T15:32:36Z

See also PyO3/maturin#2310 (comment), can probably adjust here.

ngoldbaum · 2024-11-22T19:52:26Z

Along with some changes to maturin on top of PyO3/maturin#2310 (which I'll create a separate PR for) this latest push lets more of the maturin tests pass on my local dev setup.

davidhewitt

Thanks, can confirm locally that on the PyO3 side this looks correct. I just have a few copy suggestions.

Note that maturin will also need adjusting to not build abi3 wheels in the free-threaded case, i.e. at the moment I get

📦 Built wheel for abi3 Python ≥ 3.7 to /var/folders/z2/3t02d9cn6gxg11n5yd9yk3wm0000gn/T/.tmpM81f0O/pyo3_scratch-0.1.0-cp37-abi3-macosx_11_0_arm64.whl
💥 maturin failed
  Caused by: pip install in /Users/david/.virtualenvs/pyo3-3.13t failed running ["-m", "pip", "--disable-pip-version-check", "install", "--no-deps", "--force-reinstall", "/var/folders/z2/3t02d9cn6gxg11n5yd9yk3wm0000gn/T/.tmpM81f0O/pyo3_scratch-0.1.0-cp37-abi3-macosx_11_0_arm64.whl"]: exit status: 1
--- Stdout:

--- Stderr:
ERROR: pyo3_scratch-0.1.0-cp37-abi3-macosx_11_0_arm64.whl is not a supported wheel on this platform.
---

(maybe you already did that in the maturin PR).

davidhewitt · 2024-11-23T07:22:41Z

guide/src/free-threading.md

+
+The free-threaded build uses a completely new ABI and there is not yet an
+equivalent to the limited API for the free-threaded ABI. That means if your
+crate depends on PyO3 using the `abi3` feature or an an `abi-pyxx` feature, PyO3


Suggested change

crate depends on PyO3 using the `abi3` feature or an an `abi-pyxx` feature, PyO3

crate depends on PyO3 using the `abi3` feature or an an `abi3-pyxx` feature, PyO3

davidhewitt · 2024-11-23T07:27:51Z

pyo3-build-config/src/impl_.rs

+            .unwrap(),
+            "python3_d",
+        );
+        // Python versions older than 3.13 panic if gil_disabled is true


Suggested change

// Python versions older than 3.13 panic if gil_disabled is true

// Python versions older than 3.13 don't support gil_disabled

davidhewitt · 2024-11-23T07:35:52Z

pyo3-ffi/build.rs

+            if !interpreter_config
+                .build_flags
+                .0
+                .contains(&BuildFlag::Py_GIL_DISABLED)
+            {
+                ensure!(
+                        interpreter_config.version <= versions.max || env_var("PYO3_USE_ABI3_FORWARD_COMPATIBILITY").map_or(false, |os_str| os_str == "1"),


This transformation disables the version check completely on the free-threaded build; I think this is risky given testing with 3.14 free-threading is likely to start happening soon.

I think better would be to do something like

if interpreter_config.version < versions.max { if interpreter_config.build_flags.0.contains(&BuildFlags::Py_GIL_DISABLED) { // free-threaded specific error } else if !env_var("PYO3_USE_ABI3_FORWARD_COMPATIBILITY").map_or(false, |os_str| os_str == "1") { // env var not set, raise the existing error } }

I did this and noticed we were using interpreter_config.build_flags.0.contains(&BuildFlags::Py_GIL_DISABLED) quite a bit, so I added that as a method to the InterpreterConfig struct to make the spelling a little clearer

ngoldbaum · 2024-11-23T19:20:42Z

(maybe you already did that in the maturin PR).

I think that was fixed by PyO3/maturin#2316. I also had to update all the test crates to point at a local copy of pyo3.

davidhewitt

Thanks, this looks great to me!

* update build config logic and library name generation * fix free-threaded windows clippy with --features=abi3 * use constant * add release note * apply my self-review comments * use ensure and error handling instead of panicking * skip abi3 fixup on free-threaded build * don't support PYO3_USE_ABI3_FORWARD_COMPATIBILITY on free-threaded build * don't panic in pyo3-ffi in abi3 check * document lack of limited API support * add is_free_threaded() method to InterpreterConfig * implement David's code review suggestions * remove unused imports

ngoldbaum added 3 commits November 18, 2024 14:33

update build config logic and library name generation

1ba689e

fix free-threaded windows clippy with --features=abi3

dd0142b

use constant

6098339

ngoldbaum added the free-threading label Nov 18, 2024

add release note

1ffd21c

ngoldbaum commented Nov 19, 2024

View reviewed changes

pyo3-build-config/src/impl_.rs Show resolved Hide resolved

apply my self-review comments

d3c7859

davidhewitt reviewed Nov 21, 2024

View reviewed changes

ngoldbaum mentioned this pull request Nov 21, 2024

Add free-threaded Python support PyO3/maturin#2310

Merged

davidhewitt mentioned this pull request Nov 22, 2024

release: 0.23.2 #4726

Merged

ngoldbaum added 4 commits November 22, 2024 12:35

use ensure and error handling instead of panicking

6100ff3

skip abi3 fixup on free-threaded build

1aa8b1d

don't support PYO3_USE_ABI3_FORWARD_COMPATIBILITY on free-threaded build

3241d8b

don't panic in pyo3-ffi in abi3 check

dd7cca7

document lack of limited API support

1814015

davidhewitt approved these changes Nov 23, 2024

View reviewed changes

ngoldbaum added 3 commits November 23, 2024 12:13

add is_free_threaded() method to InterpreterConfig

72a12c1

implement David's code review suggestions

139950e

remove unused imports

efc167f

davidhewitt approved these changes Nov 23, 2024

View reviewed changes

davidhewitt added this pull request to the merge queue Nov 23, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Nov 23, 2024

ngoldbaum added this pull request to the merge queue Nov 23, 2024

Merged via the queue into PyO3:main with commit 0fb3623 Nov 24, 2024
46 checks passed

ngoldbaum deleted the build-config-fixes branch November 24, 2024 00:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Free-threaded build config fixes #4719

Free-threaded build config fixes #4719

ngoldbaum commented Nov 18, 2024 •

edited

Loading

ngoldbaum Nov 19, 2024

ngoldbaum Nov 19, 2024

davidhewitt commented Nov 19, 2024

ngoldbaum commented Nov 19, 2024

davidhewitt left a comment

davidhewitt Nov 21, 2024

davidhewitt commented Nov 22, 2024

ngoldbaum commented Nov 22, 2024

davidhewitt left a comment

davidhewitt Nov 23, 2024

davidhewitt Nov 23, 2024

davidhewitt Nov 23, 2024

ngoldbaum Nov 23, 2024

ngoldbaum commented Nov 23, 2024 •

edited

Loading

davidhewitt left a comment

	crate depends on PyO3 using the `abi3` feature or an an `abi-pyxx` feature, PyO3
	crate depends on PyO3 using the `abi3` feature or an an `abi3-pyxx` feature, PyO3

	// Python versions older than 3.13 panic if gil_disabled is true
	// Python versions older than 3.13 don't support gil_disabled

Free-threaded build config fixes #4719

Free-threaded build config fixes #4719

Conversation

ngoldbaum commented Nov 18, 2024 • edited Loading

ngoldbaum Nov 19, 2024

Choose a reason for hiding this comment

ngoldbaum Nov 19, 2024

Choose a reason for hiding this comment

davidhewitt commented Nov 19, 2024

ngoldbaum commented Nov 19, 2024

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt Nov 21, 2024

Choose a reason for hiding this comment

davidhewitt commented Nov 22, 2024

ngoldbaum commented Nov 22, 2024

davidhewitt left a comment

Choose a reason for hiding this comment

davidhewitt Nov 23, 2024

Choose a reason for hiding this comment

davidhewitt Nov 23, 2024

Choose a reason for hiding this comment

davidhewitt Nov 23, 2024

Choose a reason for hiding this comment

ngoldbaum Nov 23, 2024

Choose a reason for hiding this comment

ngoldbaum commented Nov 23, 2024 • edited Loading

davidhewitt left a comment

Choose a reason for hiding this comment

ngoldbaum commented Nov 18, 2024 •

edited

Loading

ngoldbaum commented Nov 23, 2024 •

edited

Loading