Merge branch 'main' into threadsafe-err

README.md

@@ -207,7 +207,7 @@ about this topic.
 - [hyperjson](https://github.com/mre/hyperjson) _A hyper-fast Python module for reading/writing JSON data using Rust's serde-json._
 - [inline-python](https://github.com/fusion-engineering/inline-python) _Inline Python code directly in your Rust code._
 - [johnnycanencrypt](https://github.com/kushaldas/johnnycanencrypt) OpenPGP library with Yubikey support.
-- [jsonschema-rs](https://github.com/Stranger6667/jsonschema-rs/tree/master/bindings/python) _Fast JSON Schema validation library._
+- [jsonschema](https://github.com/Stranger6667/jsonschema/tree/master/crates/jsonschema-py) _A high-performance JSON Schema validator for Python._
 - [mocpy](https://github.com/cds-astro/mocpy) _Astronomical Python library offering data structures for describing any arbitrary coverage regions on the unit sphere._
 - [opendal](https://github.com/apache/opendal/tree/main/bindings/python) _A data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way._
 - [orjson](https://github.com/ijl/orjson) _Fast Python JSON library._

examples/README.md

@@ -9,9 +9,11 @@ Below is a brief description of each of these:
 | `decorator` | A project showcasing the example from the [Emulating callable objects](https://pyo3.rs/latest/class/call.html) chapter of the guide. |
 | `maturin-starter` | A template project which is configured to use [`maturin`](https://github.com/PyO3/maturin) for development. |
 | `setuptools-rust-starter` | A template project which is configured to use [`setuptools_rust`](https://github.com/PyO3/setuptools-rust/) for development. |
-| `word-count` | A quick performance comparison between word counter implementations written in each of Rust and Python. |
 | `plugin` | Illustrates how to use Python as a scripting language within a Rust application |
-| `sequential` | Illustrates how to use pyo3-ffi to write subinterpreter-safe modules |
+
+Note that there are also other examples in the `pyo3-ffi/examples`
+directory that illustrate how to create rust extensions using raw FFI calls into
+the CPython C API instead of using PyO3's abstractions.
 
 ## Creating new projects from these examples
 

guide/src/faq.md

@@ -13,9 +13,11 @@ Sorry that you're having trouble using PyO3. If you can't find the answer to you
 5. Thread A is blocked, because it waits to re-acquire the GIL which thread B still holds.
 6. Deadlock.
 
-PyO3 provides a struct [`GILOnceCell`] which works similarly to these types but avoids risk of deadlocking with the Python GIL. This means it can be used in place of other choices when you are experiencing the deadlock described above. See the documentation for [`GILOnceCell`] for further details and an example how to use it.
+PyO3 provides a struct [`GILOnceCell`] which implements a single-initialization API based on these types that relies on the GIL for locking. If the GIL is released or there is no GIL, then this type allows the initialization function to race but ensures that the data is only ever initialized once. If you need to ensure that the initialization function is called once and only once, you can make use of the [`OnceExt`] and [`OnceLockExt`] extension traits that enable using the standard library types for this purpose but provide new methods for these types that avoid the risk of deadlocking with the Python GIL. This means they can be used in place of other choices when you are experiencing the deadlock described above. See the documentation for [`GILOnceCell`] and [`OnceExt`] for further details and an example how to use them.
 
 [`GILOnceCell`]: {{#PYO3_DOCS_URL}}/pyo3/sync/struct.GILOnceCell.html
+[`OnceExt`]: {{#PYO3_DOCS_URL}}/pyo3/sync/trait.OnceExt.html
+[`OnceLockExt`]: {{#PYO3_DOCS_URL}}/pyo3/sync/trait.OnceLockExt.html
 
 ## I can't run `cargo test`; or I can't build in a Cargo workspace: I'm having linker issues like "Symbol not found" or "Undefined reference to _PyExc_SystemError"!
 

guide/src/free-threading.md

@@ -152,6 +152,60 @@ We plan to allow user-selectable semantics for mutable pyclass definitions in
 PyO3 0.24, allowing some form of opt-in locking to emulate the GIL if that is
 needed.
 
+## Thread-safe single initialization
+
+Until version 0.23, PyO3 provided only `GILOnceCell` to enable deadlock-free
+single initialization of data in contexts that might execute arbitrary Python
+code. While we have updated `GILOnceCell` to avoid thread safety issues
+triggered only under the free-threaded build, the design of `GILOnceCell` is
+inherently thread-unsafe, in a manner that can be problematic even in the
+GIL-enabled build.
+
+If, for example, the function executed by `GILOnceCell` releases the GIL or
+calls code that releases the GIL, then it is possible for multiple threads to
+try to race to initialize the cell. While the cell will only ever be intialized
+once, it can be problematic in some contexts that `GILOnceCell` does not block
+like the standard library `OnceLock`.
+
+In cases where the initialization function must run exactly once, you can bring
+the `OnceExt` or `OnceLockExt` traits into scope. The `OnceExt` trait adds
+`OnceExt::call_once_py_attached` and `OnceExt::call_once_force_py_attached`
+functions to the api of `std::sync::Once`, enabling use of `Once` in contexts
+where the GIL is held. Similarly, `OnceLockExt` adds
+`OnceLockExt::get_or_init_py_attached`. These functions are analogous to
+`Once::call_once`, `Once::call_once_force`, and `OnceLock::get_or_init` except
+they accept a `Python<'py>` token in addition to an `FnOnce`. All of these
+functions release the GIL and re-acquire it before executing the function,
+avoiding deadlocks with the GIL that are possible without using the PyO3
+extension traits. Here is an example of how to use `OnceExt` to
+enable single-initialization of a runtime cache holding a `Py<PyDict>`.
+
+```rust
+# fn main() {
+# use pyo3::prelude::*;
+use std::sync::Once;
+use pyo3::sync::OnceExt;
+use pyo3::types::PyDict;
+
+struct RuntimeCache {
+    once: Once,
+    cache: Option<Py<PyDict>>
+}
+
+let mut cache = RuntimeCache {
+    once: Once::new(),
+    cache: None
+};
+
+Python::with_gil(|py| {
+    // guaranteed to be called once and only once
+    cache.once.call_once_py_attached(py, || {
+        cache.cache = Some(PyDict::new(py).unbind());
+    });
+});
+# }
+```
+
 ## `GILProtected` is not exposed
 
 `GILProtected` is a PyO3 type that allows mutable access to static data by

guide/src/migration.md

@@ -230,7 +230,12 @@ PyO3 0.23 introduces preliminary support for the new free-threaded build of
 CPython 3.13. PyO3 features that implicitly assumed the existence of the GIL are
 not exposed in the free-threaded build, since they are no longer safe.  Other
 features, such as `GILOnceCell`, have been internally rewritten to be threadsafe
-without the GIL.
+without the GIL, although note that `GILOnceCell` is inherently racey. You can
+also use `OnceExt::call_once_py_attached` or
+`OnceExt::call_once_force_py_attached` to enable use of `std::sync::Once` in
+code that has the GIL acquired without risking a dealock with the GIL. We plan
+We plan to expose more extension traits in the future that make it easier to
+write code for the GIL-enabled and free-threaded builds of Python.
 
 If you make use of these features then you will need to account for the
 unavailability of this API in the free-threaded build. One way to handle it is

newsfragments/4665.changed.md

@@ -0,0 +1,2 @@
+* The `sequential` and `string-sum` examples have moved into a new `examples`
+  directory in the `pyo3-ffi` crate.

newsfragments/4667.added.md

@@ -0,0 +1 @@
+Add `PyList_Extend` & `PyList_Clear` to pyo3-ffi

newsfragments/4674.fixed.md

@@ -0,0 +1 @@
+Fixes unintentional `unsafe_op_in_unsafe_fn` trigger by adjusting macro hygiene.

newsfragments/4676.added.md

@@ -0,0 +1 @@
+Add `pyo3::sync::OnceExt` and `pyo3::sync::OnceLockExt` traits.

noxfile.py

@@ -65,6 +65,8 @@ def test_py(session: nox.Session) -> None:
     _run(session, "nox", "-f", "pytests/noxfile.py", external=True)
     for example in glob("examples/*/noxfile.py"):
         _run(session, "nox", "-f", example, external=True)
+    for example in glob("pyo3-ffi/examples/*/noxfile.py"):
+        _run(session, "nox", "-f", example, external=True)
 
 
 @nox.session(venv_backend="none")

pyo3-build-config/src/lib.rs

@@ -138,6 +138,10 @@ fn resolve_cross_compile_config_path() -> Option<PathBuf> {
 pub fn print_feature_cfgs() {
     let rustc_minor_version = rustc_minor_version().unwrap_or(0);
 
+    if rustc_minor_version >= 70 {
+        println!("cargo:rustc-cfg=rustc_has_once_lock");
+    }
+
     // invalid_from_utf8 lint was added in Rust 1.74
     if rustc_minor_version >= 74 {
         println!("cargo:rustc-cfg=invalid_from_utf8_lint");
@@ -175,6 +179,7 @@ pub fn print_expected_cfgs() {
     println!("cargo:rustc-check-cfg=cfg(pyo3_leak_on_drop_without_reference_pool)");
     println!("cargo:rustc-check-cfg=cfg(diagnostic_namespace)");
     println!("cargo:rustc-check-cfg=cfg(c_str_lit)");
+    println!("cargo:rustc-check-cfg=cfg(rustc_has_once_lock)");
 
     // allow `Py_3_*` cfgs from the minimum supported version up to the
     // maximum minor version (+1 for development for the next)

pyo3-ffi/examples/README.md

@@ -0,0 +1,21 @@
+# `pyo3-ffi` Examples
+
+These example crates are a collection of toy extension modules built with
+`pyo3-ffi`. They are all tested using `nox` in PyO3's CI.
+
+Below is a brief description of each of these:
+
+| Example | Description |
+| `word-count` | Illustrates how to use pyo3-ffi to write a static rust extension |
+| `sequential` | Illustrates how to use pyo3-ffi to write subinterpreter-safe modules using multi-phase module initialization |
+
+## Creating new projects from these examples
+
+To copy an example, use [`cargo-generate`](https://crates.io/crates/cargo-generate). Follow the commands below, replacing `<example>` with the example to start from:
+
+```bash
+$ cargo install cargo-generate
+$ cargo generate --git https://github.com/PyO3/pyo3 examples/<example>
+```
+
+(`cargo generate` will take a little while to clone the PyO3 repo first; be patient when waiting for the command to run.)

examples/sequential/Cargo.toml → pyo3-ffi/examples/sequential/Cargo.toml

@@ -8,6 +8,6 @@ name = "sequential"
 crate-type = ["cdylib", "lib"]
 
 [dependencies]
-pyo3-ffi = { path = "../../pyo3-ffi", features = ["extension-module"] }
+pyo3-ffi = { path = "../../", features = ["extension-module"] }
 
 [workspace]

examples/string-sum/Cargo.toml → pyo3-ffi/examples/string-sum/Cargo.toml

@@ -8,6 +8,6 @@ name = "string_sum"
 crate-type = ["cdylib"]
 
 [dependencies]
-pyo3-ffi = { path = "../../pyo3-ffi", features = ["extension-module"] }
+pyo3-ffi = { path = "../../", features = ["extension-module"] }
 
 [workspace]

pyo3-ffi/src/compat/py_3_13.rs

@@ -83,3 +83,24 @@ compat_function!(
         1
     }
 );
+
+compat_function!(
+    originally_defined_for(Py_3_13);
+
+    #[inline]
+    pub unsafe fn PyList_Extend(
+        list: *mut crate::PyObject,
+        iterable: *mut crate::PyObject,
+    ) -> std::os::raw::c_int {
+        crate::PyList_SetSlice(list, crate::PY_SSIZE_T_MAX, crate::PY_SSIZE_T_MAX, iterable)
+    }
+);
+
+compat_function!(
+    originally_defined_for(Py_3_13);
+
+    #[inline]
+    pub unsafe fn PyList_Clear(list: *mut crate::PyObject) -> std::os::raw::c_int {
+        crate::PyList_SetSlice(list, 0, crate::PY_SSIZE_T_MAX, std::ptr::null_mut())
+    }
+);

pyo3-ffi/src/lib.rs

@@ -64,8 +64,8 @@
 //! your `Cargo.toml`:
 //!
 //! ```toml
-//! [build-dependency]
-//! pyo3-build-config = "VER"
+//! [build-dependencies]
+#![doc = concat!("pyo3-build-config =\"", env!("CARGO_PKG_VERSION"),  "\"")]
 //! ```
 //!
 //! And then either create a new `build.rs` file in the project root or modify
@@ -108,104 +108,31 @@
 //! [dependencies.pyo3-ffi]
 #![doc = concat!("version = \"", env!("CARGO_PKG_VERSION"),  "\"")]
 //! features = ["extension-module"]
+//!
+//! [build-dependencies]
+//! # This is only necessary if you need to configure your build based on
+//! # the Python version or the compile-time configuration for the interpreter.
+#![doc = concat!("pyo3_build_config = \"", env!("CARGO_PKG_VERSION"),  "\"")]
 //! ```
 //!
-//! **`src/lib.rs`**
-//! ```rust
-//! use std::os::raw::c_char;
-//! use std::ptr;
-//!
-//! use pyo3_ffi::*;
-//!
-//! static mut MODULE_DEF: PyModuleDef = PyModuleDef {
-//!     m_base: PyModuleDef_HEAD_INIT,
-//!     m_name: c_str!("string_sum").as_ptr(),
-//!     m_doc: c_str!("A Python module written in Rust.").as_ptr(),
-//!     m_size: 0,
-//!     m_methods: unsafe { METHODS.as_mut_ptr().cast() },
-//!     m_slots: std::ptr::null_mut(),
-//!     m_traverse: None,
-//!     m_clear: None,
-//!     m_free: None,
-//! };
-//!
-//! static mut METHODS: [PyMethodDef; 2] = [
-//!     PyMethodDef {
-//!         ml_name: c_str!("sum_as_string").as_ptr(),
-//!         ml_meth: PyMethodDefPointer {
-//!             PyCFunctionFast: sum_as_string,
-//!         },
-//!         ml_flags: METH_FASTCALL,
-//!         ml_doc: c_str!("returns the sum of two integers as a string").as_ptr(),
-//!     },
-//!     // A zeroed PyMethodDef to mark the end of the array.
-//!     PyMethodDef::zeroed()
-//! ];
-//!
-//! // The module initialization function, which must be named `PyInit_<your_module>`.
-//! #[allow(non_snake_case)]
-//! #[no_mangle]
-//! pub unsafe extern "C" fn PyInit_string_sum() -> *mut PyObject {
-//!     PyModule_Create(ptr::addr_of_mut!(MODULE_DEF))
-//! }
+//! If you need to use conditional compilation based on Python version or how
+//! Python was compiled, you need to add `pyo3-build-config` as a
+//! `build-dependency` in your `Cargo.toml` as in the example above and either
+//! create a new `build.rs` file or modify an existing one so that
+//! `pyo3_build_config::use_pyo3_cfgs()` gets called at build time:
 //!
-//! pub unsafe extern "C" fn sum_as_string(
-//!     _self: *mut PyObject,
-//!     args: *mut *mut PyObject,
-//!     nargs: Py_ssize_t,
-//! ) -> *mut PyObject {
-//!     if nargs != 2 {
-//!         PyErr_SetString(
-//!             PyExc_TypeError,
-//!             c_str!("sum_as_string() expected 2 positional arguments").as_ptr(),
-//!         );
-//!         return std::ptr::null_mut();
-//!     }
-//!
-//!     let arg1 = *args;
-//!     if PyLong_Check(arg1) == 0 {
-//!         PyErr_SetString(
-//!             PyExc_TypeError,
-//!             c_str!("sum_as_string() expected an int for positional argument 1").as_ptr(),
-//!         );
-//!         return std::ptr::null_mut();
-//!     }
-//!
-//!     let arg1 = PyLong_AsLong(arg1);
-//!     if !PyErr_Occurred().is_null() {
-//!         return ptr::null_mut();
-//!     }
-//!
-//!     let arg2 = *args.add(1);
-//!     if PyLong_Check(arg2) == 0 {
-//!         PyErr_SetString(
-//!             PyExc_TypeError,
-//!             c_str!("sum_as_string() expected an int for positional argument 2").as_ptr(),
-//!         );
-//!         return std::ptr::null_mut();
-//!     }
-//!
-//!     let arg2 = PyLong_AsLong(arg2);
-//!     if !PyErr_Occurred().is_null() {
-//!         return ptr::null_mut();
-//!     }
-//!
-//!     match arg1.checked_add(arg2) {
-//!         Some(sum) => {
-//!             let string = sum.to_string();
-//!             PyUnicode_FromStringAndSize(string.as_ptr().cast::<c_char>(), string.len() as isize)
-//!         }
-//!         None => {
-//!             PyErr_SetString(
-//!                 PyExc_OverflowError,
-//!                 c_str!("arguments too large to add").as_ptr(),
-//!             );
-//!             std::ptr::null_mut()
-//!         }
-//!     }
+//! **`build.rs`**
+//! ```rust,ignore
+//! fn main() {
+//!     pyo3_build_config::use_pyo3_cfgs()
 //! }
 //! ```
 //!
+//! **`src/lib.rs`**
+//! ```rust
+#![doc = include_str!("../examples/string-sum/src/lib.rs")]
+//! ```
+//!
 //! With those two files in place, now `maturin` needs to be installed. This can be done using
 //! Python's package manager `pip`. First, load up a new Python `virtualenv`, and install `maturin`
 //! into it:
@@ -230,6 +157,12 @@
 //! [manually][manual_builds]. Both offer more flexibility than `maturin` but require further
 //! configuration.
 //!
+//! This example stores the module definition statically and uses the `PyModule_Create` function
+//! in the CPython C API to register the module. This is the "old" style for registering modules
+//! and has the limitation that it cannot support subinterpreters. You can also create a module
+//! using the new multi-phase initialization API that does support subinterpreters. See the
+//! `sequential` project located in the `examples` directory at the root of the `pyo3-ffi` crate
+//! for a worked example of how to this using `pyo3-ffi`.
 //!
 //! # Using Python from Rust
 //!
@@ -255,7 +188,7 @@
 #![doc = concat!("[manual_builds]: https://pyo3.rs/v", env!("CARGO_PKG_VERSION"), "/building-and-distribution.html#manual-builds \"Manual builds - Building and Distribution - PyO3 user guide\"")]
 //! [setuptools-rust]: https://github.com/PyO3/setuptools-rust "Setuptools plugin for Rust extensions"
 //! [PEP 384]: https://www.python.org/dev/peps/pep-0384 "PEP 384 -- Defining a Stable ABI"
-#![doc = concat!("[Features chapter of the guide]: https://pyo3.rs/v", env!("CARGO_PKG_VERSION"), "/features.html#features-reference \"Features Reference - PyO3 user guide\"")]
+#![doc = concat!("[Features chapter of the guide]: https://pyo3.rs/v", env!("CARGO_PKG_VERSION"), "/features.html#features-reference \"Features eference - PyO3 user guide\"")]
 #![allow(
     missing_docs,
     non_camel_case_types,

pyo3-ffi/src/listobject.rs

@@ -50,6 +50,10 @@ extern "C" {
         arg3: Py_ssize_t,
         arg4: *mut PyObject,
     ) -> c_int;
+    #[cfg(Py_3_13)]
+    pub fn PyList_Extend(list: *mut PyObject, iterable: *mut PyObject) -> c_int;
+    #[cfg(Py_3_13)]
+    pub fn PyList_Clear(list: *mut PyObject) -> c_int;
     #[cfg_attr(PyPy, link_name = "PyPyList_Sort")]
     pub fn PyList_Sort(arg1: *mut PyObject) -> c_int;
     #[cfg_attr(PyPy, link_name = "PyPyList_Reverse")]