diff --git a/README.md b/README.md index 1094a489999..a131a823f0e 100644 --- a/README.md +++ b/README.md @@ -207,7 +207,7 @@ about this topic. - [hyperjson](https://github.com/mre/hyperjson) _A hyper-fast Python module for reading/writing JSON data using Rust's serde-json._ - [inline-python](https://github.com/fusion-engineering/inline-python) _Inline Python code directly in your Rust code._ - [johnnycanencrypt](https://github.com/kushaldas/johnnycanencrypt) OpenPGP library with Yubikey support. -- [jsonschema-rs](https://github.com/Stranger6667/jsonschema-rs/tree/master/bindings/python) _Fast JSON Schema validation library._ +- [jsonschema](https://github.com/Stranger6667/jsonschema/tree/master/crates/jsonschema-py) _A high-performance JSON Schema validator for Python._ - [mocpy](https://github.com/cds-astro/mocpy) _Astronomical Python library offering data structures for describing any arbitrary coverage regions on the unit sphere._ - [opendal](https://github.com/apache/opendal/tree/main/bindings/python) _A data access layer that allows users to easily and efficiently retrieve data from various storage services in a unified way._ - [orjson](https://github.com/ijl/orjson) _Fast Python JSON library._ diff --git a/examples/README.md b/examples/README.md index baaa57b650d..3c7cc301399 100644 --- a/examples/README.md +++ b/examples/README.md @@ -9,9 +9,11 @@ Below is a brief description of each of these: | `decorator` | A project showcasing the example from the [Emulating callable objects](https://pyo3.rs/latest/class/call.html) chapter of the guide. | | `maturin-starter` | A template project which is configured to use [`maturin`](https://github.com/PyO3/maturin) for development. | | `setuptools-rust-starter` | A template project which is configured to use [`setuptools_rust`](https://github.com/PyO3/setuptools-rust/) for development. | -| `word-count` | A quick performance comparison between word counter implementations written in each of Rust and Python. | | `plugin` | Illustrates how to use Python as a scripting language within a Rust application | -| `sequential` | Illustrates how to use pyo3-ffi to write subinterpreter-safe modules | + +Note that there are also other examples in the `pyo3-ffi/examples` +directory that illustrate how to create rust extensions using raw FFI calls into +the CPython C API instead of using PyO3's abstractions. ## Creating new projects from these examples diff --git a/guide/src/faq.md b/guide/src/faq.md index 5752e14adbd..83089cf395e 100644 --- a/guide/src/faq.md +++ b/guide/src/faq.md @@ -13,9 +13,11 @@ Sorry that you're having trouble using PyO3. If you can't find the answer to you 5. Thread A is blocked, because it waits to re-acquire the GIL which thread B still holds. 6. Deadlock. -PyO3 provides a struct [`GILOnceCell`] which works similarly to these types but avoids risk of deadlocking with the Python GIL. This means it can be used in place of other choices when you are experiencing the deadlock described above. See the documentation for [`GILOnceCell`] for further details and an example how to use it. +PyO3 provides a struct [`GILOnceCell`] which implements a single-initialization API based on these types that relies on the GIL for locking. If the GIL is released or there is no GIL, then this type allows the initialization function to race but ensures that the data is only ever initialized once. If you need to ensure that the initialization function is called once and only once, you can make use of the [`OnceExt`] and [`OnceLockExt`] extension traits that enable using the standard library types for this purpose but provide new methods for these types that avoid the risk of deadlocking with the Python GIL. This means they can be used in place of other choices when you are experiencing the deadlock described above. See the documentation for [`GILOnceCell`] and [`OnceExt`] for further details and an example how to use them. [`GILOnceCell`]: {{#PYO3_DOCS_URL}}/pyo3/sync/struct.GILOnceCell.html +[`OnceExt`]: {{#PYO3_DOCS_URL}}/pyo3/sync/trait.OnceExt.html +[`OnceLockExt`]: {{#PYO3_DOCS_URL}}/pyo3/sync/trait.OnceLockExt.html ## I can't run `cargo test`; or I can't build in a Cargo workspace: I'm having linker issues like "Symbol not found" or "Undefined reference to _PyExc_SystemError"! diff --git a/guide/src/free-threading.md b/guide/src/free-threading.md index 77b2ff327a2..8100a3d45ef 100644 --- a/guide/src/free-threading.md +++ b/guide/src/free-threading.md @@ -152,6 +152,60 @@ We plan to allow user-selectable semantics for mutable pyclass definitions in PyO3 0.24, allowing some form of opt-in locking to emulate the GIL if that is needed. +## Thread-safe single initialization + +Until version 0.23, PyO3 provided only `GILOnceCell` to enable deadlock-free +single initialization of data in contexts that might execute arbitrary Python +code. While we have updated `GILOnceCell` to avoid thread safety issues +triggered only under the free-threaded build, the design of `GILOnceCell` is +inherently thread-unsafe, in a manner that can be problematic even in the +GIL-enabled build. + +If, for example, the function executed by `GILOnceCell` releases the GIL or +calls code that releases the GIL, then it is possible for multiple threads to +try to race to initialize the cell. While the cell will only ever be intialized +once, it can be problematic in some contexts that `GILOnceCell` does not block +like the standard library `OnceLock`. + +In cases where the initialization function must run exactly once, you can bring +the `OnceExt` or `OnceLockExt` traits into scope. The `OnceExt` trait adds +`OnceExt::call_once_py_attached` and `OnceExt::call_once_force_py_attached` +functions to the api of `std::sync::Once`, enabling use of `Once` in contexts +where the GIL is held. Similarly, `OnceLockExt` adds +`OnceLockExt::get_or_init_py_attached`. These functions are analogous to +`Once::call_once`, `Once::call_once_force`, and `OnceLock::get_or_init` except +they accept a `Python<'py>` token in addition to an `FnOnce`. All of these +functions release the GIL and re-acquire it before executing the function, +avoiding deadlocks with the GIL that are possible without using the PyO3 +extension traits. Here is an example of how to use `OnceExt` to +enable single-initialization of a runtime cache holding a `Py`. + +```rust +# fn main() { +# use pyo3::prelude::*; +use std::sync::Once; +use pyo3::sync::OnceExt; +use pyo3::types::PyDict; + +struct RuntimeCache { + once: Once, + cache: Option> +} + +let mut cache = RuntimeCache { + once: Once::new(), + cache: None +}; + +Python::with_gil(|py| { + // guaranteed to be called once and only once + cache.once.call_once_py_attached(py, || { + cache.cache = Some(PyDict::new(py).unbind()); + }); +}); +# } +``` + ## `GILProtected` is not exposed `GILProtected` is a PyO3 type that allows mutable access to static data by diff --git a/guide/src/migration.md b/guide/src/migration.md index 0d76d220dc9..0f56498043b 100644 --- a/guide/src/migration.md +++ b/guide/src/migration.md @@ -230,7 +230,12 @@ PyO3 0.23 introduces preliminary support for the new free-threaded build of CPython 3.13. PyO3 features that implicitly assumed the existence of the GIL are not exposed in the free-threaded build, since they are no longer safe. Other features, such as `GILOnceCell`, have been internally rewritten to be threadsafe -without the GIL. +without the GIL, although note that `GILOnceCell` is inherently racey. You can +also use `OnceExt::call_once_py_attached` or +`OnceExt::call_once_force_py_attached` to enable use of `std::sync::Once` in +code that has the GIL acquired without risking a dealock with the GIL. We plan +We plan to expose more extension traits in the future that make it easier to +write code for the GIL-enabled and free-threaded builds of Python. If you make use of these features then you will need to account for the unavailability of this API in the free-threaded build. One way to handle it is diff --git a/newsfragments/4665.changed.md b/newsfragments/4665.changed.md new file mode 100644 index 00000000000..2ebbf0c86b4 --- /dev/null +++ b/newsfragments/4665.changed.md @@ -0,0 +1,2 @@ +* The `sequential` and `string-sum` examples have moved into a new `examples` + directory in the `pyo3-ffi` crate. diff --git a/newsfragments/4667.added.md b/newsfragments/4667.added.md new file mode 100644 index 00000000000..fc2a914607e --- /dev/null +++ b/newsfragments/4667.added.md @@ -0,0 +1 @@ +Add `PyList_Extend` & `PyList_Clear` to pyo3-ffi diff --git a/newsfragments/4674.fixed.md b/newsfragments/4674.fixed.md new file mode 100644 index 00000000000..6245a6f734a --- /dev/null +++ b/newsfragments/4674.fixed.md @@ -0,0 +1 @@ +Fixes unintentional `unsafe_op_in_unsafe_fn` trigger by adjusting macro hygiene. \ No newline at end of file diff --git a/newsfragments/4676.added.md b/newsfragments/4676.added.md new file mode 100644 index 00000000000..730b2297d91 --- /dev/null +++ b/newsfragments/4676.added.md @@ -0,0 +1 @@ +Add `pyo3::sync::OnceExt` and `pyo3::sync::OnceLockExt` traits. diff --git a/noxfile.py b/noxfile.py index ce59162f120..32176240f59 100644 --- a/noxfile.py +++ b/noxfile.py @@ -65,6 +65,8 @@ def test_py(session: nox.Session) -> None: _run(session, "nox", "-f", "pytests/noxfile.py", external=True) for example in glob("examples/*/noxfile.py"): _run(session, "nox", "-f", example, external=True) + for example in glob("pyo3-ffi/examples/*/noxfile.py"): + _run(session, "nox", "-f", example, external=True) @nox.session(venv_backend="none") diff --git a/pyo3-build-config/src/lib.rs b/pyo3-build-config/src/lib.rs index 033e7b46540..642fdf1659f 100644 --- a/pyo3-build-config/src/lib.rs +++ b/pyo3-build-config/src/lib.rs @@ -138,6 +138,10 @@ fn resolve_cross_compile_config_path() -> Option { pub fn print_feature_cfgs() { let rustc_minor_version = rustc_minor_version().unwrap_or(0); + if rustc_minor_version >= 70 { + println!("cargo:rustc-cfg=rustc_has_once_lock"); + } + // invalid_from_utf8 lint was added in Rust 1.74 if rustc_minor_version >= 74 { println!("cargo:rustc-cfg=invalid_from_utf8_lint"); @@ -175,6 +179,7 @@ pub fn print_expected_cfgs() { println!("cargo:rustc-check-cfg=cfg(pyo3_leak_on_drop_without_reference_pool)"); println!("cargo:rustc-check-cfg=cfg(diagnostic_namespace)"); println!("cargo:rustc-check-cfg=cfg(c_str_lit)"); + println!("cargo:rustc-check-cfg=cfg(rustc_has_once_lock)"); // allow `Py_3_*` cfgs from the minimum supported version up to the // maximum minor version (+1 for development for the next) diff --git a/pyo3-ffi/examples/README.md b/pyo3-ffi/examples/README.md new file mode 100644 index 00000000000..f02ae4ba6b4 --- /dev/null +++ b/pyo3-ffi/examples/README.md @@ -0,0 +1,21 @@ +# `pyo3-ffi` Examples + +These example crates are a collection of toy extension modules built with +`pyo3-ffi`. They are all tested using `nox` in PyO3's CI. + +Below is a brief description of each of these: + +| Example | Description | +| `word-count` | Illustrates how to use pyo3-ffi to write a static rust extension | +| `sequential` | Illustrates how to use pyo3-ffi to write subinterpreter-safe modules using multi-phase module initialization | + +## Creating new projects from these examples + +To copy an example, use [`cargo-generate`](https://crates.io/crates/cargo-generate). Follow the commands below, replacing `` with the example to start from: + +```bash +$ cargo install cargo-generate +$ cargo generate --git https://github.com/PyO3/pyo3 examples/ +``` + +(`cargo generate` will take a little while to clone the PyO3 repo first; be patient when waiting for the command to run.) diff --git a/examples/sequential/.template/Cargo.toml b/pyo3-ffi/examples/sequential/.template/Cargo.toml similarity index 100% rename from examples/sequential/.template/Cargo.toml rename to pyo3-ffi/examples/sequential/.template/Cargo.toml diff --git a/examples/sequential/.template/pre-script.rhai b/pyo3-ffi/examples/sequential/.template/pre-script.rhai similarity index 100% rename from examples/sequential/.template/pre-script.rhai rename to pyo3-ffi/examples/sequential/.template/pre-script.rhai diff --git a/examples/sequential/.template/pyproject.toml b/pyo3-ffi/examples/sequential/.template/pyproject.toml similarity index 100% rename from examples/sequential/.template/pyproject.toml rename to pyo3-ffi/examples/sequential/.template/pyproject.toml diff --git a/examples/sequential/Cargo.toml b/pyo3-ffi/examples/sequential/Cargo.toml similarity index 67% rename from examples/sequential/Cargo.toml rename to pyo3-ffi/examples/sequential/Cargo.toml index 4500c69b597..3348595b4e9 100644 --- a/examples/sequential/Cargo.toml +++ b/pyo3-ffi/examples/sequential/Cargo.toml @@ -8,6 +8,6 @@ name = "sequential" crate-type = ["cdylib", "lib"] [dependencies] -pyo3-ffi = { path = "../../pyo3-ffi", features = ["extension-module"] } +pyo3-ffi = { path = "../../", features = ["extension-module"] } [workspace] diff --git a/examples/sequential/MANIFEST.in b/pyo3-ffi/examples/sequential/MANIFEST.in similarity index 100% rename from examples/sequential/MANIFEST.in rename to pyo3-ffi/examples/sequential/MANIFEST.in diff --git a/examples/sequential/README.md b/pyo3-ffi/examples/sequential/README.md similarity index 100% rename from examples/sequential/README.md rename to pyo3-ffi/examples/sequential/README.md diff --git a/examples/sequential/cargo-generate.toml b/pyo3-ffi/examples/sequential/cargo-generate.toml similarity index 100% rename from examples/sequential/cargo-generate.toml rename to pyo3-ffi/examples/sequential/cargo-generate.toml diff --git a/examples/sequential/noxfile.py b/pyo3-ffi/examples/sequential/noxfile.py similarity index 100% rename from examples/sequential/noxfile.py rename to pyo3-ffi/examples/sequential/noxfile.py diff --git a/examples/sequential/pyproject.toml b/pyo3-ffi/examples/sequential/pyproject.toml similarity index 100% rename from examples/sequential/pyproject.toml rename to pyo3-ffi/examples/sequential/pyproject.toml diff --git a/examples/sequential/src/id.rs b/pyo3-ffi/examples/sequential/src/id.rs similarity index 100% rename from examples/sequential/src/id.rs rename to pyo3-ffi/examples/sequential/src/id.rs diff --git a/examples/sequential/src/lib.rs b/pyo3-ffi/examples/sequential/src/lib.rs similarity index 100% rename from examples/sequential/src/lib.rs rename to pyo3-ffi/examples/sequential/src/lib.rs diff --git a/examples/sequential/src/module.rs b/pyo3-ffi/examples/sequential/src/module.rs similarity index 100% rename from examples/sequential/src/module.rs rename to pyo3-ffi/examples/sequential/src/module.rs diff --git a/examples/sequential/tests/test.rs b/pyo3-ffi/examples/sequential/tests/test.rs similarity index 100% rename from examples/sequential/tests/test.rs rename to pyo3-ffi/examples/sequential/tests/test.rs diff --git a/examples/sequential/tests/test_.py b/pyo3-ffi/examples/sequential/tests/test_.py similarity index 100% rename from examples/sequential/tests/test_.py rename to pyo3-ffi/examples/sequential/tests/test_.py diff --git a/examples/string-sum/.template/Cargo.toml b/pyo3-ffi/examples/string-sum/.template/Cargo.toml similarity index 100% rename from examples/string-sum/.template/Cargo.toml rename to pyo3-ffi/examples/string-sum/.template/Cargo.toml diff --git a/examples/string-sum/.template/pre-script.rhai b/pyo3-ffi/examples/string-sum/.template/pre-script.rhai similarity index 100% rename from examples/string-sum/.template/pre-script.rhai rename to pyo3-ffi/examples/string-sum/.template/pre-script.rhai diff --git a/examples/string-sum/.template/pyproject.toml b/pyo3-ffi/examples/string-sum/.template/pyproject.toml similarity index 100% rename from examples/string-sum/.template/pyproject.toml rename to pyo3-ffi/examples/string-sum/.template/pyproject.toml diff --git a/examples/string-sum/Cargo.toml b/pyo3-ffi/examples/string-sum/Cargo.toml similarity index 66% rename from examples/string-sum/Cargo.toml rename to pyo3-ffi/examples/string-sum/Cargo.toml index 4a48b221c60..6fb72141cdc 100644 --- a/examples/string-sum/Cargo.toml +++ b/pyo3-ffi/examples/string-sum/Cargo.toml @@ -8,6 +8,6 @@ name = "string_sum" crate-type = ["cdylib"] [dependencies] -pyo3-ffi = { path = "../../pyo3-ffi", features = ["extension-module"] } +pyo3-ffi = { path = "../../", features = ["extension-module"] } [workspace] diff --git a/examples/string-sum/MANIFEST.in b/pyo3-ffi/examples/string-sum/MANIFEST.in similarity index 100% rename from examples/string-sum/MANIFEST.in rename to pyo3-ffi/examples/string-sum/MANIFEST.in diff --git a/examples/string-sum/README.md b/pyo3-ffi/examples/string-sum/README.md similarity index 100% rename from examples/string-sum/README.md rename to pyo3-ffi/examples/string-sum/README.md diff --git a/examples/string-sum/cargo-generate.toml b/pyo3-ffi/examples/string-sum/cargo-generate.toml similarity index 100% rename from examples/string-sum/cargo-generate.toml rename to pyo3-ffi/examples/string-sum/cargo-generate.toml diff --git a/examples/string-sum/noxfile.py b/pyo3-ffi/examples/string-sum/noxfile.py similarity index 100% rename from examples/string-sum/noxfile.py rename to pyo3-ffi/examples/string-sum/noxfile.py diff --git a/examples/string-sum/pyproject.toml b/pyo3-ffi/examples/string-sum/pyproject.toml similarity index 100% rename from examples/string-sum/pyproject.toml rename to pyo3-ffi/examples/string-sum/pyproject.toml diff --git a/examples/string-sum/src/lib.rs b/pyo3-ffi/examples/string-sum/src/lib.rs similarity index 100% rename from examples/string-sum/src/lib.rs rename to pyo3-ffi/examples/string-sum/src/lib.rs diff --git a/examples/string-sum/tests/test_.py b/pyo3-ffi/examples/string-sum/tests/test_.py similarity index 100% rename from examples/string-sum/tests/test_.py rename to pyo3-ffi/examples/string-sum/tests/test_.py diff --git a/pyo3-ffi/src/compat/py_3_13.rs b/pyo3-ffi/src/compat/py_3_13.rs index 9f44ced6f3f..59289cb76ae 100644 --- a/pyo3-ffi/src/compat/py_3_13.rs +++ b/pyo3-ffi/src/compat/py_3_13.rs @@ -83,3 +83,24 @@ compat_function!( 1 } ); + +compat_function!( + originally_defined_for(Py_3_13); + + #[inline] + pub unsafe fn PyList_Extend( + list: *mut crate::PyObject, + iterable: *mut crate::PyObject, + ) -> std::os::raw::c_int { + crate::PyList_SetSlice(list, crate::PY_SSIZE_T_MAX, crate::PY_SSIZE_T_MAX, iterable) + } +); + +compat_function!( + originally_defined_for(Py_3_13); + + #[inline] + pub unsafe fn PyList_Clear(list: *mut crate::PyObject) -> std::os::raw::c_int { + crate::PyList_SetSlice(list, 0, crate::PY_SSIZE_T_MAX, std::ptr::null_mut()) + } +); diff --git a/pyo3-ffi/src/lib.rs b/pyo3-ffi/src/lib.rs index 293c5171eb5..23a5e0000b1 100644 --- a/pyo3-ffi/src/lib.rs +++ b/pyo3-ffi/src/lib.rs @@ -64,8 +64,8 @@ //! your `Cargo.toml`: //! //! ```toml -//! [build-dependency] -//! pyo3-build-config = "VER" +//! [build-dependencies] +#![doc = concat!("pyo3-build-config =\"", env!("CARGO_PKG_VERSION"), "\"")] //! ``` //! //! And then either create a new `build.rs` file in the project root or modify @@ -108,104 +108,31 @@ //! [dependencies.pyo3-ffi] #![doc = concat!("version = \"", env!("CARGO_PKG_VERSION"), "\"")] //! features = ["extension-module"] +//! +//! [build-dependencies] +//! # This is only necessary if you need to configure your build based on +//! # the Python version or the compile-time configuration for the interpreter. +#![doc = concat!("pyo3_build_config = \"", env!("CARGO_PKG_VERSION"), "\"")] //! ``` //! -//! **`src/lib.rs`** -//! ```rust -//! use std::os::raw::c_char; -//! use std::ptr; -//! -//! use pyo3_ffi::*; -//! -//! static mut MODULE_DEF: PyModuleDef = PyModuleDef { -//! m_base: PyModuleDef_HEAD_INIT, -//! m_name: c_str!("string_sum").as_ptr(), -//! m_doc: c_str!("A Python module written in Rust.").as_ptr(), -//! m_size: 0, -//! m_methods: unsafe { METHODS.as_mut_ptr().cast() }, -//! m_slots: std::ptr::null_mut(), -//! m_traverse: None, -//! m_clear: None, -//! m_free: None, -//! }; -//! -//! static mut METHODS: [PyMethodDef; 2] = [ -//! PyMethodDef { -//! ml_name: c_str!("sum_as_string").as_ptr(), -//! ml_meth: PyMethodDefPointer { -//! PyCFunctionFast: sum_as_string, -//! }, -//! ml_flags: METH_FASTCALL, -//! ml_doc: c_str!("returns the sum of two integers as a string").as_ptr(), -//! }, -//! // A zeroed PyMethodDef to mark the end of the array. -//! PyMethodDef::zeroed() -//! ]; -//! -//! // The module initialization function, which must be named `PyInit_`. -//! #[allow(non_snake_case)] -//! #[no_mangle] -//! pub unsafe extern "C" fn PyInit_string_sum() -> *mut PyObject { -//! PyModule_Create(ptr::addr_of_mut!(MODULE_DEF)) -//! } +//! If you need to use conditional compilation based on Python version or how +//! Python was compiled, you need to add `pyo3-build-config` as a +//! `build-dependency` in your `Cargo.toml` as in the example above and either +//! create a new `build.rs` file or modify an existing one so that +//! `pyo3_build_config::use_pyo3_cfgs()` gets called at build time: //! -//! pub unsafe extern "C" fn sum_as_string( -//! _self: *mut PyObject, -//! args: *mut *mut PyObject, -//! nargs: Py_ssize_t, -//! ) -> *mut PyObject { -//! if nargs != 2 { -//! PyErr_SetString( -//! PyExc_TypeError, -//! c_str!("sum_as_string() expected 2 positional arguments").as_ptr(), -//! ); -//! return std::ptr::null_mut(); -//! } -//! -//! let arg1 = *args; -//! if PyLong_Check(arg1) == 0 { -//! PyErr_SetString( -//! PyExc_TypeError, -//! c_str!("sum_as_string() expected an int for positional argument 1").as_ptr(), -//! ); -//! return std::ptr::null_mut(); -//! } -//! -//! let arg1 = PyLong_AsLong(arg1); -//! if !PyErr_Occurred().is_null() { -//! return ptr::null_mut(); -//! } -//! -//! let arg2 = *args.add(1); -//! if PyLong_Check(arg2) == 0 { -//! PyErr_SetString( -//! PyExc_TypeError, -//! c_str!("sum_as_string() expected an int for positional argument 2").as_ptr(), -//! ); -//! return std::ptr::null_mut(); -//! } -//! -//! let arg2 = PyLong_AsLong(arg2); -//! if !PyErr_Occurred().is_null() { -//! return ptr::null_mut(); -//! } -//! -//! match arg1.checked_add(arg2) { -//! Some(sum) => { -//! let string = sum.to_string(); -//! PyUnicode_FromStringAndSize(string.as_ptr().cast::(), string.len() as isize) -//! } -//! None => { -//! PyErr_SetString( -//! PyExc_OverflowError, -//! c_str!("arguments too large to add").as_ptr(), -//! ); -//! std::ptr::null_mut() -//! } -//! } +//! **`build.rs`** +//! ```rust,ignore +//! fn main() { +//! pyo3_build_config::use_pyo3_cfgs() //! } //! ``` //! +//! **`src/lib.rs`** +//! ```rust +#![doc = include_str!("../examples/string-sum/src/lib.rs")] +//! ``` +//! //! With those two files in place, now `maturin` needs to be installed. This can be done using //! Python's package manager `pip`. First, load up a new Python `virtualenv`, and install `maturin` //! into it: @@ -230,6 +157,12 @@ //! [manually][manual_builds]. Both offer more flexibility than `maturin` but require further //! configuration. //! +//! This example stores the module definition statically and uses the `PyModule_Create` function +//! in the CPython C API to register the module. This is the "old" style for registering modules +//! and has the limitation that it cannot support subinterpreters. You can also create a module +//! using the new multi-phase initialization API that does support subinterpreters. See the +//! `sequential` project located in the `examples` directory at the root of the `pyo3-ffi` crate +//! for a worked example of how to this using `pyo3-ffi`. //! //! # Using Python from Rust //! @@ -255,7 +188,7 @@ #![doc = concat!("[manual_builds]: https://pyo3.rs/v", env!("CARGO_PKG_VERSION"), "/building-and-distribution.html#manual-builds \"Manual builds - Building and Distribution - PyO3 user guide\"")] //! [setuptools-rust]: https://github.com/PyO3/setuptools-rust "Setuptools plugin for Rust extensions" //! [PEP 384]: https://www.python.org/dev/peps/pep-0384 "PEP 384 -- Defining a Stable ABI" -#![doc = concat!("[Features chapter of the guide]: https://pyo3.rs/v", env!("CARGO_PKG_VERSION"), "/features.html#features-reference \"Features Reference - PyO3 user guide\"")] +#![doc = concat!("[Features chapter of the guide]: https://pyo3.rs/v", env!("CARGO_PKG_VERSION"), "/features.html#features-reference \"Features eference - PyO3 user guide\"")] #![allow( missing_docs, non_camel_case_types, diff --git a/pyo3-ffi/src/listobject.rs b/pyo3-ffi/src/listobject.rs index 9d8b7ed6a58..881a8a8707b 100644 --- a/pyo3-ffi/src/listobject.rs +++ b/pyo3-ffi/src/listobject.rs @@ -50,6 +50,10 @@ extern "C" { arg3: Py_ssize_t, arg4: *mut PyObject, ) -> c_int; + #[cfg(Py_3_13)] + pub fn PyList_Extend(list: *mut PyObject, iterable: *mut PyObject) -> c_int; + #[cfg(Py_3_13)] + pub fn PyList_Clear(list: *mut PyObject) -> c_int; #[cfg_attr(PyPy, link_name = "PyPyList_Sort")] pub fn PyList_Sort(arg1: *mut PyObject) -> c_int; #[cfg_attr(PyPy, link_name = "PyPyList_Reverse")] diff --git a/pyo3-macros-backend/src/method.rs b/pyo3-macros-backend/src/method.rs index 019fb5e644b..f99e64562b7 100644 --- a/pyo3-macros-backend/src/method.rs +++ b/pyo3-macros-backend/src/method.rs @@ -270,9 +270,9 @@ impl FnType { ::std::convert::Into::into( #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(#py, &*(&#slf as *const _ as *const *mut _)) .downcast_unchecked::<#pyo3_path::types::PyType>() - ), + ) }; - Some(ret) + Some(quote! { unsafe { #ret }, }) } FnType::FnModule(span) => { let py = syn::Ident::new("py", Span::call_site()); @@ -283,9 +283,9 @@ impl FnType { ::std::convert::Into::into( #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(#py, &*(&#slf as *const _ as *const *mut _)) .downcast_unchecked::<#pyo3_path::types::PyModule>() - ), + ) }; - Some(ret) + Some(quote! { unsafe { #ret }, }) } FnType::FnNew | FnType::FnStatic | FnType::ClassAttribute => None, } @@ -332,6 +332,8 @@ impl SelfType { let py = syn::Ident::new("py", Span::call_site()); let slf = syn::Ident::new("_slf", Span::call_site()); let Ctx { pyo3_path, .. } = ctx; + let bound_ref = + quote! { unsafe { #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(#py, &#slf) } }; match self { SelfType::Receiver { span, mutable } => { let method = if *mutable { @@ -344,7 +346,7 @@ impl SelfType { error_mode.handle_error( quote_spanned! { *span => #pyo3_path::impl_::extract_argument::#method::<#cls>( - #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(#py, &#slf).0, + #bound_ref.0, &mut #holder, ) }, @@ -355,7 +357,7 @@ impl SelfType { let pyo3_path = pyo3_path.to_tokens_spanned(*span); error_mode.handle_error( quote_spanned! { *span => - #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(#py, &#slf).downcast::<#cls>() + #bound_ref.downcast::<#cls>() .map_err(::std::convert::Into::<#pyo3_path::PyErr>::into) .and_then( #[allow(unknown_lints, clippy::unnecessary_fallible_conversions)] // In case slf is Py (unknown_lints can be removed when MSRV is 1.75+) @@ -665,14 +667,14 @@ impl<'a> FnSpec<'a> { FnType::Fn(SelfType::Receiver { mutable: false, .. }) => { quote! {{ #(let #arg_names = #args;)* - let __guard = #pyo3_path::impl_::coroutine::RefGuard::<#cls>::new(&#pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &_slf))?; + let __guard = unsafe { #pyo3_path::impl_::coroutine::RefGuard::<#cls>::new(&#pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &_slf))? }; async move { function(&__guard, #(#arg_names),*).await } }} } FnType::Fn(SelfType::Receiver { mutable: true, .. }) => { quote! {{ #(let #arg_names = #args;)* - let mut __guard = #pyo3_path::impl_::coroutine::RefMutGuard::<#cls>::new(&#pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &_slf))?; + let mut __guard = unsafe { #pyo3_path::impl_::coroutine::RefMutGuard::<#cls>::new(&#pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &_slf))? }; async move { function(&mut __guard, #(#arg_names),*).await } }} } @@ -862,11 +864,13 @@ impl<'a> FnSpec<'a> { _args: *mut #pyo3_path::ffi::PyObject, ) -> *mut #pyo3_path::ffi::PyObject { - #pyo3_path::impl_::trampoline::noargs( - _slf, - _args, - #wrapper - ) + unsafe { + #pyo3_path::impl_::trampoline::noargs( + _slf, + _args, + #wrapper + ) + } } trampoline }, diff --git a/pyo3-macros-backend/src/module.rs b/pyo3-macros-backend/src/module.rs index 7d2c72dbdfb..5aaf7740461 100644 --- a/pyo3-macros-backend/src/module.rs +++ b/pyo3-macros-backend/src/module.rs @@ -449,7 +449,7 @@ fn module_initialization( #[doc(hidden)] #[export_name = #pyinit_symbol] pub unsafe extern "C" fn __pyo3_init() -> *mut #pyo3_path::ffi::PyObject { - #pyo3_path::impl_::trampoline::module_init(|py| _PYO3_DEF.make_module(py)) + unsafe { #pyo3_path::impl_::trampoline::module_init(|py| _PYO3_DEF.make_module(py)) } } }); } diff --git a/pyo3-macros-backend/src/params.rs b/pyo3-macros-backend/src/params.rs index ccf725d3760..67054458c98 100644 --- a/pyo3-macros-backend/src/params.rs +++ b/pyo3-macros-backend/src/params.rs @@ -79,7 +79,7 @@ pub fn impl_arg_params( .collect(); return ( quote! { - let _args = #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &_args); + let _args = unsafe { #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &_args) }; let _kwargs = #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr_or_opt(py, &_kwargs); #from_py_with }, @@ -301,9 +301,10 @@ pub(crate) fn impl_regular_arg_param( } } else { let holder = holders.push_holder(arg.name.span()); + let unwrap = quote! {unsafe { #pyo3_path::impl_::extract_argument::unwrap_required_argument(#arg_value) }}; quote_arg_span! { #pyo3_path::impl_::extract_argument::extract_argument( - #pyo3_path::impl_::extract_argument::unwrap_required_argument(#arg_value), + #unwrap, &mut #holder, #name_str )? diff --git a/pyo3-macros-backend/src/pymethod.rs b/pyo3-macros-backend/src/pymethod.rs index d825609cd77..1254a8d510b 100644 --- a/pyo3-macros-backend/src/pymethod.rs +++ b/pyo3-macros-backend/src/pymethod.rs @@ -1182,25 +1182,26 @@ fn extract_object( let Ctx { pyo3_path, .. } = ctx; let name = arg.name().unraw().to_string(); - let extract = - if let Some(from_py_with) = arg.from_py_with().map(|from_py_with| &from_py_with.value) { - quote! { - #pyo3_path::impl_::extract_argument::from_py_with( - #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &#source_ptr).0, - #name, - #from_py_with as fn(_) -> _, - ) - } - } else { - let holder = holders.push_holder(Span::call_site()); - quote! { - #pyo3_path::impl_::extract_argument::extract_argument( - #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &#source_ptr).0, - &mut #holder, - #name - ) - } - }; + let extract = if let Some(from_py_with) = + arg.from_py_with().map(|from_py_with| &from_py_with.value) + { + quote! { + #pyo3_path::impl_::extract_argument::from_py_with( + unsafe { #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &#source_ptr).0 }, + #name, + #from_py_with as fn(_) -> _, + ) + } + } else { + let holder = holders.push_holder(Span::call_site()); + quote! { + #pyo3_path::impl_::extract_argument::extract_argument( + unsafe { #pyo3_path::impl_::pymethods::BoundRef::ref_from_ptr(py, &#source_ptr).0 }, + &mut #holder, + #name + ) + } + }; let extracted = extract_error_mode.handle_error(extract, ctx); quote!(#extracted) diff --git a/src/lib.rs b/src/lib.rs index b4fcf918fae..25c88143609 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -123,14 +123,25 @@ //! - `nightly`: Uses `#![feature(auto_traits, negative_impls)]` to define [`Ungil`] as an auto trait. // //! ## `rustc` environment flags -//! -//! PyO3 uses `rustc`'s `--cfg` flags to enable or disable code used for different Python versions. -//! If you want to do this for your own crate, you can do so with the [`pyo3-build-config`] crate. -//! -//! - `Py_3_7`, `Py_3_8`, `Py_3_9`, `Py_3_10`: Marks code that is only enabled when -//! compiling for a given minimum Python version. +//! - `Py_3_7`, `Py_3_8`, `Py_3_9`, `Py_3_10`, `Py_3_11`, `Py_3_12`, `Py_3_13`: Marks code that is +//! only enabled when compiling for a given minimum Python version. //! - `Py_LIMITED_API`: Marks code enabled when the `abi3` feature flag is enabled. +//! - `Py_GIL_DISABLED`: Marks code that runs only in the free-threaded build of CPython. //! - `PyPy` - Marks code enabled when compiling for PyPy. +//! - `GraalPy` - Marks code enabled when compiling for GraalPy. +//! +//! Additionally, you can query for the values `Py_DEBUG`, `Py_REF_DEBUG`, +//! `Py_TRACE_REFS`, and `COUNT_ALLOCS` from `py_sys_config` to query for the +//! corresponding C build-time defines. For example, to conditionally define +//! debug code using `Py_DEBUG`, you could do: +//! +//! ```rust,ignore +//! #[cfg(py_sys_config = "Py_DEBUG")] +//! println!("only runs if python was compiled with Py_DEBUG") +//! ``` +//! To use these attributes, add [`pyo3-build-config`] as a build dependency in +//! your `Cargo.toml` and call `pyo3_build_config::use_pyo3_cfgs()` in a +//! `build.rs` file. //! //! # Minimum supported Rust and Python versions //! diff --git a/src/sealed.rs b/src/sealed.rs index cc835bee3b8..0a2846b134a 100644 --- a/src/sealed.rs +++ b/src/sealed.rs @@ -53,3 +53,5 @@ impl Sealed for ModuleDef {} impl Sealed for PyNativeTypeInitializer {} impl Sealed for PyClassInitializer {} + +impl Sealed for std::sync::Once {} diff --git a/src/sync.rs b/src/sync.rs index 65a81d06bd5..0845eaf8cec 100644 --- a/src/sync.rs +++ b/src/sync.rs @@ -5,10 +5,17 @@ //! //! [PEP 703]: https://peps.python.org/pep-703/ use crate::{ + ffi, + sealed::Sealed, types::{any::PyAnyMethods, PyAny, PyString}, Bound, Py, PyResult, PyTypeCheck, Python, }; -use std::{cell::UnsafeCell, marker::PhantomData, mem::MaybeUninit, sync::Once}; +use std::{ + cell::UnsafeCell, + marker::PhantomData, + mem::MaybeUninit, + sync::{Once, OnceState}, +}; #[cfg(not(Py_GIL_DISABLED))] use crate::PyVisit; @@ -473,6 +480,139 @@ where } } +#[cfg(rustc_has_once_lock)] +mod once_lock_ext_sealed { + pub trait Sealed {} + impl Sealed for std::sync::OnceLock {} +} + +/// Helper trait for `Once` to help avoid deadlocking when using a `Once` when attached to a +/// Python thread. +pub trait OnceExt: Sealed { + /// Similar to [`call_once`][Once::call_once], but releases the Python GIL temporarily + /// if blocking on another thread currently calling this `Once`. + fn call_once_py_attached(&self, py: Python<'_>, f: impl FnOnce()); + + /// Similar to [`call_once_force`][Once::call_once_force], but releases the Python GIL + /// temporarily if blocking on another thread currently calling this `Once`. + fn call_once_force_py_attached(&self, py: Python<'_>, f: impl FnOnce(&OnceState)); +} + +// Extension trait for [`std::sync::OnceLock`] which helps avoid deadlocks between the Python +/// interpreter and initialization with the `OnceLock`. +#[cfg(rustc_has_once_lock)] +pub trait OnceLockExt: once_lock_ext_sealed::Sealed { + /// Initializes this `OnceLock` with the given closure if it has not been initialized yet. + /// + /// If this function would block, this function detaches from the Python interpreter and + /// reattaches before calling `f`. This avoids deadlocks between the Python interpreter and + /// the `OnceLock` in cases where `f` can call arbitrary Python code, as calling arbitrary + /// Python code can lead to `f` itself blocking on the Python interpreter. + /// + /// By detaching from the Python interpreter before blocking, this ensures that if `f` blocks + /// then the Python interpreter cannot be blocked by `f` itself. + fn get_or_init_py_attached(&self, py: Python<'_>, f: F) -> &T + where + F: FnOnce() -> T; +} + +struct Guard(*mut crate::ffi::PyThreadState); + +impl Drop for Guard { + fn drop(&mut self) { + unsafe { ffi::PyEval_RestoreThread(self.0) }; + } +} + +impl OnceExt for Once { + fn call_once_py_attached(&self, py: Python<'_>, f: impl FnOnce()) { + if self.is_completed() { + return; + } + + init_once_py_attached(self, py, f) + } + + fn call_once_force_py_attached(&self, py: Python<'_>, f: impl FnOnce(&OnceState)) { + if self.is_completed() { + return; + } + + init_once_force_py_attached(self, py, f); + } +} + +#[cfg(rustc_has_once_lock)] +impl OnceLockExt for std::sync::OnceLock { + fn get_or_init_py_attached(&self, py: Python<'_>, f: F) -> &T + where + F: FnOnce() -> T, + { + // this trait is guarded by a rustc version config + // so clippy's MSRV check is wrong + #[allow(clippy::incompatible_msrv)] + // Use self.get() first to create a fast path when initialized + self.get() + .unwrap_or_else(|| init_once_lock_py_attached(self, py, f)) + } +} + +#[cold] +fn init_once_py_attached(once: &Once, _py: Python<'_>, f: F) +where + F: FnOnce() -> T, +{ + // Safety: we are currently attached to the GIL, and we expect to block. We will save + // the current thread state and restore it as soon as we are done blocking. + let ts_guard = Guard(unsafe { ffi::PyEval_SaveThread() }); + + once.call_once(move || { + drop(ts_guard); + f(); + }); +} + +#[cold] +fn init_once_force_py_attached(once: &Once, _py: Python<'_>, f: F) +where + F: FnOnce(&OnceState) -> T, +{ + // Safety: we are currently attached to the GIL, and we expect to block. We will save + // the current thread state and restore it as soon as we are done blocking. + let ts_guard = Guard(unsafe { ffi::PyEval_SaveThread() }); + + once.call_once_force(move |state| { + drop(ts_guard); + f(state); + }); +} + +#[cfg(rustc_has_once_lock)] +#[cold] +fn init_once_lock_py_attached<'a, F, T>( + lock: &'a std::sync::OnceLock, + _py: Python<'_>, + f: F, +) -> &'a T +where + F: FnOnce() -> T, +{ + // SAFETY: we are currently attached to a Python thread + let ts_guard = Guard(unsafe { ffi::PyEval_SaveThread() }); + + // this trait is guarded by a rustc version config + // so clippy's MSRV check is wrong + #[allow(clippy::incompatible_msrv)] + // By having detached here, we guarantee that `.get_or_init` cannot deadlock with + // the Python interpreter + let value = lock.get_or_init(move || { + drop(ts_guard); + f() + }); + + value +} + #[cfg(test)] mod tests { use super::*; @@ -589,4 +729,56 @@ mod tests { }); }); } + + #[test] + #[cfg(not(target_arch = "wasm32"))] // We are building wasm Python with pthreads disabled + fn test_once_ext() { + // adapted from the example in the docs for Once::try_once_force + let init = Once::new(); + std::thread::scope(|s| { + // poison the once + let handle = s.spawn(|| { + Python::with_gil(|py| { + init.call_once_py_attached(py, || panic!()); + }) + }); + assert!(handle.join().is_err()); + + // poisoning propagates + let handle = s.spawn(|| { + Python::with_gil(|py| { + init.call_once_py_attached(py, || {}); + }); + }); + + assert!(handle.join().is_err()); + + // call_once_force will still run and reset the poisoned state + Python::with_gil(|py| { + init.call_once_force_py_attached(py, |state| { + assert!(state.is_poisoned()); + }); + + // once any success happens, we stop propagating the poison + init.call_once_py_attached(py, || {}); + }); + }); + } + + #[cfg(rustc_has_once_lock)] + #[cfg(not(target_arch = "wasm32"))] // We are building wasm Python with pthreads disabled + #[test] + fn test_once_lock_ext() { + let cell = std::sync::OnceLock::new(); + std::thread::scope(|s| { + assert!(cell.get().is_none()); + + s.spawn(|| { + Python::with_gil(|py| { + assert_eq!(*cell.get_or_init_py_attached(py, || 12345), 12345); + }); + }); + }); + assert_eq!(cell.get(), Some(&12345)); + } } diff --git a/tests/test_declarative_module.rs b/tests/test_declarative_module.rs index a911702ce20..93e0e1366f0 100644 --- a/tests/test_declarative_module.rs +++ b/tests/test_declarative_module.rs @@ -1,9 +1,11 @@ #![cfg(feature = "macros")] +use std::sync::Once; + use pyo3::create_exception; use pyo3::exceptions::PyException; use pyo3::prelude::*; -use pyo3::sync::GILOnceCell; +use pyo3::sync::{GILOnceCell, OnceExt}; #[path = "../src/tests/common.rs"] mod common; @@ -149,9 +151,17 @@ mod declarative_module2 { fn declarative_module(py: Python<'_>) -> &Bound<'_, PyModule> { static MODULE: GILOnceCell> = GILOnceCell::new(); - MODULE - .get_or_init(py, || pyo3::wrap_pymodule!(declarative_module)(py)) - .bind(py) + static ONCE: Once = Once::new(); + + // Guarantee that the module is only ever initialized once; GILOnceCell can race. + // TODO: use OnceLock when MSRV >= 1.70 + ONCE.call_once_py_attached(py, || { + MODULE + .set(py, pyo3::wrap_pymodule!(declarative_module)(py)) + .expect("only ever set once"); + }); + + MODULE.get(py).expect("once is completed").bind(py) } #[test] diff --git a/tests/ui/forbid_unsafe.rs b/tests/ui/forbid_unsafe.rs index 9b62886b650..660f5fa36c0 100644 --- a/tests/ui/forbid_unsafe.rs +++ b/tests/ui/forbid_unsafe.rs @@ -1,4 +1,5 @@ #![forbid(unsafe_code)] +#![forbid(unsafe_op_in_unsafe_fn)] use pyo3::*;