Can python objects reference the Rust heap?

[ShaneMurphy2]

Could I, say, leak a Vec and use it as a ptr for the method PyBytes::from_ptr?

[birkenfeld]

In the specific case you mention, yes, since that method will copy the memory into the new Bytes object (those always have their allocation on the Python heap).

The way to use arbitrary memory is with buffers/memoryviews, and those are specifically meant for representing non Python data, so no problem referencing a leaked Rust object.

[ShaneMurphy2]

Gotcha, I see that now in the C API docs. Is there even a concept of a Rust heap when building an extension module?

[adamreichold]

Python does not consider how Rust objects are allocated. There is also more than just the heap they are coming from, i.e. a Vec<T> must not just be freed using C's free function even it was originally allocated by malloc by the standard library. It must be dropped as a Vec<T> which implies that Rust code must be executed to properly free memory allocated by Rust code.

Maybe to extend on the above, since buffers are meant to support foreign data, they also support backing objects via PyBuffer::obj custom destructors via bf_releasebuffer. This can be used to make them own data on Rust's heap, e.g. allocate it in Rust code, use it to back the buffer and provide it with a custom destructor which calls e.g. Vec::from_raw_parts to properly drop the backing data.

We use the backing object approach in rust-numpy for example, where we have a custom pyclass called PySliceContainer with the only purpose of dropping Rust-allocated Vec<T> or Box<[T]> when being used as the base object for NumPy arrays wrapping these types.

[ShaneMurphy2]

Gotcha, that makes sense. Thanks!