[Python] set_memory_pool() is misleading

[pitrou]

Describe the bug, including details regarding any error messages, version, and platform.

The PyArrow set_memory_pool() function does not set the C++ default memory pool, but a separate PyArrow-specific default memory pool (which does not seem to be used everywhere, see #47266 (comment)). So in many cases, calling it does not change the underlying allocator for later operations.

We could either fix this function to set the default C++ memory pool (but we would need to add an API for that), or deprecate it.

Component(s)

Parquet, Python

[pitrou]

@rok @xhochy

[rok]

Do we know how much and where this is used right now?

[pitrou]

No, perhaps a code search could give insights.