Allow custom allocator to be injected in more places

[paleolimbot]

When developing https://github.com/paleolimbot/duckdb-nanoarrow it was apparent that we need the ability to inject an ArrowBufferAllocator into the ArrowIpcArrayStreamReader for clients that need tight control over memory usage.

There is also at least one place in the device API where we assume a buffer allocator:

• arrow-nanoarrow/src/nanoarrow/nanoarrow_device.h

  Lines 220 to 223 in e54b7df

  	ArrowErrorCode (*buffer_init)(struct ArrowDevice* device_src,
  	struct ArrowBufferView src,
  	struct ArrowDevice* device_dst, struct ArrowBuffer* dst,
  	void* stream);

In those two examples specifically a lack of custom allocator isn't blocking any real-world case (in DuckDB we can also do a better job coordinating IO if we skip the ArrowIpcArrayStreamReader entirely and go through the lower-level ArrowIpcDecoder; in cudf they have their own code for copying arrays and don't use the device API); however, the option should still be there.

[paleolimbot]

@vyasr @zeroshade I had some feedback that the new cudf Arrow converter doesn't allow a caller-specified allocator. I know you use this already on the GPU side, but does cudf have an CPU allocator abstraction for this somewhere that you can plug in?

[vyasr]

No, not at the moment. Pre-nanoarrow our legacy arrow layer had some internal controls over how memory was allocated, but there was nothing publicly exposed and that didn't make it into the new implementations. As you noted, the new functions do support providing a device allocator in the form of an rmm memory resource. We could certainly consider adding something similar for host allocation if there are use cases.

[paleolimbot]

Thanks for the note! I did have one complaint 🙂 (but I'll let them raise the feature request if it comes up for them).