(rust) How to use opendal to load data into DuckDB or Polars LazyFrame?

[chitralverma]

I’m trying to load data from a remote storage backend (e.g., S3, local FS) using opendal, and I want to pass that data into either:

• a DuckDB table, or
• a Polars LazyFrame for further processing.

My current idea is to read the data into memory using op.read() and then wrap it with Cursor, but I’m unsure if this is the best or most idiomatic way to integrate with these libraries, especially for streaming large datasets.

Has anyone successfully done this? Are there preferred patterns or best practices for interoperability between opendal and DuckDB or Polars?

Thanks!

[Xuanwo]

Hi, we have https://github.com/apache/opendal/blob/main/bindings/python/examples/polars.ipynb and https://github.com/apache/opendal/blob/main/bindings/python/examples/pandas.ipynb as basic examples. Would these be helpful to you?

[chitralverma]

these are examples in python, i was wondering if theres a solution in rust directly

[Xuanwo]

For Polars, one possible approach is to use object_store_opendal, which allows using OpenDAL as an implementation of ObjectStore. However, the Polars Rust API is relatively closed and doesn't offer a way to inject custom implementations. For instance, their ParquetAsyncReader only supports the from_uri method.

cc @reswqa for more help, can we use opendal in polars directly?

---

As for DuckDB, I haven't examined it closely, but from what I remember, there isn't a straightforward way to interact with the DuckDB table class from Rust. Please correct me if that has changed.

[chitralverma]

Yes for Polars I've raised pola-rs/polars#22154, hopefully this can be seen as an enhancement.

[chitralverma]

@Xuanwo @reswqa please see one of the comments from a collaborator on the issue above, might need your support as im not very familiar with the internals of opendal

That is unfortunately not possible. Our internal API is based on memory maps, this does allow non-eager reading. AsyncRead and Streams are very hard for us to support efficiently, since many file formats rely on seeking across the file or would require significant buffering anyway.

[chitralverma]

@Xuanwo I had one idea to make this work for all 3 - pyarrow, polars and duckdb.

Instead of doing things on rust/ c side in each project, I saw that the python bindings all all 3 projects support fsspec based file systems - pyarrow, duckdb and polars

So if the opendal team sees it as a good fit, we can provide an opendal backed implementation of the fsspec.AsyncFileSystem + maybe pyarrow.fs.FSSpecHandler, directly in the python bindings for end users to use (maybe behind an extra/ group??).

Beyond polars and duckdb, also this will help users to use opendal backends through these interfaces.

Someone is also partially maintaining this in a separate repo, might as well do it officially. wdyt?

[Xuanwo]

Hi @chitralverma, I started https://github.com/fsspec/opendalfs to explore whether it would be better to build an fsspec implementation directly in Rust. However, I've realized that it duplicates much of the work we've already done on the Python side. It might be better to use the OpenDAL Python binding directly in opendalfs. I'll start a discussion about this in opendalfs instead.

For pyarrow.fs.FSSpecHandler, maybe we can build a new integration like we do for other crates.

[chitralverma]

awesome, that's what i thought as well.

but my suggestion was to ship this as a part of the python bindings themselves, i assume this will reduce a lot of package management, version compatibility, backend service support and release stuff as well if its in the same repo