chore(rust): make Arrow version selection more flexible

[felipecrv]

The driver manager doesn't have to necessarily require the same version that the datafusion driver requires (due to datafusion's specific arrow version requirements).

Fixes #2524

[lidavidm]

Hmm, should we set up CI to make sure things (minus Datafusion) build with a few versions of the Rust Arrow crate?

[mbrobbel]

Testing this in CI won't work when the datafusion driver is in the workspace (because it locks the arrow version in the workspace).

[felipecrv]

Testing this in CI won't work when the datafusion driver is in the workspace (because it locks the arrow version in the workspace).

That's what I suspected. It's a tricky issue. We could discuss on the issue page the best way to decouple adbc_core from driver/datafusion.

[mbrobbel]

Testing this in CI won't work when the datafusion driver is in the workspace (because it locks the arrow version in the workspace).

That's what I suspected. It's a tricky issue. We could discuss on the issue page the best way to decouple adbc_core from driver/datafusion.

We could maybe move the datafusion driver to the datafusion repo or use two workspaces in this repo (one with the core crate and the other with the drivers)?

[felipecrv]

The 2-workspace solution sounds good. Moving to datafusion sounds even better.

Does Rust tooling assume one-repo==one-workspace anywhere?

[mbrobbel]

Moving to datafusion sounds even better.

+1

Does Rust tooling assume one-repo==one-workspace anywhere?

No.

[lidavidm]

I think we need to split workspaces for another reason: currently we claim we only need rust 1.80 but DataFusion needs rust 1.81. So I was unable to upload v0.17.0 to crates.io.

error: failed to verify package tarball

Caused by:
  rustc 1.80.1 is not supported by the following packages:
    [email protected] requires rustc 1.81.0