Don't require polars-u64-idx, or somehow allow both normal and u64

[IlanCosman]

Requiring polars-u64-idx means that I can't use normal polars elsewhere in my codebase (whichever of polars or polars-u64-idx was installed more recently will override the other).

It doesn't feel great to sacrifice performance / memory usage in my local polars code just because sometimes I use polars cloud.

Would appreciate if there was some way to say which one to use. I guess this is sort of a more general problem with the whole polars vs. polars-u64-idx, since the same argument could be applied that I shouldn't have to use polars-u64-idx everywhere when only some specific parts of my code need it.

But maybe there's a more clever solution using polars cloud? Where I could use normal polars for my local, and then u64-idx would automatically be used for the cloud jobs?

[ritchie46]

This is anticipated and in the works. We are changing OSS Polars to resolve polars[u32] and polars[u64] in the same environment.

In polars-cloud, polars[u64] is needed. It runs on the streaming engine and there you don't pay for the idxsize in performance nor memory.

In ~2 weeks polars will automatically resolve to polars[u64] if installed, by default it will resolve to polars[u32]. If you want local code to specifically use u32 index, you can then force it by importing polars32 as pl.

So I think that meets your request?

[IlanCosman]

Yes, that plan should resolve all of my concerns!