Manifest Splitting

[dcherian]

• does the config get serialized properly?
• Add ManifestSplitCondition.AnyArray
• python tests for Or, And
• Add docs
• real-world benchmark; test with ERA5
• add ndim based condition (3D vs 4D) (if someone asks for it)

Minimal docs here: https://icechunk--767.org.readthedocs.build/en/767/icechunk-python/performance/

I rewrote the ERA5 manifests to put 1 year per manifest (~9000 chunks); This gets us 3X speedup.

[dcherian]

I don't like this, but it is certainly tied to ndim.

[paraseba]

Maybe ManifestSplits is an enum to avoid this?

enum ManifestSplits {
   Single,
   Multiple(Vec<ManifestExtents>)
}

What I don't like is the empty vector. I wonder if Rust has a NonEmptyVec type, otherwise, a trick people use is:

...
   Multiple{ first: ManifestExtents, rest: Vec<ManifestExtents>}

[dcherian]

Ok I don't need the default any more. It was an artifact that appeared because I implemented the core logic before wiring up the config. Now the default gets set when parsing the config using the Array Metadata

[paraseba]

We need to start checking on writes that indexes have the proper size for the metadata

[paraseba]

ping