Implement Zarr v3 and sharding codec

[frcroth]

URL of deployed dev instance (used for testing):

• https://zarr.webknossos.xyz

Steps to test:

• Explore and view an unsharded and a sharded zarr v3 dataset.

Questions

• Name classes zarr3 or zarrV3? -> zarr3
• Storage transformers: There are currently no storage transformers proposed (or I didn't find them). Should this implementation already allow for these, or is that not relevant? -> Not relevant
• Are there any more codecs that work like sharding (wrapping other codecs in its implementation) or is it just that one. If this is the only one (that should be supported anyway), it could be treated as a special case. Otherwise, codec parsing and handling probably needs to be overhauled significantly -> Current assumption is only support one layer of sharding

Todos

• Implement suffix length range requests for gcs/s3

Issues:

• fixes Implement Zarr v3 and sharding #6993

---

(Please delete unneeded items, merge only when none are left open)

• Updated changelog
• Needs datastore update after deployment

[fm3]

Consider me impressed! This is good stuff, I only added minor comments to the code. Thanks for getting through all of this :-)

Some things I noticed during testing:

• Got execution exception while exploring https://static.webknossos.org/data/zarr_v3/l4_sample_no_sharding/color (which is not the right uri, should have been https://static.webknossos.org/data/zarr_v3/l4_sample_no_sharding/color/1 ) but should be Fox.failure (so the other explorers can be tried)

• I get different segment ids (pretty large numbers) for https://static.webknossos.org/data/zarr_v3/l4dense_motta_et_al_demo/segmentation/1, as compared to https://webknossos.org/datasets/scalable_minds/l4dense_motta_et_al_demo/view#2786,4254,1706,0,1.074
  (segment positions seem right, but the ids are different. possibly something with endianness? or could it be because there is no ZarrV3SegmentationLayer case class yet?)
  
  
  (Edit: looks like this is a problem of the written dataset, not wk reading it)

• Minor: Explore does not work for https://static.webknossos.org/data/zarr_v3/l4_sample_no_sharding/color/1/zarr.json (I think the other explorers can handle the uri to the json file (there .zarray) being supplied directly?)

[frcroth]

The wrong segment ids are an endian issue. Where would be the best place to switch the endianness?

[fm3]

Thanks for addressing the feedback! As discussed in person, the endianness issue for the segmentation layer may be in the data, we should be re-checking that.

I “unresolved” two conversations above that I think are still open, and turned some notes into checkboxes in my last comment. Could you have a look at those? I’d say this is already very close to complete :)

[fm3]

Niice!

I took the liberty to change a few lines in the json validation, where I still got bare exceptions. This works for me now :)

If more issues come up with future datasets we can still fix them then.

Good to go!