Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Harden concurrency assertions for Zarr writing #1080

Open
normanrz opened this issue May 27, 2024 · 1 comment
Open

Harden concurrency assertions for Zarr writing #1080

normanrz opened this issue May 27, 2024 · 1 comment
Assignees
Milestone

Comments

@normanrz
Copy link
Member

We want to switch our default data format to zarr3. In contrast to wkw, zarr3 doesn't allow concurrent writes (as uncompressed wkw does) at all. Therefore, we should be more careful when writing partial shards. Perhaps, we should elevate some of the warnings to assertions to better enforce shard-aligned writing.

@normanrz normanrz mentioned this issue Oct 9, 2024
4 tasks
@normanrz normanrz added this to the 2.x milestone Oct 9, 2024
@hotzenklotz hotzenklotz changed the title Harden concurrency assertions Harden concurrency assertions for Zarr writing Jan 8, 2025
@hotzenklotz
Copy link
Member

hotzenklotz commented Jan 8, 2025

After discussing this, we figured it is very hard to do in the libs. We don't have a support good idea yet. Likely, this needs to be checked and asserted in VX.

This issue becomes important for once we switch to Zarr3 as the default dataset format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants