Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support partial zarr download: asset path could point within zarr #1461

Open
yarikoptic opened this issue Jul 18, 2024 · 1 comment
Open

Comments

@yarikoptic
Copy link
Member

ATM path would need to stop at .zarr asset boundary, and if provided subpath within zarr -- it would fail:

❯ dandi download dandi://dandi/000719/sub-1214579789_ses-1214621812_icephys/sub-1214579789_ses-1214621812_icephys.nwb.zarr/acquisition/data_00002_AD0/data/576
2024-07-18 08:19:12,994 [    INFO] Logs saved in /home/yoh/.local/state/dandi-cli/log/2024.07.18-18.19.10Z-3643693.log
Error: No asset at path 'sub-1214579789_ses-1214621812_icephys/sub-1214579789_ses-1214621812_icephys.nwb.zarr/acquisition/data_00002_AD0/data/576'

I think we should implement "path upscending" logic to go up the path until we hit some path known to be an asset (or dandiset boundary - and then just report on "no asset"), and if asset is a zarr -- take the remainder of the path as the path within zarr, which might lead to folder or a file within zarr, and download that locally.

Relates to

  • params in dandiset URL dandi-archive#1460
    as would need full path within zarr but I think as individual file within zarr makes little sense by itself, for partial zarr download we should always reproduce its leading directory structure locally.

Original use-case/user request:

@kabilar
Copy link
Member

kabilar commented Jul 31, 2024

Related issue: #1075

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants