Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sidecar JSON file is not recognized by dandi upload when the dataset_description.json file is missing #1464

Open
kabilar opened this issue Jul 23, 2024 · 5 comments
Labels

Comments

@kabilar
Copy link
Member

kabilar commented Jul 23, 2024

Description

  1. When uploading a new or edited JSON file that accompanies a Zarr file in a BIDS inspired dataset, the JSON file does not get recognized by the dandi upload command when the dataset_description.json file is missing at the top-level directory of the Dandiset. Do you know why this might be the case?
  2. The dataset_description.json file is required for a BIDS dataset (see BIDS Specification), but we don't specify that this is a requirement for a Dandiset.

Thank you.

cc @dstansby @aaronkanzer

@yarikoptic
Copy link
Member

yarikoptic commented Jul 23, 2024

If it isn't bids dataset, then we upload only nwb, zarr, and video files "by default"... Eg as documented at

By default all .nwb, .zarr, and .ngff assets in the Dandiset (ignoring
. What's the use case here?

@kabilar
Copy link
Member Author

kabilar commented Jul 23, 2024

Thanks @yarikoptic. For the LINC project, we are working towards uploading BIDS compliant datasets.

  1. Sometimes, I have forgotten to add the dataset_description.json file for a new dataset/Dandiset.
  2. Other times, a user wants to change the metadata in the sidecar JSON file.
    1. They don't have the entire dataset/Dandiset stored locally because the Zarr files are >1 TB.
    2. So they download the dandiset.yaml file, download the sidecar JSON file, create the dataset/Dandiset directory tree, place the JSON file in the correct location within the tree, and modify the JSON file.
    3. They then try to upload the modified JSON file but dandi upload doesn't recognize the (changed) JSON file, since they didn't know that they need to download the dataset_description.json file.

@yarikoptic
Copy link
Member

heh, we might want to change behaviors here, but should also discuss since .json is nohow specific here, and in general such partial uploads forbid validation of the entire dandiset (unless we validate against full list/content in the archive). Meanwhile workaround is 'ad-hoc' - should do smth like

DANDI_DEVEL=1 dandi upload --allow-any-path whatever/path/to.json

since --allow-any-path is available only when DANDI_DEVEL=1

@kabilar
Copy link
Member Author

kabilar commented Jul 23, 2024

Thanks for the workaround, Yarik.

@yarikoptic
Copy link
Member

For now, as long as we do not decide that "DANDI layout accepts/expects sidecar .json files", I think there is nothing to act on here to allow for download of those. But I think we might want to explicitly inform user , hence

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants