error installing 000026

[anhknguyen96]

The dl.install(source='https://github.com/dandi/dandisets.git', path=root_folder, recursive=True) command returned an error on 000026:

[INFO ] https://github.com/dandisets/000026.git/config download failed: Not Found [INFO ] access to 1 dataset sibling dandi-dandisets-dropbox not auto-enabled, enable with: | datalad siblings -d "/tmp/dandisets/000026" enable -s dandi-dandisets-dropbox [13 similar messages have been suppressed; disable with datalad.ui.suppress-similar-results=off] install(error): /tmp/dandisets/000026 (dataset) [ValueError(Target commit 39e4afc1 does not exist in the clone, and a fetch that commit from remote failed)]
More details:

install (error: 1, ok: 185) Traceback (most recent call last): File "/home/anh/Documents/gsoc/DANDIArchiveShowcase/nwb_table_readme.py", line 260, in <module> create_dandiset_summary() File "/home/anh/Documents/gsoc/DANDIArchiveShowcase/nwb_table_readme.py", line 27, in create_dandiset_summary dl.install(source='https://github.com/dandi/dandisets.git', path=root_folder, recursive=True) File "/home/anh/.conda/envs/nwb_explorer/lib/python3.9/site-packages/datalad/interface/utils.py", line 447, in eval_func return return_func(*args, **kwargs) File "/home/anh/.conda/envs/nwb_explorer/lib/python3.9/site-packages/datalad/interface/utils.py", line 439, in return_func results = list(results) File "/home/anh/.conda/envs/nwb_explorer/lib/python3.9/site-packages/datalad/interface/utils.py", line 424, in generator_func raise IncompleteResultsError( datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed: [{'action': 'install', 'contains': [PosixPath('/tmp/dandisets/000026')], 'exception': Target commit 39e4afc1 does not exist in the clone, and a fetch that commit from remote failed [clone.py:clone_dataset:742,clone.py:postclone_checkout_commit:820], 'exception_traceback': '[clone.py:clone_dataset:742,clone.py:postclone_checkout_commit:820]', 'message': 'ValueError(Target commit 39e4afc1 does not exist in the clone, ' 'and a fetch that commit from remote failed)', 'path': '/tmp/dandisets/000026', 'refds': None, 'source': {'default_destpath': '000026', 'giturl': 'https://github.com/dandisets/000026.git', 'source': 'https://github.com/dandisets/000026.git', 'type': 'giturl', 'version': None}, 'status': 'error', 'type': 'dataset'}]

I think it could just be that the pointer to 000026 on the dandisets repo is broken (404 error), since individually installing 000026 was successful. I'd appreciate if this gets fixed soon.

[yarikoptic]

yes, hang there a bit more. We need to address #244 .

NB out of curiosity, why are you interested in the entire hierarchy of the dandisets?

[yarikoptic]

the issue is mitigated for now, I pushed updated states of both 000026 and dandisets "super-dataset"

(dandisets) dandi@drogon:/tmp$ datalad install https://github.com/dandi/dandisets
[INFO   ] Remote origin not usable by git-annex; setting annex-ignore                                                            
[INFO   ] https://github.com/dandi/dandisets/config download failed: Not Found                                                   
[INFO   ] access to 1 dataset sibling dandi-dandisets-dropbox not auto-enabled, enable with:
|               datalad siblings -d "/tmp/dandisets" enable -s dandi-dandisets-dropbox 
install(ok): /tmp/dandisets (dataset)
(dandisets) dandi@drogon:/tmp$ cd dandisets
(dandisets) dandi@drogon:/tmp/dandisets$ datalad install 000026
[INFO   ] scanning for annexed files (this may take some time)                                                                   
[INFO   ] Remote origin not usable by git-annex; setting annex-ignore                                                            
[INFO   ] https://github.com/dandisets/000026.git/config download failed: Not Found 
[INFO   ] access to 1 dataset sibling dandi-dandisets-dropbox not auto-enabled, enable with:
|               datalad siblings -d "/tmp/dandisets/000026" enable -s dandi-dandisets-dropbox 
install(ok): /tmp/dandisets/000026 (dataset) [Installed subdataset in order to get /tmp/dandisets/000026]

[anhknguyen96]

Thanks for the quick resolution of the issue!

NB out of curiosity, why are you interested in the entire hierarchy of the dandisets?

I'm working on a GSoC project, mentored by @pgleeson and @sanjayankur31, in which I explore compatibility issues between existing NWB dandisets and NWB explorer. As I wish to do that systematically, datalad is one of the tools I'm using to go through the hierarchy of the dandisets.

[yarikoptic]

That is great -- thank you @anhknguyen96 and good luck with the project! Do not hesitate to reach out if you encounter any other difficulties or have any other questions answers to which would expedite your progress on the project!

[pgleeson]

Yes @yarikoptic, this is a project which we needed to do for NWBE, but should be of interest in general to DANDI, as means to bulk analyse the contents and automate tests on validity etc.
We're long overdue a chat with the DANDI team to discuss wider interactions between DANDI and NWBE/Open Source Brain, and well get in contact once this all is in a stable state to chat.

[satra]

@anhknguyen96 and @pgleeson - if you use the dandiset metadata assetsSummary to restrict the dandisets, it would act as a better filter for NWB data. 26 actually doesn't contain any NWB files.

[pgleeson]

@satra It would be good for us to keep an eye on other non nwb dandisets too, as we'll be keen to know the other common formats being used. BIDS should be on our list to support on OSB some time too...

[satra]

keep an eye on other non nwb dandisets

in that case, sounds good.