Error running 000004_demo_analysis.ipynb

[dnkennedy]

Hi, I was trying to run 000004_demo_analysis.ipynb through Google Colab and received the following error (I get the same error running locally as well, for what that's worth).

On the cell:

import numpy as np
from pathlib import Path
import pandas as pd
import os
import sys
from pynwb import NWBHDF5IO
import matplotlib.pyplot as plt
import seaborn as sns
from dandi.download import download as dandi_download
import RutishauserLabtoNWB.events.newolddelay.python.analysis.helper as helper
import RutishauserLabtoNWB.events.newolddelay.python.analysis.single_neuron as single_neuron

I get the following error:

AttributeError Traceback (most recent call last)
in <cell line: 6>()
4 import os
5 import sys
----> 6 from pynwb import NWBHDF5IO
7 import matplotlib.pyplot as plt
8 import seaborn as sns

10 frames
/usr/local/lib/python3.10/dist-packages/ruamel/yaml/main.py in error_deprecation(fun, method, arg, comment)
1037 raise AttributeError(s)
1038 else:
-> 1039 raise AttributeError(s, name=None)
1040
1041

AttributeError:
"safe_load()" has been removed, use

yaml = YAML(typ='safe', pure=True)
yaml.load(...)

instead of file "/usr/local/lib/python3.10/dist-packages/hdmf/spec/namespace.py", line 194

        d = yaml.safe_load(stream)

Any suggestions?

[kabilar]

Thanks for the report, @dnkennedy. Please try upgrading the pynwb version. I filed an issue in the RutishauserLabtoNWB repository for a more permanent solution - rutishauserlab/recogmem-release-NWB#7.

[kabilar]

Hi @bendichter, when running the subsequent cell I am receiving the error below. Any idea on how to resolve this? Thank you.

Command

url = "https://gui.dandiarchive.org/#/folder/5e72b6ac3da50caa9adb0498"
dandi_download([url], os.getcwd(), existing="skip")

Error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[4], line 4
      1 # Input Path to the NWB file here:
      3 url = "https://gui.dandiarchive.org/#/folder/5e72b6ac3da50caa9adb0498"
----> 4 dandi_download([url], os.getcwd(), existing="skip")
      7 pathtoNWBFile = 'sub-P9HMH_ses-20060301_obj-1otd1m8_ecephys+image.nwb'
      8 nwbBasePath = Path(pathtoNWBFile)

File /opt/conda/lib/python3.11/site-packages/dandi/download.py:110, in download(urls, output_dir, format, existing, jobs, jobs_per_zarr, get_metadata, get_assets, preserve_tree, sync, path_type)
    104 if not urls:
    105     # if no paths provided etc, we will download dandiset path
    106     # we are at, BUT since we are not git -- we do not even know
    107     # on which instance it exists!  Thus ATM we would do nothing but crash
    108     raise NotImplementedError("No URLs were provided.  Cannot download anything")
--> 110 parsed_urls = [parse_dandi_url(u, glob=path_type is PathType.GLOB) for u in urls]
    112 # dandi.cli.formatters are used in cmd_ls to provide switchable
    113 pyout_style = pyouts.get_style(hide_if_missing=False)

File [/opt/conda/lib/python3.11/site-packages/dandi/download.py:110](https://hub.dandiarchive.org/opt/conda/lib/python3.11/site-packages/dandi/download.py#line=109), in <listcomp>(.0)
    104 if not urls:
    105     # if no paths provided etc, we will download dandiset path
    106     # we are at, BUT since we are not git -- we do not even know
    107     # on which instance it exists!  Thus ATM we would do nothing but crash
    108     raise NotImplementedError("No URLs were provided.  Cannot download anything")
--> 110 parsed_urls = [parse_dandi_url(u, glob=path_type is PathType.GLOB) for u in urls]
    112 # dandi.cli.formatters are used in cmd_ls to provide switchable
    113 pyout_style = pyouts.get_style(hide_if_missing=False)

File [/opt/conda/lib/python3.11/site-packages/dandi/dandiarchive.py:754](https://hub.dandiarchive.org/opt/conda/lib/python3.11/site-packages/dandi/dandiarchive.py#line=753), in _dandi_url_parser.parse(cls, url, map_instance, glob)
    752 new_url = cls.follow_redirect(url)
    753 if new_url != url:
--> 754     return cls.parse(new_url, map_instance=map_instance, glob=glob)
    755 if handle_redirect == "pass":
    756     # We used to issue warning in such cases, but may be it got implemented
    757     # now via reverse proxy and we had added a new regex? let's just
    758     # continue with a debug msg
    759     lgr.debug("Redirection did not happen for %s", url)

File [/opt/conda/lib/python3.11/site-packages/dandi/dandiarchive.py:761](https://hub.dandiarchive.org/opt/conda/lib/python3.11/site-packages/dandi/dandiarchive.py#line=760), in _dandi_url_parser.parse(cls, url, map_instance, glob)
    759         lgr.debug("Redirection did not happen for %s", url)
    760     else:
--> 761         raise RuntimeError(
    762             f"{url} did not redirect to another location which dandi client would"
    763             f" know how to handle."
    764         )
    765 elif settings.get("map_instance"):
    766     if map_instance:

RuntimeError: https://dandiarchive.org/#/folder/5e72b6ac3da50caa9adb0498 did not redirect to another location which dandi client would know how to handle.

[satra]

@waxlamp - is the redirect for gui.dandiarchive.org still in place, and does it handle url's like the one in this notebook?

@kabilar - best would be to update the notebook. gui.dandiarchive.org shouldn't have been used in the first place, but many folks did that as that was the only way to download a folder at that time.

[waxlamp]

is the redirect for gui.dandiarchive.org still in place

It seems to be, in the sense that the URL in question "properly" redirects to dandiarchive.org, but...

does it handle url's like the one in this notebook?

...short answer: no.

The URL in the notebook looks like one of the old-school Girder URLs, and as such I don't know how to interpret it as a modern DANDI URL. Is there any information about which Dandiset the script was referencing?

[kabilar]

Is there any information about which Dandiset the script was referencing?

Thanks Roni - https://dandiarchive.org/dandiset/000004

[waxlamp]

It seems that the correct URL is https://api.dandiarchive.org/api/assets/757c3982-8e4c-466d-9a9f-73793262268c/download/ (as can be seen in the folder here: https://dandiarchive.org/dandiset/000004/0.220126.1852/files?location=sub-P9HMH&page=1). @kabilar, can you replace the breaking URL in the notebook with this one and see if things work?

[kabilar]

It seems that the correct URL is https://api.dandiarchive.org/api/assets/757c3982-8e4c-466d-9a9f-73793262268c/download/ (as can be seen in the folder here: https://dandiarchive.org/dandiset/000004/0.220126.1852/files?location=sub-P9HMH&page=1). @kabilar, can you replace the breaking URL in the notebook with this one and see if things work?

Thanks @waxlamp. That worked. I filed #107.

There were some additional errors:

1. Previously discussed pywnb and hdmf issue. See fix in Upgrade pynwb and hdmf rutishauserlab/recogmem-release-NWB#8
2. Issue with data type. See fix in Fix data type error rutishauserlab/recogmem-release-NWB#9
3. Issue with the last cell in generating PSTH plots. See traceback below. I was not able to resolve.

   ---------------------------------------------------------------------------
   ValueError                                Traceback (most recent call last)
   Cell In[6], [line 11]
         [9] #Plot all the Neurons
        [10] for neuron in neurons:
   ---> [11]     neuron.raster_psth(cell_type='visual', bin_size = 150)
        [12]     neuron.raster_psth(cell_type = 'memory', bin_size = 150)
   
   File ~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:129, in Neuron.raster_psth(self, height_light_range, xlim, cell_type, bin_size, smooth, saveFig)
       [124](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:124) color_mapping = [colors1[trial.category_id - 1] for trial in trials]
       [125](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:125) trials_timestamps = [self.spike_timestamps[(self.spike_timestamps > (-1000000 + trial.stim_on)) &
       [126](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:126)                                            (self.spike_timestamps < (trial.delay2_off))]-(-1000000+trial.stim_on)
       [127](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:127)                      for trial in trials]
   --> [129](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:129) axs[0].eventplot(np.asarray(trials_timestamps) / 1000, colors=color_mapping)
       [130](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:130) axs[0].axvspan(height_light_range[0], height_light_range[1], color='grey', alpha=0.1)
       [131](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:131) axs[0].set_xlim(xlim[0], xlim[1])
   
   ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (100,) + inhomogeneous part.

[kabilar]

3. Issue with the last cell in generating PSTH plots. See traceback below. I was not able to resolve.

   ---------------------------------------------------------------------------
   ValueError                                Traceback (most recent call last)
   Cell In[6], [line 11]
         [9] #Plot all the Neurons
        [10] for neuron in neurons:
   ---> [11]     neuron.raster_psth(cell_type='visual', bin_size = 150)
        [12]     neuron.raster_psth(cell_type = 'memory', bin_size = 150)
   
   File ~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:129, in Neuron.raster_psth(self, height_light_range, xlim, cell_type, bin_size, smooth, saveFig)
       [124](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:124) color_mapping = [colors1[trial.category_id - 1] for trial in trials]
       [125](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:125) trials_timestamps = [self.spike_timestamps[(self.spike_timestamps > (-1000000 + trial.stim_on)) &
       [126](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:126)                                            (self.spike_timestamps < (trial.delay2_off))]-(-1000000+trial.stim_on)
       [127](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:127)                      for trial in trials]
   --> [129](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:129) axs[0].eventplot(np.asarray(trials_timestamps) / 1000, colors=color_mapping)
       [130](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:130) axs[0].axvspan(height_light_range[0], height_light_range[1], color='grey', alpha=0.1)
       [131](~/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:131) axs[0].set_xlim(xlim[0], xlim[1])
   
   ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (100,) + inhomogeneous part.

Hi @bendichter, could you please take a look at this last issue with the notebook? Thanks.

[bendichter]

@nandchandravadia could you take a look at this, since you put this notebook together?

[sczcheng]

Hi @kabilar @bendichter

Please see the fix in Commit 42aad89

We will release an update soon. Thank you.

[kabilar]

Thank you @sczcheng. I encountered a similar error with cell_type = 'memory', so I submitted rutishauserlab/recogmem-release-NWB#10.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[8], [line 12]
     [10] for neuron in neurons:
     [11]     neuron.raster_psth(cell_type='visual', bin_size = 150)
---> [12]     neuron.raster_psth(cell_type = 'memory', bin_size = 150)

File ~/miniconda3/envs/dandiexample4/lib/python3.11/site-packages/RutishauserLabtoNWB/events/newolddelay/python/analysis/single_neuron.py:247, in Neuron.raster_psth(self, height_light_range, xlim, cell_type, bin_size, smooth, saveFig)
    [242] color_mapping = [colors1[int(trial.response > 3)] for trial in trials]
    [243] trials_timestamps = [self.spike_timestamps[(self.spike_timestamps > (-1000000 + trial.stim_on)) &
    [244]                                            (self.spike_timestamps < (trial.delay2_off))]-(-1000000+trial.stim_on)
    [245]                      for trial in trials]
--> [247] axs[0].eventplot(np.asarray(trials_timestamps) / 1000, colors=color_mapping)
    [248] axs[0].axvspan(height_light_range[0], height_light_range[1], color='grey', alpha=0.1)
    [249] axs[0].set_xlim(xlim[0], xlim[1])

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (100,) + inhomogeneous part.

[kabilar]

Hi @sczcheng, I filed rutishauserlab/recogmem-release-NWB#11 to request a new release on PyPI. Thanks.