Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert to nwb #44

Merged
merged 88 commits into from
May 9, 2022
Merged
Show file tree
Hide file tree
Changes from 79 commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
720eb00
Mult rootdirs. If sess_dir, check fullpath. Give OpenEphys fullpath.
CBroz1 Nov 10, 2021
f3dd8d8
add draft convert to nwb
bendichter Dec 15, 2021
8d75a61
* add lfp from source
bendichter Jan 5, 2022
470b20a
Update element_array_ephys/export/nwb.py
bendichter Jan 12, 2022
3298341
Update element_array_ephys/export/nwb.py
bendichter Jan 12, 2022
e837452
Update element_array_ephys/export/nwb.py
bendichter Jan 12, 2022
d2b93f2
add documentation
bendichter Jan 14, 2022
d924d57
* optimize imports
bendichter Jan 14, 2022
b4d9c0e
Merge remote-tracking branch 'origin/convert_to_nwb' into convert_to_nwb
bendichter Jan 14, 2022
a7f846b
add missing nwbfile arg
bendichter Jan 25, 2022
b35ee72
add some docstrings
bendichter Jan 25, 2022
19f78b7
add readme for exporting to NWB
bendichter Jan 25, 2022
013ae7d
refactor to include requirements for nwb conversion
bendichter Jan 30, 2022
1363214
relative import of ephys
bendichter Jan 30, 2022
d1f3dab
add datetime import
bendichter Jan 30, 2022
365b43b
add optional session keys to ecephys_session_to_nwb
bendichter Jan 30, 2022
96c57f7
fix nwbfile_kwargs logic
bendichter Jan 30, 2022
eb47ee5
fix nwbfile_kwargs logic
bendichter Jan 30, 2022
1b67629
import from workflow pipeline
bendichter Jan 30, 2022
5de22e0
import from workflow pipeline
bendichter Jan 30, 2022
1dccb68
import session_to_nwb
bendichter Jan 30, 2022
379ae11
import probe
bendichter Jan 30, 2022
e992478
ephys.get_ephys_root_data_dir
bendichter Jan 30, 2022
8d3df71
standardize slashes
bendichter Jan 30, 2022
ba3f86a
import os
bendichter Jan 30, 2022
654d567
ephys.find_full_path
bendichter Jan 30, 2022
7deb00f
correctly set conversion and channel_conversion
bendichter Feb 4, 2022
7953662
refactor into gains_helper
bendichter Feb 4, 2022
4cab8c8
add tests
bendichter Feb 4, 2022
f8027cc
Merge branch 'main' into convert_to_nwb
bendichter Feb 8, 2022
9374d94
Update element_array_ephys/export/nwb/nwb.py
bendichter Feb 8, 2022
6338daf
update import path
bendichter Feb 9, 2022
63b545d
Merge remote-tracking branch 'origin/convert_to_nwb' into convert_to_nwb
bendichter Feb 9, 2022
a557b17
Rebase, squashed. See Details
kabilar Sep 27, 2021
2595dae
Avoid linking_module issues. See details.
CBroz1 Feb 11, 2022
bf5e82a
nwb function specification in linking_module
Feb 15, 2022
6d7ad7c
Merge pull request #1 from ttngu207/chris-nwb
CBroz1 Feb 15, 2022
b4ffe1d
import the correct ephys module that has been activated
Feb 16, 2022
2beb41b
Merge pull request #2 from ttngu207/chris-nwb
CBroz1 Feb 16, 2022
782a4a5
adjust imports in __init__ and nwb.py
CBroz1 Feb 16, 2022
fafdde1
add tests for getting lfp data from datajoint
bendichter Feb 22, 2022
a7f4624
import GenericDataChunkIterator from hdmf
bendichter Feb 28, 2022
806684f
rmv subject_id
bendichter Feb 28, 2022
5b942b3
Merge remote-tracking branch 'cbroz1/ben' into convert_to_nwb
bendichter Feb 28, 2022
c63c8c4
rmv tests (they are moved to the ephys workflow)
bendichter Mar 1, 2022
cc14671
fix imports
bendichter Mar 4, 2022
b373b26
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 21, 2022
e973743
Update element_array_ephys/ephys_acute.py
bendichter Mar 21, 2022
3ae6e2d
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 21, 2022
c68f2ca
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 21, 2022
584c738
Update element_array_ephys/export/nwb/README.md
bendichter Mar 21, 2022
a1fb193
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 21, 2022
24ac2c0
Update element_array_ephys/export/nwb/README.md
bendichter Mar 21, 2022
abccafa
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 21, 2022
00c3691
Update element_array_ephys/export/nwb/README.md
bendichter Mar 21, 2022
12974ff
add docstring for gain_helper
bendichter Mar 21, 2022
ba6cbcf
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 21, 2022
49fac08
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 21, 2022
daccfc4
specify releases for dependencies
bendichter Mar 21, 2022
826335b
Merge remote-tracking branch 'origin/convert_to_nwb' into convert_to_nwb
bendichter Mar 21, 2022
b3779e5
Merge branch 'main' into convert_to_nwb
kabilar Mar 21, 2022
9e72773
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 21, 2022
2ee2bd5
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 21, 2022
c200699
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 21, 2022
acdb5f9
fix docstring for get_electrodes_mapping
bendichter Mar 21, 2022
a7b2abb
Merge remote-tracking branch 'origin/convert_to_nwb' into convert_to_nwb
bendichter Mar 21, 2022
cbefcde
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 21, 2022
a01ee9c
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 22, 2022
2e621c0
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 22, 2022
15044cf
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 22, 2022
2526812
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 22, 2022
766f4eb
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 22, 2022
842beec
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 22, 2022
3971fe6
add explanation in docstring of add_ephys_units_to_nwb
bendichter Mar 22, 2022
bc54009
Merge remote-tracking branch 'origin/convert_to_nwb' into convert_to_nwb
bendichter Mar 22, 2022
707adff
add explanation for index parameter
bendichter Mar 22, 2022
82c8655
fix insertion record
bendichter Mar 22, 2022
0152c5d
rmv units_query
bendichter Mar 22, 2022
c2004ea
trying clustering_query..proj()
bendichter Mar 22, 2022
068ea3d
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 22, 2022
59028cb
Update element_array_ephys/export/nwb/nwb.py
bendichter Mar 22, 2022
d94453b
Update element_array_ephys/export/nwb/nwb.py
bendichter Apr 18, 2022
fb98327
Update element_array_ephys/export/nwb/nwb.py
bendichter Apr 18, 2022
9f9872c
Update element_array_ephys/export/nwb/nwb.py
bendichter Apr 18, 2022
3e07c61
remove ephys_no_curation.py
bendichter Apr 27, 2022
6fc51b0
Merge remote-tracking branch 'origin/convert_to_nwb' into convert_to_nwb
bendichter Apr 27, 2022
441cfe2
Update element_array_ephys/export/nwb/nwb.py
bendichter Apr 27, 2022
9ee6088
Update element_array_ephys/export/nwb/nwb.py
bendichter Apr 27, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions element_array_ephys/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
# ephys_acute as default
import element_array_ephys.ephys_acute as ephys
1 change: 0 additions & 1 deletion element_array_ephys/ephys_acute.py
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,6 @@ def make(self, key):
shank, shank_col, shank_row, _ = spikeglx_recording.apmeta.shankmap['data'][recorded_site]
electrode_keys.append(probe_electrodes[(shank, shank_col, shank_row)])
elif acq_software == 'Open Ephys':

session_dir = find_full_path(get_ephys_root_data_dir(),
get_session_directory(key))

Expand Down
1,024 changes: 1,024 additions & 0 deletions element_array_ephys/ephys_no_curation.py

Large diffs are not rendered by default.

Empty file.
55 changes: 55 additions & 0 deletions element_array_ephys/export/nwb/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Exporting data to NWB

The `export/nwb/nwb.py` module maps from the element-array-ephys data structure to NWB.
The main function is `ecephys_session_to_nwb`, which contains flags to control calling the following functions,
which can be called independently:

1. `session.export.nwb.session_to_nwb`: Gathers session-level metadata


2. `add_electrodes_to_nwb`: Add electrodes table to NWBFile. This is needed for any ElectricalSeries, including
raw source data and LFP.


ephys.InsertionLocation -> ElectrodeGroup.location

probe.Probe::probe -> device.name
probe.Probe::probe_comment -> device.description
probe.Probe::probe_type -> device.manufacturer

probe.ProbeType.Electrode::electrode -> electrodes["id_in_probe"]
probe.ProbeType.Electrode::y_coord -> electrodes["rel_y"]
probe.ProbeType.Electrode::x_coord -> electrodes["rel_x"]
probe.ProbeType.Electrode::shank -> electrodes["shank"]
probe.ProbeType.Electrode::shank_col -> electrodes["shank_col"]
probe.ProbeType.Electrode::shank_row -> electrodes["shank_row"]

3. `add_ephys_recording_to_nwb`: Read voltage data directly from source files and iteratively transfer them to the
NWB file. Automatically applies lossless compression to the data, so the final file might be smaller than the original, but there is no
data loss. Currently supports Neuropixels data acquired with SpikeGLX or Open Ephys, and relies on SpikeInterface to read the data.


source data -> acquisition["ElectricalSeries"]

4. `add_ephys_units_to_nwb`: Add spiking data from CuratedClustering to NWBFile.


ephys.CuratedClustering.Unit::unit -> units.id
ephys.CuratedClustering.Unit::spike_times -> units["spike_times"]
ephys.CuratedClustering.Unit::spike_depths -> units["spike_depths"]
ephys.CuratedClustering.Unit::cluster_quality_label -> units["cluster_quality_label"]

ephys.WaveformSet.PeakWaveform::peak_electrode_waveform -> units["waveform_mean"]

5. `add_ephys_lfp_from_dj_to_nwb`: Read LFP data from the data in element-array-ephys and convert to NWB.


ephys.LFP.Electrode::lfp -> processing["ecephys"].lfp.electrical_series["ElectricalSeries{insertion_number}"].data
ephys.LFP::lfp_time_stamps -> processing["ecephys"].lfp.electrical_series["ElectricalSeries{insertion_number}"].timestamps

6. `add_ephys_lfp_from_source_to_nwb`: Read the LFP data directly from the source file. Currently, only works for
SpikeGLX data. Can be used instead of `add_ephys_lfp_from_dj_to_nwb`.


source data -> processing["ecephys"].lfp.electrical_series["ElectricalSeries{insertion_number}"].data
source data -> processing["ecephys"].lfp.electrical_series["ElectricalSeries{insertion_number}"].timestamps
1 change: 1 addition & 0 deletions element_array_ephys/export/nwb/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
from .nwb import ecephys_session_to_nwb, write_nwb
Loading