Do files need to be NWB? Linking derived data to existing data.

[AngCamp]

I am uploading some data I've curated, its just a set of events and some descriptive metadata. Time interval of the event and other metadata about each event is provided, I don't feel NWB is needed or appropriate in this case and I'd just like to upload the data as CSVs or TSVs. The data comes from the ABI and IBL datasets so it can be related back to other data in their API using the time data provided.

Will DANDI host this data?

[CodyCBakerPhD]

Given their importance in experiments, TimeIntervals are one the most core data types of NWB files

See this simple tutorial for adding them to an NWB file: https://pynwb.readthedocs.io/en/latest/tutorials/general/plot_timeintervals.html#sphx-glr-tutorials-general-plot-timeintervals-py

DANDI does not accept arbitrarily structured CSV/TSV containing experiment data

I also recommend storing the data in the file in a robust way that is API-independent, as this is a common complaint we receive from data reusers (since knowing which API to use, how to use it, and differences between APIs can all lead to confusion)

[yarikoptic]

I am uploading some data I've curated, its just a set of events and some descriptive metadata.

do you mean it is purely behavioral data without any neural data? DANDI itself might not be the most appropriate archive for it ATM, but given lack of behavioral archive -- might be ok.

Then could be pure behavioral BIDS dataset -- checkout https://bids-specification.readthedocs.io/en/stable/modality-specific-files/behavioral-experiments.html . In effect it would be a collection of .tsv files with minimal .json to describe them. It seems that there are no purely behavioral examples in bids-examples but those might give you examples on the ones with both neuroimaging and behavioural: https://github.com/bids-standard/bids-examples?tab=readme-ov-file#behavioral

[AngCamp]

No it is sharp wave ripples in the ABI and IBL datasets, its more accurately described as reprocessed or curated data. I've just represented them in a way people can use what I created very easily. I was overly permissive with my thresholding (different people will use different criteria) and I've suggest proper thresholds (number of units in ca1 on a probe, minimum peak threshold, artifacts etc etc). So each event is a start, stop time, peak threshold value and a time of peak. I also have similar data for artifacts like gamma bursts or potential motion artifacts. This should save people having to reprocess the LFP themselves.

All the files are tagged with the session, probe, and channel ids so in principle if they wanted the can go back and redo everything but there should be no need to.

[AngCamp]

One thing I would be curious about would be would it be possible to merge the NWB files I make with the NWB files of the original data, so users can easily works with them as one file?

[yarikoptic]

since it is derived data, and various people could derive even primitive derivatives differently, I would recommend to keep them separate from original "raw" data. But may be with @magland's https://github.com/NeurodataWithoutBorders/lindi some magic could be done? i.e. in your dandiset .nwb.json would state that original file is from the original dandiset and yours is just an overlay on top so if someone is opening that .nwb.json they would "seamlessly" fetch data from the remote original .nwb?

then LINDI should probably gain support for some "mapping" of remote URLs to local ones, happen I have those .nwb files locally.

WDYT @magland about such use cases ?

[AngCamp]

Currently I have included the session id in the folder names so this would become the new name for the NWB object for each session. Then I have files for each probe and identified their origin by including the probe_id and channel_id (or channel index on the probe for the IBL, which does not have unique channel ids) from which each table of events was computed. In addition to the SWRs themselves there are gamma band events from the same CA1 channel and motion artifacts taken from outside hippocampus. So each probe gives me 4 tables, an events table and gamma band events from CA1 then two motion artifacts tables from two non-Hp channels.

This should allow people to relate everything back to the source data in the LFP. Not sure how to realize this in NWB, could these prehaps be provided as metadata? I assume LINDI would handle that somehow.

[magland]

That's right @AngCamp @yarikoptic

Lindi is designed to solve the problem where you can form a new NWB file that references the original and includes additional new neurodata objects.

[AngCamp]

@yarikoptic @CodyCBakerPhD So this is my file/directory structure I would appreciate your guys input on what would be best for the NWB file. There are probe level events, which are the detected ripple band events on the probes themselves. Since some of the detected events may spread across hippocampus I also have a global event file, which is meant to represent all events across all probes, though it only includes filtered ones (probe level events that did not overlap in time with a gamma burst or a motion artifact). The global events specifies which probes it was detected on and which probe had the peak ripple band power, and what time that peak occurred on.

There are also tables for gamma band events for each probe, and each probe had two non-hippocampal channels for detecting motion artifacts as well. I labelled probe level events as "overlapping with gamma" or "overlapping with movement" but for completeness I think I should include the gamma band events. In retrospect I also think including the gamma band power may have been a good idea, so users could set their own thresholds or used it themselves for another analysis.

So all of these can be intervals objects, and can the peak_times and max_zscore be metadata objects? Or would they be series?
The probe and channel id's should be metadata I think, and we would link the whole object to the appropriate session obviously.

Also @magland I saw that LINDI is still under construction, are there any issues I should be aware of when I use it? There is no rush to do this, I'm using OSF for now but I'd prefer to host on DANDI eventually. I support NWB, having one file standard is obviously preferable. I also think tis worth noting that I scanned the LINDI docs and I am wondering if there is a way to automate this step...

import pynwb
import lindi

# URL of the remote NWB file
h5_url = "https://api.dandiarchive.org/api/assets/11f512ba-5bcf-4230-a8cb-dc8d36db38cb/download/"

# Set up a local cache
local_cache = lindi.LocalCache(cache_dir='lindi_cache')

# Create the h5py-like client
client = lindi.LindiH5pyFile.from_hdf5_file(h5_url, local_cache=local_cache)

# Open using pynwb
with pynwb.NWBHDF5IO(file=client, mode="r") as io:
    nwbfile = io.read()
    print(nwbfile)

# The downloaded data will be cached locally, so subsequent reads will be faster

Is there a function to call the urls from DANDI given a session ID and a DANDI set? Or do I need to write something to deal with the html script myself?

[AngCamp]

Do my NWB files need a timeseries, or can I just point to the original NWB?

[AngCamp]

I've written some code, needs to be tested and debugged. But it should loop through my files and automatically convert them to NWB, hopefully I haven't done anything glaringly wrong. I'm mostly concerned that I'm not instantiating the NWB objects correctly. How could I link objects within the session together than come from the same channel and/or probe? Note the hierarchy should go session -> probe -> channel and there will also be global level on the same level as the probe I guess.

How could I incorperate lindi into this? @magland

from datetime import datetime
from uuid import uuid4
import pandas as pd
from dateutil.tz import tzlocal
from pynwb import NWBFile, NWBHDF5IO
from pynwb.epoch import TimeIntervals
import os
import re

# Function to extract session ID from folder name
def extract_session_id(folder_name):
    match = re.search(r'swrs_session_(\d+)', folder_name)
    return match.group(1) if match else None

# Function to extract probe and channel IDs from file name
def extract_probe_channel_ids(file_name):
    match = re.search(r'probe_(\d+)_channel_(\d+)', file_name)
    return match.groups() if match else (None, None)

# Function to read CSV file into a DataFrame
def process_csv_file(file_path):
    return pd.read_csv(file_path)

# Function to create an NWB file with metadata
def create_nwb_file(session_description, experimenter, lab, institution, experiment_description, session_id):
    return NWBFile(
        session_description=session_description,
        identifier=session_id,  # Use session ID as the identifier
        session_start_time=datetime.now(tzlocal()),
        experimenter=experimenter,
        lab=lab,
        institution=institution,
        experiment_description=experiment_description,
        session_id=session_id,
    )

# Function to add time intervals from a CSV DataFrame to the NWB file
def add_intervals_from_csv(nwbfile, df, name, description, probe_id, channel_id, additional_columns=None):
    # Create a TimeIntervals object
    time_intervals = TimeIntervals(
        name=name,
        description=description,
    )

    # Add columns for probe and channel IDs
    time_intervals.add_column(name="probe_id", description="ID of the probe")
    time_intervals.add_column(name="channel_id", description="ID of the channel")

    # Add additional columns if provided
    if additional_columns:
        for col_name, col_desc in additional_columns.items():
            time_intervals.add_column(name=col_name, description=col_desc)

    # Add rows to the TimeIntervals object
    for _, row in df.iterrows():
        kwargs = {col: row[col] for col in df.columns if col in additional_columns}
        time_intervals.add_row(
            start_time=row['start_time'],
            stop_time=row['end_time'],
            probe_id=probe_id,
            channel_id=channel_id,
            **kwargs
        )

    # Add the TimeIntervals object to the NWB file
    nwbfile.add_time_intervals(time_intervals)

# Function to process all CSV files in a session folder and create an NWB file
def process_session_folder(folder_path, output_path, session_description, experimenter, lab, institution, experiment_description):
    # Extract session ID from the folder name
    session_id = extract_session_id(os.path.basename(folder_path))

    # Create NWB file with metadata
    nwbfile = create_nwb_file(session_description, experimenter, lab, institution, experiment_description, session_id)

    # Initialize a set to keep track of unique probe IDs
    probe_ids = set()

    # First pass: collect unique probe IDs
    for file in os.listdir(folder_path):
        if file.endswith('.csv'):
            probe_id, channel_id = extract_probe_channel_ids(file)
            if probe_id:
                probe_ids.add(probe_id)

    # Second pass: process each CSV file for each probe ID
    for probe_id in probe_ids:
        for file in os.listdir(folder_path):
            if file.endswith('.csv') and f'probe_{probe_id}' in file:
                file_path = os.path.join(folder_path, file)
                df = process_csv_file(file_path)
                probe_id, channel_id = extract_probe_channel_ids(file)

                # Determine the type of event based on the filename and add intervals accordingly
                if 'karlsson_detector' in file:
                    add_intervals_from_csv(
                        nwbfile, df, name="ripples",
                        description="Ripple events",
                        probe_id=probe_id,
                        channel_id=channel_id,
                        additional_columns={
                            'mean_zscore': "Mean z-score of the event",
                            'median_zscore': "Median z-score of the event",
                            'max_zscore': "Max z-score of the event",
                            'min_zscore': "Min z-score of the event",
                            'Peak_time': "Peak time of the event",
                            'Overlaps_with_gamma': "Overlaps with gamma events",
                            'Overlaps_with_movement': "Overlaps with movement events"
                        }
                    )
                elif 'movement_artifacts' in file:
                    add_intervals_from_csv(
                        nwbfile, df, name="motion_artifacts",
                        description="Motion artifact events",
                        probe_id=probe_id,
                        channel_id=channel_id,
                        additional_columns={
                            'mean_zscore': "Mean z-score of the event",
                            'median_zscore': "Median z-score of the event",
                            'max_zscore': "Max z-score of the event",
                            'min_zscore': "Min z-score of the event",
                            'Peak_time': "Peak time of the event",
                        }
                    )
                elif 'gamma_band_events' in file:
                    add_intervals_from_csv(
                        nwbfile, df, name="gamma_events",
                        description="Gamma band events",
                        probe_id=probe_id,
                        channel_id=channel_id
                    )

    # Save the NWB file
    with NWBHDF5IO(output_path, 'w') as io:
        io.write(nwbfile)

# Example usage
folder_path = '/mnt/data/swrs_session_1234'  # Path to the session folder
output_path = '/mnt/data/output_file.nwb'  # Path to save the NWB file
session_description = "your session description"  # Description of the session
experimenter = "your name"  # Name of the experimenter
lab = "your lab"  # Name of the lab
institution = "your institution"  # Name of the institution
experiment_description = "your experiment description"  # Description of the experiment

# Process the session folder and create the NWB file
process_session_folder(folder_path, output_path, session_description, experimenter, lab, institution, experiment_description)