[BEP032]: add microelectrode electrophysiology example dataset

[bendichter]

This example dataset demonstrates the BIDS microelectrode electrophysiology
(microephys) specification for both extracellular (ecephys) and intracellular
(icephys) recordings.

Features Demonstrated

This dataset demonstrates the following key features:

• Registration to Allen CCF v3 space: Electrode positions are provided in
  Allen Common Coordinate Framework v3 (see space-AllenCCFv3 files)

• Multiple probes in a single subject: sub-mouse01 uses two probes
  simultaneously (probe01 and probe02) targeting different brain regions

• Probe defined in ProbeInterface Library: probe01 uses the
  A1x32-Poly3-10mm-50-177 model from the ProbeInterface library, referenced
  via TermURL in the probes.json sidecar

• Probe defined using a custom ProbeInterface definition: probe02 uses a
  custom probe definition stored in the probes/ directory, referenced via
  BIDS URI (bids::probes/customprobe1.json)

added by @yarikoption now that #515 is merged:

• BIDS Specification PR: [ENH]: BEP032 formal review bids-specification#2307
• BIDS Validator PR: NONE

[ree-gupta]

Hi Ben, thank you for contributing this! could you please rename the dataset directory to something more specific such that we can distinguish it from more future datasets? Perhaps, microephys_toy or anything else that is suitable?

Linking my PRs here to the @bids-standard/bep032 branch for completeness:

• Updating existing icephys data on this branch: [BEP032] Add updated old icephys dataset #490 - it passes validation but a review from others would be nice
• Updating/adding existing ecephys reach2grasp dataset: [BEP032] Add reach to grasp ecephys dataset #491

[bendichter]

The good news is this almost works validating against the compiled BEP32 PR: bids-standard/bids-specification#1705. The bad news is it does not work 100% and I don't know why:

% bst -v export --schema src/schema --output src/schema.json

% bids-validator -s file:///Users/bdichter/dev/bids-specification/src/schema.json /Users/bdichter/dev/bids-examples/microephys/                  
        [ERROR] MICROEPHYS_SPACE_MISMATCH The space entity in electrodes.tsv must match the space entity in the
corresponding coordsystem.json file for microelectrode electrophysiology data.

                /sub-mouse01/ecephys/sub-mouse01_space-AllenCCFv3_electrodes.tsv

        Please visit https://neurostars.org/search?q=MICROEPHYS_SPACE_MISMATCH for existing conversations about this issue.

        [ERROR] MICROEPHYS_ELECTRODES_SPACE_REQUIRED If a coordsystem.json file with a space entity is present for microelectrode
electrophysiology data, a corresponding electrodes.tsv file with the same
space entity must also be present.

                /sub-mouse01/ecephys/sub-mouse01_space-AllenCCFv3_coordsystem.json

        Please visit https://neurostars.org/search?q=MICROEPHYS_ELECTRODES_SPACE_REQUIRED for existing conversations about this issue.

        [ERROR] EMPTY_FILE Empty files not allowed.
                /sub-mouse01/ses-01/ecephys/sub-mouse01_ses-01_task-rest_ecephys.nwb
                /sub-mouse01/ses-01/ecephys/sub-mouse01_ses-01_task-reach_ecephys.nwb

                1 more files with the same issue

        Please visit https://neurostars.org/search?q=EMPTY_FILE for existing conversations about this issue.


          Summary:                         Available Tasks:                 Available Modalities:
          24 Files, 15.6 kB                Reach to Grasp                   microephys           
          2 - Subjects 1 - Sessions        Resting State                                         
                                           IV Curve Characterization                             

        If you have any questions, please post on https://neurostars.org/tags/bids.

The empty files are not an issue-- this repo specifically requests them. The problem is the space validation. I'm pretty sure I added those correctly but they are not validating.

@yarikoptic and @ree-gupta , you both know the schema validation stuff better than I do. Any ideas?

[ree-gupta]

The good news is this almost works validating against the compiled BEP32 PR: bids-standard/bids-specification#1705. The bad news is it does not work 100% and I don't know why:

Hi @bendichter , thanks again. It was to do with the associations files. The original validation failed because the coordsystem association lacked entities: - space in its target. Without this, the association lookup ignored the space entity when finding coordsystem files - so an electrodes.tsv with space-AllenCCFv3 could match ANY coordsystem.json regardless of its space entity, causing false MICROEPHYS_SPACE_MISMATCH
errors.

Adding entities: - space ensures the association only finds coordsystem files with matching space values. The check associations.coordsystem != null now implicitly verifies both existence AND space matching.

I also improved the comments in the schema files.

I cloned your fork, and checked, it passes the validation now 🎉

[bendichter]

yay! I can confirm the fix worked 💯

[bendichter]

From my perspective this is ready

[ree-gupta]

From my perspective this is ready

I get a couple of errors:

• TSV_COLUMN_ORDER_INCORRECT Some TSV columns are in the incorrect order: we did not explicitly discuss this. Some modalities use a specific order and I had done the same but without very strong rationale. We could either simply align with iEEG or leave it out, that is, not specify any order.
• PharmaceuticalName/PharmaceuticalDoseUnits should be strings (not arrays) : should we discuss?
• SampleEnvironment needs valid enum ("in vivo" not "in-vivo")
• issue with the custom probe - I am looking into this, will fix this one

[bendichter]

Huh I didn't see these in the valuation errors

[ree-gupta]

Huh I didn't see these in the valuation errors

Surprisingly the probes error went away when I used, deno cache --reload jsr:@bids/validator with no changes to the schema at all. Could you please run the validator one more time and report what you see?

[bendichter]

ok I recompiled and now I am getting the errors you mentioned. I will let you know when they are all fixed.

[bendichter]

@ree-gupta ok, fixed!

[ree-gupta]

We need to remove dimension_unit use (ref bids-standard/bids-specification@ad50350), otherwise looks good to me!

[yarikoptic]

Note that I have added testing against our BEP032 PR schema changes. Unfortunately testing fails

	Please visit https://neurostars.org/search?q=SIDECAR_KEY_RECOMMENDED for existing conversations about this issue.

	[ERROR] TSV_COLUMN_MISSING A required column is missing
		electrode_name
		/sub-mouse01/ecephys/sub-mouse01_channels.tsv - Required initial column 2 not found.
		/sub-mouse02/icephys/sub-mouse02_channels.tsv - Required initial column 2 not found.

	Please visit https://neurostars.org/search?q=TSV_COLUMN_MISSING for existing conversations about this issue.

	[ERROR] TSV_ADDITIONAL_COLUMNS_MUST_DEFINE Additional TSV columns must be defined in the associated JSON sidecar for this file type
		dimension_unit
		/sub-mouse01/ecephys/sub-mouse01_probes.tsv

	Please visit https://neurostars.org/search?q=TSV_ADDITIONAL_COLUMNS_MUST_DEFINE for existing conversations about this issue.


          Summary:                         Available Tasks:                 Available Modalities:
          26 Files, 21.9 kB                Reach to Grasp                   microephys           
          2 - Subjects 1 - Sessions        Resting State                                         
                                           IV Curve Characterization                             

[ree-gupta]

Also take a look here: #491 - it is reviewed and complete and passes the validator.