Preprocessing for AIND ephys pipeline

aind-ephys-preprocessing

Description

This capsule is designed to preprocess data for the AIND pipeline.

This capsule preprocessed the ephys stream with the following steps:

• Phase shift correction (see phase_shift)
• High-pass filtering
• Bad channel removal (using the method developed by IBL)
• Denoising: using one of the following methods
  • Common Median Referenc (CMR)
  • High-pass spatial filtering (see highpass_spatial_filter())
• (optional) Drift correction (estimation and interpolation)

Inputs

The data/ folder must include the session data and at least one JSON file generated by the aind-ephys-job-dispatch capsule is required (1 is recommended).

Parameters

The code/run script takes the following arguments:

  --debug               Whether to run in DEBUG mode
  --denoising {cmr,destripe}
                        Which denoising strategy to use. Can be 'cmr' or 'destripe'
  --no-remove-out-channels
                        Whether to remove out channels
  --no-remove-bad-channels
                        Whether to remove bad channels
  --max-bad-channel-fraction MAX_BAD_CHANNEL_FRACTION
                        Maximum fraction of bad channels to remove. If more than this fraction, processing is skipped
  --motion {skip,compute,apply}
                        How to deal with motion correction. Can be 'skip', 'compute', or 'apply'
  --motion-preset {nonrigid_accurate,kilosort_like,nonrigid_fast_and_accurate}
                        What motion preset to use. Can be 'nonrigid_accurate', 'kilosort_like', or 'nonrigid_fast_and_accurate'
  --debug-duration DEBUG_DURATION
                        Duration of clipped recording in debug mode. Default is 30 seconds. Only used if debug is enabled
  --n-jobs N_JOBS       Number of jobs to use for parallel processing. Default is -1 (all available cores). It can also be a float between 0 and 1 to use a fraction of available cores
  --params-file PARAMS_FILE
                        Optional json file with parameters
  --params-str PARAMS_STR
                        Optional json string with parameters

A full list of parameters can be found in the code/params.json file and is reported here:

{
    "job_kwargs": {
        "chunk_duration": "1s",
        "progress_bar": false
    },
    "preprocessing": {
        "denoising_strategy": "cmr",
        "min_preprocessing_duration": 120,
        "highpass_filter": {
            "freq_min": 300.0,
            "margin_ms": 5.0
        },
        "phase_shift": {
            "margin_ms": 100.0
        },
        "detect_bad_channels": {
            "method": "coherence+psd",
            "dead_channel_threshold": -0.5,
            "noisy_channel_threshold": 1.0,
            "outside_channel_threshold": -0.3,
            "outside_channels_location": "top",
            "n_neighbors": 11,
            "seed": 0
        },
        "remove_out_channels": true,
        "remove_bad_channels": true,
        "max_bad_channel_fraction": 0.5,
        "common_reference": {
            "reference": "global",
            "operator": "median"
        },
        "highpass_spatial_filter": {
            "n_channel_pad": 60,
            "n_channel_taper": null,
            "direction": "y",
            "apply_agc": true,
            "agc_window_length_s": 0.01,
            "highpass_butter_order": 3,
            "highpass_butter_wn": 0.01
        }
    },
    "motion_correction": {
        "preset": "nonrigid_fast_and_accurate",
        "detect_kwargs": {},
        "select_kwargs": {},
        "localize_peaks_kwargs": {},
        "estimate_motion_kwargs": {
            "time_horizon_s": 120
        },
        "interpolate_motion_kwargs": {}
    }
}

Output

The output of this capsule is the following:

• results/preprocessed_{recording_name} folder, containing the preprocessed data saved to binary
• results/motion_{recording_name} file, containing the motion estimation data (if motion is estimate or apply)
• results/preprocessed_{recording_name}.json file, containing the JSON file to reload the processing chain
• results/preprocessedviz_{recording_name}.json file, all the information required to make a visualization of the preprocessing downstream
• results/data_process_preprocessing_{recording_name}.json file, a JSON file containing a DataProcess object from the aind-data-schema package.