Establish full Bpod ingestion

[CBroz1]

MATLAB is the only environment officially supported by Bpod developers. SciPy has a lot of support for loading mat files. In the current draft, SciPy is used to import Bpod .mat files as embedded dictionaries. Micheal Wulf shared the Bpod files that are currently available via djarchive as workflow-trial, revision 0.0.0b1. They were shared as a diverse set of examples. Under element_trial/readers/, there is a reader draft in need of further development, Bpod.py, and eventual inclusion in element_interface. It handles general ingestion for fields that are consistent across examples, but there are many differences across files. A fully realized loader may need to pull directly from Bpod 'raw' fields to reconstruct the experiment, as many other fields are organized differently across trials.

Ecosystem: The SanWorks team (git repositories here) offers no offical python support, suggesting instead that users export to JSON. The PyBpod project (github, docs) offers a python-based GUI alternative for running Bpod hardware. So far as I could tell, they do not offer any useful ingestion functions we would want to incorporate.

from workflow_trial.pipeline import subject, session, trial
from workflow_trial.ingest import ingest_subjects, ingest_sessions
ingest_subjects(); _ = ingest_sessions()

# Download from djarchive, set root trial directory in workflow
from djarchive_client import client; c = client()
c.download('workflow-trial', '0.0.0b1', '<local-path>', create_target=True)
def get_trial_root_dir(): # also defined in workflow_trial.paths
    return '<local-path>'

# List of files
bpods_filepath=[
    "Bpod_Behavior_Data/BalbC_Ph_W1_LH_Randdelay_changeover_Aug08_2021_Session1.mat",
    "Bpod_Behavior_Data/1119A_3_LoomingThreat_Dec26_2019_Session1.mat",
    "Bpod_Behavior_Data_and_TrialEvents/NeuroPixels/TQ03_Dual2AFC_Jun18_2021_Session1.mat",
    "Bpod_Behavior_Data_and_TrialEvents/NeuroPixels/TrialEvents.mat",
    "Bpod_Behavior_Data_and_TrialEvents/Neuralynx/TP24_Dual2AFC_Feb12_2019_Session1.mat",
    "Bpod_Behavior_Data_and_TrialEvents/Neuralynx/TrialEvents.mat"]

import scipy.io as spio
def load_bpod_matfile(matlab_filepath):
    """Loading routine for behavioral file, bpod .mat """
    SessionData = spio.loadmat(matlab_filepath.as_posix(),
                               squeeze_me=True,simplify_cells=True,
                               struct_as_record=False)['SessionData']
    return SessionData # functionally a dictionary

# See how files differ
for f in bpods_filepath:
    filepath = find_full_path(get_trial_root_dir(),f)
    print(str(filepath).split("/")[-1])
    data = load_bpod_matfile(filepath)
    try: print(data['RawEvents']['Trial'][0]) # replace with other fields
    except KeyError: pass

[CBroz1]

Early bpod loader draft, nonfunctional

import pathlib
import scipy.io as spio


class BPod:
    """ Parse a bpod file into the following objects
    fields: top-level bpod fields
    number_trials: total number of trials
    trial_start_times: list of floats, seconds relative to session start
    trial_types: array of tinyint designating condition number
    """
    def __init__(self, full_dir):
        try:  # check for file in path
            self.filepath = next(pathlib.Path(full_dir).glob('*.mat'))
        except StopIteration:
            raise FileNotFoundError(f'No .mat file found at: {full_dir}')
        self.data = load_bpod_matfile(self.filepath)  # see helper below

    @property
    def fields(self):
        if self._fields is None:
            self._fields = list(self.data.keys())
        return self._fields

    @property
    def number_trials(self):
        if self._number_trials is None:
            self._number_trials = self.data['nTrials']
        return self._number_trials

    @property
    def trial_start_times(self):
        if self._trial_start_times is None:
            self._trial_start_times = list(self.data['TrialStartTimestamp'])
        return self._trial_start_times

    @property
    def trial_types(self):
        if self._trial_types is None and 'TrialTypes' in self.fields:
            self._trial_types = self.data['TrialTypes']
        return self._trial_types

    '''
    @property
    def [each relevant bpod property](self)]:
        if self.[relevant property] is None:
            self.[relevant property] = self.file.[specific structure]
    '''

# --------------------- HELPER LOADER FUNCTIONS -----------------


def load_bpod_matfile(mat_filepath):
    """
    Loading routine for behavioral file, bpod .mat
    """
    # loadmat optionally takes mdict, existing dictionary which it loads into
    # simplify_cells returns a simplified dict structure, instead of reversible
    #                mat-like files
    # squeeze_me compresses matrix dimensions
    mat_file = spio.loadmat(mat_filepath.as_posix(),squeeze_me=True,
                            struct_as_record=False,simplify_cells=True)
    # bpod files load as dict with the following keys
    # __header__    : mat version, flatform, creation date
    # __version__   : file version
    # __globals__
    # SessionData   : mat_struct
    if 'SessionData' in mat_file.keys():
        return mat_file['SessionData']
    else:
        raise FileNotFoundError('.mat file missing SessionData'
                                + f'field at: {mat_filepath}')

[CBroz1]

djarchive has a bucket for bpod data: <bucket>/workflow-trial/0.0.0b1