Add table definitions

.gitignore

@@ -123,3 +123,4 @@ docker-compose.y*ml
 
 # notes
 temp*
+*/temp*

CHANGELOG.md

@@ -2,10 +2,10 @@
 
 Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
 
-## [0.1.0c0] - 2021-11-03
+## [0.1.0b0] - [unreleased]
 ### Added
-+ First draft begins
++ First beta release
 
-## [0.1.0b0] - 2021-00-00
+## [0.1.0a0] - 2021-11-15
 ### Added
-+ First beta release
++ First draft begins

CONTRIBUTING.md

@@ -0,0 +1,3 @@
+# Contribution Guidelines
+
+This project follows the [DataJoint Contribution Guidelines](https://docs.datajoint.io/python/community/02-Contribute.html). Please reference the link for more full details.

README.md

@@ -1,33 +1,62 @@
-# element-trial-behavior
-This repository is a work in progress. It serves as a draft of a DataJoints element for trial-based behavior for our U24 itiative.
-
-## Notes:
-I looked at the structure for `element-array-ephys` for general principle on how to call and load files. I mirrored the main DataJoint implementation as split from 'readers'. I incorporated feedback from project-specific `behavior.py` elsewhere in table development.
-
-## To do:
-- [ ] Support functions
-   - [ ] Other elements/workflows pull `find_full_path` and `find_root_directory` either from their own `__init__.py` files or from `element-data-loader.utils`. Which is best practice?
-   - [ ] `workflow-array-ephys` relies on the linking module for functions to get root and session directories, but the MAP project defines these internally.  Which is best practice?
-- [ ] Table definitions: Discuss table structure
-- [ ] Decide supported filetypes
-   -  [ ] BPOD
-   -  [ ] Kepec standard, TBD
-   -  [ ] Generalizable CSV with user-determined column name to DJ variable name correspondence?
-- [ ] Contact the [BPod team](https://github.com/sanworks/)
-   - [ ] Already an implementation of loading to Python?
-   - [ ] Create joint sustainability roadmap
-- [ ] Contact Kepec team - joint sustainability roadmap
-- [ ] Analysis package
-   - [ ] Load processed data to table structure
-   - [ ] Trigger analysis on raw data import
-- [ ] Quality control metrics
-- [ ] GitHub Actions for PyPI release
-- [ ] example workflow
-   - [ ] Integration tests with pytest
-   - [ ] Tutorials in text format (i.e. Jupyter notebook)
-   - [ ] Tutorial in video format
-   - [ ] Docker for tests
-   - [ ] Example dataset(s) for public release, in DJ Archive
-   - [ ] NWB export
-   - [ ] README
-- [ ] RRID
+# DataJoint Element - Experimental trials
+This repository is a work in progress not yet ready for public release.
+It serves as a draft of a DataJoint element for trialized experiments behavior
+for our U24 itiative.
+
+Work in progress.
+
+## Element architecture
+
+In both of the following diagrams, the trial element starts immediately downstream from ***Session***. In one case, Sessions are first segmented into trials, and then segmented into events. This might be appropriate, for example, in a paradigm with repeated conditions and response behaviors associated with different conditions. In the next, Sessions are directly upstream from Events. This might be appropropriate for a paradigm that recorded events within naturalistic free behavior.
+We  provide an [example workflow](https://github.com/datajoint/workflow-trial/) with a
+[pipeline script](https://github.com/datajoint/workflow-trial/blob/main/workflow_trial/pipeline.py) that models combining this Element with the corresponding [Element-Session](https://github.com/datajoint/element-session).
+
+
+<!---
+![element-trial diagram](images/attached_trial_element_trialized.svg)
+![element-trial diagram](images/attached_trial_element_events.svg)
+-->
+
+## Installation
+
++ Install `element-trial`
+    ```
+    pip install element-trial
+    ```
+
++ Upgrade `element-trial` previously installed with `pip`
+    ```
+    pip install --upgrade element-trial
+    ```
+
+<!---
++ Install `element-interface`
+
+    + `element-interface` is a dependency of `element-trial`, however it is not contained within `requirements.txt`.
+
+    ```
+    pip install "element-interface @ git+https://github.com/datajoint/element-interface"
+    ```
+-->
+
+## Usage
+
+### Element activation
+
+To activate the `element-trial`, one need to provide:
+
+1. Schema names for the event or trial module
+
+2. Upstream Session table: A set of keys identifying a recording session (see [Element-Session](https://github.com/datajoint/element-session)).
+3. Utility functions. See example definitions here](https://github.com/datajoint/workflow-trial/blob/main/workflow_trial/paths...
+
+For more detail, check the docstring of the `element-trial`:
+```python
+from element_trial import event, trial
+help(event.activate)
+help(trial.activate)
+```
+
+### Example usage
+
+See [this project](https://github.com/datajoint/workflow-trial) for an example usage of this Array Electrophysiology Element.

element_trial/__init__.py

@@ -0,0 +1,4 @@
+"""
+Two schemas, trial and event. Trial for fully trialized, segmented.
+Event for events independent of trials, like an act of naturaistic behavior.
+"""

element_trial/event.py

@@ -0,0 +1 @@
+# Copy from trial, remove 'EventTrialized'

element_trial/export/__init__.py

@@ -0,0 +1,7 @@
+
+
+__author__ = "DataJoint"
+__date__ = "November, 2021"
+__version__ = "0.1.0c0"
+
+__all__ = ['__author__', '__version__', '__date__']

element_trial/export/nwb.py

@@ -0,0 +1,83 @@
+from pynwb import NWBFile
+
+from element_session import session
+from element_trial import trial
+
+
+def trial_to_nwb(trial_key):
+    scan_query = session.Session & trial.TrialEvent & trial_key
+    # https://github.com/nwb-extensions/ndx-events-record
+    return NWBFile(scan_query)
+
+
+""" From CHEN 2017 https://github.com/vathes/DJ-NWB-Chen-2017
+
+
+# ===========================================================================
+# ============================= BEHAVIOR TRIALS =============================
+# ===========================================================================
+
+# =============== TrialSet ====================
+# NWB 'trial' (of type dynamic table) by default comes with three mandatory
+# attributes: 'start_time' and 'stop_time'. Other trial-related information
+# needs to be added in to the trial-table as additional columns (with column
+# name and column description)
+
+dj_trial = experiment.SessionTrial * experiment.BehaviorTrial
+skip_adding_columns = experiment.Session.primary_key + ['trial_uid', 'trial']
+
+if experiment.SessionTrial & session_key:
+    # Get trial descriptors from TrialSet.Trial and TrialStimInfo
+    trial_columns = [{'name': tag,
+              'description': re.sub('\s+:|\s+', ' ', re.search(
+                f'(?<={tag})(.*)', str(dj_trial.heading)).group()).strip()}
+             for tag in dj_trial.heading.names
+             if tag not in skip_adding_columns + ['start_time', 'stop_time']]
+
+    # Add new table columns to nwb trial-table for trial-label
+    for c in trial_columns:
+        nwbfile.add_trial_column(**c)
+
+    # Add entry to the trial-table
+    for trial in (dj_trial & session_key).fetch(as_dict=True):
+        trial['start_time'] = float(trial['start_time'])
+        trial['stop_time'] = (float(trial['stop_time']) if
+                              trial['stop_time'] else 5.0)
+        [trial.pop(k) for k in skip_adding_columns]
+        trial['early_lick'] = True if trial['early_lick'] == 'early' else False
+        nwbfile.add_trial(**trial)
+
+# ===========================================================================
+# ============================= BEHAVIOR TRIAL EVENTS =======================
+# ===========================================================================
+
+behavior = nwbfile.create_processing_module(
+    'behavior', 'Time of behavioral events in this session')
+behav_event = pynwb.behavior.BehavioralEvents(name='BehavioralEvents')
+behavior.add_data_interface(behav_event)
+
+for trial_event_type in \
+        (experiment.TrialEventType & \
+            experiment.TrialEvent & session_key).fetch('trial_event_type'):
+    event_times, trial_starts = \
+        (experiment.TrialEvent * experiment.SessionTrial
+            & session_key & {'trial_event_type': trial_event_type}).fetch(
+        'trial_event_time', 'start_time')
+
+    if trial_event_type == 'sample':
+        description = 'Timestamps: beginning of the sampling on each trial.'
+    elif trial_event_type == 'delay':
+        description = 'Timestamps: beginning of the delay on each trial.'
+    elif trial_event_type == 'go':
+        description = 'Time stamps of the go cue signal on each trial.'
+
+    if len(event_times) > 0:
+        event_times = np.hstack(event_times.astype(float)
+                                + trial_starts.astype(float))
+        behav_event.create_timeseries(
+            name=trial_event_type, unit='a.u.', conversion=1.0,
+            data=np.full_like(event_times, 1),
+            timestamps=event_times,
+            description=description)
+
+"""

element_trial/readers/bpod.py

@@ -0,0 +1,76 @@
+''' bpod_loader.py
+For eventual inclusion in element-data-loader
+'''
+
+import pathlib
+import scipy.io as spio
+
+
+class BPod:
+    """ Parse a bpod file into the following objects
+    fields: top-level bpod fields
+    number_trials: total number of trials
+    trial_start_times: list of floats, seconds relative to session start
+    trial_types: array of tinyint designating condition number
+    """
+    def __init__(self, full_dir):
+        try:  # check for file in path
+            self.filepath = next(pathlib.Path(full_dir).glob('*.mat'))
+        except StopIteration:
+            raise FileNotFoundError(f'No .mat file found at: {full_dir}')
+        self.data = load_bpod_matfile(self.filepath)  # see helper below
+
+    @property
+    def fields(self):
+        if self._fields is None:
+            self._fields = list(self.data.keys())
+        return self._fields
+
+    @property
+    def number_trials(self):
+        if self._number_trials is None:
+            self._number_trials = self.data['nTrials']
+        return self._number_trials
+
+    @property
+    def trial_start_times(self):
+        if self._trial_start_times is None:
+            self._trial_start_times = list(self.data['TrialStartTimestamp'])
+        return self._trial_start_times
+
+    @property
+    def trial_types(self):
+        if self._trial_types is None and 'TrialTypes' in self.fields:
+            self._trial_types = self.data['TrialTypes']
+        return self._trial_types
+
+    '''
+    @property
+    def [each relevant bpod property](self)]:
+        if self.[relevant property] is None:
+            self.[relevant property] = self.file.[specific structure]
+    '''
+
+# --------------------- HELPER LOADER FUNCTIONS -----------------
+
+
+def load_bpod_matfile(mat_filepath):
+    """
+    Loading routine for behavioral file, bpod .mat
+    """
+    # loadmat optionally takes mdict, existing dictionary which it loads into
+    # simplify_cells returns a simplified dict structure, instead of reversible
+    #                mat-like files
+    # squeeze_me compresses matrix dimensions
+    mat_file = spio.loadmat(mat_filepath.as_posix(),squeeze_me=True,
+                            struct_as_record=False,simplify_cells=True)
+    # bpod files load as dict with the following keys
+    # __header__    : mat version, flatform, creation date
+    # __version__   : file version
+    # __globals__
+    # SessionData   : mat_struct
+    if 'SessionData' in mat_file.keys():
+        return mat_file['SessionData']
+    else:
+        raise FileNotFoundError('.mat file missing SessionData'
+                                + f'field at: {mat_filepath}')

element_trial/readers/bpod_fields_notes.py

@@ -0,0 +1,102 @@
+'''Temporary file for keeping track of bpod structures
+
+The RAW prefix in bpod structures seems to always have data elsewhere
+	Raw is where it is initially stored, and then ported elsewhere during export
+
+Inconsistent across examples:
+	BalbC top level: LeftValveTime, RightValveTime, LeftAmount, RightAmount,changebridge, bridgepos, subject, delays
+	1119a top level: Notes, MarkerCodes
+
+Custom is determined by user:
+	1119a Custom: contrastLevels, dbSplTrialType3, dbSplTrialType6,startSessionDatetime, startSessionTimeStamp,TQ03_Dual2AFC_Jun18_2021_Session1.mat, BlockNumber,BlockTrial, ChoiceLeft, ChoiceCorrect, Feedback, FeedbackTimeFixBroke, EarlyWithdrawal, FixDur, MT, CatchTrial, OdorFracA,OdorID, OdorPair, ST, ResolutionTime, Rewarded,RewardMagnitude, number_trials, LaserTrial,LaserTrialTrainStart, AuditoryTrial, ClickTask,OlfactometerStartup, PsychtoolboxStartup, AuditoryOmega,LeftClickRate, RightClickRate, LeftClickTrain,RightClickTrain, LeftRewarded, DV, Rig, Subject,PulsePalParamStimulus, PulsePalParamFeedback, StimDelay,FeedbackDelay, MinSampleAud,
+	TP24 Custom: BlockNumber, BlockTrial, ChoiceLeft, ChoiceCorrect, Feedback,FeedbackTime, FixBroke, EarlyWithdrawal, FixDur, MT,CatchTrial, OdorFracA, OdorID, OdorPair, ST, Rewarded,RewardMagnitude, number_trials, LaserTrial, LaserTrialTrainStart,AuditoryTrial, ClickTask, OlfactometerStartup,PsychtoolboxStartup, AuditoryOmega, LeftClickRate,RightClickRate, LeftClickTrain, RightClickTrain, LeftRewarded,DV, Rig, Subject, PulsePalParamStimulus, PulsePalParamFeedback,StimDelay, FeedbackDelay, MinSampleAud
+
+Settings is inconsistently used:
+	BalcC: Nothing
+	TQ03 : same as TrialSettings
+
+TrialSettings is also file-specific:
+	BalbC TrialSettings: Subject, RewardAmount, RewardDelay, ChangeOver, block, depleft, depright, MaxTrials, randomDelay, Lange, Bridge, Drugs
+	1119A TrialSettings: MaxTrialNum, StartPhaseLength, TrialType1, TrialType2,TrialType3, TrialType4, TrialType5, TrialType6, TrialType7, RewardAmount, ContrastLevelMin, ContrastLevelMax, UseBonsai, UsePulsePal, PulsePalComPort, PulsePalParameterFile, UseOptoGeneticPulseLength, NumberOfPulses, PulseFrequency,Bpod_BNC_Ch_Opto, PulsePal_Trig_Ch_Opto, PulsePal_Out_Ch_Opto, Opto_Stim_In_TrialType2, Opto_Stim_In_TrialType3, Opto_Stim_In_TrialType5, Opto_Stim_In_TrialType6, UseSoundStimulation, dB_SPL_min, dB_SPL_max, dB_SPL_step, Photometry, DbleFibers, Isobestic405, RedChannel,PhotometryVersion, Modulation, NidaqDuration, NidaqSamplingRate, DecimateFactor, LED1_Name, LED1_Amp, LED1_Freq, LED2_Name, LED2_Amp, LED2_Freq, LED1b_Name, LED1b_Amp, LED1b_Freq
+	TQ03_ TrialSettings: Aud_Levels, Aud_NoEvidence, Aud_Ramp, Aud_SamplingRate, Aud_ToneDuration, Aud_ToneOverlap, Aud_UseMiddleOctave, Aud_Volume, Aud_maxFreq, Aud_minFreq, Aud_nFreq, AuditoryAlpha, AuditoryStimulusTime, AuditoryStimulusType, BaselineBegin, BaselineEnd, BlockTable, CatchError, CenterWaitMax, ChoiceDeadLine, DbleFibers, DecimateFactor, DrinkingGrace, DrinkingTime, FeedbackDelay, FeedbackDelayDecr, FeedbackDelayGrace, FeedbackDelayIncr, FeedbackDelayMax, FeedbackDelayMin, FeedbackDelaySelection, FeedbackDelayTau, ITI, IncorrectChoiceFeedbackType, Isobestic405, LED1_Amp, LED1_Freq, LED1_Name, LED1b_Amp, LED1b_Freq, LED1b_Name, LED2_Amp, LED2_Freq, LED2_Name, LaserAmp, LaserFeedback, LaserITI, LaserMov, LaserPreStim, LaserPulseDuration_ms, LaserRampDuration_ms, LaserRew, LaserSoftCode, LaserStim, LaserStimFreq, LaserTimeInvestment, LaserTrainDuration_ms, LaserTrainRandStart, LaserTrainStartMax_s, LaserTrainStartMin_s, LaserTrials, LeftBiasAud, MaxSessionTime, MinSampleAud, MinSampleAudAutoincrement, MinSampleAudDecr, MinSampleAudIncr, MinSampleAudMax, MinSampleAudMin, Modulation, NidaqDuration, NidaqMax, NidaqMin, NidaqSamplingRate, OdorA_bank, OdorB_bank, OdorStimulusTimeMin, OdorTable, Percent50Fifty, PercentAuditory, PercentCatch, PhotoPlotReward, PhotoPlotSidePokeIn, PhotoPlotSidePokeLeave, Photometry, PhotometryVersion, PortLEDs, Ports_LMR, PreITI, RedChannel, RewardAmount, ShowFeedback, ShowFix, ShowPsycAud, ShowPsycOlf, ShowST, ShowTrialRate, ShowVevaiometric, SkippedFeedbackFeedbackType, StartEasyTrials, StimDelay, StimDelayAutoincrement, StimDelayDecr, StimDelayIncr, StimDelayMax, StimDelayMin, SumRates, TimeMax, TimeMin, TimeOutBrokeFixation, TimeOutEarlyWithdrawal, TimeOutIncorrectChoice, TimeOutSkippedFeedback, TrialSelection, VevaiometricMinWT, VevaiometricNBin, VevaiometricShowPoints, VideoTrials, Wire1VideoTrigger, nidaqDev
+	TP24_ TrialSettings: Aud_Levels, Aud_NoEvidence, Aud_Ramp, Aud_SamplingRate, Aud_ToneDuration, Aud_ToneOverlap, Aud_UseMiddleOctave, Aud_Volume, Aud_maxFreq, Aud_minFreq, Aud_nFreq, AuditoryAlpha, AuditoryStimulusTime, AuditoryStimulusType, BlockTable, CatchError, ChoiceDeadLine, FeedbackDelay, FeedbackDelayDecr, FeedbackDelayGrace, FeedbackDelayIncr, FeedbackDelayMax, FeedbackDelayMin, FeedbackDelaySelection, FeedbackDelayTau, ITI, IncorrectChoiceFeedbackType, LaserFeedback, LaserITI, LaserMov, LaserPreStim, LaserPulseDuration_ms, LaserRew, LaserStim, LaserStimFreq, LaserTimeInvestment, LaserTrainDuration_ms, LaserTrainRandStart, LaserTrainStartMax_s, LaserTrainStartMin_s, LaserTrials, LeftBiasAud, MaxSessionTime, MinSampleAud, MinSampleAudAutoincrement, MinSampleAudDecr, MinSampleAudIncr, MinSampleAudMax, MinSampleAudMin, OdorA_bank, OdorB_bank, OdorStimulusTimeMin, OdorTable, Percent50Fifty, PercentAuditory, PercentCatch, PortLEDs, Ports_LMR, RewardAmount, ShowFeedback, ShowFix, ShowPsycAud, ShowPsycOlf, ShowST, ShowTrialRate, ShowVevaiometric, SkippedFeedbackFeedbackType, StartEasyTrials, StimDelay, StimDelayAutoincrement, StimDelayDecr, StimDelayIncr, StimDelayMax, StimDelayMin, SumRates, TimeOutBrokeFixation, TimeOutEarlyWithdrawal, TimeOutIncorrectChoice, TimeOutSkippedFeedback, TrialSelection, VevaiometricMinWT, VevaiometricNBin, VevaiometricShowPoints, Wire1VideoTrigger
+
+TrialTypes: array of condition? Not consistent across all trials
+
+['SessionData'].RawEvents.Trial[0].States
+BalbC_Ph_W1_LH_Randdelay_changeover_Aug08_2021_Session1.mat
+RawEvents
+RawData
+
+Bonn bpod SessionData structure
+	TrialTypes - 1,2,3,1,2,3
+	TrialTypeNames - Visibile,Visible,Fading
+	Info
+		StateMachineVersion
+		SessionDate
+		SessionStartTime_UTC
+		SessionStartTime_MATLAB
+	nTrials (# trials in session, here 54)
+	RawEvents (timestamps for each trial's state transitions/recorded events)
+		Trial{1,n}.States #Which of these are important?
+			WaitForPosTriggerSoftCode
+			CueDelay
+			WaitForResponse
+			Port2RewardDelay
+			Port2Reward
+			CloseValves
+			Drinking
+			Port1RewardDelay
+			Port3RewardDelay
+			Port4RewardDelay
+			Port5RewardDelay
+			Port6RewardDelay
+			Port7RewardDelay
+			Port8RewardDelay
+			Port1Reward
+			Port3Reward
+			Port4Reward
+			Port5Reward
+			Port6Reward
+			Port7Reward
+			Port8Reward
+			Punish
+			Punishexit
+			EarlyWithdrawal
+		Trial{1,n}.Events
+			Port4In
+			Port4Out
+			SoftCode10
+			Tup
+			Port2In
+			Port2Out
+	RawData (copy of raw data from state machine)
+	TrialStartTimestamp (time when trial started on Bpod's clock)
+		Note: Timestamps in RawEvents are relative to each trial's start
+	TrialEndTimestamp
+	SettingsFile (the settings file you selected in the launch manager)
+	Notes
+	MarkerCodes
+	CurrentSubjectName
+	TrialSettings
+		GUI
+		GUIMeta
+		GUIPanels
+		polling
+		debug
+		debugvis
+		Data
+		arm_number
+		arm_baited_orig
+		arm_baited
+		SF
+		rotation
+		position
+		StimAlpha
+	StimPos
+		TriggerLocPix
+		TriggerLocOptitrackHitbox
+		TriggerLocOptitrackCenter
+		TriggerLocOptitrackCircleHitRadius
+		tform
+'''