LORIS Release v24.0.0

[driusan]

This release contains user interface improvements, code clean up, bug fixes and new features across all modules in LORIS.

Full list of changes

Some of the major changes in this release are:

Core

Features

• Data tables may now stream data as they're loading rather than waiting
  until all data has loaded. (PR [Data Tables] Add ability for data tables to stream data #6853)
• Added support for Amazon S3 (PR Add S3 Support to LORIS #7963)
• Added ability for scripts to bulk load instrument data (PR [Instrument] Add method for loading multiple instruments data at once #6869)
• New data dictionary framework and module (PR New Data Dictionary framework and module #6936)
• New classes to describe a data dictionary (PR Add Scope, Cardinality, Type, and DictionaryItem classes #6938)

Updates and Improvements

• Module-specific permissions added for Survey Accounts, Imaging Behavioural
  Quality Control, and Behavioural Quality Control. (PR Add module-specific permissions for Survey, Imaging QC, Behavioural QC #6041)
• Addition of a new account_request_date in users table that will be used when
  requesting a new account and will be displayed in the User Accounts module (PR [users] Add 'request account date' to users #6191)
• Candidate's age can be retrieved from the Candidate class in days, months, or years (PR [Candidate] Add getAge function #5945)
• Addition of autoSelect property to React SelectElement allows for auto-selection of
  only available select option (PR [new_profile] Auto-select only site option #6156)
• An AcquisitionDate field has been added to the files table (PR Add AcquisitionDate field in the files table and back populate it #6892)
• Field Data_entry_completion_status given its own column in flag, and renamed to
  Required_elements_completed (PR [Instrument] Rename Data_entry_completion_status & move to its own column in flag #6876)
• Addition of changeProject and resetProject helper functions to the
  LorisIntegrationTest class to help test project permissions (PR [Integration] changeUserProject helper function, candidate_profile tests #6912)
• Unit tests added for the NDB_BVL_Instrument, NDB_Page, NDB_Factory,
  User, and UserPermissions libraries (PR [Test] unittests for NDB_BVL_Instrument library #6819, PR [Test] unittests for NDB_Page.class.inc #6804, PR [Test] unittests for NDB_Factory.class.inc #6776, PR [Test] unittests for User.class.inc, UserPermissions.class.inc #6765)
• The name of a Standard Date field in a LINST instrument must now end with the
  string _date. Otherwise, a LORIS exception is thrown. There is no restriction
  on the naming format of a Basic Date or MonthYear field. (PR [LINST] Remove '_date' only for Standard Format #6923)
• React Form Select Element now has the ability to set an option in the element
  as a disabled option. (PR [issue_tracker] Only show inactive user if already selected in form #7306)
• Pending accounts in Dashboard now includes DCC users (PR [user_accounts] Remove DDC restriction for pending accounts in dashboard #7054)
• Subproject filter added to Behavioural QC module (PR [Behavioural QC] Add subproject filter #7430)
• Addition of date_format as a DataType in ConfigSettings (PR [configuration] Add date_format as DataType in ConfigSettings #6719)
• Addition of new tables to store PET HRRT data (PR [SQL/imaging] Adding SQL code for the new HRRT PET insertion scripts #6142)
• Modification of the parameter_file table's Value field type to longtext (PR change parameter_file Value to longtext #7392)
• Addition of 4 configuration settings for the minc2bids converter (PR [Configuration] Adding mnc2bids converter configuration settings #7488)
• A PSR3 compatible logging interface was added (PR Add PSR3 Logger #7509)
• Multiple classes of errors flagged by phan are now fixed (multiple PRs)
• A LorisInstance class was added to represent an installed LORIS instance (PR [Core] Add LorisInstance class #6118)
• Validation for DateElement (JS). (PR [Core] Date field js validation #7266)
• Session Current_stage default value changed for Not Started (PR Session Current_stage - Not Started as default value #7102)
• Permissions were reorganized and categorized in the user accounts module (PR [permissions] Renaming and organizing permissions phase 0 #7327)
• The EchoTime field has been added to the following tables: MRICandidateErrors,
  and mri_violations_log. EchoTime is necessary to distiguish MINC files for
  multi-echo aquisitions (PR Added EchoTime field in the violation tables #7515).- The Center_name field in the mri_protocol table has been replaced by CenterID
  from the psc table. The default value of CenterID is NULL. Previously, the
  default for Center_name was AAAA or ZZZZ. (PR Replace Center_name in mri_protocol with CenterID from psc #7525)

Bug Fixes

• A LINST instrument Date field name now appears correctly (not truncated) on the
  instrument if it includes the string _date. (PR [LINST] Remove '_date' only for Standard Format #6923)
• A subtest with only static and/or static score fields (i.e. no values to submit)
  can be saved with no errors so that the instrument scoring script can be called. (PR [NDB_BVL_Instrument] Only update instrument table if $values not empty #7124)
• When a superuser edits another user, the labels for each permission is correctly
  displayed (PR [User Accounts] Permissions panel fix #7451)
• The imaging insertion pipeline (LORIS-MRI scripts) now starts automatically if you have the imaging uploader
  auto-launch set to true and your current upload overwrites an existing file (PR Fixed imaging uploader that would not automatically start when overwriting a file #7084).
• Script CouchDB_MRI_Importer now computes the correct names from the data dictionary
  entries associated to MRI comments (PR Fixes and improvements for the CouchDB_MRI_Importer. #7082).
• Candidate library now allows a null sex in the select() function to accommodate
  scanner candidates. This prevents an error from being thrown in the candidate parameters module. (PR [Candidate] Allow null Sex for scanner candidates #7058)
• Fix public pages with missing title (PR [Core] Public page missing title fix #7121)
• Fix "Go to main page" broken link (PR [Core] Error page - Main page link fix #7258)
• Download CSV fix to remove duplicates and entries that partially match the filtering criteria (PR [Core] Download CSV issues #7242)
• Partially fix instrument escaping issues by reloading instrument and its data upon successful save (PR [Instruments] Escaping issue sub-optimal - Redirect on save success to load defaults from database #7776)
• Fix recognition of null sessionID in NDB_BVL_Instrument (PR [NDB_BVL_Instrument] Fix null recognition of getSessionID #8031)

Modules

API

• Creation of a new version of the API under development (v0.0.4-dev) (PR Transition to API version 0.0.4 #6944)
• Deletion of support for the oldest version of the API (v0.0.2) (PR Transition to API version 0.0.4 #6944)
• Addition of a PATCH request for /candidates/$CandID/$VisitLabel to start next
  stage when the payload contains a "Visit" stage with "In Progress" as Status,
  when the current status of the Visit stage is "Not Started". (PR [API] Visit patch request to start next stage #7479)
• Handle characters that must be urlencoded (such as a space) in the API path for
  visit labels and projects. (PR [API] Add support for visit label name with space #7478 & [API] Add support for project names with space #7463)
• Markdown API documentation was moved into the module docs directory (PR [Docs] Move API docs into the API module #6151)

API Documentation (New Module)

• New module mostly intended for developers, this module provides a user
  interface to inspect and try LORIS modules API.

Behavioural QC

• Fix a fatal error if the datatable is filtered with "All instruments" (PR [Behavioural QC] - Instruments filter fixes #6945)
• Fix for the Instrument filter to keep track of the selected value (PR [Behavioural QC] - Instruments filter fixes #6945)

BrainBrowser

• Now uses Loris API to download imaging files (PR [brainbrowser] Fully remove ajax directory #7824)

Candidate Parameters

• Consents may now be grouped in UI of consent tab (PR [Consent] Add ParentID to consent for consent grouping #6042, PR [Consent] Group consents + Improve UI #6044)
• Fix to prevent titles cut off (PR [Candidate parameters] Titles cut off in Candidate Info #6731)

Candidate Profile

• New integration test class to test project permissions (PR [Integration] changeUserProject helper function, candidate_profile tests #6912)

Configuration

• Addition of configuration settings for the DICOM to BIDS insertion pipeline (PR Add config for the python pipeline config file into the Config module #7937)
• Addition of configuration settings for the MINC to BIDS converter script (PR [Configuration] Adding mnc2bids converter configuration settings #7488)

Conflict Resolver

• Changes are now saved automatically, one by one. Once a conflict is resolved
  the cell that contains the input field will glow green. It is possible to change
  the resolved conflicts to a new value until the page is refreshed. (PR [conflict_resolver] api + dataframework (version 3) #7558)
• This module's API is now described in a Open API Specification file (schema.yml)
  that can be loaded in the new API Documentation module.
• Change the display of multi select values from "value1{@}value2" to "value1, value2"
  in the Correct Answer column of Unresolved and Resolved Conflicts. (PR [Conflict Resolver] Multiselect values #7239)

Create timepoint

• Fix a reindexing of the languages array which caused a database insert error. (PR [Create timepoint] Language dropdown index fix #7145)
• Reorganization of the fields on the page to a more logical order, clean up of
  inefficient code and conditional display of select options based on previous selections (PR [create_timepoint] Code refactor to display proper select options based on previous selections #7825)

Data Dictionary

• Changed instrument filter to multiselect (PR [Data Dictionary] Change filter to multiselect + Update TestPlan #7040)

DQT

• The dataquery module user interface has been completely redesigned. (PR [dqt] import CSV to Define Filters (Feature for leigh) #6908)

EEG Browser

• Signal Visualization, Events and Electrode map (PR [EEG Browser] Signal Visualization, Events, Electrode map and EEG split files support #7387)
• Site/Project/subproject filters only displays entries user has permission for. (PR [EEG] Sites/Project/subproject filters and permission #7400)
• Addition of tables in the SQL schema, a filter on the main page of the module, and a download button
  on the session page to support new annotation features (PR [New Feature] SQL Patch and functionality for EEG BIDS Derivatives #7345)
• New integration test class added to this module (PR [EEG Browser] Integration testing for EEG Browser module #6922)

Genomic Browser

• CNV/CPG records added for candidates to view and test the CNV and Methylation tabs in the Genomic Browser (PR [Raisinbread] Added CNV/CPG records for candidates with SNP data #6900)

Help Editor

• Cleaned up the deprecated column Parent Topic (PR [Help editor] Remove parentID #7025)

Imaging Uploader

• Fixed a bug that would prevent the imaging uploader from starting automatically even if you had the imaging uploader
  auto-launch set to true and your current upload overwrote an existing file (PR Fixed imaging uploader that would not automatically start when overwriting a file #7084)
• Use Loris API to view files (PR [imaging_browser] Use API for thumbnail and download #7816)

Instrument Builder

• Fix for error 'Max value must be larger than min value' when clicking 'Add Row'. (PR [instrument builder] Min/max null default value #6810)

Instruments

• General help text added for instrument data entry (PR [Help Doc] Added help file for instrument data entry #6902)
• Fix to avoid select with required option in group fields to display as multiselect (PR [Instrument] Select required option displays a multiselect fix (group … #7254)
• Fixes to insert JSON instruments (PR [instrument] insert JSON intrument fix #7155)

Issue Tracker

• Readability of comments and history was improved. (PR [issue_tracker] Improve readability of comments and history #6138)
• Update validation to allow NULL Site (For All Sites issues) ([issue_tracker] Update validation to allow NULL Site #6526)
• Fixing redirect and error reporting when creating a new issue (PR [Issue Tracker] Redirect and error warning fixes #7323)
• Show inactive users in the list of assignees as a disabled option. If the
  inactive user had already been previously assigned the issue, the disabled option
  appears but cannot be reselected. Inactive users can no longer be assigned new
  issues. (PR [issue_tracker] Only show inactive user if already selected in form #7306)

Login

• Add option to toggle visibility on password input types. (PR [Login] Toggle password visibility (#5935) #6210, [Login] Password expiry - Password visibility button #7043)

Media

• Fix to display the file name when editing a file (PR [core] fileElement multitype value fix #7381)

Publication

• Display all filterable columns in datatable ([Publication] Display all filterable columns in list #7277)
• Fix for file deletion (PR [Publication] File delete not working (#6195 regression) #7284)

User Accounts

• Fix a false positive validation error when a new LORIS user is added with "Make
  user name match email address" and "Generate new password". (PR [User accounts] New user email with pwd generation fix #6803)
• Fix to allow a superuser to create new users with customizable permissions. ([User Accounts] Superuser: add user permission fix #6770)
• Indicate required fields ([User Accounts] Form improvements #6617)
• Fix to show superuser the labels of each permission when editing a user (PR [User Accounts] Permissions panel fix #7451)

Tools

• Fix fatal errors in delete_candidate.php tool. (PR [Tools] delete_candidate CanID Fix #6805, [Tools] delete_candidate fix #7275)
• Fix fatal errors in fix_candidate_age.php (PR [Tools] Fix to prevent fatal in fix_candidate_age.php #7546)
• New tool generate_candidate_externalids.php to fill external IDs for all
  candidates where a NULL value is found. (PR [Tools] Generate Candidate External IDs #7095)
• New tool populate_visits.php to back-populate visits from the config.xml,
  session table and Visit_Windows table into the visit and visit_project_subproject_rel ([Visits Architecture (1 of 4)] - Population tool for visit table #7663)
• Deprecation of the populate_visit_windows.php tool in favour of populate_visits.php ([Visits Architecture (1 of 4)] - Population tool for visit table #7663)
• Fixes a bug in the way that the data dictionary entries associated to MRI comments were named in the CouchDB database (PR Fixes and improvements for the CouchDB_MRI_Importer. #7082).

Clean Up

• Removal of unused variables and unnecessary branching from getBattery() and
  getBatteryVerbose() functions (PR [instrument_list/battery] Clean-up battery retrieval function #7167)
• Removal of the violated_scans_edit permission (PR [SQL Cleanup Patch] Remove violated_scans_edit permission #6747)
• Removal for the need of the VisitLabel section of the config.xml file. All
  visit configurations and their association to projects are now in the database.
  The visits can be removed from the config.xml file after running the
  populate_visits.php script only. ([Visits Architecture (1 of 4)] - Population tool for visit table #7663 & [Visits Architecture (2 of 4)] - Making config.xml unnecessary #7729)
• Removal of references to Reliability module in Raisinbread (PR [Raisinbread] Removed references to Reliability module #6895)
• Raisinbread visit stage inconsistency changed (PR [Raisinbread] Change Visit Stage to match Instrument data #6896)
• HRRT patch sourced to Raisinbread (PR [Raisinbread] SQL patch for HRRT not applied to Raisinbread #6897)
• Improved consistency of RB data: field UploadLocation of table mri_upload now has an appropriate value (PR Improved consistency of RB data for table mri_upload. #7086)
• The default value of the ScannerID field of the mri_protocol table is now
  NULL instead of 0. This means that if a protocol is valid on all the study's
  scanners, then ScannerID of the protocol should be set to NULL (PR [SQL] Setting default value of ScannerID in mri_protocol to be NULL instead of 0 #7496)

Notes For Existing Projects

• The RegistrationProjectID column of the candidate table and the ProjectID column of the session table
  in the database are no longer nullable. This means that a value must be set in these fields BEFORE running
  the release SQL patch or it will fail.
• New function Candidate::getSubjectForMostRecentVisit replaces Utility::getSubprojectIDUsingCandID,
  adding the ability to determine which subproject a candidate belongs to given their most recent visit.
• LINST instrument class was modified to implement the getFullName() and getSubtestList()
  functions thus making entries in the test_names and instrument_subtests tables
  unnecessary for LINST instruments (PR [LINST] Implement getFullName() and getSubtestList() and fix numeric type #7169)
• The script tools/single_use/remove_publication_users_edit_perm_rel_duplicates.php
  should be run before upgrading LORIS.
• The Data_entry_completion_status column of instrument tables has been migrated
  to its own column in the flag table, and renamed to Required_elements_completed. After script
  Set_Required_elements_completed_flag.php is run, projects will need to delete the
  Data_entry_completion_status column of instrument tables. This can be accomplished
  by running Remove_Data_entry_completion_status_instr_column.php, and then sourcing
  the patch generated by this script. (PR [Instrument] Rename Data_entry_completion_status & move to its own column in flag #6876)
• If _setDataEntryCompletionStatus, _determineDataEntryCompletionStatus, and/or
  updateDataEntryCompletionStatus are called in any overrides, make sure to replace
  all instances with their newly named counterparts, _setRequiredElementsCompletedFlag,
  _determineRequiredElementsCompletedFlag, updateRequiredElementsCompletedFlag. (PR [Instrument] Rename Data_entry_completion_status & move to its own column in flag #6876)
• Deprecation of begintable and endtable elements in LINST instruments (PR Instrument class and LINST class cleanup #7183)
• Deletion of dateTimeFields and monthYearFields variables in instrument class.
  All references to these variables should be removed from project instruments. (PR Instrument class and LINST class cleanup #7183)
• Visit definitions is no longer done in the config.xml. An importer tool (populate_visits.php) is
  available to import the current setup into the visit table of the database.
  Make sure the visits displayed after these changes are what you expect.
  The visits can be removed from the config.xml file after the importer tool is run. ([Visits Architecture (1 of 4)] - Population tool for visit table #7663 & [Visits Architecture (2 of 4)] - Making config.xml unnecessary #7729)
• There is a new abstract getDataDictionary() function in the instrument class
  for the new data dictionary framework. This is already implemented for LINST instruments,
  and existing instruments using LorisForm can use the \LorisFormDictionaryImpl
  trait to extract it in the same way as lorisform_parser.php did for the old datadict module.
• The name of a Standard Date field in a LINST instrument must now end with the
  string _date. Otherwise, a LORIS exception is thrown. There is no restriction
  on the naming format of a Basic Date or MonthYear field. (PR [LINST] Remove '_date' only for Standard Format #6923)
• Run tools/single_use/Remove_duplicate_examiners.php to remove duplicate
  examiners that may have been created before bugfix. Make sure to run this script
  before running the SQL/New_patches/2021-09-28-Unique_examiners.sql. (PR [user_accounts] Use userID to update examiners table #7462)
• Deletion of support for the oldest version of the API (v0.0.2) (PR Transition to API version 0.0.4 #6944)
• Refer to instructions for clean up scripts meant to remove unused data in the SQL/Archive/24.0/Cleanup directory
• all instruments in the test_names module must have a corresponding instrument on the filesystem for the media module to load
• the instrument class's static loadInstanceData function has been replaced with a non-static getInstanceData, calls from existing instruments must be updated appropriately.

Notes For Developers

• PHPCS enabled for tools/ and test/ and must now pass on new PRs (Various PRs)
• Auto fix PHPCS support added (npm run lintfix:php) ([Test/Tools] Option in run-php-linter.sh to auto fix PHPCS #6825)

Upgrade Process

Note: the minimum version of PHP is now PHP 7.4

1. Get the 24.0.0 source code
2. Run the following scripts:
   1. php tools/configuration_check.php
      This script is used to validate the configuration of LORIS. It checks configuration settings and server paths to make sure that the application and underlying architecture are set up to allow LORIS to run properly.
   2. php tools/single_use/SaveUserIDToInstrumentData.php confirm
      This script is intended for a one-time use only to restore the value of the UserID column of instrument tables and the UserID key of the instrument JSON Data in the flag table.
   3. php tools/single_use/remove_publication_users_edit_perm_rel_duplicates.php
      This script deletes duplicate rows in publication_users_edit_perm_rel to make sure the patch '2020-01-07-publication_users_edit_perm_rel_pk.sql' will execute successfully
   4. php tools/single_use/Cleanup_Special_Chars_Document_Repository.php
      This script is written to clean up the files with special characters from the document repository data table as well as clean up the quotes appearing as %22 in the file names in the file system
   5. php tools/single_use/Cleanup_Special_Chars_Media.php
      This script is written to clean up the files with special characters from the media data table as well as clean up the quotes appearing as %22 in the file names in the file system
   6. php tools/single_use/Remove_duplicate_examiners.php
      This script is written to clean up duplicate userIDs in the examiners table
3. Make sure the RegistrationProjectID column in the candidate table of the dataabse does not contain any NULL values. If it does, either assign an existing project to the candidate/scanner or create a new project for these entries before running the release patch (see "Notes for existing projects" for more details).
4. Source the SQL/Release_patches/23.0_To_24.0_upgrade.sql SQL file into your database.
5. Run the following scripts:
   1. php tools/single_use/populate_visits.php
      This script populates the visits table based on the config.xml, session table and Visit_Windows table.
   2. php tools/single_use/Set_Required_elements_completed_flag.php confirm
      This script is written to migrate the Data_entry_completion_status field from the instrument table (or Data column of the flag table) to it's own column in the flag table called Required_elements_completed. Note that this script can take a long time to complete on large projects.
6. Run make all on the LORIS root directory
7. OPTIONAL: Source SQL patches from the SQL/Archive/24.0/Cleanup directory at your discretion to remove superfluous or deprecated entries in the database.
8. OPTIONAL: Run the tools/single_use/Remove_Data_entry_completion_status_instr_column.php PHP script to generate an SQL patch containing all DROP COLUMN statements for the now unused Data_entry_completion_status field in instruments. This script will generate the SQL patch and store it in the SQL/Archive/autogenerated/single_use/ directory under the name Remove_Data_entry_completion_status.sql.
9. OPTIONAL: Remove the <visitLabel subprojectID="X"> tags in the Config.xml file

NOTE: The tools/single_use/SaveUserIDToInstrumentData.php can take several hours up to a day to run depending on the size and age of the databse and number of instruments used

---

This discussion was created from the release LORIS Release v24.0.0.