v2.0.0

Feature

• #618 by tznind.
  IsIdentifiableReviewer
• Added progress when loading large files (with cancellation support)
• Now groups outstanding failures by column
• Fixed rules being flagged as 'Identical' when classifying different
  input columns
• #634 by rkm. Convert to
  single entry-point app
• Breaking: Existing scripts and processes which reference the old
  applications
• #636 by howff. Improvements
  to Python scripts, tests, documentation
• #647 by howff. Improved
  testing of SR anonymiser with a standalone stub
• #662 by tznind. Added ability
  to ignore multiple failures at once in Is Identifiable Reviewer

Bugfix

• #597 by tznind. Fixed
  ConsensusRules not being run
• #619 by jas88. Reduce memory
  consumption in nerd
• #632 by rkm. Normalise the
  IsIdentifiableReviewer namespace to match the other Applications
• #646 by howff. Call the
  garbage collector (and report progress) every 1000 records processed from a
  database source in IsIdentifiable.
• #650 by howff. Create python
  package called SmiServices so a wheel can be created.
• Rename all the imports for the new package name
• Remove references to PYTHONPATH Replace all paths and relative imports
  to be independent of current directory Fixes to python tests
• #656 by howff. SRAnonTool
  updated to handle the output from the latest SemEHR anonymiser and ignore
  None-type annotations
• #661 by tznind. Fixed layout
  of main window so it no longer obscures classification/type
• #665 by tznind. Fixed tree
  view loosing selected index when updating/ignoring a failure (in tree view)
• #666 by howff. Silence
  deprecation warning from newer Python as noted by the azure pipeline test

Meta

• #588 by rkm. Prevent
  additional language packs being included in published packages. Reduces
  overall package size a bit.
• #592 by rkm. Manually update
  csvhelper to 25.0.0, fo-dicom to 4.0.7
• #616 by rkm. Check for
  clobbered files during package build
• #620 by rkm. Replace the
  legacy SecurityCodeScan with SecurityCodeScan.VS2019
• #637 by rkm. Change to
  tracking PR changes in news fragment files. Add a script to auto-update the
  CHANGELOG from these files.
• #648 by rkm. Remove the
  temporary reference to BadMedicine added in
  #592
• #654 by howff. Add Azure
  Pipelines test and packaging for the Python scripts

v1.15.1

Fixed

• #610 by howff. Fixed CTP logging

v1.15.0

Changed

• #575 by rkm. Standardised logging setup and Program entries across whole solution
  • Breaking: YAML config change required
  • Removes the SMI_LOGS_ROOT variable - now in YAML config
  • Removes the --trace-logging CLI option - now in YAML config
  • All invokations of IsIdentifiable now require a YAML config to ensure logging is properly configured
• #577 by rkm. Simplify branch workflow by dropping develop

Fixed

• #581 by rkm. Fixed a bug where newlines would never be correctly parsed from the config option in CohortPackager
• #597 by tznind. Fixed ConsensusRules not being run

Dependencies

• Bump CsvHelper from 22.1.1 to 22.1.2
• Bump HIC.RDMP.Plugin from 4.2.3 to 4.2.4
• Bump HIC.RDMP.Plugin.Test from 4.2.3 to 4.2.4
• Bump Magick.NET-Q16-AnyCPU from 7.23.1 to 7.23.2
• Bump SecurityCodeScan from 3.5.3 to 3.5.4
• Bump System.Drawing.Common from 5.0.0 to 5.0.1
• Bump System.IO.Abstractions from 13.2.9 to 13.2.11
• Bump System.IO.Abstractions.TestingHelpers from 13.2.9 to 13.2.11
• Bump jansi from 2.2.0 to 2.3.1
• Bump junit from 4.13.1 to 4.13.2

v1.14.1

• #576 Fixup Windows package build

v1.14.0

Added

• Added total job duration to extraction report header
• IsIdentifiableReviewer rule review screen
• Added CSV input support for IsIdentifiable, use verb file from command line
• Updater microservice now audits performance of queries (cumulative affected rows, queries executed etc)
• Added -f option to DicomTagReader to read a single zip/dicom file
• Added some Python library code

Changed

• Clarified the CLI help text for --format in CohortPackager
• CTP calls an external program to anonymise Structured Reports
• Requires an addition to default.yaml: CTPAnonymiserOptions.SRAnonTool (adding this does not break existing programs).
• Consolidate System.IO.Abstractions.TestingHelpers package references into the Smi.Common.Tests package
• Tidy common csproj options into Directory.Build.props files for all, src, and test projects

Fixed

• Fixed a bug where newlines would never be correctly parsed from the config option in CohortPackager
• CohortPackager: Don't try and create the jobId file when recreating an existing report
• CohortPackager.Tests: Fix a flaky test caused by NUnit setup/teardown code when running tests in parallel
• CohortPackager.Tests: Fix a flaky test caused by using the same MongoDB database name when running tests in parallel
• Fixed the existing CTPAnonymiser tests which had not been updated for the SRAnonTool changes
• Fixed executable name on UpdateValues microservice

Dependencies

• Bump CsvHelper from 17.0.1 to 22.1.1
• Bump HIC.RDMP.Dicom from 2.1.11 to 2.2.2
• Bump HIC.RDMP.Plugin from 4.1.9 to 4.2.3
• Bump HIC.RDMP.Plugin.Test from 4.1.9 to 4.2.3
• Bump Magick.NET-Q16-AnyCPU 7.22.2.2 to 7.23.1
• Bump Microsoft.NET.Test.Sdk 16.8.0 to 16.8.3
• Bump Moq from 4.15.2 to 4.16.0
• Bump NUnit from 3.12.0 to 3.13.1
• Bump NunitXml.TestLogger from 2.1.80 to 3.0.91
• Bump System.IO.Abstractions from 13.2.2 to 13.2.9
• Bump System.IO.Abstractions.TestingHelpers from 13.2.2 to 13.2.9
• Bump YamlDotNet from 9.1.0 to 9.1.4

v1.13.0

Changelog

Added

• Added new command line application TriggerUpdates for detecting and issuing UpdateValuesMessages (e.g. ECHI mapping changes)
• Added new service UpdateValues which propagates changes (e.g. ECHI mapping changes) throughout the deployed database tables.
• ConsensusRule for combining 2+ other rules e.g. SocketRules (See IsIdentifiable Readme.md for more details)
• Added runtime and total failures count to IsIdentifiable logs
• Added NoSuffixProjectPathResolver which generates anonymous image path names that do not contain "-an" (which is the default behaviour).
  • To use, set CohortExtractorOptions.ProjectPathResolverType to Microservices.CohortExtractor.Execution.ProjectPathResolvers.NoSuffixProjectPathResolver
  • For identifiable extractions, the NoSuffixProjectPathResolver is now used
• Validation reports can now be created as either "Combined" (single report as before" or "Split" (a pack of reports including CSVs suitable for post-processing). This is configurable in the YAML config and can also be specified on the CLI when recreating reports for an extraction
• Added JobCompletedAt to the validation reports
• IsIdentifiable: Add support for ignoring OCR output less than n characters in length
• IsIdentifiable: Add a test case for burned-in image text

Changed

• Update docs and make more keywords links to the relevant docs (#440)
• Reduce memory usage on long-running microservices even when .Net assumes RAM is plentiful
• Validation reports are now written to the project reports directory, instead of to a central reports directory

Fixed

• Fix mismatch in Java/C# messages for ExtractionModality
• ExtractionFileCopier: Copy files relative to the extraction root not the global filesystem root
• Fix implementation of minimum OCR length (before being reported) #471

Dependencies

• Bump CsvHelper from 17.0.0 to 17.0.1
• Bump System.IO.Abstractions from 13.2.1 to 13.2.2
• Bump Moq from 4.15.1 to 4.15.2
• Bump System.IO.Abstractions.TestingHelpers from 13.2.1 to 13.2.2
• Bump CsvHelper from 16.2.0 to 17.0.0
• Bump JetBrains.Annotations from 2020.1.0 to 2020.3.0
• Bump jackson-dataformat-yaml from 2.11.3 to 2.12.0
• Bump jackson-databind from 2.11.3 to 2.12.0

v1.12.2

Changelog

Fixed

• Fix missing JSON fields from CTP output

v1.12.1

Changelog

• Remove reference to MongoDB.Driver in Smi.Common.MongoDb.csproj since it caused a version conflict in the output packages

v1.12.0

NOTE - package smi-services-linux-x64-v1.12.0.tar.gz below contains a broken version of the MongoDBPopulator service. Please use a version of this in a later release

Changelog

Added

• [breaking] Add identifiable extraction support
  • New service "FileCopier" which sits in place of CTP for identifiable extractions and copies source files to their output dirs
  • Changes to MongoDB extraction schema, but backwards compatibility has been tested
  • RabbitMQ extraction config has been refactored. Queues and service config files need to be updated
• Add SecurityCodeScan tool to build chain for .NET code
• Add "no filters" extraction support. If specified when running ExtractorCLI, no file rejection filters will be applied by CohortExtractor. True by default for identifiable extractions
• Added caching of values looked up in NLP/rulesbase for IsIdentifiable tool
• Added new rejector that throws out values (e.g. patient IDs) whose IDs are stored in a database table. Set RejectColumnInfos option in yaml to enable this
• Added a check to QueryToExecuteResult for RejectReason being null when Reject is true.

Changed

• [breaking] Environment variables are no longer required. Previous settings now appear in configuration file
  • Environment variable SMI_LOGS_ROOT is now GlobalOptions.LogsRoot
  • Environment variable MONGO_SERVICE_PASSWORD is now MongoDbOptions.Password
  • Removed ISIDENTIFIABLE_NUMTHREADS as it didn't work correctly anyway
• Extraction report: Group PixelData separately and sort by length
• IsIdentifiable Reviewer 'Symbols' rule factory now supports digits only or characters only mode (e.g. use \d for digits but leave characters verbatim)
• IsIdentifiable Reviewer 'symbols' option when building Regex now builds capture groups and matches only the failing parts of the input string not the full ProblemValue. For example MR Head 12-11-20 would return (\d\d-\d\d-\d\d)$

Fixed

• Fix the extraction output directory to be <projId>/extractions/<extractname>

Dependencies

• Bump fo-dicom.Drawing from 4.0.5 to 4.0.6
• Bump fo-dicom.NetCore from 4.0.5 to 4.0.6
• Bump HIC.BadMedicine.Dicom from 0.0.6 to 0.0.7
• Bump HIC.DicomTypeTranslation from 2.3.0 to 2.3.1
• Bump HIC.FAnsiSql from 1.0.2 to 1.0.5
• Bump HIC.RDMP.Dicom from 2.1.6 to 2.1.10
• Bump HIC.RDMP.Plugin from 4.1.6 to 4.1.8
• Bump HIC.RDMP.Plugin.Test from 4.1.6 to 4.1.8
• Bump Microsoft.CodeAnalysis.CSharp.Scripting from 3.6.0 to 3.7.0
• Bump Microsoft.Extensions.Caching.Memory from 3.1.6 to 3.1.8
• Bump Microsoft.NET.Test.Sdk from 16.6.1 to 16.7.1
• Bump MongoDB.Driver from 2.11.0 to 2.11.2
• Bump System.IO.Abstractions from 12.1.1 to 12.1.9
• Bump System.IO.Abstractions.TestingHelpers from 12.1.1 to 12.1.9
• Bump Terminal.Gui from 0.81.0 to 0.89.4

v1.11.1: Hotfix - publish trimmed (#348)

* Disable publishtrimmed since it breaks things

* bump for v1.11.0

Co-authored-by: James A Sutherland <[email protected]>