Version 3.6.3 Release

CHANGELOG

@@ -6,6 +6,40 @@ All notable changes to this project will be documented in this file.
 The format is based on `Keep a Changelog <http://keepachangelog.com/>`_
 and this project adheres to `Semantic Versioning <http://semver.org/>`_
 
+3.6.3 - 2023-08-15
+------------------
+
+Fixed
+~~~~~
+- Bug in computing confidence intervals when performance was always the same,
+  results now in a np.nan confidence interval.
+- Error catched and message added when you provide images and/or features in the
+  test set, but not labels.
+- SimpleWORC and BasicWORC now detect whether user has provided a separate training
+  and test set and thus bootstrapping should be used.
+- Bug in PREDICT was fixed that mixed up the mode in shape feature extraction (2D / 2.5D)
+- Bug in performance calculation of multiclass classification.
+- Bugs in statistical feature testing.
+
+Changed
+~~~~~~~
+- Statistical test feature selection before PCA: otherwise when combined,
+  it will select PCA components, not features.
+
+Added
+~~~~~
+- Histogram equalization to preprocessing.
+- Recursive feature elimination (RFE) feature selection.
+- Workflow to provide evaluate a trained model on new data
+- Option to set a fixed random seed in the hyperoptimization
+  for reproducibility.
+- Various FAQs
+- Updated user manual with more extensive debugging guide.
+- Thoroughly updated user manual documentation on different data flows, e.g.
+  train-test setups, multiple segmentations per patient.
+- function in SimpleWORC to add parameter files for elastix.
+
+
 3.6.2 - 2023-03-14
 ------------------
 

README.md

@@ -1,4 +1,4 @@
-# WORC v3.6.2
+# WORC v3.6.3
 ## Workflow for Optimal Radiomics Classification
 
 ## Information
@@ -37,12 +37,12 @@ When using WORC, please cite this repository and the paper describing WORC as as
 
 ```bibtex
 @article{starmans2021reproducible,
-   title={Reproducible radiomics through automated machine learning validated on twelve clinical applications}, 
-   author={Martijn P. A. Starmans and Sebastian R. van der Voort and Thomas Phil and Milea J. M. Timbergen and Melissa Vos and Guillaume A. Padmos and Wouter Kessels and David    Hanff and Dirk J. Grunhagen and Cornelis Verhoef and Stefan Sleijfer and Martin J. van den Bent and Marion Smits and Roy S. Dwarkasing and Christopher J. Els and Federico Fiduzi and Geert J. L. H. van Leenders and Anela Blazevic and Johannes Hofland and Tessa Brabander and Renza A. H. van Gils and Gaston J. H. Franssen and Richard A. Feelders and Wouter W. de Herder and Florian E. Buisman and Francois E. J. A. Willemssen and Bas Groot Koerkamp and Lindsay Angus and Astrid A. M. van der Veldt and Ana Rajicic and Arlette E. Odink and Mitchell Deen and Jose M. Castillo T. and Jifke Veenland and Ivo Schoots and Michel Renckens and Michail Doukas and Rob A. de Man and Jan N. M. IJzermans and Razvan L. Miclea and Peter B. Vermeulen and Esther E. Bron and Maarten G. Thomeer and Jacob J. Visser and Wiro J. Niessen and Stefan Klein},
-   year={2021},
-   eprint={2108.08618},
-   archivePrefix={arXiv},
-   primaryClass={eess.IV}
+   title          = {Reproducible radiomics through automated machine learning validated on twelve clinical applications}, 
+   author         = {Martijn P. A. Starmans and Sebastian R. van der Voort and Thomas Phil and Milea J. M. Timbergen and Melissa Vos and Guillaume A. Padmos and Wouter Kessels and David    Hanff and Dirk J. Grunhagen and Cornelis Verhoef and Stefan Sleijfer and Martin J. van den Bent and Marion Smits and Roy S. Dwarkasing and Christopher J. Els and Federico Fiduzi and Geert J. L. H. van Leenders and Anela Blazevic and Johannes Hofland and Tessa Brabander and Renza A. H. van Gils and Gaston J. H. Franssen and Richard A. Feelders and Wouter W. de Herder and Florian E. Buisman and Francois E. J. A. Willemssen and Bas Groot Koerkamp and Lindsay Angus and Astrid A. M. van der Veldt and Ana Rajicic and Arlette E. Odink and Mitchell Deen and Jose M. Castillo T. and Jifke Veenland and Ivo Schoots and Michel Renckens and Michail Doukas and Rob A. de Man and Jan N. M. IJzermans and Razvan L. Miclea and Peter B. Vermeulen and Esther E. Bron and Maarten G. Thomeer and Jacob J. Visser and Wiro J. Niessen and Stefan Klein},
+   year           = {2021},
+   eprint         = {2108.08618},
+   archivePrefix  = {arXiv},
+   primaryClass   = {eess.IV}
 }
 
 @software{starmans2018worc,
@@ -140,9 +140,9 @@ Make sure you add the executable to the PATH when prompted.
 ### Elastix
 Image registration is included in WORC through [elastix and transformix](http://elastix.isi.uu.nl/).
 In order to use elastix, please download the binaries and place them in your
-fastr.config.mounts['apps'] path. Check the elastix tool description for the correct
-subdirectory structure. For example, on Linux, the binaries and libraries should be in "../apps/elastix/4.8/install/"  and
-"../apps/elastix/4.8/install/lib" respectively.
+``fastr.config.mounts['apps']`` path. Check the elastix tool description for the correct
+subdirectory structure. For example, on Linux, the binaries and libraries should be in ``"../apps/elastix/4.8/install/"``  and
+``"../apps/elastix/4.8/install/lib"`` respectively.
 
 Note: optionally, you can tell WORC to copy the metadata from the image file
 to the segmentation file before applying the deformation field. This requires

README.rst

@@ -1,4 +1,4 @@
-WORC v3.6.2
+WORC v3.6.3
 ===========
 
 Workflow for Optimal Radiomics Classification
@@ -7,11 +7,12 @@ Workflow for Optimal Radiomics Classification
 Information
 -----------
 
-+---------------------+---------------------+---------------------+---------------+
-| Unit test           | Documentation       | PyPi                | Citing WORC   |
-+=====================+=====================+=====================+===============+
-| |image0|            | |image1|            | |image2|            | |image3|      |
-+---------------------+---------------------+---------------------+---------------+
++-------------------+------------------+------------------+------------+
+| Unit test         | Documentation    | PyPi             | Citing     |
+|                   |                  |                  | WORC       |
++===================+==================+==================+============+
+| |image1|          | |image2|         | |image3|         | |image4|   |
++-------------------+------------------+------------------+------------+
 
 Introduction
 ============
@@ -38,25 +39,25 @@ WORC as as follows:
 
 .. code:: bibtex
 
-    @article{starmans2021reproducible,
-       title={Reproducible radiomics through automated machine learning validated on twelve clinical applications}, 
-       author={Martijn P. A. Starmans and Sebastian R. van der Voort and Thomas Phil and Milea J. M. Timbergen and Melissa Vos and Guillaume A. Padmos and Wouter Kessels and David    Hanff and Dirk J. Grunhagen and Cornelis Verhoef and Stefan Sleijfer and Martin J. van den Bent and Marion Smits and Roy S. Dwarkasing and Christopher J. Els and Federico Fiduzi and Geert J. L. H. van Leenders and Anela Blazevic and Johannes Hofland and Tessa Brabander and Renza A. H. van Gils and Gaston J. H. Franssen and Richard A. Feelders and Wouter W. de Herder and Florian E. Buisman and Francois E. J. A. Willemssen and Bas Groot Koerkamp and Lindsay Angus and Astrid A. M. van der Veldt and Ana Rajicic and Arlette E. Odink and Mitchell Deen and Jose M. Castillo T. and Jifke Veenland and Ivo Schoots and Michel Renckens and Michail Doukas and Rob A. de Man and Jan N. M. IJzermans and Razvan L. Miclea and Peter B. Vermeulen and Esther E. Bron and Maarten G. Thomeer and Jacob J. Visser and Wiro J. Niessen and Stefan Klein},
-       year={2021},
-       eprint={2108.08618},
-       archivePrefix={arXiv},
-       primaryClass={eess.IV}
-    }
-
-    @software{starmans2018worc,
-      author       = {Martijn P. A. Starmans and Thomas Phil and Sebastian R. van der Voort and Stefan Klein},
-      title        = {Workflow for Optimal Radiomics Classification (WORC)},
-      year         = {2018},
-      publisher    = {Zenodo},
-      doi          = {10.5281/zenodo.3840534},
-      url          = {https://github.com/MStarmans91/WORC}
-    }
-
-For the DOI, visit |image4|.
+   @article{starmans2021reproducible,
+      title          = {Reproducible radiomics through automated machine learning validated on twelve clinical applications}, 
+      author         = {Martijn P. A. Starmans and Sebastian R. van der Voort and Thomas Phil and Milea J. M. Timbergen and Melissa Vos and Guillaume A. Padmos and Wouter Kessels and David    Hanff and Dirk J. Grunhagen and Cornelis Verhoef and Stefan Sleijfer and Martin J. van den Bent and Marion Smits and Roy S. Dwarkasing and Christopher J. Els and Federico Fiduzi and Geert J. L. H. van Leenders and Anela Blazevic and Johannes Hofland and Tessa Brabander and Renza A. H. van Gils and Gaston J. H. Franssen and Richard A. Feelders and Wouter W. de Herder and Florian E. Buisman and Francois E. J. A. Willemssen and Bas Groot Koerkamp and Lindsay Angus and Astrid A. M. van der Veldt and Ana Rajicic and Arlette E. Odink and Mitchell Deen and Jose M. Castillo T. and Jifke Veenland and Ivo Schoots and Michel Renckens and Michail Doukas and Rob A. de Man and Jan N. M. IJzermans and Razvan L. Miclea and Peter B. Vermeulen and Esther E. Bron and Maarten G. Thomeer and Jacob J. Visser and Wiro J. Niessen and Stefan Klein},
+      year           = {2021},
+      eprint         = {2108.08618},
+      archivePrefix  = {arXiv},
+      primaryClass   = {eess.IV}
+   }
+
+   @software{starmans2018worc,
+     author       = {Martijn P. A. Starmans and Thomas Phil and Sebastian R. van der Voort and Stefan Klein},
+     title        = {Workflow for Optimal Radiomics Classification (WORC)},
+     year         = {2018},
+     publisher    = {Zenodo},
+     doi          = {10.5281/zenodo.3840534},
+     url          = {https://github.com/MStarmans91/WORC}
+   }
+
+For the DOI, visit |image5|.
 
 Disclaimer
 ----------
@@ -79,16 +80,16 @@ The publicly released WORC database is described in the following paper:
 
 .. code:: bibtex
 
-    @article {Starmans2021WORCDatabase,
-        author = {Starmans, Martijn P.A. and Timbergen, Milea J.M. and Vos, Melissa and Padmos, Guillaume A. and Gr{\"u}nhagen, Dirk J. and Verhoef, Cornelis and Sleijfer, Stefan and van Leenders, Geert J.L.H. and Buisman, Florian E. and Willemssen, Francois E.J.A. and Koerkamp, Bas Groot and Angus, Lindsay and van der Veldt, Astrid A.M. and Rajicic, Ana and Odink, Arlette E. and Renckens, Michel and Doukas, Michail and de Man, Rob A. and IJzermans, Jan N.M. and Miclea, Razvan L. and Vermeulen, Peter B. and Thomeer, Maarten G. and Visser, Jacob J. and Niessen, Wiro J. and Klein, Stefan},
-        title = {The WORC database: MRI and CT scans, segmentations, and clinical labels for 930 patients from six radiomics studies},
-        elocation-id = {2021.08.19.21262238},
-        year = {2021},
-        doi = {10.1101/2021.08.19.21262238},
-        URL = {https://www.medrxiv.org/content/early/2021/08/25/2021.08.19.21262238},
-        eprint = {https://www.medrxiv.org/content/early/2021/08/25/2021.08.19.21262238.full.pdf},
-        journal = {medRxiv}
-    }
+   @article {Starmans2021WORCDatabase,
+       author = {Starmans, Martijn P.A. and Timbergen, Milea J.M. and Vos, Melissa and Padmos, Guillaume A. and Gr{\"u}nhagen, Dirk J. and Verhoef, Cornelis and Sleijfer, Stefan and van Leenders, Geert J.L.H. and Buisman, Florian E. and Willemssen, Francois E.J.A. and Koerkamp, Bas Groot and Angus, Lindsay and van der Veldt, Astrid A.M. and Rajicic, Ana and Odink, Arlette E. and Renckens, Michel and Doukas, Michail and de Man, Rob A. and IJzermans, Jan N.M. and Miclea, Razvan L. and Vermeulen, Peter B. and Thomeer, Maarten G. and Visser, Jacob J. and Niessen, Wiro J. and Klein, Stefan},
+       title = {The WORC database: MRI and CT scans, segmentations, and clinical labels for 930 patients from six radiomics studies},
+       elocation-id = {2021.08.19.21262238},
+       year = {2021},
+       doi = {10.1101/2021.08.19.21262238},
+       URL = {https://www.medrxiv.org/content/early/2021/08/25/2021.08.19.21262238},
+       eprint = {https://www.medrxiv.org/content/early/2021/08/25/2021.08.19.21262238.full.pdf},
+       journal = {medRxiv}
+   }
 
 The code to download the WORC database and reproduce our experiments can
 be found at https://github.com/MStarmans91/WORCDatabase.
@@ -107,19 +108,19 @@ The package can be installed through pip:
 
 ::
 
-      pip install WORC
+     pip install WORC
 
 Alternatively, you can directly install WORC from this repository:
 
 ::
 
-      python setup.py install
+     python setup.py install
 
 Make sure you install the requirements first:
 
 ::
 
-      pip install -r requirements.txt
+     pip install -r requirements.txt
 
 3rd-party packages used in WORC:
 --------------------------------
@@ -172,7 +173,7 @@ sure you install graphviz. On Ubuntu, simply run
 
 ::
 
-      apt install graphiv
+     apt install graphiv
 
 On Windows, follow the installation instructions provided on the
 graphviz website. Make sure you add the executable to the PATH when
@@ -184,10 +185,10 @@ Elastix
 Image registration is included in WORC through `elastix and
 transformix <http://elastix.isi.uu.nl/>`__. In order to use elastix,
 please download the binaries and place them in your
-fastr.config.mounts['apps'] path. Check the elastix tool description for
-the correct subdirectory structure. For example, on Linux, the binaries
-and libraries should be in "../apps/elastix/4.8/install/" and
-"../apps/elastix/4.8/install/lib" respectively.
+``fastr.config.mounts['apps']`` path. Check the elastix tool description
+for the correct subdirectory structure. For example, on Linux, the
+binaries and libraries should be in ``"../apps/elastix/4.8/install/"``
+and ``"../apps/elastix/4.8/install/lib"`` respectively.
 
 Note: optionally, you can tell WORC to copy the metadata from the image
 file to the segmentation file before applying the deformation field.
@@ -205,13 +206,13 @@ to XNAT. We advise you to specify your account settings in a .netrc file
 when using this feature for your own datasets, such that you do not need
 to input them on every request.
 
-.. |image0| image:: https://github.com/MStarmans91/WORC/workflows/Unit%20test/badge.svg
+.. |image1| image:: https://github.com/MStarmans91/WORC/workflows/Unit%20test/badge.svg
    :target: https://github.com/MStarmans91/WORC/actions?query=workflow%3A%22Unit+test%22
-.. |image1| image:: https://readthedocs.org/projects/worc/badge/?version=latest
+.. |image2| image:: https://readthedocs.org/projects/worc/badge/?version=latest
    :target: https://worc.readthedocs.io/en/latest/?badge=latest
-.. |image2| image:: https://badge.fury.io/py/WORC.svg
+.. |image3| image:: https://badge.fury.io/py/WORC.svg
    :target: https://badge.fury.io/py/WORC
-.. |image3| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.3840534.svg
-   :target: https://zenodo.org/badge/latestdoi/92295542
 .. |image4| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.3840534.svg
    :target: https://zenodo.org/badge/latestdoi/92295542
+.. |image5| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.3840534.svg
+   :target: https://zenodo.org/badge/latestdoi/92295542

WORC/IOparser/config_WORC.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 
-# Copyright 2016-2020 Biomedical Imaging Group Rotterdam, Departments of
+# Copyright 2016-2023 Biomedical Imaging Group Rotterdam, Departments of
 # Medical Informatics and Radiology, Erasmus MC, Rotterdam, The Netherlands
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -37,14 +37,15 @@ def load_config(config_file_path):
     settings = configparser.ConfigParser()
     settings.read(config_file_path)
 
-    settings_dict = {'ImageFeatures': dict(), 'General': dict(),
-                     'SVMFeatures': dict()}
+    settings_dict = {'Preprocessing': dict(), 'ImageFeatures': dict(), 'General': dict(),
+                     'SVMFeatures': dict(), 'Ensemble': dict(),
+                     'Labels': dict()}
 
     settings_dict['ImageFeatures']['image_type'] =\
         str(settings['ImageFeatures']['image_type'])
 
     settings_dict['General']['FeatureCalculators'] =\
-        [str(item).strip() for item in
+        [str(item).strip('[]') for item in
          settings['General']['FeatureCalculators'].split(',')]
 
     settings_dict['General']['Preprocessing'] =\
@@ -55,5 +56,29 @@ def load_config(config_file_path):
 
     settings_dict['General']['Segmentix'] =\
         settings['General'].getboolean('Segmentix')
+
+    # Settings for ensembling
+    settings_dict['Ensemble']['Method'] =\
+        str(settings['Ensemble']['Method'])
+    settings_dict['Ensemble']['Size'] =\
+        int(settings['Ensemble']['Size'])
+
+   # Label settings
+    settings_dict['Labels']['label_names'] =\
+        [str(item).strip() for item in
+         settings['Labels']['label_names'].split(',')]
+
+    settings_dict['Labels']['modus'] =\
+        str(settings['Labels']['modus'])
+
+    # Whether to use some methods or not
+    settings_dict['General']['ComBat'] =\
+        str(settings['General']['ComBat'])
+
+    settings_dict['General']['Fingerprint'] =\
+        str(settings['General']['Fingerprint'])
 
+    settings_dict['Preprocessing']['Resampling'] =\
+        settings['Preprocessing'].getboolean('Resampling')
+
     return settings_dict

WORC/IOparser/config_io_classifier.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 
-# Copyright 2016-2022 Biomedical Imaging Group Rotterdam, Departments of
+# Copyright 2016-2023 Biomedical Imaging Group Rotterdam, Departments of
 # Medical Informatics and Radiology, Erasmus MC, Rotterdam, The Netherlands
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -93,6 +93,29 @@ def load_config(config_file_path):
         [int(str(item).strip()) for item in
          settings['Featsel']['SelectFromModel_n_trees'].split(',')]
 
+    settings_dict['Featsel']['RFE'] =\
+        settings['Featsel'].getfloat('RFE')
+
+    settings_dict['Featsel']['RFE_lasso_alpha'] =\
+        [float(str(item).strip()) for item in
+         settings['Featsel']['RFE_lasso_alpha'].split(',')]
+
+    settings_dict['Featsel']['RFE_estimator'] =\
+        [str(item).strip() for item in
+         settings['Featsel']['RFE_estimator'].split(',')]
+
+    settings_dict['Featsel']['RFE_n_trees'] =\
+        [int(str(item).strip()) for item in
+         settings['Featsel']['RFE_n_trees'].split(',')]
+
+    settings_dict['Featsel']['RFE_n_features_to_select'] =\
+        [float(str(item).strip()) for item in
+         settings['Featsel']['RFE_n_features_to_select'].split(',')]
+
+    settings_dict['Featsel']['RFE_step'] =\
+        [int(str(item).strip()) for item in
+         settings['Featsel']['RFE_step'].split(',')]
+
     settings_dict['Featsel']['GroupwiseSearch'] =\
         [str(item).strip() for item in
          settings['Featsel']['GroupwiseSearch'].split(',')]
@@ -375,7 +398,7 @@ def load_config(config_file_path):
     settings_dict['CrossValidation']['fixed_seed'] =\
         settings['CrossValidation'].getboolean('fixed_seed')
 
-    # Genetic settings
+    # Label settings
     settings_dict['Labels']['label_names'] =\
         [str(item).strip() for item in
          settings['Labels']['label_names'].split(',')]
@@ -404,6 +427,8 @@ def load_config(config_file_path):
         settings['HyperOptimization'].getboolean('refit_validation_workflows')
     settings_dict['HyperOptimization']['memory'] = \
         str(settings['HyperOptimization']['memory'])
+    settings_dict['HyperOptimization']['fix_random_seed'] = \
+        settings['HyperOptimization'].getboolean('fix_random_seed')
 
     # Settings for SMAC
     settings_dict['SMAC']['use'] =\

WORC/IOparser/config_io_combat.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 
-# Copyright 2016-2020 Biomedical Imaging Group Rotterdam, Departments of
+# Copyright 2016-2023 Biomedical Imaging Group Rotterdam, Departments of
 # Medical Informatics and Radiology, Erasmus MC, Rotterdam, The Netherlands
 #
 # Licensed under the Apache License, Version 2.0 (the "License");