Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version 3.6.3 Release #83

Merged
merged 28 commits into from
Aug 15, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
a467157
Two bugfixes
MStarmans91 Mar 21, 2023
8fbea3e
Facades now detect separate train/test set
MStarmans91 Mar 27, 2023
84e8560
Add some new features, see changelog
MStarmans91 Apr 7, 2023
0ca596a
Fix syntax error
MStarmans91 Apr 7, 2023
73b8cdb
Another syntax fix
MStarmans91 Apr 7, 2023
1d4553a
Bugfixes in RFE
MStarmans91 Apr 19, 2023
c3dcef2
Minor bugfix
MStarmans91 Apr 19, 2023
75277c7
See ChangeLog
MStarmans91 Jun 1, 2023
ece43bc
Update WORCTutorialSimple_unittest_regression.py
MStarmans91 Jun 8, 2023
065f348
Update WORCTutorialSimple_unittest_multiclass.py
MStarmans91 Jun 8, 2023
b460aa5
Update WORCTutorialSimple_unittest_multiclass.py
MStarmans91 Jun 9, 2023
66e6e6f
fix bugs and typos;
lyhyl Jun 20, 2023
35bae2c
See Changelog
MStarmans91 Aug 14, 2023
9d1dbd0
Add facade function on elastix parameter files, documentation updates
MStarmans91 Aug 14, 2023
85d4826
Minor doc updates
MStarmans91 Aug 14, 2023
93cd593
Update documentation
MStarmans91 Aug 14, 2023
ae2ef53
Update faq.rst
MStarmans91 Aug 14, 2023
67067d4
Merge branch 'development' into master
MStarmans91 Aug 14, 2023
3e49286
Merge pull request #81 from lyhyl/master
MStarmans91 Aug 14, 2023
96e867c
Fix bug in statistical test when feature labels are lower case
MStarmans91 Aug 14, 2023
ab2d115
fix bug in Readme RST
MStarmans91 Aug 14, 2023
be84252
Fix issues in Readme
MStarmans91 Aug 14, 2023
3ab8784
Fix bug in statistical test features.
Aug 15, 2023
0fef447
Update requirements.txt
MStarmans91 Aug 15, 2023
eed5ae3
Prepare for 3.6.3 release
MStarmans91 Aug 15, 2023
bc8a1e9
Some more bug fixes in statistical tests
Aug 15, 2023
3717d52
Typo fix
Aug 15, 2023
3d8aaa6
Prepare for 3.6.3 release
MStarmans91 Aug 15, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions CHANGELOG
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,40 @@ All notable changes to this project will be documented in this file.
The format is based on `Keep a Changelog <http://keepachangelog.com/>`_
and this project adheres to `Semantic Versioning <http://semver.org/>`_

3.6.3 - 2023-08-15
------------------

Fixed
~~~~~
- Bug in computing confidence intervals when performance was always the same,
results now in a np.nan confidence interval.
- Error catched and message added when you provide images and/or features in the
test set, but not labels.
- SimpleWORC and BasicWORC now detect whether user has provided a separate training
and test set and thus bootstrapping should be used.
- Bug in PREDICT was fixed that mixed up the mode in shape feature extraction (2D / 2.5D)
- Bug in performance calculation of multiclass classification.
- Bugs in statistical feature testing.

Changed
~~~~~~~
- Statistical test feature selection before PCA: otherwise when combined,
it will select PCA components, not features.

Added
~~~~~
- Histogram equalization to preprocessing.
- Recursive feature elimination (RFE) feature selection.
- Workflow to provide evaluate a trained model on new data
- Option to set a fixed random seed in the hyperoptimization
for reproducibility.
- Various FAQs
- Updated user manual with more extensive debugging guide.
- Thoroughly updated user manual documentation on different data flows, e.g.
train-test setups, multiple segmentations per patient.
- function in SimpleWORC to add parameter files for elastix.


3.6.2 - 2023-03-14
------------------

Expand Down
20 changes: 10 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# WORC v3.6.2
# WORC v3.6.3
## Workflow for Optimal Radiomics Classification

## Information
Expand Down Expand Up @@ -37,12 +37,12 @@ When using WORC, please cite this repository and the paper describing WORC as as

```bibtex
@article{starmans2021reproducible,
title={Reproducible radiomics through automated machine learning validated on twelve clinical applications},
author={Martijn P. A. Starmans and Sebastian R. van der Voort and Thomas Phil and Milea J. M. Timbergen and Melissa Vos and Guillaume A. Padmos and Wouter Kessels and David Hanff and Dirk J. Grunhagen and Cornelis Verhoef and Stefan Sleijfer and Martin J. van den Bent and Marion Smits and Roy S. Dwarkasing and Christopher J. Els and Federico Fiduzi and Geert J. L. H. van Leenders and Anela Blazevic and Johannes Hofland and Tessa Brabander and Renza A. H. van Gils and Gaston J. H. Franssen and Richard A. Feelders and Wouter W. de Herder and Florian E. Buisman and Francois E. J. A. Willemssen and Bas Groot Koerkamp and Lindsay Angus and Astrid A. M. van der Veldt and Ana Rajicic and Arlette E. Odink and Mitchell Deen and Jose M. Castillo T. and Jifke Veenland and Ivo Schoots and Michel Renckens and Michail Doukas and Rob A. de Man and Jan N. M. IJzermans and Razvan L. Miclea and Peter B. Vermeulen and Esther E. Bron and Maarten G. Thomeer and Jacob J. Visser and Wiro J. Niessen and Stefan Klein},
year={2021},
eprint={2108.08618},
archivePrefix={arXiv},
primaryClass={eess.IV}
title = {Reproducible radiomics through automated machine learning validated on twelve clinical applications},
author = {Martijn P. A. Starmans and Sebastian R. van der Voort and Thomas Phil and Milea J. M. Timbergen and Melissa Vos and Guillaume A. Padmos and Wouter Kessels and David Hanff and Dirk J. Grunhagen and Cornelis Verhoef and Stefan Sleijfer and Martin J. van den Bent and Marion Smits and Roy S. Dwarkasing and Christopher J. Els and Federico Fiduzi and Geert J. L. H. van Leenders and Anela Blazevic and Johannes Hofland and Tessa Brabander and Renza A. H. van Gils and Gaston J. H. Franssen and Richard A. Feelders and Wouter W. de Herder and Florian E. Buisman and Francois E. J. A. Willemssen and Bas Groot Koerkamp and Lindsay Angus and Astrid A. M. van der Veldt and Ana Rajicic and Arlette E. Odink and Mitchell Deen and Jose M. Castillo T. and Jifke Veenland and Ivo Schoots and Michel Renckens and Michail Doukas and Rob A. de Man and Jan N. M. IJzermans and Razvan L. Miclea and Peter B. Vermeulen and Esther E. Bron and Maarten G. Thomeer and Jacob J. Visser and Wiro J. Niessen and Stefan Klein},
year = {2021},
eprint = {2108.08618},
archivePrefix = {arXiv},
primaryClass = {eess.IV}
}

@software{starmans2018worc,
Expand Down Expand Up @@ -140,9 +140,9 @@ Make sure you add the executable to the PATH when prompted.
### Elastix
Image registration is included in WORC through [elastix and transformix](http://elastix.isi.uu.nl/).
In order to use elastix, please download the binaries and place them in your
fastr.config.mounts['apps'] path. Check the elastix tool description for the correct
subdirectory structure. For example, on Linux, the binaries and libraries should be in "../apps/elastix/4.8/install/" and
"../apps/elastix/4.8/install/lib" respectively.
``fastr.config.mounts['apps']`` path. Check the elastix tool description for the correct
subdirectory structure. For example, on Linux, the binaries and libraries should be in ``"../apps/elastix/4.8/install/"`` and
``"../apps/elastix/4.8/install/lib"`` respectively.

Note: optionally, you can tell WORC to copy the metadata from the image file
to the segmentation file before applying the deformation field. This requires
Expand Down
97 changes: 49 additions & 48 deletions README.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
WORC v3.6.2
WORC v3.6.3
===========

Workflow for Optimal Radiomics Classification
Expand All @@ -7,11 +7,12 @@ Workflow for Optimal Radiomics Classification
Information
-----------

+---------------------+---------------------+---------------------+---------------+
| Unit test | Documentation | PyPi | Citing WORC |
+=====================+=====================+=====================+===============+
| |image0| | |image1| | |image2| | |image3| |
+---------------------+---------------------+---------------------+---------------+
+-------------------+------------------+------------------+------------+
| Unit test | Documentation | PyPi | Citing |
| | | | WORC |
+===================+==================+==================+============+
| |image1| | |image2| | |image3| | |image4| |
+-------------------+------------------+------------------+------------+

Introduction
============
Expand All @@ -38,25 +39,25 @@ WORC as as follows:

.. code:: bibtex

@article{starmans2021reproducible,
title={Reproducible radiomics through automated machine learning validated on twelve clinical applications},
author={Martijn P. A. Starmans and Sebastian R. van der Voort and Thomas Phil and Milea J. M. Timbergen and Melissa Vos and Guillaume A. Padmos and Wouter Kessels and David Hanff and Dirk J. Grunhagen and Cornelis Verhoef and Stefan Sleijfer and Martin J. van den Bent and Marion Smits and Roy S. Dwarkasing and Christopher J. Els and Federico Fiduzi and Geert J. L. H. van Leenders and Anela Blazevic and Johannes Hofland and Tessa Brabander and Renza A. H. van Gils and Gaston J. H. Franssen and Richard A. Feelders and Wouter W. de Herder and Florian E. Buisman and Francois E. J. A. Willemssen and Bas Groot Koerkamp and Lindsay Angus and Astrid A. M. van der Veldt and Ana Rajicic and Arlette E. Odink and Mitchell Deen and Jose M. Castillo T. and Jifke Veenland and Ivo Schoots and Michel Renckens and Michail Doukas and Rob A. de Man and Jan N. M. IJzermans and Razvan L. Miclea and Peter B. Vermeulen and Esther E. Bron and Maarten G. Thomeer and Jacob J. Visser and Wiro J. Niessen and Stefan Klein},
year={2021},
eprint={2108.08618},
archivePrefix={arXiv},
primaryClass={eess.IV}
}

@software{starmans2018worc,
author = {Martijn P. A. Starmans and Thomas Phil and Sebastian R. van der Voort and Stefan Klein},
title = {Workflow for Optimal Radiomics Classification (WORC)},
year = {2018},
publisher = {Zenodo},
doi = {10.5281/zenodo.3840534},
url = {https://github.com/MStarmans91/WORC}
}

For the DOI, visit |image4|.
@article{starmans2021reproducible,
title = {Reproducible radiomics through automated machine learning validated on twelve clinical applications},
author = {Martijn P. A. Starmans and Sebastian R. van der Voort and Thomas Phil and Milea J. M. Timbergen and Melissa Vos and Guillaume A. Padmos and Wouter Kessels and David Hanff and Dirk J. Grunhagen and Cornelis Verhoef and Stefan Sleijfer and Martin J. van den Bent and Marion Smits and Roy S. Dwarkasing and Christopher J. Els and Federico Fiduzi and Geert J. L. H. van Leenders and Anela Blazevic and Johannes Hofland and Tessa Brabander and Renza A. H. van Gils and Gaston J. H. Franssen and Richard A. Feelders and Wouter W. de Herder and Florian E. Buisman and Francois E. J. A. Willemssen and Bas Groot Koerkamp and Lindsay Angus and Astrid A. M. van der Veldt and Ana Rajicic and Arlette E. Odink and Mitchell Deen and Jose M. Castillo T. and Jifke Veenland and Ivo Schoots and Michel Renckens and Michail Doukas and Rob A. de Man and Jan N. M. IJzermans and Razvan L. Miclea and Peter B. Vermeulen and Esther E. Bron and Maarten G. Thomeer and Jacob J. Visser and Wiro J. Niessen and Stefan Klein},
year = {2021},
eprint = {2108.08618},
archivePrefix = {arXiv},
primaryClass = {eess.IV}
}

@software{starmans2018worc,
author = {Martijn P. A. Starmans and Thomas Phil and Sebastian R. van der Voort and Stefan Klein},
title = {Workflow for Optimal Radiomics Classification (WORC)},
year = {2018},
publisher = {Zenodo},
doi = {10.5281/zenodo.3840534},
url = {https://github.com/MStarmans91/WORC}
}

For the DOI, visit |image5|.

Disclaimer
----------
Expand All @@ -79,16 +80,16 @@ The publicly released WORC database is described in the following paper:

.. code:: bibtex

@article {Starmans2021WORCDatabase,
author = {Starmans, Martijn P.A. and Timbergen, Milea J.M. and Vos, Melissa and Padmos, Guillaume A. and Gr{\"u}nhagen, Dirk J. and Verhoef, Cornelis and Sleijfer, Stefan and van Leenders, Geert J.L.H. and Buisman, Florian E. and Willemssen, Francois E.J.A. and Koerkamp, Bas Groot and Angus, Lindsay and van der Veldt, Astrid A.M. and Rajicic, Ana and Odink, Arlette E. and Renckens, Michel and Doukas, Michail and de Man, Rob A. and IJzermans, Jan N.M. and Miclea, Razvan L. and Vermeulen, Peter B. and Thomeer, Maarten G. and Visser, Jacob J. and Niessen, Wiro J. and Klein, Stefan},
title = {The WORC database: MRI and CT scans, segmentations, and clinical labels for 930 patients from six radiomics studies},
elocation-id = {2021.08.19.21262238},
year = {2021},
doi = {10.1101/2021.08.19.21262238},
URL = {https://www.medrxiv.org/content/early/2021/08/25/2021.08.19.21262238},
eprint = {https://www.medrxiv.org/content/early/2021/08/25/2021.08.19.21262238.full.pdf},
journal = {medRxiv}
}
@article {Starmans2021WORCDatabase,
author = {Starmans, Martijn P.A. and Timbergen, Milea J.M. and Vos, Melissa and Padmos, Guillaume A. and Gr{\"u}nhagen, Dirk J. and Verhoef, Cornelis and Sleijfer, Stefan and van Leenders, Geert J.L.H. and Buisman, Florian E. and Willemssen, Francois E.J.A. and Koerkamp, Bas Groot and Angus, Lindsay and van der Veldt, Astrid A.M. and Rajicic, Ana and Odink, Arlette E. and Renckens, Michel and Doukas, Michail and de Man, Rob A. and IJzermans, Jan N.M. and Miclea, Razvan L. and Vermeulen, Peter B. and Thomeer, Maarten G. and Visser, Jacob J. and Niessen, Wiro J. and Klein, Stefan},
title = {The WORC database: MRI and CT scans, segmentations, and clinical labels for 930 patients from six radiomics studies},
elocation-id = {2021.08.19.21262238},
year = {2021},
doi = {10.1101/2021.08.19.21262238},
URL = {https://www.medrxiv.org/content/early/2021/08/25/2021.08.19.21262238},
eprint = {https://www.medrxiv.org/content/early/2021/08/25/2021.08.19.21262238.full.pdf},
journal = {medRxiv}
}

The code to download the WORC database and reproduce our experiments can
be found at https://github.com/MStarmans91/WORCDatabase.
Expand All @@ -107,19 +108,19 @@ The package can be installed through pip:

::

pip install WORC
pip install WORC

Alternatively, you can directly install WORC from this repository:

::

python setup.py install
python setup.py install

Make sure you install the requirements first:

::

pip install -r requirements.txt
pip install -r requirements.txt

3rd-party packages used in WORC:
--------------------------------
Expand Down Expand Up @@ -172,7 +173,7 @@ sure you install graphviz. On Ubuntu, simply run

::

apt install graphiv
apt install graphiv

On Windows, follow the installation instructions provided on the
graphviz website. Make sure you add the executable to the PATH when
Expand All @@ -184,10 +185,10 @@ Elastix
Image registration is included in WORC through `elastix and
transformix <http://elastix.isi.uu.nl/>`__. In order to use elastix,
please download the binaries and place them in your
fastr.config.mounts['apps'] path. Check the elastix tool description for
the correct subdirectory structure. For example, on Linux, the binaries
and libraries should be in "../apps/elastix/4.8/install/" and
"../apps/elastix/4.8/install/lib" respectively.
``fastr.config.mounts['apps']`` path. Check the elastix tool description
for the correct subdirectory structure. For example, on Linux, the
binaries and libraries should be in ``"../apps/elastix/4.8/install/"``
and ``"../apps/elastix/4.8/install/lib"`` respectively.

Note: optionally, you can tell WORC to copy the metadata from the image
file to the segmentation file before applying the deformation field.
Expand All @@ -205,13 +206,13 @@ to XNAT. We advise you to specify your account settings in a .netrc file
when using this feature for your own datasets, such that you do not need
to input them on every request.

.. |image0| image:: https://github.com/MStarmans91/WORC/workflows/Unit%20test/badge.svg
.. |image1| image:: https://github.com/MStarmans91/WORC/workflows/Unit%20test/badge.svg
:target: https://github.com/MStarmans91/WORC/actions?query=workflow%3A%22Unit+test%22
.. |image1| image:: https://readthedocs.org/projects/worc/badge/?version=latest
.. |image2| image:: https://readthedocs.org/projects/worc/badge/?version=latest
:target: https://worc.readthedocs.io/en/latest/?badge=latest
.. |image2| image:: https://badge.fury.io/py/WORC.svg
.. |image3| image:: https://badge.fury.io/py/WORC.svg
:target: https://badge.fury.io/py/WORC
.. |image3| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.3840534.svg
:target: https://zenodo.org/badge/latestdoi/92295542
.. |image4| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.3840534.svg
:target: https://zenodo.org/badge/latestdoi/92295542
.. |image5| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.3840534.svg
:target: https://zenodo.org/badge/latestdoi/92295542
33 changes: 29 additions & 4 deletions WORC/IOparser/config_WORC.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env python

# Copyright 2016-2020 Biomedical Imaging Group Rotterdam, Departments of
# Copyright 2016-2023 Biomedical Imaging Group Rotterdam, Departments of
# Medical Informatics and Radiology, Erasmus MC, Rotterdam, The Netherlands
#
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down Expand Up @@ -37,14 +37,15 @@ def load_config(config_file_path):
settings = configparser.ConfigParser()
settings.read(config_file_path)

settings_dict = {'ImageFeatures': dict(), 'General': dict(),
'SVMFeatures': dict()}
settings_dict = {'Preprocessing': dict(), 'ImageFeatures': dict(), 'General': dict(),
'SVMFeatures': dict(), 'Ensemble': dict(),
'Labels': dict()}

settings_dict['ImageFeatures']['image_type'] =\
str(settings['ImageFeatures']['image_type'])

settings_dict['General']['FeatureCalculators'] =\
[str(item).strip() for item in
[str(item).strip('[]') for item in
settings['General']['FeatureCalculators'].split(',')]

settings_dict['General']['Preprocessing'] =\
Expand All @@ -55,5 +56,29 @@ def load_config(config_file_path):

settings_dict['General']['Segmentix'] =\
settings['General'].getboolean('Segmentix')

# Settings for ensembling
settings_dict['Ensemble']['Method'] =\
str(settings['Ensemble']['Method'])
settings_dict['Ensemble']['Size'] =\
int(settings['Ensemble']['Size'])

# Label settings
settings_dict['Labels']['label_names'] =\
[str(item).strip() for item in
settings['Labels']['label_names'].split(',')]

settings_dict['Labels']['modus'] =\
str(settings['Labels']['modus'])

# Whether to use some methods or not
settings_dict['General']['ComBat'] =\
str(settings['General']['ComBat'])

settings_dict['General']['Fingerprint'] =\
str(settings['General']['Fingerprint'])

settings_dict['Preprocessing']['Resampling'] =\
settings['Preprocessing'].getboolean('Resampling')

return settings_dict
29 changes: 27 additions & 2 deletions WORC/IOparser/config_io_classifier.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env python

# Copyright 2016-2022 Biomedical Imaging Group Rotterdam, Departments of
# Copyright 2016-2023 Biomedical Imaging Group Rotterdam, Departments of
# Medical Informatics and Radiology, Erasmus MC, Rotterdam, The Netherlands
#
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down Expand Up @@ -93,6 +93,29 @@ def load_config(config_file_path):
[int(str(item).strip()) for item in
settings['Featsel']['SelectFromModel_n_trees'].split(',')]

settings_dict['Featsel']['RFE'] =\
settings['Featsel'].getfloat('RFE')

settings_dict['Featsel']['RFE_lasso_alpha'] =\
[float(str(item).strip()) for item in
settings['Featsel']['RFE_lasso_alpha'].split(',')]

settings_dict['Featsel']['RFE_estimator'] =\
[str(item).strip() for item in
settings['Featsel']['RFE_estimator'].split(',')]

settings_dict['Featsel']['RFE_n_trees'] =\
[int(str(item).strip()) for item in
settings['Featsel']['RFE_n_trees'].split(',')]

settings_dict['Featsel']['RFE_n_features_to_select'] =\
[float(str(item).strip()) for item in
settings['Featsel']['RFE_n_features_to_select'].split(',')]

settings_dict['Featsel']['RFE_step'] =\
[int(str(item).strip()) for item in
settings['Featsel']['RFE_step'].split(',')]

settings_dict['Featsel']['GroupwiseSearch'] =\
[str(item).strip() for item in
settings['Featsel']['GroupwiseSearch'].split(',')]
Expand Down Expand Up @@ -375,7 +398,7 @@ def load_config(config_file_path):
settings_dict['CrossValidation']['fixed_seed'] =\
settings['CrossValidation'].getboolean('fixed_seed')

# Genetic settings
# Label settings
settings_dict['Labels']['label_names'] =\
[str(item).strip() for item in
settings['Labels']['label_names'].split(',')]
Expand Down Expand Up @@ -404,6 +427,8 @@ def load_config(config_file_path):
settings['HyperOptimization'].getboolean('refit_validation_workflows')
settings_dict['HyperOptimization']['memory'] = \
str(settings['HyperOptimization']['memory'])
settings_dict['HyperOptimization']['fix_random_seed'] = \
settings['HyperOptimization'].getboolean('fix_random_seed')

# Settings for SMAC
settings_dict['SMAC']['use'] =\
Expand Down
2 changes: 1 addition & 1 deletion WORC/IOparser/config_io_combat.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env python

# Copyright 2016-2020 Biomedical Imaging Group Rotterdam, Departments of
# Copyright 2016-2023 Biomedical Imaging Group Rotterdam, Departments of
# Medical Informatics and Radiology, Erasmus MC, Rotterdam, The Netherlands
#
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down
Loading
Loading