Update main branch for Release 2.2.0

Merge all changes from "develop" into "main" to prepare the main branch for the next release

Author: tennlee

.github/workflows/python-app.yml

@@ -28,7 +28,7 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         pip install .[all]
-    - name: Test with pytest
+    - name: Test with pytest, with dask installed
       run: |
         # editable is necessary as pytest will run against the installed
         # package rather than the local files creating a coverage report of 0%
@@ -37,6 +37,13 @@ jobs:
     - name: Test tutorial notebooks with pytest
       run: |
         pytest --nbmake tutorials/
+    - name: Test again with without dask installed
+      run: |
+        # editable is necessary as pytest will run against the installed
+        # package rather than the local files creating a coverage report of 0%
+        pip install -e .
+        pip uninstall dask -y
+        pytest tests/ 
     - name: Coveralls GitHub Action
       uses: coverallsapp/[email protected]
       with:

.pre-commit-config.yaml

@@ -1,13 +1,13 @@
 ---
 repos:
   - repo: https://github.com/psf/black
-    rev: 24.1.0
+    rev: 24.10.0
     hooks:
       - id: black
         name: black
         entry: black --check
   - repo: https://github.com/PyCQA/bandit
-    rev: 1.7.8
+    rev: 1.8.0
     hooks:
       - id: bandit
         args: [--skip, "B101", --recursive, src/scores]
@@ -25,10 +25,6 @@ repos:
     hooks:
       - id: isort
         entry: isort --profile=black
-  - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.13.0
-    hooks:
-      - id: mypy
   - repo: local
     hooks:
       - id: pytest

README.md

@@ -18,12 +18,13 @@ Below is a **curated selection** of the metrics, tools and statistical tests inc
 
 |                       	| **Description** 	| **Selection of Included Functions** 	|
 |-----------------------	|-----------------	|--------------	|
-| **[Continuous](https://scores.readthedocs.io/en/stable/included.html#continuous)**        	|Scores for evaluating single-valued continuous forecasts.                  	|MAE, MSE, RMSE, Additive Bias, Multiplicative Bias, Percent Bias, Pearson's Correlation Coefficient, Kling-Gupta Efficiency, Flip-Flop Index, Quantile Loss, Quantile Interval Score, Interval Score, Murphy Score, and threshold weighted scores for expectiles, quantiles and Huber Loss.             	|
-| **[Probability](https://scores.readthedocs.io/en/stable/included.html#probability)**        |Scores for evaluating forecasts that are expressed as predictive distributions, ensembles, and probabilities of binary events.                   |Brier Score, Continuous Ranked Probability Score (CRPS) for Cumulative Density Functions (CDF) and ensembles (including threshold weighted versions), Receiver Operating Characteristic (ROC), Isotonic Regression (reliability diagrams).               |
+| **[Continuous](https://scores.readthedocs.io/en/stable/included.html#continuous)**        	|Scores for evaluating single-valued continuous forecasts.                  	|MAE, MSE, RMSE, Additive Bias, Multiplicative Bias, Percent Bias, Pearson's Correlation Coefficient, Spearman's Correlation Coefficient, Kling-Gupta Efficiency, Flip-Flop Index, Quantile Loss, Quantile Interval Score, Interval Score, and threshold weighted scores for expectiles, quantiles and Huber Loss.             	|
+| **[Probability](https://scores.readthedocs.io/en/stable/included.html#probability)**        |Scores for evaluating forecasts that are expressed as predictive distributions, ensembles, and probabilities of binary events.                   |Brier Score, Continuous Ranked Probability Score (CRPS) for Cumulative Density Functions (CDF) and ensembles (including threshold weighted versions), and Isotonic Regression (reliability diagrams).               |
 | **[Categorical](https://scores.readthedocs.io/en/stable/included.html#categorical)**       	|Scores for evaluating forecasts of categories.                	|18 binary contingency table (confusion matrix) metrics, the FIxed Risk Multicategorical (FIRM) Score, and the SEEPS score.               	|
 | **[Spatial](https://scores.readthedocs.io/en/stable/included.html#spatial)** 	|Scores that take into account spatial structure.                 	|Fractions Skill Score.              	|
 | **[Statistical Tests](https://scores.readthedocs.io/en/stable/included.html#statistical-tests)** 	|Tools to conduct statistical tests and generate confidence intervals.                 	|Diebold Mariano.              	|
 | **[Processing Tools](https://scores.readthedocs.io/en/stable/included.html#processing-tools-for-preparing-data)**        	|Tools to pre-process data.                 	|Data matching, Discretisation, Block Bootstrapping, and Cumulative Density Function Manipulation.              	|
+| **[Plotting Data](https://scores.readthedocs.io/en/stable/included.html#plotting-data)**        	|Tools to generate data for plotting.               	|ROC curves, Murphy diagrams, and Q-Q plots.             	|
 | **[Emerging](https://scores.readthedocs.io/en/stable/included.html#emerging)**        	|Emerging scores that are still undergoing mathematical peer review. They may change in line with the peer review process.                 	|Risk Matrix Score.             	|
 
 `scores` not only includes common scores (e.g., MAE, RMSE), it also includes novel scores not commonly found elsewhere (e.g., FIRM, Flip-Flop Index), complex scores (e.g., threshold weighted CRPS), and statistical tests (e.g., the Diebold Mariano test). Additionally, it provides pre-processing tools for preparing data for scores in a variety of formats including cumulative distribution functions (CDF). `scores` provides its own implementations where relevant to avoid extensive dependencies.

docs/api.md

@@ -21,6 +21,7 @@
 .. autofunction:: scores.continuous.flip_flop_index
 .. autofunction:: scores.continuous.flip_flop_index_proportion_exceeding
 .. autofunction:: scores.continuous.correlation.pearsonr
+.. autofunction:: scores.continuous.correlation.spearmanr
 .. autofunction:: scores.continuous.multiplicative_bias
 .. autofunction:: scores.continuous.pbias
 .. autofunction:: scores.continuous.kge
@@ -36,6 +37,7 @@
 .. autofunction:: scores.continuous.tw_expectile_score
 .. autofunction:: scores.continuous.quantile_interval_score
 .. autofunction:: scores.continuous.interval_score
+.. autofunction:: scores.continuous.qq
 ```
 
 ## scores.probability
@@ -106,6 +108,14 @@
 .. autofunction:: scores.processing.cdf.cdf_envelope
 ```
 
+## scores.plotdata
+```{eval-rst}
+.. autofunction:: scores.plotdata.murphy_score
+.. autofunction:: scores.plotdata.murphy_thetas
+.. autofunction:: scores.plotdata.qq
+.. autofunction:: scores.plotdata.roc
+```
+
 ## scores.pandas
 ```{eval-rst}
 .. autofunction:: scores.pandas.continuous.mse

docs/conf.py

@@ -9,7 +9,7 @@
 
 project = "scores"
 copyright = "Licensed under Apache 2.0 - https://www.apache.org/licenses/LICENSE-2.0"
-release = "2.1.0"
+release = "2.2.0"
 
 version = __version__
 

docs/included.md

@@ -125,6 +125,10 @@
   - [API](api.md#scores.continuous.quantile_score)
   - [Tutorial](project:./tutorials/Quantile_Loss.md)
   - [Gneiting (2011) - Theorem 9](https://doi.org/10.1198/jasa.2011.r10138)
+* - Quantile-Quantile (QQ) Plots
+  - [API](api.md#scores.continuous.qq)
+  - [Tutorial](project:./tutorials/Q-Q_plots.md)
+  - [Déqué (2011)](https://doi.org/10.1002/9781119960003.ch5)
 * - Quantile Score, *see Quantile Loss*
   - —
   - —
@@ -137,6 +141,10 @@
   - [API](api.md#scores.continuous.rmse)
   - [Tutorial](project:./tutorials/Root_Mean_Squared_Error.md)
   - [Wikipedia](https://en.wikipedia.org/wiki/Root-mean-square_deviation)
+* - Spearman's Correlation Coefficient
+  - [API](api.md#scores.continuous.correlation.spearmanr)
+  - [Tutorial](project:./tutorials/Spearmans_Correlation.md)
+  - [Spearman (1904)](https://doi.org/10.2307/1412159)
 * - Threshold Weighted Absolute Error
   - [API](api.md#scores.continuous.tw_absolute_error)
   - [Tutorial](project:./tutorials/Threshold_Weighted_Scores.md)
@@ -731,6 +739,44 @@
   - —   
 ```
 
+## Plotting Data
+
+```{list-table}
+:header-rows: 1
+
+* - Name (Alphabetical order)
+  - API
+  - Tutorial
+  - Reference(s)
+* - Murphy Diagrams (Plots of the Murphy Score)
+  - 
+  - 
+  - 
+* -  
+    - Murphy Score (Mean Elementary Score)
+  -
+    [API](api.md#scores.plotdata.murphy_score)
+  - 
+    [Tutorial](project:./tutorials/Murphy_Diagrams.md)
+  - 
+    [Ehm et al. (2016) - Theorem 1](https://doi.org/10.1111/rssb.12154); [Taggart (2022) - Theorem 5.3](https://doi.org/10.1214/21-ejs1957)
+* -  
+    - Murphy Score (Mean Elementary Score) - Theta Values
+  -
+    [API](api.md#scores.plotdata.murphy_thetas)
+  - 
+    [Tutorial](project:./tutorials/Murphy_Diagrams.md)
+  - 
+    [Ehm et al. (2016) - Corollary 2 (p.521)](https://doi.org/10.1111/rssb.12154); [Taggart (2022) - Corollary 5.6](https://doi.org/10.1214/21-ejs1957)
+* - Quantile-Quantile (QQ) Plots
+  - [API](api.md#scores.plotdata.qq)
+  - [Tutorial](project:./tutorials/Q-Q_plots.md)
+  - [Déqué (2011)](https://doi.org/10.1002/9781119960003.ch5)
+* - Receiver (Relative) Operating Characteristic (ROC)
+  - [API](api.md#scores.plotdata.roc)   
+  - [Tutorial](project:./tutorials/ROC.md)
+  - [Fawcett and Niculescu-Mizil (2007)](https://doi.org/10.1007/s10994-007-5011-0); [Gneiting and Vogel (2022)](https://doi.org/10.1007/s10994-021-06115-2); [Hand (2009)](https://doi.org/10.1007/s10994-009-5119-5); [Hand and Anagnostopoulos (2013)](https://doi.org/10.1016/j.patrec.2012.12.004)); [Hand and Anagnostopoulos (2023)](https://doi.org/10.1007/s11634-021-00490-3); [Pesce et al. (2010)](https://doi.org/10.1016/j.acra.2010.04.001)
+```
 ## Pandas
 
 ```{list-table}

docs/maintainer.md

@@ -9,6 +9,9 @@ Information relevant for package maintenance
 2. In the full changelog URL, change “develop” to the correct version number, i.e (https://github.com/nci/scores/compare/(X-1).(Y-1).(Z-1)...X.Y.Z)
 3. Delete any unused headers
 
+### Immediately before making a release, validate the .zenodo.json file
+1. Run the .zenodo.json file contents through a JSON validator 
+
 ### On the pre-release branch
 1. Run through some of the notebooks manually
 2. Run the unit tests
@@ -48,9 +51,10 @@ Information relevant for package maintenance
 ### Confirm Zenodo correctness
 
 Link to Zenodo archive: [https://doi.org/10.5281/zenodo.12697241](https://doi.org/10.5281/zenodo.12697241)
-1. Confirm license
-2. Confirm authors
+1. Confirm license is correct
+2. Confirm author list is correct
 3. Scan everything else
+4. Edit the metadata - add the introductory paragraph describing `scores`, and the links to the documentation, source code and journal article.
 
 ## This section covers how to format and prepare release notes
 
@@ -61,9 +65,9 @@ Link to Zenodo archive: [https://doi.org/10.5281/zenodo.12697241](https://doi.or
 
 For a list of all changes in this release, see the [full changelog](https://github.com/nci/scores/compare/(X-1).(Y-1).(Z-1)...X.Y.Z). Below are the changes we think users may wish to be aware of.
 
-### Features
 ### Breaking Changes
 ### Deprecations
+### Features
 ### Bug Fixes
 ### Documentation
 ### Internal Changes
@@ -98,6 +102,8 @@ In .zenodo.json, please add your details at the bottom of the “creators” sec
 3. “name”. Format: surname, given name(s).
 ```
 
+Each time the .zenodo.json file is updated, run the file contents through a JSON validator.
+
 ## This section gives guidance for maintaining compatibility with old versions of Python and packages
 
 tldr; about 3 years old is OK, longer if painless