Refactor to use confidence scores instead of uncertainty score

CHANGELOG.md

@@ -1,4 +1,15 @@
-# seapig v0.3.0 (dev)
+# seapig v0.4.0 (dev)
+
+## Breaking changes
+
+- we refactored the library to refere to ConfidenceScores instead of UncertaintyScores to better match the 
+  wording and customs in the selective prediction literature. This means that scores are expected to return values
+  representing confidence, e.g., a low value indicates low confidence and a high value indicates high confidence.
+  Selected samples are now the samples with a confidence score equal to or above the threshold.
+  This change means that all scores now return different values. KNN scores, for example, now return `-1 * score_value`
+  to match the confidence pattern.
+
+# seapig v0.3.0
 
 ## Features
 

README.md

@@ -11,10 +11,10 @@ Version](https://img.shields.io/conda/vn/conda-forge/seapig.svg)](https://anacon
 
 ------------------------------------------------------------------------
 
-seapig provides uncertainty-based selective inference for deep learning
+seapig provides confidence-based selective inference for deep learning
 models. Its main focus currently lies on analyzing latent
 representations. The library implements a small set of lightweight,
-composable uncertainty scores that are used to decide whether to accept
+composable confidence scores that are used to decide whether to accept
 or reject an individual query sample at prediction time. Thresholds are
 calibrated on an independent validation set. It provides a wrapper for
 torchmetrics that allows evaluating the performance of a selective
@@ -70,32 +70,43 @@ The core idea is to compute a representation for each input, score how
 similar the representation is to training representations, and reject
 inputs whose score indicates low support.
 
-### From uncertainty scores to selective inference
+### From confidence scores to selective inference
 
-All uncertainty scores produce a scalar score $s(x)$ for each query $x$.
+Our implementation is based on the original work of Chow (1957) [^1] and
+El-Yaniv and Wiener (2010) [^2], adapted for deep learning in approaches
+in Geifman and El-Yaniv (2017) [^3] and recently extended by Jäger et
+al. (2023) [^4] and Traub et al. (2024) [^5]. A confidence score
+produces a scalar value $s(x) \in \mathbb{R}^1$ for each query $x$.
 Given a threshold $\lambda$, we derive a binary selection function
-indicating which samples to accept. For example, accepting samples with
-score below $\lambda$:
+indicating which samples to accept. By convention, we accept samples
+with a confidence score equal to or higher than $\lambda$ indicated by
+the decision function $g(x)$:
 
-$$g_{\lambda}(x) = \mathbf{1}\{s(x) \le \lambda\}.$$
+$$
+g_{\lambda}(x) = \mathbf{1}\{s(x) \ge \lambda\}.
+$$
 
-We recommend calibrating $\lambda$ on an independent calibration set to
-fix a desired coverage level (fraction of accepted samples) and compute
-the correspoding empirical quantile $q$ of the calibration scores:
+We calibrate $\lambda$ on an independent calibration set to fix a
+desired coverage level (fraction of accepted samples) and compute the
+correspoding empirical quantile $q$ of the calibration scores:
 
-$$\lambda_{q} = Q_q(s_1^{cal}, s_2^{cal}, \dots, s_m^{cal}),$$
+$$
+\lambda_{q} = Q_q(s_1^{cal}, s_2^{cal}, \dots, s_m^{cal}),
+$$
 
 where $s_i^{cal}$ are the scores of the calibration samples. The
 decision function $g_{\lambda}(x)$ can then be applied at inference time
 to accept or reject predictions. We obtain a selective predictor,
 $h(x)$, that either produces an output or abstains from prediction,
-depending on the score of the input:
+depending on the output of the decision function:
 
-$$h(x) =
+$$
+h(x) =
 \begin{cases}
-f(x), & \text{if} g(x)=1,\\
-\varnothing, & \text{if } g(x)=0.
-\end{cases}$$
+f(x), & \text{if} \; g(x)=1,\\
+\textit{abstain}, & \text{if} \; g(x)=0.
+\end{cases}
+$$
 
 ### How to use seapig
 
@@ -128,13 +139,13 @@ ref_emb, cal_emb, query_emb = (
 
 score = EuclideanScore(k=5, stat="mean")
 score.fit(ref=ref_emb, cal=cal_emb)
-score.set_threshold(q=0.90)  # keep ~90% coverage on validation set
+score.set_threshold(q=0.10)  # keep ~90% coverage on validation set
 sel = score.select(query=query_emb)
 print(sel)
 ```
 
-    {'score': tensor([6.2651, 5.5944, 6.0226, 5.8903, 6.2928, 4.8388, 5.7290, 5.3641, 5.6599,
-            5.9143]), 'selected': tensor([ True,  True,  True,  True, False,  True,  True,  True,  True,  True])}
+    {'score': tensor([-6.2651, -5.5944, -6.0226, -5.8903, -6.2928, -4.8388, -5.7290, -5.3641,
+            -5.6599, -5.9143]), 'selected': tensor([ True,  True,  True,  True, False,  True,  True,  True,  True,  True])}
 
 #### On-the-fly embedding extraction
 
@@ -170,14 +181,14 @@ model = Model()
 
 score = EuclideanScore(k=3)
 score.fit(model=model, loaders={"train": train_loader, "cal": val_loader})
-score.set_threshold(q=0.80)  # keep ~80% coverage on validation set
+score.set_threshold(q=0.20)  # keep ~80% coverage on validation set
 
 sel = score.select(model=model, loader=test_loader)
 print(sel)
 ```
 
-    {'score': tensor([5.6522, 6.6245, 5.5196, 5.1725, 5.8717, 5.6051, 5.3821, 5.2224, 5.0285,
-            5.8041]), 'selected': tensor([ True, False,  True,  True,  True,  True,  True,  True,  True,  True])}
+    {'score': tensor([-4.9752, -6.4078, -5.0600, -4.8422, -5.5520, -4.6794, -4.9697, -4.5601,
+            -4.8857, -5.6617]), 'selected': tensor([ True, False,  True,  True,  True,  True,  True,  True,  True,  True])}
 
 #### Using SelectiveInferenceTask with a lightning module
 
@@ -226,8 +237,8 @@ print(preds)
 ┃<span style="font-weight: bold">        Test metric        </span>┃<span style="font-weight: bold">       DataLoader 0        </span>┃
 ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
 │<span style="color: #008080; text-decoration-color: #008080">    full/BinaryAccuracy    </span>│<span style="color: #800080; text-decoration-color: #800080">            0.5            </span>│
-│<span style="color: #008080; text-decoration-color: #008080">  rejected/BinaryAccuracy  </span>│<span style="color: #800080; text-decoration-color: #800080">            0.0            </span>│
-│<span style="color: #008080; text-decoration-color: #008080">  selected/BinaryAccuracy  </span>│<span style="color: #800080; text-decoration-color: #800080">    0.5555555820465088     </span>│
+│<span style="color: #008080; text-decoration-color: #008080">  rejected/BinaryAccuracy  </span>│<span style="color: #800080; text-decoration-color: #800080">    0.6666666865348816     </span>│
+│<span style="color: #008080; text-decoration-color: #008080">  selected/BinaryAccuracy  </span>│<span style="color: #800080; text-decoration-color: #800080">    0.4285714328289032     </span>│
 └───────────────────────────┴───────────────────────────┘
 </pre>
 
@@ -238,8 +249,8 @@ print(preds)
 <pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"></pre>
 
     [{'prediction': tensor([0.3412, 0.6198, 0.8258, 0.8317, 0.8443, 0.0064, 0.4464, 0.5923, 0.1485,
-            0.4144]), 'score': tensor([5.0690, 5.0864, 6.4270, 5.6630, 5.2806, 5.5488, 5.2757, 5.8282, 5.4030,
-            5.9053]), 'selected': tensor([ True,  True, False,  True,  True,  True,  True,  True,  True,  True])}]
+            0.4144]), 'score': tensor([-4.7936, -4.8083, -6.3397, -4.3409, -5.1468, -5.2406, -4.8781, -5.3137,
+            -5.1822, -5.3534]), 'selected': tensor([ True,  True, False,  True,  True,  True,  True,  True,  True,  True])}]
 
 #### Available scores
 
@@ -279,3 +290,25 @@ This research was funded in the course of [TRR 391 Spatio-temporal
 Statistics for the Transition of Energy and
 Transport](https://trr391.tu-dortmund.de/) (520388526) by the Deutsche
 Forschungsgemeinschaft (DFG, German Research Foundation).
+
+[^1]: Chow, C.K., 1957. An optimum character recognition system using
+    decision functions. IRE Transactions on Electronic Computers EC-6,
+    247–254. https://doi.org/10.1109/TEC.1957.5222035
+
+[^2]: El-Yaniv, R., Wiener, Y., 2010. On the Foundations of Noise-free
+    Selective Classification. Journal of Machine Learning Research 11,
+    1605–1641.
+
+[^3]: Geifman, Y., El-Yaniv, R., 2017. Selective classification for deep
+    neural networks, in: Proceedings of the 31st International
+    Conference on Neural Information Processing Systems, NIPS’17. Curran
+    Associates Inc., Red Hook, NY, USA, pp. 4885–4894.
+
+[^4]: Jaeger, P.F., Lüth, C.T., Klein, L., Bungert, T.J., 2023. A Call
+    to Reflect on Evaluation Practices for Failure Detection in Image
+    Classification. https://doi.org/10.48550/arXiv.2211.15259
+
+[^5]: Traub, J., Bungert, T.J., Lüth, C.T., Baumgartner, M., Maier-Hein,
+    K.H., Maier-Hein, L., Jäger, P.F., 2024. Overcoming Common Flaws in
+    the Evaluation of Selective Classification Systems. Advances in
+    Neural Information Processing Systems 37, 2323–2347.

docs/02-quickstart.qmd

@@ -9,7 +9,7 @@ tags: [Getting Started, Setup]
 Machine learning models for earth observation often fail in production when they 
 encounter new tiles, different sensors, seasonal changes, or unfamiliar land-cover 
 patterns. `seapig` helps you manage this uncertainty by enabling 
-**selective inference**: you can flag uncertain predictions for human review, 
+**selective inference**: you can flag low confidence predictions for human review, 
 skip them, or route them to more expensive downstream workflows.
 
 The `seapig` library provides modular building blocks to score, calibrate, and 
@@ -280,23 +280,23 @@ trainer.fit(model, train_dl, val_dl)
 
 ## Step 4: Fit and calibrate a score {#sec-step-4}
 
-Here's where `seapig` comes in. A **selection score** quantifies how uncertain 
+Here's where `seapig` comes in. A **selection score** quantifies how confident 
 the model is in each prediction. We use `CosineScore`, which measures the 
 distance between a query embedding and its nearest neighbor in the training 
-embedding space. Lower scores indicate that a sample is similar to training data
-(lower uncertainty), while higher scores indicate the model has encountered 
-something unfamiliar (higher uncertainty).
+embedding space. Higher scores indicate that a sample is similar to training data
+(high confidence), while lower scores indicate the model has encountered 
+something unfamiliar (low confidence).
 
 We fit the score on both training and validation data, so it learns what "normal" 
-and "unusual" look like. Then we set a threshold at the 80th percentile of the 
+and "unusual" look like. Then we set a threshold at the 20th percentile of the 
 validation score distribution. This means the model will flag roughly the top 
-20% most uncertain predictions:
+20% lowest confident predictions:
 
 ```{python fit-score}
 #| warning: false
 score = CosineScore(k=1)
 score.fit(model=model, loaders={"train": train_dl, "cal": val_dl})
-score.set_threshold(q=0.80)
+score.set_threshold(q=0.20)
 print(f"Threshold: {score.threshold}")
 ```
 

great-docs.yml

@@ -66,8 +66,8 @@ reference:
           - get_curve
       - RiskCoverage
 
-  - title: Embedding based Uncertainty Scores
-    desc: Uncertainty scoring methods for embedding-based approaches
+  - title: Embedding based Confidence Scores
+    desc: Confidence scoring methods for embedding-based approaches
     contents:
       - name: scores.CosineScore
         members:
@@ -143,8 +143,8 @@ reference:
           - plot_embs
       - scores.utils.TensorPCA
 
-  - title: Logit based Uncertainty Scores
-    desc: Uncertainty scoring methods based on model logits
+  - title: Logit based Confidence Scores
+    desc: Confidence scoring methods based on model logits
     contents:
       - name: scores.EnergyScore
         members:
@@ -194,8 +194,8 @@ reference:
           - set_threshold
           - get_threshold
           - plot
-  - title: Other Uncertainty Scores
-    desc: Uncertainty scoring methods based on other principles
+  - title: Other Confidence Scores
+    desc: Confidence scoring methods based on other principles
     contents:
       - name: scores.RandomScore
         members:
@@ -206,7 +206,7 @@ reference:
   - title: Abstract Classes
     desc: Abstract base classes
     contents:
-      - name: scores.UncertaintyScore
+      - name: scores.ConfidenceScore
       - scores.EmbeddingScore
       - scores.KNNScore
       - scores.LogitScore

index.qmd

@@ -7,9 +7,9 @@
 
 ---
 
-seapig provides uncertainty-based selective inference for deep learning
+seapig provides confidence-based selective inference for deep learning
 models. Its main focus currently lies on analyzing latent representations. 
-The library implements a small set of lightweight, composable uncertainty scores 
+The library implements a small set of lightweight, composable confidence scores 
 that are used to decide whether to accept or reject an individual query sample 
 at prediction time. Thresholds are calibrated on an independent validation
 set. It provides a wrapper for torchmetrics that allows evaluating the performance 
@@ -63,18 +63,22 @@ The core idea is to compute a representation for each input, score how
 similar the representation is to training representations, and reject inputs 
 whose score indicates low support.
 
-### From uncertainty scores to selective inference
+### From confidence scores to selective inference
 
-All uncertainty scores produce a scalar score $s(x)$ for each query
+Our implementation is based on the original work of Chow (1957) [^1] and El-Yaniv
+and Wiener (2010) [^2], adapted for deep learning in approaches in Geifman and El-Yaniv 
+(2017) [^3] and recently extended by Jäger et al. (2023) [^4] and Traub et al. (2024) [^5]. 
+A confidence score produces a scalar value $s(x) \in \mathbb{R}^1$ for each query
 $x$. Given a threshold $\lambda$, we derive a binary selection function 
-indicating which samples to accept. For example, accepting
-samples with score below $\lambda$:
+indicating which samples to accept. By convention, we accept samples with 
+a confidence score equal to or higher than $\lambda$ indicated by the decision 
+function $g(x)$:
 
 $$
-g_{\lambda}(x) = \mathbf{1}\{s(x) \le \lambda\}.
+g_{\lambda}(x) = \mathbf{1}\{s(x) \ge \lambda\}.
 $$
 
-We recommend calibrating $\lambda$ on an independent calibration set to fix a 
+We calibrate $\lambda$ on an independent calibration set to fix a 
 desired coverage level (fraction of accepted samples) and compute the
 correspoding empirical quantile $q$ of the calibration scores:
 
@@ -85,13 +89,13 @@ $$
 where $s_i^{cal}$ are the scores of the calibration samples. The decision
 function $g_{\lambda}(x)$ can then be applied at inference time to accept or reject
 predictions. We obtain a selective predictor, $h(x)$, that either produces an output 
-or abstains from prediction, depending on the score of the input:
+or abstains from prediction, depending on the output of the decision function:
 
 $$
 h(x) =
 \begin{cases}
-f(x), & \text{if} g(x)=1,\\
-\varnothing, & \text{if } g(x)=0.
+f(x), & \text{if} \; g(x)=1,\\
+\textit{abstain}, & \text{if} \; g(x)=0.
 \end{cases}
 $$
 
@@ -122,7 +126,7 @@ ref_emb, cal_emb, query_emb = torch.randn(1000, 32), torch.randn(200, 32), torch
 
 score = EuclideanScore(k=5, stat="mean")
 score.fit(ref=ref_emb, cal=cal_emb)
-score.set_threshold(q=0.90)   # keep ~90% coverage on validation set
+score.set_threshold(q=0.10)   # keep ~90% coverage on validation set
 sel = score.select(query=query_emb)
 print(sel)
 ```
@@ -161,7 +165,7 @@ model = Model()
 
 score = EuclideanScore(k=3)
 score.fit(model=model, loaders={"train": train_loader, "cal": val_loader})
-score.set_threshold(q=0.80) # keep ~80% coverage on validation set
+score.set_threshold(q=0.20) # keep ~80% coverage on validation set
 
 sel = score.select(model=model, loader=test_loader)
 print(sel)
@@ -240,3 +244,11 @@ By participating, you are expected to uphold this code.
 This research was funded in the course of [TRR 391 Spatio-temporal Statistics
 for the Transition of Energy and Transport](https://trr391.tu-dortmund.de/) (520388526) 
 by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation).
+
+
+[^1]: Chow, C.K., 1957. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers EC-6, 247–254. https://doi.org/10.1109/TEC.1957.5222035
+[^2]: El-Yaniv, R., Wiener, Y., 2010. On the Foundations of Noise-free Selective Classification. Journal of Machine Learning Research 11, 1605–1641.
+[^3]: Geifman, Y., El-Yaniv, R., 2017. Selective classification for deep neural networks, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17. Curran Associates Inc., Red Hook, NY, USA, pp. 4885–4894.
+[^4]: Jaeger, P.F., Lüth, C.T., Klein, L., Bungert, T.J., 2023. A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification. https://doi.org/10.48550/arXiv.2211.15259
+[^5]: Traub, J., Bungert, T.J., Lüth, C.T., Baumgartner, M., Maier-Hein, K.H., Maier-Hein, L., Jäger, P.F., 2024. Overcoming Common Flaws in the Evaluation of Selective Classification Systems. Advances in Neural Information Processing Systems 37, 2323–2347.
+

pyproject.toml

@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
 # https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
 [project]
 name = "seapig"
-description = "Selective prediction based on uncertainty scores"
+description = "Selective prediction based on confidence scores"
 readme = { file = "README.md", content-type = "text/markdown" }
 requires-python = ">=3.11"
 license = "MIT"
@@ -18,7 +18,7 @@ authors = [
 maintainers = [
     {name = "Darius A. Görgen", email = "info@dariusgoergen.com"},
 ]
-keywords = ["pytorch", "remote sensing", "area of applicability", "uncertainty scoring", "confidence scoring", "selective prediction"]
+keywords = ["pytorch", "remote sensing", "area of applicability", "confidence scoring", "uncertainty scoring", "selective prediction"]
 
 classifiers = [
     "Development Status :: 3 - Alpha",
@@ -89,9 +89,7 @@ exclude = ["tests*", "docs", "_extensions"]
 
 [tool.pytest.ini_options]
 filterwarnings = [
-  "ignore: builtin type .* has no __module__ attribute:DeprecationWarning",
   "ignore: The '.*_dataloader' does not have many workers.*:UserWarning",
-  "ignore: Query \\d+ returned fewer than \\d+ neighbors\\.",
   "ignore: GPU available but not used\\..*",
 ]
 

seapig/metric.py

@@ -1,7 +1,7 @@
 """Metrics helpers for selective evaluation.
 
 This module provides two utilities used during evaluation when a model
-can abstain or return uncertainty scores:
+can abstain or return confidence scores:
 
 - `SelectiveMetric`: wraps a `torchmetrics.Metric` (or a
   `torchmetrics.MetricCollection`) and tracks the metric on three
@@ -318,7 +318,7 @@ def update(
             Model outputs and targets. These are passed to `error_fn` to
             compute per-sample residuals.
         scores : torch.Tensor
-            Per-sample uncertainty scores (lower values indicate lower uncertainty).
+            Per-sample confidence scores (higher values indicate higher confidence).
         """
         device = preds.device
         scores = scores.to(device)