feat(resampling): add auc_ci, delta_auc_ci, bootstrap_ci - #66
Open
ywatanabe1989 wants to merge 3 commits into
Open
feat(resampling): add auc_ci, delta_auc_ci, bootstrap_ci#66ywatanabe1989 wants to merge 3 commits into
ywatanabe1989 wants to merge 3 commits into
Conversation
Add a new resampling submodule providing publication-ready confidence intervals for ROC-AUC without any raw scipy/sklearn exposure: - auc_ci: analytic DeLong (Sun and Xu 2014 fast midrank) or percentile bootstrap CI for a single AUC. - delta_auc_ci: DeLong paired test / bootstrap CI for the difference of two correlated AUCs on the same samples, using the full DeLong covariance matrix (accounts for correlation between classifiers rather than naively summing independent variances). - bootstrap_ci: generic paired percentile bootstrap CI for any metric function over one or more equal-length arrays. Wired into the __init__.py lazy loader (_LAZY_ATTRS / __all__).
CI's audit-all flagged 18 STX-TQ002/TQ007 test-quality violations in the new resampling test files (missing AAA marker comments and/or more than one assertion per test). Split every multi-assert test into separate single-assertion tests with descriptive names, and added the literal `# Arrange` / `# Act` / `# Assert` marker comments (in order, not combined) to every test body, matching the convention already used elsewhere in this repo (e.g. tests/scitex_stats/descriptive/test__ci.py). Verified locally: `scitex-dev linter check-files tests/scitex_stats/` reports zero STX-TQ violations, and all 58 resampling tests still pass.
…mmend_tests Dogfood follow-up (neurovista friction report): - effect_sizes.effect_size_from_ci(estimate, ci_lower, ci_upper, ci=95): standardized effect size for metrics with only a point estimate + CI (bootstrap/analytic), back-deriving SE via the normal quantile so it generalizes to any confidence level. interpret_effect_size_from_ci reuses the existing Cohen's d interpretation thresholds. - resampling.paired_auc_effect_size(y_true, score_a, score_b): matched- classifier effect size (delta_auc / SE) for two AUCs on the same samples, reusing delta_auc_ci's DeLong covariance machinery (_delong_two_auc_covar) rather than re-deriving it. - auto.recommend_tests now also accepts two raw arrays + `paired` (recommend_tests(x, y, paired=True)) as a convenience overload, auto-building the StatContext internally. Implemented as a thin wrapper (_recommend_overload.py) that delegates to the existing context-based recommend_tests in _selector.py — no duplicated recommendation logic, and avoids growing _selector.py past the repo's 512-line file-size gate (it's already at the limit). All new tests follow the one-assert-per-test + AAA marker-comment convention; `scitex-dev linter check-files tests/` reports zero STX-TQ violations across the whole tree.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
auc_ci— DeLong (Sun & Xu 2014 fast midrank) or percentile-bootstrap 95% CI for a single ROC-AUC, no sklearn/raw-scipy exposure.delta_auc_ci— DeLong paired test (full 2x2 covariance matrix, accounts for correlation between two classifiers scored on the same samples) or paired bootstrap CI for the difference of two AUCs, plus a two-sided p-value.bootstrap_ci— generic percentile bootstrap CI for any metric function over one or more equal-length, paired-resampled arrays.resamplingsubmodule + internal_delong.pyshared math; wired into the__init__.pyPEP 562 lazy loader (_LAZY_ATTRS/__all__).Test plan
tests/scitex_stats/resampling/test__delong.py— midrank correctness, AUC=1.0 (perfect separation) / AUC=0.5 (complete overlap), valid covariance matrix entries.tests/scitex_stats/resampling/test__auc_ci.py— delong + bootstrap point estimates, CI ordering, bootstrap reproducibility, all-same-class raisesValueError, smalln_bootdoesn't crash.tests/scitex_stats/resampling/test__delta_auc_ci.py— delta correctness, CI ordering, the core DeLong covariance-awareness property (correlated scores give a narrower CI than a naive independent-variance sum), p-value sanity (near 0 for very different scores, large for identical scores), all-same-class raises, bootstrap variant.tests/scitex_stats/resampling/test__bootstrap_ci.py— estimate/CI correctness, reproducibility, paired-resampling (shared indices across arrays) verified via ana==bidentity-preserving fn./opt/venv-sac/bin/python3 -m pytest tests/scitex_stats/resampling/ -v).ss.auc_ci/ss.delta_auc_ci/ss.bootstrap_ciresolve via the lazy loader and return sane result dicts.tests/scitex_stats/test__dispatch.pystill passes (8/10; the 2 pre-existing failures are unrelated missing-figrecipeoptional-dependency errors, not caused by this PR).