A pytest plugin that fails a test when a tracked ML metric regresses past a fixed tolerance.
A notebook can report whatever accuracy it wants; nobody re-runs it before merging. The version of "did the model get worse" that actually matters is the one checked automatically, on every pull request, against a number saved from a run everyone agreed was good. Without that check, small metric drops slip in one at a time until the model is quietly worse than it was a month ago and nobody can point at the commit that did it.
pytest-metricguard adds one fixture, metric_guard, that compares a metric
against a baseline stored in a JSON file and fails the test if it dropped
more than the tolerance allows.
def test_model_quality(metric_guard):
auc = train_and_score()
metric_guard.check("roc_auc", auc, tol=0.01)$ pytest
FAILED test_model.py::test_model_quality - metric 'roc_auc' regressed beyond tolerance
$ pytest --update-metric-baselines # accept the new number on purposeThe first run with no baseline records one automatically (with a warning),
same idea as --snapshot-update in snapshot-testing plugins.
$ pip install pytest-metricguard
$ uv add --dev pytest-metricguard # if you use uvNo dependencies beyond pytest itself (plus tomli on Python 3.10, where the
standard library has no TOML reader yet).
def test_model_quality(metric_guard):
metric_guard.check("roc_auc", 0.847)check() takes the metric name, the value from this run, and optional
per-call overrides:
metric_guard.check("roc_auc", auc, tol=0.01, direction="higher-is-better")
metric_guard.check("rmse", rmse, rel_tol=0.02, direction="lower-is-better")tol— absolute tolerance: the metric may move by at most this much before the test fails.rel_tol— relative tolerance, as a fraction of the baseline value. When both are given, whichever allows the larger drift wins (the same rulepytest.approxuses forabsandrel).direction—"higher-is-better"(the default) or"lower-is-better". Decides whether a rising or falling value counts as an improvement.
If neither tol nor rel_tol is set, the metric must match the baseline
exactly, which is rarely what you want — set at least one.
Baselines are recorded per test id and metric name, so two different tests
can each track a metric called roc_auc without overwriting one another.
| Situation | Result |
|---|---|
| No baseline recorded yet | Passes, records the value, emits a warning |
| Value matches the baseline | Passes |
| Value improved | Passes, warns that the baseline was not updated |
| Value is worse, within tolerance | Passes |
| Value is worse, beyond tolerance | Fails, with baseline, new value, delta and tolerance in the message |
$ pytest --update-metric-baselinesRewrites every checked baseline with the value from this run, whatever the
verdict would otherwise have been. Use it the same way you'd use
--snapshot-update: after reviewing the change and deciding it is correct.
At the end of the run, every checked metric is listed with its baseline, new value and verdict:
-------------------------------- metric-guard --------------------------------
test metric baseline new verdict
tests/test_model.py::test_quality roc_auc 0.8500 0.8470 worse (ok)
tests/test_model.py::test_quality f1 0.6000 0.6000 equal
Everything below is optional; the defaults are a baseline file named
.metric_baselines.json at the rootdir, no default tolerance (so set one
per call), and higher-is-better.
Via pytest ini options (pytest.ini, tox.ini, setup.cfg, or a
project's [tool.pytest.ini_options]):
[pytest]
metricguard_baseline_path = .metric_baselines.json
metricguard_default_tol = 0.01
metricguard_default_rel_tol = 0.02
metricguard_default_direction = higher-is-betterOr via a dedicated table in pyproject.toml, which also supports
per-metric overrides:
[tool.pytest-metricguard]
baseline_path = ".metric_baselines.json"
default_tol = 0.01
default_direction = "higher-is-better"
[tool.pytest-metricguard.metrics.rmse]
tol = 0.5
direction = "lower-is-better"Precedence, lowest to highest: built-in defaults, ini options, the
[tool.pytest-metricguard] table, its per-metric overrides, then whatever
is passed directly to check().
Both tools exist to stop a metric regression from merging quietly, but they answer different questions.
evalgate decides whether a
difference between two eval runs is real or just sampling noise, using a
significance test over the number of examples involved. It is the right
tool when your eval set is small enough that a couple of flipped examples
would move the score, and you don't want CI to flap on noise.
pytest-metricguard does not know what noise is. It compares two numbers
against a fixed tolerance you chose, inside your existing test suite, and
fails when the drop is bigger than that. It is the simple belt: cheap to
add to any test that already computes a metric, with no statistics
involved. It is a poor fit for a five-example smoke eval, where a single
example flipping is a huge percentage swing that a fixed tolerance cannot
tell apart from a real regression — that is exactly the case evalgate
is for.
In practice: reach for pytest-metricguard on metrics computed over data
large and stable enough that a real regression clearly separates from
run-to-run wobble. Reach for evalgate when it doesn't, and you need a test
to tell you which one you're looking at.
It compares one number against another and applies a threshold you set; it has no notion of statistical significance and will happily fail a test over a difference that is well within normal run-to-run variance if your tolerance is too tight, or let a real regression through if it's too loose. Baselines are only as trustworthy as the run that produced them — recording one from a bad run bakes the bad number in as the standard to protect.
MIT. See LICENSE.