mutmut reports false survivors: the cached test-selection map is partial and never repaired

[t41372]

Summary

mutmut run reports mutants as survived that the test suite actually kills. The cause is not a coverage gap: mutmut's cached test-selection map (mutants/mutmut-stats.json) can be partial, and once it exists mutmut never repairs it — so every mutant of an affected function is checked against a too-small subset of tests.

Separately, the nightly mutation workflow on main has failed on at least four consecutive nights (runner shutdown at ~1.5–2 h, ~65 % through the mutant list), so AGENTS.md principle 5's "zero surviving mutants" is currently unverified on main.

Evidence — false survivors

During PR #34 review follow-up I ran targeted mutation testing and re-verified every reported survivor by running the suite against the live mutant:

cd mutants && MUTANT_UNDER_TEST=<name> PYTHONPATH=src \
  uv run --project .. python -m pytest <relevant test files> -x -q

Two batches, every reported survivor re-checked:

batch	mutmut said "survived"	actually killed by the suite
argstate / store / flows, round 1	37	21 (57 %)
same modules, round 2	8	8 (100 %)

Concrete case — skit.argstate.x__purge_secret_locked__mutmut_16 (doc["values"] = _strip_secrets(...) → doc["values"] = None):

• mutmut results → survived
• MUTANT_UNDER_TEST=…__mutmut_16 pytest tests/test_argstate_mut.py → FAILED test_purge_secret_reports_names_removed_across_values_and_presets

That killing test is absent from the cached map:

$ python -c "import json; d=json.load(open('mutants/mutmut-stats.json')); \
    print(len(d['tests_by_mangled_function_name']['skit.argstate.x__purge_secret_locked']))"
11    # and test_purge_secret_reports_names_removed_across_values_and_presets is not among them

Re-running mutmut's own StatsCollector over that one file records 6/6 purge tests, including the killer — so the test does execute the function; the cached map is simply missing it.

Root cause

mutmut/__main__.py:

def collect_or_load_stats(runner):
    did_load = load_stats()
    if not did_load:
        run_stats_collection(runner)            # full collection
    else:
        ...                                     # incremental
        new_tests = <test ids not already in duration_by_test>
        run_stats_collection(runner, tests=new_tests)

and the collection itself runs pytest with -x:

pytest_args = ["-x", "-q", "-p", "no:randomly", "-p", "no:random-order"]

So:

1. One failing test aborts stats collection mid-suite. Every test after the abort point contributes nothing to tests_by_mangled_function_name. (Reproduced: an abort at tests/test_childenv.py leaves everything alphabetically after it unmapped.)
2. The partial map is then permanent. load_stats() succeeds on the next run, so mutmut takes the incremental path and only re-collects for test IDs it has never seen. An unchanged test that now exercises a newly added or newly extracted function is never re-mapped — load_stats merges with |=, so the map only ever grows by whole new test IDs, never gets corrected.
3. Mutants of those functions are then checked against whatever subset happened to be recorded, and anything the missing tests would have killed is reported as survived.

Ruled out while diagnosing: max_stack_depth truncation in record_trampoline_hit (defaults to -1, i.e. disabled — we do not set it), and a wholly aborted stats phase (duration_by_test held all 5 940 test IDs).

Why CI is a separate question

The nightly workflow does not cache mutants/ (enable-cache: true is uv's package cache), so each run starts with no stats file and does a full collection. That path is only vulnerable via -x if a test fails during the stats phase. Main's nightly failures are a timeout/shutdown problem, not obviously this one — but the 1 472 survivors it reported before dying have not been verified either way, and the same "no repair" design applies to any run that aborts.

Suggested actions

• Invalidate the stats cache on source change. A new/renamed/extracted function makes the cached map wrong for that function; deleting mutants/mutmut-stats.json is the current workaround and should be documented in AGENTS.md next to uv run mutmut run.
• Verify survivors before trusting them. Add a helper (script or CI step) that re-runs the suite against each reported survivor with MUTANT_UNDER_TEST set and reports only the ones that really survive. scripts/check_mutation_stats.py is the natural home.
• Make the nightly finish. It is currently killed at ~1.5–2 h, ~65 % through. Options: shard the mutant list across jobs, raise timeout-minutes, or run incrementally against changed files.
• Consider reporting upstream that collect_or_load_stats's incremental path cannot repair a partial map, and that -x during stats collection silently produces one.

Impact

Not a product bug — no shipped behaviour is wrong. But the mutation gate is currently reporting noise, and a survivor list that is >50 % false is a gate people learn to ignore. The verification loop above is cheap and makes the gate trustworthy again.