Let tests say "I did not verify this"

[b-macker]

Summary

Five times recently an assertion passed while the property it named was untrue — grep '"key_fingerprint":"' matching an empty value, an attestation-count equality against a counter incremented on non-attesting paths, a taint total asserted on the wrong scope, a baseline inflated 2x by a telemetry defect, and a test-phase split that ran 19 fewer tests while printing clean summaries. None were caught by review; all were caught by running the degraded case.

Four accidental discoveries is a poor sampling method, so this audits the set: 360 assertion sites in the example's run.sh and 2294 across 235 shell suites.

Most of it is already disciplined

Worth saying first, because it shaped the scope. Absence checks generally pair with a positive existence check, a fail-safe default (${X:-1}, where unset means fail), or a named control — TR-01 is labelled "the anti-regression", PU-01 fails outright if its scenario produced fewer than six samples, C-02 is guarded by C-01 asserting presence. 43 of governance_v4's 69 candidates already pair with a positive check. Those were left alone.

The systematic finding

Seven security suites define only ok() and fail() — no skip(). Unable to say "inconclusive", they were forced to record every "executor not available" branch as a PASS:

test_gov_comment_styles_gov002.sh
  before:  Results: 4/4 passed
  after:   Results: 0/4 passed, 4 skipped (unverified)

That suite verified nothing at all — every check unverifiable in this environment, the Python polyglot block evaluating to null so the probe never satisfies — while reporting four passes. 19 sites across 7 files, plus a SKIP count in each summary.

Five individual findings

	before
H01	health verdict inferred from stderr containing the string "governance", which every governed run prints. The living script emits its verdict as its last act, so the line is absent exactly when a run died early — a gk_fail-severity check passing hardest in the case that matters most. IMPAIRED still fails; absence is now inconclusive.
L24-03	trivially true at zero attestations; non-vacuous only because a neighbouring assertion stayed strict. Now self-guarding.
DB-01, DB-03	initialiser 0 indistinguishable from "the compliant scenario never ran" → sentinel -1
DB-04	fallback "1.00 0 0" made a missing result file read as a confirmed known-negative
E04	asserted a line was absent from a dashboard that a failed run never produced

Deliberately unchanged

T9 greps stable source; A05 genuinely tests config acceptance; B02 requires the value present; FM-01/FM-02 treat a missing result file as uncaught → fail. tests/gorilla (62 candidates) is out of scope — older scenario tests, weakest claims.

Test Plan

• Ran bash run-all-tests.sh with no new failures — 441 tests, 0 unexpected failures
• Added/updated tests for new functionality — n/a; this removes false passes rather than adding checks
• bash tests/security/test_error_msg_leaks.sh — 874 checks, 0 failures
• Tested manually in the REPL — n/a, test harness

Every fix verified by degraded case. H01 three ways:

scenario	old	new
run died early, stderr mentions "governance"	PASS "healthy"	SKIP
genuinely healthy run	PASS	PASS
IMPAIRED in stderr	FAIL	FAIL

DB-01/DB-04 likewise skip on absent measurements while the real run still passes 4/4.

Note test_r24_fixes.sh fails 2/10 locally (exit 127 — naab-scanner/naab-gov not built in this container). Pre-existing and identical before this change, and run-all-tests.sh does not invoke it.

Related Issues

Follow-up to #109–#112. docs/governance-campaign-findings.md records the audit and the rule it produced: an assertion about an absence needs the thing it is absent from to exist. A missing line in a document nobody wrote, a zero from a counter nobody incremented, and a clean result are the same observation until something distinguishes them.

---

Generated by Claude Code

[github-actions]

NAAb Governance Report

Metric	Count
Files checked	16
Passed	16
Failed	0

All governance checks passed!

_{Generated by NAAb Governance Engine v4.0}