tests: unit shard intermittently fails subscription-status, turn-scope and answer-delivery liveness together #3256

Context

Observed on PR #3245, head b80acd2a1624d16e532e8750b874d5cc272cfaf6, base a84d9bfbcb4e608b504aa0a9feb1144ad9a0b77e: the unit (3) shard failed (check run 125037245269, started 2026-11-02T00:54:46Z) with 16 failures in ui-server and ui-react tests. The same shard passed on the previous head 830dbfbdc367efd674e20cdf7bf9b02d6fa0c7d7 about 45 minutes earlier. The two trees differ only in docs/integration-contract.md (prose only), so the PR's change cannot explain the failures.

Failing tests in that shard:

• packages/ui-server/tests/subscription-status.test.ts: all six /api/status's subscription tests that need a completed turn. Example: subscription-status.test.ts:307 expected provenBy "turn" and received null, so the turn did not record as a successful subscription turn.
• packages/ui-server/tests/turn-scope.test.ts: freeze/resume and storage-accounting tests, for example turn-scope.test.ts:41 and :73 (toMatchObject mismatches) and IGNORED_BYTES_ONCE_PER_PATH_SCOPE_COUNTER.
• packages/ui-react/tests/answer-delivery-runtime.test.ts:976: "Liveness B: a blackholed idle socket is probed after 15 s…", which failed with Liveness native event did not complete.

Locally, subscription-status.test.ts passes. In a sandbox without the native tmpfs runtime, some turn-scope.test.ts cases fail on both this branch and main. That is not the CI symptom.

Related, but not the same symptom: #2551 (low-space failures in running-turn fixtures) and #2989 (answer-delivery storage refusal).

Scope

Scoped on 2026-11-02 (see Acceptance criteria). Original note: needs scoping. First establish whether these tests fail together under runner load or resource pressure (shared worker directories under .brain-unit-workers-*, disk, timing). Then establish whether one shared fixture or harness cause explains all three files.

Out of scope

Raising timeouts or retrying to get green without a diagnosed cause.

Acceptance criteria

Attributed on 2026-11-02 from the second occurrence. Hosted run 41505209464, unit (1), had 2,665,242,624 bytes free on the root filesystem that holds the fixture brains. The turn floor max(5% of 76.9 GB, 2 GiB) is 3,844,357,734.4 bytes, so turns stopped at the floor. A preload that fakes those statfs figures reproduces the CI failure lines on an unchanged base.

• Hosted unit shards begin proof with free space on the fixtures' filesystem of at least the turn-fixture floor plus the proof's own consumption. This is a provisioning fix: it reclaims or measures what fills the runner. It does not change the floor formula.
• A runner that cannot provide that headroom fails in environment setup, as a named infrastructure prerequisite, before any test runs.
• Each proof records the fixtures' filesystem (total, free at start, minimum, free at end, floor). A dip below the floor produces a named annotation that attributes turn-dependent failures to runner disk exhaustion.
• The floor's protective purpose is unchanged. The production check and requireTurnFreeSpace keep max(5%, SCOPE_FREE_BYTES), and a test pins the CI copy of the formula to it.
• Failing-first and mutation receipts are recorded. Off-main qualification is recorded on the PR (proof-system lane): hosted unit shards with the measured runner disk, none of which reaches the floor.
• Coordination with tests: attribute low-space failures in remaining running-turn fixtures #2551. Per-fixture requireTurnFreeSpace calls in the remaining suites, including subscription-status.test.ts, stay in tests: attribute low-space failures in remaining running-turn fixtures #2551's scope. This issue covers the runner and the job-level attribution.

The first occurrence's turn-scope.test.ts and answer-delivery liveness symptoms have no disk figures. If they recur on a runner with recorded headroom above the floor, they get a new issue.

Contract impact

None expected.

Triage notes (2026-11-02, main 3b9f3e84b)

No local reproduction was attempted in triage. The single natural receipt above, from a shard that passed on the neighbouring head with a prose-only tree difference, is the only evidence. It establishes the failure but not its cause.

What would complete the reproduction. Any one of these would do:

• A second natural occurrence with the shard's runner resource figures (free disk and inode counts in the worker and .brain-unit-workers-* directories, and load).
• A controlled local run that pins the three files to one worker under constrained disk or CPU and reproduces at least two of the three symptoms together.
• A per-file run showing that the symptoms are independent.

The provenBy: null at subscription-status.test.ts:307 is the most specific symptom. Establishing why the completed turn did not record as a subscription turn is the first narrowing step.

Acceptance once attributed. A named cause per symptom, or one shared cause shown to explain all three. A fixture or harness repair scoped to that cause, with a controlled failing-first receipt. No raised deadline or retry.

Labels: added priority: p2 and needs: repro. Left standalone. #736, which owns non-browser verification determinism, is at GitHub's 100 direct sub-issue limit, so this is related to it by reference only. #2753 covers only browser runs.