Windows full-suite failing set is not stable run-to-run (7/8/9 files at the same commit)

[bradygaster]

Summary

The set of failing test files on Windows varies between identical full-suite runs on the same commit. Three runs at 369bba8f produced three different sets. Every suite involved passes when run in isolation.

This matters because the Windows baseline (#1797 / .squad/e2e/windows-test-baseline.md) exists so that tomorrow's E2E day can distinguish a real regression from expected noise. A failing set that moves on its own defeats a single-number baseline: a flaky suite is indistinguishable from a real regression.

Measurement

Same machine, same commit 369bba8f, npm test each time.

Run	Files failed	Notable
Baseline	9	incl. promote-insider-tag, patch-esm-imports
With #1794 fix applied	8	scheduler gone (genuinely fixed); template-sync + consumer-imports appeared
Control — #1794 change stashed, tree restored	7	template-sync, consumer-imports, promote-insider-tag, patch-esm-imports all absent

Isolation check, with the #1794 change applied:

npx vitest run test/template-sync.test.ts test/consumer-imports.test.ts test/scheduler.test.ts
  Test Files  3 passed (3)
       Tests  340 passed (340)

So the suites are not individually broken. The failure depends on what else is running.

Suites that move

• test/template-sync.test.ts
• test/consumer-imports.test.ts
• test/scripts/promote-insider-tag.test.ts
• test/scripts/patch-esm-imports.test.ts

Stable failing set for contrast: check-changeset-drift, cli-packaging-smoke, plugin-extensibility, repl-ux, acceptance, cli/watch-capabilities.

Mechanism — one confirmed, the rest unknown

template-sync — confirmed. This is the #1796 race. The suite byte-compares .github/agents/squad.agent.md against .squad-templates/squad.agent.md while test/init-scaffolding.test.ts, running in a parallel worker, rewrites that same tracked file via stampVersion(). Whether it fails depends purely on worker interleaving — which is exactly why the file already carries two beforeAll re-syncs described in its own comments as "minimising the window to near-zero". PR #1798 removes the writer and should stabilise this one.

The other three — UNKNOWN. Stated plainly rather than guessed. Plausible candidates, none verified:

• Parallel-worker contention on shared on-disk state (dist/, test-fixtures/, temp dirs).
• Interaction with the check-changeset-drift.test.ts silently runs zero tests on Windows (shebang + CRLF) #1788 CRLF dead-gate class — promote-insider-tag and patch-esm-imports are both in that set, and a suite that loads zero tests reports differently from one that fails an assertion, so their apparent presence/absence may partly be a reporting artifact rather than a real behavioural change.

That second possibility is worth ruling out explicitly, because it would mean the count is unreliable for a reason we already have a name for.

Suggested investigation

1. Run the full suite 5× at a fixed commit, capturing the full file list each time, and confirm the moving set.
2. Re-run with --pool=forks --poolOptions.forks.singleFork=true (or --no-file-parallelism) to remove worker interleaving. If the set stabilises, it is contention; if not, look at reporting.
3. For promote-insider-tag / patch-esm-imports specifically, record whether each run shows an assertion failure or a zero-tests load error — those look nearly identical in summary output and are completely different in consequence (check-changeset-drift.test.ts silently runs zero tests on Windows (shebang + CRLF) #1788).
4. Re-measure after fix(test): stop init-scaffolding from mutating tracked repo files (#1796) #1798 and fix(ci): force LF on *.mjs so shebanged scripts stay loadable by vitest #1790 merge, since both remove known sources of cross-suite interference.

Success criteria

• Five consecutive full runs at one commit produce an identical failing set.
• Any remaining variance has a named, proven mechanism — not "probably parallelism".
• .squad/e2e/windows-test-baseline.md can state a single expected number again rather than a 7–9 range.

Interim mitigation (already landed in #1797)

The baseline doc now states a range and, more importantly, tells the reader to treat the failing set as the signal rather than the count, and lists which suites are known to move. That is a workaround, not a fix — and per tonight's repeated lesson, a workaround that hides intermittency is how bugs survive.

Not a blocker for tomorrow

Documented and usable as-is. Filing so the variance is tracked rather than absorbed as folklore.