Run the full unit suite in CI instead of Jest's affected-tests mode - #2255
Conversation
The `unit-tests` job ran `jest --changedSince=origin/<base>` on PRs. Jest builds that selection from the module dependency graph, so a spec that reads a source file as text (`fs.readFileSync`) rather than importing it has no edge to that file and is never selected by a change to the file it guards. This repo leans on that pattern heavily: 69 specs under `tests/unit/` read source as text, and 52 of them import nothing but `fs` and `path`. Measured against the real resolver, `--findRelatedTests shared/database/src/schema.ts` selects 250 suites and only 3 of the 31 `tests/unit/database/schema.*` specs written to guard that file; `shared/authentication/src/auth.definition.ts` selects 1 suite and none of its 5 guards. Volume is not the signal — a big green number is exactly what the failure mode looks like. That is how PR #2246 merged green while breaking 5 assertions in `tests/unit/database/schema.fk-cascades.test.ts`, the spec that holds #2239's per-constraint decision in place. The nightly full-suite run was the only backstop, one day late. The optimization was not buying time either. `unit-tests` gates no other job (`Integration-Tests` needs `[detect-changes, lint-and-typecheck]`), and it runs beside `lint-and-typecheck`, which is consistently about twice as long — 45-91s against 113-154s across recent runs. The largest observed affected run was 252 suites / 5002 tests in 38.3s; the full suite is 472 suites / 8181 tests in ~32s locally, so roughly +30s of runner time inside a minute of existing slack. The repo is public, so those minutes are free. Also drops `fetch-depth: 0` from the checkout, which existed only to give `--changedSince` history, and `--passWithNoTests`, which is only meaningful under selection. An allowlist of text-reading suites was considered and rejected: it reintroduces the list-somebody-must-remember that let a second set of five FKs drift underneath BS#1126's hardcoded regression spec, which is why #2239 shipped a general guard instead. `tests/unit/scripts/ci-unit-tests-full-suite.test.ts` pins the decision, asserting the job carries no selection flag and that the npm scripts it calls are unfiltered. Verified it fails when `--changedSince` is put back.
|
CI green on the first run, and the cost prediction held on the runner rather than my laptop:
Dropping |
…ctory comment
Review follow-ups on the BS#2249 guard.
`extractUnitTestsJob()` terminated on the *next job key*, but every job here is introduced by its own two-space-indented `#` comment block, so that block landed inside the returned slice — verified: the slice ended with `Integration-Tests`' "service containers but harmless" note. That is wrong in both directions. A future comment above `Integration-Tests` mentioning `--changedSince` or `github.event_name`/`pull_request` would fail these assertions spuriously, and `toContain('npm run test:unit:coverage')` could be satisfied by text belonging to a different job. Terminating on the first line at two-space indent that is not part of this job's body fixes it, since every line of a job body is indented four or more. Added an `it` pinning the slice boundary, and verified both directions: reintroducing `--changedSince` still fails 3 assertions, and putting `--changedSince` into the neighbouring job's comment no longer fails anything.
The missing-job check ran in the `describe` body via `expect`, so a rename would surface as a whole-file load failure rather than a named assertion. It now throws an Error naming the file and pointing at the branch-protection required-check name, which also has to move on a rename.
Also removed `# Unit tests - runs affected tests only` from directly above the new block, which says the opposite. A reader skimming the job header hit the stale claim first.
|
Review follow-ups pushed in 1037f6d. CI green. Job-slicing boundary — confirmed by probing the extractor directly: the slice ended with
Describe-time throw — the missing-job check now throws a named Stale comment — On the out-of-scope note: the |
Closes #2249.
The defect
The
unit-testsjob ranjest --changedSince=origin/<base>on PRs. Jest builds that selection from the module dependency graph, so a spec that reads a source file as text (fs.readFileSync) rather thanimporting it has no edge to that file and is never selected by a change to the file it guards.The audit (#2249 acceptance criterion 2)
69 specs under
tests/unit/read source as text. 52 of them import nothing butfsandpath— zero edges to the code they guard:shared/database/src/schema.tsshared/authentication/src/auth.definition.tsdev_env/docker-compose.yml,Dockerfile.*,.github/workflows/test.ymlapps/backend/services/*.ts,jobs/*/job.ts,scripts/*.mjsMeasured against the real resolver:
Volume is not the signal. A change to
schema.tsselects 250 suites and still misses 28 of the 31 specs written to guardschema.ts— a big green number is exactly what the failure mode looks like. That is how #2246 merged green while breaking 5 assertions intests/unit/database/schema.fk-cascades.test.ts, the spec that holds #2239's per-constraint decision in place. The nightly full-suite run was the only backstop, one day late.Proof the fix closes it (#2249 acceptance criterion 1)
Re-added
{ onDelete: 'cascade' }toshow_djs.show_id— reverting #2250's fix — and ran both commands against that tree:jest --changedSince=origin/main --coverage --passWithNoTestsnpm run test:unit:coverageThe failure is the assertion that should have caught it:
schema.tswas restored afterwards; it is untouched in this diff.Why option 1, and what it costs
unit-testsgates no other job —Integration-Testsneeds[detect-changes, lint-and-typecheck]— and it runs besidelint-and-typecheck, which is consistently about twice as long:unit-tests, recent runslint-and-typecheck, same runsSo the change adds roughly +30 s of runner time inside a minute of slack on a job that is not on the critical path, and the repo is public, so those minutes are free. The run still ends when
Integration-Testsdoes, ~3 minutes later.The issue's options 2 (union in a fixed list) and 3 (give the specs a real import edge) were rejected: both reintroduce a list somebody has to remember to maintain — the same rot that let a second set of five FKs drift underneath BS#1126's hardcoded regression spec, which is why #2239 shipped a general guard instead of another allowlist. Neither buys anything, because there is no time to protect.
Changes
.github/workflows/test.yml— the job runsnpm run test:unit:coverageunconditionally; dropsfetch-depth: 0(existed only to give--changedSincehistory) and--passWithNoTests(only meaningful under selection). A comment records the measurements so the optimization is not reintroduced as an obvious win.tests/unit/scripts/ci-unit-tests-full-suite.test.ts— new guard asserting the job carries no selection flag, does not branch ongithub.event_name, and thattest:unit/test:unit:coverageare unfiltered. Verified it fails (3 assertions) when--changedSinceis put back. It is itself a text-reading spec, which is now safe precisely because the full suite always runs.docs/testing.md— corrects the CI description, which still described affected-tests mode.Local verification
format:checkclean ·lint0 errors (917 pre-existing warnings) ·typecheckclean · full unit suite 472 suites / 8181 tests green.One unrelated observation worth its own ticket:
Install Dependenciesrannpm cion 13 of 13 recent distinct branches, including branches that never touchedpackage-lock.json. Thenode_modulescache never restores, costing 25–34 s per run — more than the entire optimization this PR removes was saving.Related