Repository navigation
Epic: tests give faster feedback with less redundant work #1849
Description
Outcome
Offline CI and local tests provide the same meaningful product, security and compatibility evidence with less elapsed time and repeated work. Each guarantee is tested at the lowest sufficient level, with representative integration proof connecting detailed lower-level coverage to the real application. Environment lifetimes match test intent and safely amortize setup through sharing, private cloning or complete reset. Scenario matrices are justified by independent failure modes; theme parity is covered centrally in Storybook rather than multiplying unrelated runtime scenarios. Fast feedback, expensive final confirmation and endurance have separate execution controls; endurance can run alongside ordinary proof.
Why now
The maintainer requested a full duration catalog, then selected separating connection recovery from light/dark parity and asked for an optimization epic with scoped issues, without implementing changes yet. The offline local profile at 5cf1d548f7bd3a314eb36394dc5db0268db97da6 measured a recording-cap test at 602.65 seconds and four equivalent ten-disconnect scenarios at 92.86–94.57 seconds each. Successful exhaustive CI run 38549344893 took 27m40s across selected jobs and about 157.6 runner-minutes. Individual test-body time, setup/build time, job wall time and aggregate critical path are distinct measurements.
Current state — 2026-11-02
The implementation children from the earlier optimization pass are closed. The delivered work and its attributed receipts are recorded on each child and in the dated comments here. In summary: the recovery suite runs two responsive paths without a theme multiplier (#1851/#1905); Storybook owns light/dark parity; recording limit correctness is separated from the ten-minute endurance run (#1854/#1860/#1862); runtime packages build once per test invocation (#2458, superseding #2319); the 87 deliberate wait windows are inventoried and justified (#1855 through #2627-#2639 and #2647); browser and unit weights were refreshed (#2417, #2736). Each delivered receipt is scoped to its own source and runner; none is a controlled whole-CI benchmark.
The earlier pass still has two documentation/acceptance obligations:
- docs: correct browser weight refresh prerequisites #2550 corrects the contributor weight-refresh recipe, which still describes two browser jobs while the browser domain has three shards (
PROOF_SHARDS,scripts/proof-shards.ts:3) and the refresher requires a successful attributed run and every fresh shard (refreshBrowserCosts,scripts/refresh-browser-costs.ts:15-29). - ci: audit the test-cost epic's coverage and performance acceptance #2763 audits this epic's definition of done: named before/after source snapshots, controlled and uncontrolled comparisons kept apart, the coverage map for every bullet, and an explicit per-bullet completion verdict.
The 2026-10-25 test-value audit added 15 separately scoped children: #2961, #2962, #2964, #2965, #2966, #2967, #2973, #2974, #2975, #2976, #2977, #2978, #2979, #2980 and #2981. All 15 are now closed as completed through merged #3164, squash e09ae24b9cdc6322586f75a13493212e7ac22cdb, present on inspected main. That delivery consolidates repeated work and moves detailed markup/query matrices to cheaper owners while retaining the named integration boundaries. Its focused proof and historical mutation/source limits remain attributed to that PR; this tracker reconciliation does not independently rerun them or establish a whole-CI speedup. #2763 still owns the final coverage/performance acceptance audit.
The current native list has 89 direct children, with four open: #2550, #3238, #3329 and #3332. #2763 is closed as completed (audit comment of 2026-11-02). #2550's recipe correction is merged into the epic/1849-test-cost integration branch (#3308) and closes when that batch lands on main; #3332 has two slices on the same branch (#3362, #3395); #3329's PR #3345 is open against it. #3238 owns complete local unit execution through the existing workers, preserving offline guards, complete inventory and child cleanup; it is a separate implementation outcome, not completed by the earlier test-value reduction.
The epic closes when the original obligations, the new children and the definition of done are satisfied and the maintainer accepts the audit's verdicts. The 2026-10-25 source audit does not itself fulfill #2763's before/after performance acceptance.
Shared context
- The delivered recovery suite retains two responsive paths (
const cells,packages/ui-react/tests/connection-continuity-runtime.test.ts:327-330); its invariants include DOM identity, geometry, scroll, focus, selection, attachments and follow-ups (function same, same file:738-754). Width selects different rendered trackers (function drawn, same file:348-351). The four-cell timings in Why now describe the original profile, before ui-react: remove theme duplication from connection-continuity recovery tests #1851/test(ui-react): run continuity recovery once per viewport #1905's delivered theme reduction. - Storybook already supplies two theme projects (
const storyProject,packages/ui-kit/vitest.config.ts:61-83; story project calls at 215-216) and an accessibility error gate (a11y,packages/ui-kit/.storybook/preview.ts:68). Rendering twice does not alone prove all geometry and interaction parity. - The delivered recording split retains sustained native encoding, persistence, full-length decode and cleanup (
test("ten-minute native endurance,packages/ui-react/tests/browser/recordings.endurance.tsx:12-70). Fast limit correctness is separate; ci: separate fast feedback, final confirmation and parallel endurance suites #1854 now delivers standalone execution policy through chore(ci): gate candidate confirmation and isolate endurance proof #1862, with the separately verified actual-main proof linked below. - Binding context:
ROADMAP.md(What binds future work),docs/process/testing.md,docs/decisions/ci-utility.md, design-kit D9/D21, D32/D39 anddocs/decisions/example-corpus.md. - Related findings outside this epic: ci: include layout scenarios in measured browser shard weights #1836 (layout shard weights) and ci: attribute and reduce repeated workspace builds in UI runtime tests #1837 (repeated workspace-build attribution) keep their existing ownership; cross-reference their results rather than duplicate them. This epic owns test cost; Epic: Keyless verification completes deterministically and isolates host state #736 and its nested browser epic Epic: browser verification is deterministic and attributable #2753 own deterministic isolation and failure attribution.
Constraints and non-goals
Preserve unique behavioral assertions, justified responsive/pointer/auth/backend cases, offline guards, native browser proof, lifecycle cleanup, production limits, contract envelopes, pinned images, default discovery and required proof gates. Remove redundant parameter combinations only with an assertion-to-coverage mapping. Any change to a currently required endurance obligation or decision record must explicitly reconcile that obligation, not silently skip it. Do not add a blanket passing-test cache, shorten product deadlines, increase concurrency or shard counts without evidence, or make provider calls. Paid opt-in image-provider tests remain cataloged as unmeasured.
Breakdown
The native sub-issue list is the plan and the progress record. The delivered test-value children retain their assertion-owner maps for the final audit. #3238 supplies the remaining complete local unit execution path. #2550 owns the documentation correction. #2763 delivered the coverage/performance acceptance audit as the dated 2026-11-02 comment on this epic; the maintainer's decision on it added #3332 and moved theme loops in browser/layout/visual projects to standalone #3331. Any further unmet definition-of-done bullet becomes a new child here.
Definition of done
- Every native child is complete and its assertion mapping explains what was retained, moved or deduplicated. Remaining genuine theme-specific checks state a concrete reason.
- Shared/cloned/reset environments have measured benefit and proven case independence, dirty/failure cleanup and resource ownership. Tests whose intent is startup, isolation or teardown retain the required fresh lifecycle.
- Detailed variants use unit/component/small integration proof where sufficient; a documented guarantee/boundary map explains how representative E2E coverage completes the proof. No unique guarantee disappears through sampling or weaker doubles.
- Storybook owns reusable light/dark layout and interaction parity, with meaningful failing-first/mutation evidence and accessibility checks in both themes.
- Runtime suites preserve independent failure-mode coverage without an automatic Cartesian theme multiplier.
- Every deliberate wall-clock wait has a named temporal assertion, minimum justified duration and reason controlled time/completion signals cannot provide equivalent proof; expensive real-time confirmation follows the explicit tier policy.
- Recording limit correctness has fast proof; any remaining real-time endurance coverage has an explicit purpose and reviewed execution policy.
- Intermediate pushes use the fast tier; expensive final confirmation proves the exact candidate. Endurance is independently discoverable/scheduled, runs alongside other proof under measured resource bounds, and does not distort ordinary sharding.
- Comparable before/after results record source, runtime, resources, individual durations, setup, total runner consumption and CI critical path. Report savings and remaining bottlenecks without treating historical different-SHA runs as a controlled benchmark.
- All required selected checks pass on the delivered source. Closed children alone do not establish performance improvement or retained coverage.
Unscheduled tracking work; no release milestone is selected.
2026-10-24: rewrote the body to the current outcome and remaining obligations; added #2763 to own the coverage/performance acceptance audit that no child owned; the dated reconciliations that were in the body are preserved in a comment.
2026-10-24 triage at main af6300478a07379178bd30221a31c0b917df2283: Refreshed moved current-source pointers by inspecting their named anchors. Historical receipts retain their source identities; scope, rulings, priorities and outstanding prerequisites are preserved. No runtime result or readiness is inferred from this navigation repair.
2026-10-26: reconciled the 15 native test-value follow-ups from the 2026-10-25 review. Earlier implementation receipts remain historical evidence; current open work and final acceptance are distinguished. No source/test behavior or measured CI result changed in this tracker update.
2026-10-29 triage at main cb5db6e901e9d3d312bd3a125e03be6140faf98f: Verified the current named source anchors and repaired navigation; historical receipts and unresolved prerequisites retain their source identities. No runtime qualification or epic completion is inferred.
2026-11-02 triage at main a84d9bfbcb4e608b504aa0a9feb1144ad9a0b77e: Reconciled the 15 test-value closures with merged #3164 and the remaining native #3238 scope, and refreshed moved current-source anchors. Historical receipts, proof limits and the full definition of done remain; no new runtime or aggregate speedup is claimed.
Triage 2026-11-03 (main 3d3516a02): reconciled the current-state and breakdown paragraphs with #2763's delivered audit, the 2026-11-02 decision (#3329/#3332 attached, #3331 standalone) and the integration-branch state of #2550/#3329/#3332.
Activity
181 remaining items
schlessera commentedon Oct 25, 2026
Test-value audit follow-up — 2026-10-26
Created and attached 15 detailed implementation children from the 2026-10-25 multi-agent review: #2961, #2962, #2964, #2965, #2966, #2967 and #2973 through #2981. All remain open and unscheduled; no test implementation changed.
Suggested first wave by observable repeated work: #2977 (58 duplicate native button cases), #2973 (25 MCP round trips moved off the transport path), #2965/#2964/#2966 (22 duplicate or relocatable CLI launches), then #2967 (eight candidate PNG renders, requiring native qualification). These are operation counts, not elapsed savings. Smaller cleanup and markup-relocation children can proceed independently while retaining their exact alternate owners.
#2550 retains documentation ownership; #2763 retains final coverage/performance acceptance after delivery. The body now distinguishes earlier delivered work from these new findings. Assertion-quality findings are grouped separately in #2963, and invented production-path proof in #2540.
docs(ci): record the integration-lane and hosted draft feedback rulin…
chore(archive): record the test-cost epic acceptance audit evidence f…
schlessera commentedon Nov 2, 2026
Acceptance audit — 2026-11-02 (#2763)
This comment is #2763's deliverable: before/after snapshots, comparisons, the coverage map and a verdict for each definition-of-done bullet. Evidence is archived at
1c377ca76ac98409937e9c97774f8a11377bc2c8onarchive/measurements(README, jobs, profiled tests). Every before/after figure below is uncontrolled. The two runs differ in source (19 days, about 3,900 more unit cases), runner image, Chrome and shard layout.1. Snapshots
5cf1d548f7bd3a314eb36394dc5db0268db97da68e9855e706ab43dc1bb5b822296f171431e2546d(main, contains #3164e09ae24b9)workflow_dispatch, exhaustive, successschedule, exhaustive, successThe main tip
c23a5811chas a green main-push run, 41490214079. It reused earlier proof and skipped every product job, so it is not a comparison snapshot. The epic's integration branch (3bee06dce, #2550) is not yet on main.2. Comparisons (hosted, uncontrolled)
layout (2))bun run buildinpackunit (2)26.83 min → proofunit (2)17.32 min → aggregateRuntime-hook builds run inside each proof step and are not separately timed in either run.
Originally profiled tests:
unit (1): 93.21, 92.88, 94.52, 94.88 s = 375.49 sthe 10-minute-5-second fixture stops at the hard duration cap…602.80 s insidelayout (2); its file took 617.37 s for 41 casesten-minute native endurance…604.90 s in its ownendurancejob, ending at +13.20 min, off the critical path. Fast limit correctness inrecording-limits.offline.tsx: 14 cases, 4.39 s.recordings.offline.tsx: 39 cases, 15.43 s.layout (2)jobThe original local profile (602.65 s; 92.86–94.57 s) was not re-measured. The only available host was shared with concurrent workers (load average 20–27 on 16 logical CPUs), and the original profile records no host or resource identity to match. The hosted figures above are the comparable pair. For a controlled comparison, run both SHAs back to back on one idle host.
The critical path dropped from 27.75 to 18.42 min, mostly from unit workers (#2121) and moving endurance out of layout. Total runner-minutes rose by 9.15. Two new parallel jobs (endurance 12.42, runtime_host 10.23) and a third browser shard account for the rise, while unit inventory grew from about 13,195 to 17,055 cases. None of this is a controlled saving.
Remaining bottlenecks at the comparison source:
unit (2)at 17.32 min sets the critical path, withlayout (1)andunit (3)at 16.77 min.3–4. Coverage and completion verdicts
docs/process/theme-parity.md(continuity, replay, account/journal, artifact), #3164's owner/mutation table for the 15 test-value children. The theme gap in runtime suites is filed as #3329. The browser-project scope is ruling Q1.e09ae24b9) gives every moved assertion a retained owner and a mutation receipt. #2977–#2981 deduplicated layout registrations or moved markup matrices to component tests while keeping native boundaries.storybookandstorybook-lighteach execute 1,011 cases under the previewa11y: "error"gate.answer-delivery-runtime.test.ts:337-340(#1896) andturn-journal-runtime.test.ts:101(#2214). That is 8 cases, 22.33 s (table).docs/process/test-waits.mdand its inventory at5dc2595c, with #1855, #2627–#2639 and #2647 (#1912 stopped by ruling). The same census expression matches 662 lines in 251 files at5dc2595cand 1,310 lines in 443 files at the comparison source. 197 files match only at the comparison source, 183 of them newly added (list). These are unreviewed matches, not proven violations.c14846a66(#3315); its tipc23a5811cis green by reused proof.Questions for the maintainer
Q1 — Does the theme-reason bullet cover browser/layout/pointer/visual projects? 55
packages/ui-react/tests/browser/*files and 34packages/ui-kit/tests/visual/*files loop over both themes. Theme-named cases are 4,609 of 10,713 hosted browser cases, andtheme-parity.mdgives reasons only for the families it maps.Q2 — Is the wait bullet a snapshot or a standing invariant?
5dc2595cinventory.The epic closes after #2550, #3238 and #3329 land on main with a green exhaustive run, and after Q1/Q2 are ruled (B on Q2 adds one more child). #2763 makes no closure decision.
schlessera commentedon Nov 2, 2026
Decision (maintainer, 2026-11-02): Q1: option C. Theme loops in the browser, layout and visual projects are outside #1849; standalone issue #3331 tracks them. Q2: option B. The wait rule is standing, and child #3332 re-censuses the later waits before this epic closes.
chore(archive): record the real-time wait re-census and classificatio…
Merge pull request #3517 from schlessera/epic/1849-test-cost