Add perf regression test harnesses by juliusmarminge · Pull Request #1669 · pingdotgg/t3code

juliusmarminge · 2026-04-01T20:59:32Z

Summary

Add a synthetic perf provider adapter and registry for deterministic Codex runtime event playback.
Seed realistic orchestration state for perf scenarios, including large thread fixtures and assistant-streaming coverage.
Add web perf harnesses, scenario catalogs, and thresholds for virtualization and websocket regression tests.
Extend server startup and shared perf scenario definitions to support the new perf flows.
Add integration and unit coverage for the perf seeding and provider adapter paths.

Testing

Not run in this context.
New tests added for perf state seeding and PerfProviderAdapter event emission.
Web perf test entrypoints and Vitest perf config were added for regression coverage.

Note

Medium Risk
Adds new perf-only seeding, provider, and harness code paths and conditionally swaps the server provider/registry via env flags, which could affect startup/provider behavior if misconfigured.

Overview
Adds a local performance regression harness that seeds deterministic scenarios (e.g. large_threads, burst_base) via the real event store + projection pipeline and writes run artifacts/logs under artifacts/.

Introduces a perf-only Codex provider stack (PerfProviderAdapter, PerfProviderRegistryLive, PerfProviderLayerLive) that replays timed runtime events for websocket stress scenarios, and updates adapter streamEvents to be a getter so each access returns a fresh Stream.

Adds dedicated perf test suites and runners: server websocket/RPC latency benchmarks (integration/perf) and built-web Playwright/Vitest perf tests (virtualization + websocket application), plus new test:perf:*/perf:open* scripts, configs, and docs (docs/perf-benchmarks.md).

^{Reviewed by Cursor Bugbot for commit e9fa2b7. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add performance regression test harnesses for server and web

Adds seedPerfState (apps/server/integration/perf/seedPerfState.ts) to build reusable seeded state directories for two scenarios: large_threads and burst_base, projecting events through a real SQLite stack and snapshotting the read model.
Adds startServerPerfHarness (apps/server/integration/perf/serverPerfHarness.ts) that spawns a live server against seeded state, connects via WebSocket RPC, and writes logs and JSON latency artifacts per run.
Adds startPerfAppHarness (apps/web/test/perf/appHarness.ts) that boots the built server, launches headless Chromium via Playwright, injects a browser perf collector, and records action timings, long tasks, and RAF gaps.
Adds a PerfProviderAdapter (apps/server/src/perf/PerfProviderAdapter.ts) that simulates the codex provider with deterministic event streams; activated at runtime via T3CODE_PERF_PROVIDER=1 and T3CODE_PERF_SCENARIO.
Adds perf test suites for virtualization (apps/web/test/perf/virtualization.perf.test.ts), WebSocket streaming (apps/web/test/perf/websocket-application.perf.test.ts), and server RPC latency (apps/server/integration/perf/serverLatency.perf.test.ts), each asserting against thresholds defined in test/perf/support/thresholds.ts.
Adds a scripts/open-perf-app.ts CLI to seed state and launch the built server interactively with optional live provider mode.

^{Macroscope summarized e9fa2b7.}

coderabbitai · 2026-04-01T20:59:39Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 926c6bb6-bd04-4dc7-8d77-11c2bcafcba7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch t3code/performance-regression-tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

test/perf/support/browserMetrics.ts

apps/server/integration/perf/seedPerfState.test.ts

macroscopeapp · 2026-04-01T21:03:27Z

Approvability

Verdict: Needs human review

2 blocking correctness issues found. Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

^{You can customize Macroscope's approvability policy. Learn more.}

cursor

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for all 3 issues found in the latest run.

✅ Fixed: Reset creates duplicate animation frame loops corrupting metrics
- Stored the rAF handle and added cancelAnimationFrame before starting a new loop in reset() to prevent duplicate concurrent animation frame chains.
✅ Fixed: Generated test artifacts accidentally committed to repository
- Removed the committed artifact JSON files via git rm --cached and added artifacts/ to .gitignore to prevent future accidental commits.
✅ Fixed: Percentile returns null for zero target value
- Added Math.max(0, ...) to clamp the computed index so that target=0 returns sorted[0] (the minimum) instead of sorted[-1] (undefined).

Or push these changes by commenting:

@cursor push 0e9b05e984

Preview (0e9b05e984)

diff --git a/.gitignore b/.gitignore
--- a/.gitignore
+++ b/.gitignore
@@ -21,3 +21,4 @@
 .vitest-*
 __screenshots__/
 .tanstack
+artifacts/

diff --git a/artifacts/perf/virtualization-large_threads-1775077017735/virtualization-large_threads.json b/artifacts/perf/virtualization-large_threads-1775077017735/virtualization-large_threads.json
deleted file mode 100644
--- a/artifacts/perf/virtualization-large_threads-1775077017735/virtualization-large_threads.json
+++ /dev/null
@@ -1,141 +1,0 @@
-{
-  "suite": "virtualization",
-  "scenarioId": "large_threads",
-  "startedAt": "2026-04-01T20:57:00.745Z",
-  "completedAt": "2026-04-01T20:57:01.524Z",
-  "thresholds": {
-    "maxMountedTimelineRows": 140,
-    "threadSwitchP50Ms": 250,
-    "threadSwitchP95Ms": 500,
-    "maxLongTaskMs": 120,
-    "maxRafGapMs": 120,
-    "burstCompletionMs": 5000,
-    "longTasksOver50MsMax": 2
-  },
-  "summary": {
-    "maxMountedTimelineRows": 23,
-    "threadSwitchP50Ms": 49.5,
-    "threadSwitchP95Ms": 117,
-    "maxLongTaskMs": 0,
-    "longTasksOver50Ms": 0,
-    "maxRafGapMs": 25,
-    "burstCompletionMs": null
-  },
-  "browserMetrics": {
-    "actions": [
-      {
-        "name": "thread-switch-warmup-a",
-        "durationMs": 117,
-        "startedAtMs": 591,
-        "endedAtMs": 708
-      },
-      {
-        "name": "thread-switch-1",
-        "durationMs": 46.5,
-        "startedAtMs": 709.7999999523163,
-        "endedAtMs": 756.2999999523163
-      },
-      {
-        "name": "thread-switch-2",
-        "durationMs": 49.5,
-        "startedAtMs": 787.6000000238419,
-        "endedAtMs": 837.1000000238419
-      },
-      {
-        "name": "thread-switch-3",
-        "durationMs": 70.69999992847443,
-        "startedAtMs": 840.7000000476837,
-        "endedAtMs": 911.3999999761581
-      },
-      {
-        "name": "thread-switch-4",
-        "durationMs": 45.89999997615814,
-        "startedAtMs": 916,
-        "endedAtMs": 961.8999999761581
-      },
-      {
-        "name": "thread-switch-5",
-        "durationMs": 46.699999928474426,
-        "startedAtMs": 964.6000000238419,
-        "endedAtMs": 1011.2999999523163
-      },
-      {
-        "name": "thread-switch-6",
-        "durationMs": 49.89999997615814,
-        "startedAtMs": 1047.8999999761581,
-        "endedAtMs": 1097.7999999523163
-      }
-    ],
-    "longTasks": [],
-    "rafGapsMs": [
-      0, 8, 0, 17, 0, 7.300000000000068, 0, 8.799999999999955, 0, 8.200000000000045, 0,
-      9.099999999999909, 0, 7.7000000000000455, 0, 8.399999999999977, 0, 8.800000000000068, 0, 16,
-      0, 8.399999999999977, 0, 9, 0, 8.299999999999955, 0, 7.600000000000023, 0, 17.100000000000023,
-      0, 8.600000000000023, 0, 8.299999999999955, 0, 16.200000000000045, 0, 7.899999999999977, 0,
-      9.299999999999955, 0, 7.399999999999977, 0, 9.200000000000045, 0, 16.700000000000045, 0, 8, 0,
-      8.299999999999955, 0, 7.899999999999977, 0, 8.700000000000045, 0, 17.100000000000023, 0,
-      8.199999999999932, 0, 16, 0, 16.600000000000023, 0, 8.399999999999977, 0, 8.899999999999977,
-      0, 7.800000000000068, 0, 17.399999999999977, 0, 8.299999999999955, 0, 8.300000000000068, 0,
-      7.699999999999932, 0, 16.700000000000045, 0, 8.100000000000023, 0, 17.399999999999977, 0,
-      7.7999999999999545, 0, 8.600000000000136, 0, 8.199999999999818, 0, 8.900000000000091, 0,
-      8.099999999999909, 0, 16.200000000000045, 0, 9.100000000000136, 0, 7.399999999999864, 0, 8.5,
-      0, 9, 0, 25, 0, 7.600000000000136, 0, 9.199999999999818, 0
-    ],
-    "mountedRowSamples": [
-      {
-        "label": "heavy-a-open",
-        "count": 17,
-        "capturedAtMs": 708.5
-      },
-      {
-        "label": "thread-switch-1-rows",
-        "count": 17,
-        "capturedAtMs": 782
-      },
-      {
-        "label": "thread-switch-2-rows",
-        "count": 17,
-        "capturedAtMs": 837.7000000476837
-      },
-      {
-        "label": "thread-switch-3-rows",
-        "count": 17,
-        "capturedAtMs": 914.2000000476837
-      },
-      {
-        "label": "thread-switch-4-rows",
-        "count": 17,
-        "capturedAtMs": 962.5
-      },
-      {
-        "label": "thread-switch-5-rows",
-        "count": 17,
-        "capturedAtMs": 1034.1000000238419
-      },
-      {
-        "label": "thread-switch-6-rows",
-        "count": 17,
-        "capturedAtMs": 1098.3999999761581
-      },
-      {
-        "label": "scroll-start",
-        "count": 17,
-        "capturedAtMs": 1113.3999999761581
-      },
-      {
-        "label": "scroll-top",
-        "count": 23,
-        "capturedAtMs": 1148.3999999761581
-      },
-      {
-        "label": "scroll-bottom",
-        "count": 17,
-        "capturedAtMs": 1164.2000000476837
-      }
-    ]
-  },
-  "serverMetrics": null,
-  "metadata": {
-    "heavyThreadMessageCount": 2000
-  }
-}
\ No newline at end of file

diff --git a/artifacts/perf/websocket-application-burst_base-1775077022750/websocket-application-dense_assistant_stream.json b/artifacts/perf/websocket-application-burst_base-1775077022750/websocket-application-dense_assistant_stream.json
deleted file mode 100644
--- a/artifacts/perf/websocket-application-burst_base-1775077022750/websocket-application-dense_assistant_stream.json
+++ /dev/null
@@ -1,139 +1,0 @@
-{
-  "suite": "websocket-application",
-  "scenarioId": "dense_assistant_stream",
-  "startedAt": "2026-04-01T20:57:03.910Z",
-  "completedAt": "2026-04-01T20:57:07.151Z",
-  "thresholds": {
-    "maxMountedTimelineRows": 140,
-    "threadSwitchP50Ms": 250,
-    "threadSwitchP95Ms": 500,
-    "maxLongTaskMs": 120,
-    "maxRafGapMs": 120,
-    "burstCompletionMs": 5000,
-    "longTasksOver50MsMax": 2
-  },
-  "summary": {
-    "maxMountedTimelineRows": 0,
-    "threadSwitchP50Ms": 51.199999928474426,
-    "threadSwitchP95Ms": 52.39999997615814,
-    "maxLongTaskMs": 0,
-    "longTasksOver50Ms": 0,
-    "maxRafGapMs": 16.200000000000045,
-    "burstCompletionMs": 3135
-  },
-  "browserMetrics": {
-    "actions": [
-      {
-        "name": "thread-switch-burst-nav",
-        "durationMs": 51.199999928474426,
-        "startedAtMs": 1348.3999999761581,
-        "endedAtMs": 1399.5999999046326
-      },
-      {
-        "name": "thread-switch-burst-return",
-        "durationMs": 52.39999997615814,
-        "startedAtMs": 1401.1999999284744,
-        "endedAtMs": 1453.5999999046326
-      },
-      {
-        "name": "burst-completion",
-        "durationMs": 3135,
-        "startedAtMs": 332.2999999523163,
-        "endedAtMs": 3467.2999999523163
-      }
-    ],
-    "longTasks": [],
-    "rafGapsMs": [
-      0, 7.899999999999977, 0, 9, 0, 7.300000000000011, 0, 8.899999999999977, 0, 7.800000000000011,
-      0, 9.300000000000011, 0, 8.100000000000023, 0, 7.699999999999989, 0, 9.199999999999989, 0,
-      7.800000000000011, 0, 7.899999999999977, 0, 9.300000000000011, 0, 7.300000000000011, 0,
-      9.099999999999966, 0, 7.7000000000000455, 0, 8.199999999999989, 0, 8.399999999999977, 0,
-      8.300000000000011, 0, 8.399999999999977, 0, 8.5, 0, 9.100000000000023, 0, 7.7000000000000455,
-      0, 8.199999999999932, 0, 8.600000000000023, 0, 8.799999999999955, 0, 7.7000000000000455, 0,
-      8.200000000000045, 0, 8.399999999999977, 0, 9.100000000000023, 0, 7.5, 0, 8.199999999999932,
-      0, 9.300000000000068, 0, 7.399999999999977, 0, 8.600000000000023, 0, 8.399999999999977, 0,
-      8.899999999999977, 0, 8.299999999999955, 0, 7.800000000000068, 0, 8, 0, 8.399999999999977, 0,
-      8.799999999999955, 0, 7.7000000000000455, 0, 9.299999999999955, 0, 7.400000000000091, 0,
-      9.299999999999955, 0, 8.299999999999955, 0, 7.7000000000000455, 0, 8.100000000000023, 0,
-      8.899999999999977, 0, 8.700000000000045, 0, 7.2999999999999545, 0, 9.399999999999977, 0,
-      8.200000000000045, 0, 8.100000000000023, 0, 7.699999999999932, 0, 9.300000000000068, 0,
-      8.199999999999932, 0, 8.300000000000068, 0, 7.699999999999932, 0, 8.700000000000045, 0,
-      8.699999999999932, 0, 7.600000000000023, 0, 8.100000000000023, 0, 8.399999999999977, 0,
-      8.700000000000045, 0, 8, 0, 8.699999999999932, 0, 8.200000000000045, 0, 8.799999999999955, 0,
-      8.5, 0, 7.800000000000068, 0, 8.5, 0, 8.600000000000023, 0, 8.399999999999977, 0,
-      7.899999999999977, 0, 8.100000000000023, 0, 8.100000000000023, 0, 8.399999999999977, 0,
-      9.299999999999955, 0, 8.100000000000023, 0, 7.5, 0, 9.200000000000045, 0, 8.5, 0,
-      7.899999999999864, 0, 8.800000000000182, 0, 8.199999999999818, 0, 7.800000000000182, 0, 8.5,
-      0, 8.199999999999818, 0, 8.900000000000091, 0, 8.299999999999955, 0, 7.900000000000091, 0,
-      7.899999999999864, 0, 9.300000000000182, 0, 7.599999999999909, 0, 8.400000000000091, 0, 8, 0,
-      8.399999999999864, 0, 9.200000000000045, 0, 7.400000000000091, 0, 9.299999999999955, 0,
-      8.200000000000045, 0, 7.5, 0, 9.299999999999955, 0, 8.099999999999909, 0, 7.600000000000136,
-      0, 9.299999999999955, 0, 8.200000000000045, 0, 8.399999999999864, 0, 7.5, 0,
-      8.400000000000091, 0, 8.599999999999909, 0, 7.7999999999999545, 0, 8.5, 0, 9.300000000000182,
-      0, 7.2999999999999545, 0, 9.399999999999864, 0, 7.5, 0, 8.100000000000136, 0, 8.5, 0,
-      8.099999999999909, 0, 9.100000000000136, 0, 8.599999999999909, 0, 7.5, 0, 9.099999999999909,
-      0, 7.600000000000136, 0, 8.399999999999864, 0, 8.100000000000136, 0, 8.399999999999864, 0,
-      8.200000000000045, 0, 9.299999999999955, 0, 8, 0, 16.200000000000045, 0, 9.200000000000045, 0,
-      8.299999999999955, 0, 8.299999999999955, 0, 8.400000000000091, 0, 7.900000000000091, 0,
-      8.599999999999909, 0, 8.5, 0, 8.099999999999909, 0, 7.7000000000000455, 0, 8.700000000000045,
-      0, 8.5, 0, 8.5, 0, 7.599999999999909, 0, 8.900000000000091, 0, 8.599999999999909, 0,
-      8.400000000000091, 0, 8.299999999999955, 0, 7.7999999999999545, 0, 8.600000000000136, 0,
-      8.200000000000045, 0, 8.799999999999955, 0, 7.599999999999909, 0, 8.100000000000136, 0,
-      8.599999999999909, 0, 8.299999999999955, 0, 8.100000000000136, 0, 8.599999999999909, 0,
-      9.099999999999909, 0, 8.200000000000045, 0, 7.400000000000091, 0, 8.899999999999864, 0, 8.5,
-      0, 8.5, 0, 7.900000000000091, 0, 8.799999999999955, 0, 7.400000000000091, 0,
-      9.299999999999955, 0, 7.400000000000091, 0, 9.299999999999955, 0, 8.299999999999955, 0,
-      8.299999999999955, 0, 7.400000000000091, 0, 8.899999999999864, 0, 8.800000000000182, 0,
-      7.599999999999909, 0, 8.099999999999909, 0, 8.200000000000045, 0, 9.400000000000091, 0,
-      7.7000000000000455, 0, 8.899999999999864, 0, 7.400000000000091, 0, 8.399999999999864, 0,
-      8.900000000000091, 0, 8, 0, 8.5, 0, 8.799999999999955, 0, 7.7000000000000455, 0,
-      8.200000000000045, 0, 8.700000000000045, 0, 7.899999999999864, 0, 8.799999999999955, 0,
-      8.400000000000091, 0, 8.099999999999909, 0, 8.200000000000045, 0, 8.200000000000045, 0,
-      8.200000000000045, 0, 8.799999999999955, 0, 7.7999999999999545, 0, 8.600000000000136, 0,
-      8.199999999999818, 0, 9.200000000000045, 0, 7.500000000000227, 0, 8.199999999999818, 0,
-      9.099999999999909, 0, 7.700000000000273, 0, 8.899999999999636, 0, 7.600000000000364, 0,
-      8.399999999999636, 0, 9.300000000000182, 0, 7.400000000000091, 0, 8.299999999999727, 0,
-      9.300000000000182, 0, 8.300000000000182, 0, 7.799999999999727, 0, 8.900000000000091, 0,
-      7.699999999999818, 0, 8.800000000000182, 0, 8.300000000000182, 0, 7.599999999999909, 0,
-      8.799999999999727, 0, 7.900000000000091, 0, 9.099999999999909, 0, 8.200000000000273, 0, 8, 0,
-      8.799999999999727, 0, 7.600000000000364, 0, 9.299999999999727, 0, 8.200000000000273, 0,
-      7.699999999999818, 0, 8, 0, 8.599999999999909, 0, 9.099999999999909, 0, 7.300000000000182, 0,
-      8.900000000000091, 0, 8.799999999999727, 0, 8.100000000000364, 0, 7.899999999999636, 0, 8, 0,
-      8.400000000000091, 0, 9.099999999999909, 0, 7.5, 0, 8.800000000000182, 0, 7.900000000000091,
-      0, 9.299999999999727, 0, 7.5, 0, 9.100000000000364, 0, 8.099999999999909, 0,
-      7.799999999999727, 0, 8.700000000000273, 0, 7.799999999999727, 0, 9.300000000000182, 0,
-      8.300000000000182, 0, 8, 0, 8.699999999999818, 0, 7.599999999999909, 0, 9.099999999999909, 0,
-      8.200000000000273, 0, 7.599999999999909, 0, 9.199999999999818, 0, 8.200000000000273, 0,
-      8.400000000000091, 0, 8.299999999999727, 0, 8.200000000000273, 0, 8.5, 0, 8.299999999999727,
-      0, 8.400000000000091, 0, 8.300000000000182, 0, 7.5, 0, 8.5, 0, 8.099999999999909, 0,
-      9.299999999999727, 0, 7.400000000000091, 0, 9, 0, 8, 0, 8.900000000000091, 0,
-      8.300000000000182, 0, 8.299999999999727, 0, 7.5, 0, 8.599999999999909, 0, 8, 0,
-      9.300000000000182, 0, 7.400000000000091, 0, 9.299999999999727, 0, 8.300000000000182, 0,
-      7.599999999999909, 0, 8.800000000000182, 0, 8.099999999999909, 0, 8, 0, 8.800000000000182, 0,
-      8.5, 0, 8.5, 0, 8.099999999999909, 0, 8.5, 0, 8, 0, 8.199999999999818, 0, 7.900000000000091,
-      0, 8.599999999999909, 0, 8.599999999999909, 0, 8, 0, 9.100000000000364, 0, 8.199999999999818,
-      0, 8.5, 0, 8, 0, 7.800000000000182, 0, 8.899999999999636, 0, 7.800000000000182, 0,
-      9.099999999999909, 0, 8.300000000000182, 0, 7.799999999999727, 0, 9, 0, 8.300000000000182, 0,
-      7.5, 0, 9.199999999999818, 0, 7.400000000000091, 0, 9.300000000000182, 0, 8, 0,
-      7.799999999999727, 0, 9.200000000000273, 0, 8, 0, 7.599999999999909, 0, 8.400000000000091, 0,
-      9.299999999999727, 0, 7.300000000000182, 0, 8.5, 0, 9.199999999999818, 0, 8.200000000000273,
-      0, 8.400000000000091, 0, 7.5, 0, 9.199999999999818, 0, 8.300000000000182, 0,
-      7.399999999999636, 0, 8.400000000000091, 0, 9, 0, 8.599999999999909, 0, 8.200000000000273, 0,
-      7.5, 0, 9.299999999999727, 0, 8.200000000000273, 0, 8.400000000000091, 0, 7.399999999999636,
-      0, 8.300000000000182, 0, 9.400000000000091, 0, 8, 0, 8.5, 0, 8.099999999999909, 0, 8, 0, 8, 0,
-      8.900000000000091, 0, 7.900000000000091, 0, 8.299999999999727, 0, 8.300000000000182, 0, 8.5,
-      0, 8.699999999999818, 0, 8.099999999999909, 0, 9, 0, 8.300000000000182, 0, 7.400000000000091,
-      0, 9.099999999999909, 0, 7.800000000000182, 0, 9.099999999999909, 0, 8.199999999999818, 0,
-      8.400000000000091, 0, 8.300000000000182, 0, 7.899999999999636, 0, 8.700000000000273, 0, 8.5,
-      0, 7.400000000000091, 0, 8.299999999999727, 0, 8.300000000000182, 0, 9.299999999999727, 0,
-      8.300000000000182, 0, 8.199999999999818, 0
-    ],
-    "mountedRowSamples": []
-  },
-  "serverMetrics": null,
-  "metadata": {
-    "burstSeedThreadId": "perf-thread-burst",
-    "navigationThreadId": "perf-thread-light-01",
-    "sentinelText": "PERF_STREAM_SENTINEL:dense_assistant_stream:completed"
-  }
-}
\ No newline at end of file

diff --git a/test/perf/support/artifact.ts b/test/perf/support/artifact.ts
--- a/test/perf/support/artifact.ts
+++ b/test/perf/support/artifact.ts
@@ -68,7 +68,10 @@
   }
   const sorted = values.toSorted((left, right) => left - right);
   const clampedTarget = Math.min(Math.max(target, 0), 1);
-  const index = Math.min(sorted.length - 1, Math.ceil(sorted.length * clampedTarget) - 1);
+  const index = Math.min(
+    sorted.length - 1,
+    Math.max(0, Math.ceil(sorted.length * clampedTarget) - 1),
+  );
   return sorted[index] ?? null;
 }
 

diff --git a/test/perf/support/browserMetrics.ts b/test/perf/support/browserMetrics.ts
--- a/test/perf/support/browserMetrics.ts
+++ b/test/perf/support/browserMetrics.ts
@@ -30,15 +30,16 @@
   const rafGapsMs: number[] = [];
   const mountedRowSamples: Array<BrowserPerfMetrics["mountedRowSamples"][number]> = [];
   let previousAnimationFrameTs = 0;
+  let rafHandle = 0;
 
   const animationFrameLoop = (timestampMs: number) => {
     if (previousAnimationFrameTs > 0) {
       rafGapsMs.push(timestampMs - previousAnimationFrameTs);
     }
     previousAnimationFrameTs = timestampMs;
-    window.requestAnimationFrame(animationFrameLoop);
+    rafHandle = window.requestAnimationFrame(animationFrameLoop);
   };
-  window.requestAnimationFrame(animationFrameLoop);
+  rafHandle = window.requestAnimationFrame(animationFrameLoop);
 
   if (typeof PerformanceObserver !== "undefined") {
     try {
@@ -103,7 +104,8 @@
       rafGapsMs.length = 0;
       mountedRowSamples.length = 0;
       previousAnimationFrameTs = 0;
-      window.requestAnimationFrame(animationFrameLoop);
+      window.cancelAnimationFrame(rafHandle);
+      rafHandle = window.requestAnimationFrame(animationFrameLoop);
     },
   };
 }

_{You can send follow-ups to this agent here.}

test/perf/support/browserMetrics.ts

artifacts/perf/virtualization-large_threads-1775077017735/virtualization-large_threads.json

test/perf/support/artifact.ts

Dismissing prior approval to re-evaluate 18f19a6

test/perf/support/artifact.ts

apps/web/test/perf/appHarness.ts

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: Perf provider always enabled even without provider scenario
- Moved PERF_PROVIDER_ENV inside the providerScenarioId conditional so it is only set when a provider scenario is specified, matching the behavior in open-perf-app.ts.

Or push these changes by commenting:

@cursor push e26c2abd97

Preview (e26c2abd97)

diff --git a/apps/web/test/perf/appHarness.ts b/apps/web/test/perf/appHarness.ts
--- a/apps/web/test/perf/appHarness.ts
+++ b/apps/web/test/perf/appHarness.ts
@@ -268,8 +268,12 @@
   const env = {
     ...process.env,
     T3CODE_AUTO_BOOTSTRAP_PROJECT_FROM_CWD: "false",
-    [PERF_PROVIDER_ENV]: "1",
-    ...(options.providerScenarioId ? { [PERF_SCENARIO_ENV]: options.providerScenarioId } : {}),
+    ...(options.providerScenarioId
+      ? {
+          [PERF_PROVIDER_ENV]: "1",
+          [PERF_SCENARIO_ENV]: options.providerScenarioId,
+        }
+      : {}),
   };
 
   let stdoutBuffer = "";

_{You can send follow-ups to this agent here.}

apps/web/test/perf/appHarness.ts

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

✅ Fixed: Failed template creation cached permanently as rejected promise
- Added a .catch handler on the createTemplateDir promise that evicts the entry from templateDirPromises on rejection, allowing subsequent calls to retry.
✅ Fixed: Duplicated utility functions across harness and CLI script
- Extracted pickFreePort, waitForServerReady, stopChildProcess, cleanupPerfRunDir, verifyBuiltArtifacts, and parsePerfSeededState into a shared test/perf/support/perfProcess.ts module and updated both consumers to import from it.

Or push these changes by commenting:

@cursor push 89497fed15

Preview (89497fed15)

diff --git a/apps/server/integration/perf/seedPerfState.ts b/apps/server/integration/perf/seedPerfState.ts
--- a/apps/server/integration/perf/seedPerfState.ts
+++ b/apps/server/integration/perf/seedPerfState.ts
@@ -546,7 +546,10 @@
   if (existing) {
     return existing;
   }
-  const created = createTemplateDir(scenarioId);
+  const created = createTemplateDir(scenarioId).catch((error: unknown) => {
+    templateDirPromises.delete(scenarioId);
+    throw error;
+  });
   templateDirPromises.set(scenarioId, created);
   return created;
 }

diff --git a/apps/web/test/perf/appHarness.ts b/apps/web/test/perf/appHarness.ts
--- a/apps/web/test/perf/appHarness.ts
+++ b/apps/web/test/perf/appHarness.ts
@@ -1,6 +1,5 @@
-import { spawn, type ChildProcess } from "node:child_process";
-import { access, mkdir, rm, writeFile } from "node:fs/promises";
-import { createServer } from "node:net";
+import { spawn } from "node:child_process";
+import { mkdir, writeFile } from "node:fs/promises";
 import { join, resolve } from "node:path";
 import { fileURLToPath } from "node:url";
 import { once } from "node:events";
@@ -18,6 +17,14 @@
   installBrowserPerfCollector,
   PERF_BROWSER_GLOBAL,
 } from "../../../../test/perf/support/browserMetrics";
+import {
+  pickFreePort,
+  waitForServerReady,
+  stopChildProcess,
+  cleanupPerfRunDir,
+  verifyBuiltArtifacts,
+  parsePerfSeededState,
+} from "../../../../test/perf/support/perfProcess";
 import type { PerfThresholdProfile } from "../../../../test/perf/support/thresholds";
 import type {
   PerfProviderScenarioId,
@@ -35,8 +42,6 @@
 const serverClientIndexPath = resolve(repoRoot, "apps/server/dist/client/index.html");
 const PERF_ARTIFACT_DIR_ENV = "T3CODE_PERF_ARTIFACT_DIR";
 const PERF_HEADFUL_ENV = "T3CODE_PERF_HEADFUL";
-const PERF_SEED_JSON_START = "__T3_PERF_SEED_JSON_START__";
-const PERF_SEED_JSON_END = "__T3_PERF_SEED_JSON_END__";
 
 interface PerfSeedThreadSummary {
   readonly id: string;
@@ -104,84 +109,6 @@
   }>;
 }
 
-async function pickFreePort(): Promise<number> {
-  return await new Promise<number>((resolvePort, reject) => {
-    const server = createServer();
-    server.on("error", reject);
-    server.listen(0, "127.0.0.1", () => {
-      const address = server.address();
-      if (!address || typeof address === "string") {
-        reject(new Error("Unable to resolve a free localhost port."));
-        return;
-      }
-      const { port } = address;
-      server.close((closeError) => {
-        if (closeError) {
-          reject(closeError);
-          return;
-        }
-        resolvePort(port);
-      });
-    });
-  });
-}
-
-async function waitForServerReady(url: string, process: ChildProcess): Promise<void> {
-  const startedAt = Date.now();
-  const timeoutMs = 45_000;
-  const requestTimeoutMs = 1_000;
-
-  while (Date.now() - startedAt < timeoutMs) {
-    if (process.exitCode !== null) {
-      throw new Error(`Perf server exited early with code ${process.exitCode}.`);
-    }
-    try {
-      const response = await fetch(url, {
-        redirect: "manual",
-        signal: AbortSignal.timeout(requestTimeoutMs),
-      });
-      if (response.ok) {
-        return;
-      }
-    } catch {
-      // Ignore connection races while the server is still starting.
-    }
-    await new Promise((resolveDelay) => setTimeout(resolveDelay, 200));
-  }
-
-  throw new Error(`Timed out waiting for perf server readiness at ${url}.`);
-}
-
-async function verifyBuiltArtifacts(): Promise<void> {
-  await Promise.all([access(serverBinPath), access(serverClientIndexPath)]).catch(() => {
-    throw new Error(
-      `Built perf artifacts are missing. Expected ${serverBinPath} and ${serverClientIndexPath}. Run bun run test:perf:web or build the app first.`,
-    );
-  });
-}
-
-async function stopChildProcess(process: ChildProcess): Promise<void> {
-  if (process.exitCode !== null) {
-    return;
-  }
-
-  process.kill("SIGTERM");
-  const exited = await new Promise<boolean>((resolveExited) => {
-    const timer = setTimeout(() => resolveExited(false), 5_000);
-    process.once("exit", () => {
-      clearTimeout(timer);
-      resolveExited(true);
-    });
-  });
-
-  if (!exited && process.exitCode === null) {
-    process.kill("SIGKILL");
-    await new Promise<void>((resolveExited) => {
-      process.once("exit", () => resolveExited());
-    });
-  }
-}
-
 async function ensureArtifactDir(suite: string, scenarioId: string): Promise<string> {
   const baseArtifactDir = resolve(
     process.env[PERF_ARTIFACT_DIR_ENV] ?? join(repoRoot, "artifacts/perf"),
@@ -192,10 +119,6 @@
   return artifactDir;
 }
 
-async function cleanupPerfRunDir(runParentDir: string): Promise<void> {
-  await rm(runParentDir, { recursive: true, force: true });
-}
-
 async function writeServerLogs(
   artifactDir: string,
   stdout: string,
@@ -231,22 +154,10 @@
   );
 }
 
-function parsePerfSeededState(stdout: string): PerfSeededState {
-  const startIndex = stdout.lastIndexOf(PERF_SEED_JSON_START);
-  const endIndex = stdout.lastIndexOf(PERF_SEED_JSON_END);
-
-  if (startIndex !== -1 && endIndex !== -1 && endIndex > startIndex) {
-    const payload = stdout.slice(startIndex + PERF_SEED_JSON_START.length, endIndex).trim();
-    return JSON.parse(payload) as PerfSeededState;
-  }
-
-  return JSON.parse(stdout) as PerfSeededState;
-}
-
 export async function startPerfAppHarness(
   options: StartPerfAppHarnessOptions,
 ): Promise<PerfAppHarness> {
-  await verifyBuiltArtifacts();
+  await verifyBuiltArtifacts([serverBinPath, serverClientIndexPath]);
 
   const seededState = await (async () => {
     const seedProcess = spawn(
@@ -270,7 +181,7 @@
     if (exitCode !== 0) {
       throw new Error(`Perf seed command failed with code ${exitCode ?? "unknown"}.\n${stderr}`);
     }
-    return parsePerfSeededState(stdout);
+    return parsePerfSeededState<PerfSeededState>(stdout);
   })();
   const artifactDir = await ensureArtifactDir(options.suite, options.seedScenarioId);
   const port = await pickFreePort();

diff --git a/apps/web/tsconfig.json b/apps/web/tsconfig.json
--- a/apps/web/tsconfig.json
+++ b/apps/web/tsconfig.json
@@ -27,6 +27,7 @@
     "test",
     "../../test/perf/support/artifact.ts",
     "../../test/perf/support/browserMetrics.ts",
+    "../../test/perf/support/perfProcess.ts",
     "../../test/perf/support/serverSampler.ts",
     "../../test/perf/support/thresholds.ts"
   ]

diff --git a/scripts/open-perf-app.ts b/scripts/open-perf-app.ts
--- a/scripts/open-perf-app.ts
+++ b/scripts/open-perf-app.ts
@@ -1,17 +1,22 @@
-import { spawn, type ChildProcess } from "node:child_process";
+import { spawn } from "node:child_process";
 import { once } from "node:events";
-import { access, rm } from "node:fs/promises";
-import { createServer } from "node:net";
 import { resolve } from "node:path";
 import { fileURLToPath } from "node:url";
 
+import {
+  pickFreePort,
+  waitForServerReady,
+  stopChildProcess,
+  cleanupPerfRunDir,
+  verifyBuiltArtifacts,
+  parsePerfSeededState,
+} from "../test/perf/support/perfProcess";
+
 const repoRoot = fileURLToPath(new URL("../", import.meta.url));
 const serverBinPath = resolve(repoRoot, "apps/server/dist/bin.mjs");
 const serverClientIndexPath = resolve(repoRoot, "apps/server/dist/client/index.html");
 const PERF_PROVIDER_ENV = "T3CODE_PERF_PROVIDER";
 const PERF_SCENARIO_ENV = "T3CODE_PERF_SCENARIO";
-const PERF_SEED_JSON_START = "__T3_PERF_SEED_JSON_START__";
-const PERF_SEED_JSON_END = "__T3_PERF_SEED_JSON_END__";
 
 type PerfSeedScenarioId = "large_threads" | "burst_base";
 type PerfProviderScenarioId = "dense_assistant_stream";
@@ -151,47 +156,6 @@
   };
 }
 
-async function pickFreePort(): Promise<number> {
-  return await new Promise<number>((resolvePort, reject) => {
-    const server = createServer();
-    server.on("error", reject);
-    server.listen(0, "127.0.0.1", () => {
-      const address = server.address();
-      if (!address || typeof address === "string") {
-        reject(new Error("Unable to resolve a free localhost port."));
-        return;
-      }
-      server.close((closeError) => {
-        if (closeError) {
-          reject(closeError);
-          return;
-        }
-        resolvePort(address.port);
-      });
-    });
-  });
-}
-
-async function verifyBuiltArtifacts(): Promise<void> {
-  await Promise.all([access(serverBinPath), access(serverClientIndexPath)]).catch(() => {
-    throw new Error(
-      `Built perf artifacts are missing. Expected ${serverBinPath} and ${serverClientIndexPath}. Run bun run test:perf:web or build the app first.`,
-    );
-  });
-}
-
-function parsePerfSeededState(stdout: string): PerfSeededState {
-  const startIndex = stdout.lastIndexOf(PERF_SEED_JSON_START);
-  const endIndex = stdout.lastIndexOf(PERF_SEED_JSON_END);
-
-  if (startIndex === -1 || endIndex === -1 || endIndex <= startIndex) {
-    throw new Error(`Perf seed command did not emit the expected JSON markers.\n${stdout}`);
-  }
-
-  const payload = stdout.slice(startIndex + PERF_SEED_JSON_START.length, endIndex).trim();
-  return JSON.parse(payload) as PerfSeededState;
-}
-
 async function seedPerfState(scenarioId: PerfSeedScenarioId): Promise<PerfSeededState> {
   const seedProcess = spawn("bun", ["run", "apps/server/scripts/seedPerfState.ts", scenarioId], {
     cwd: repoRoot,
@@ -213,61 +177,9 @@
     throw new Error(`Perf seed command failed with code ${exitCode ?? "unknown"}.\n${stderr}`);
   }
 
-  return parsePerfSeededState(stdout);
+  return parsePerfSeededState<PerfSeededState>(stdout);
 }
 
-async function waitForServerReady(url: string, process: ChildProcess): Promise<void> {
-  const startedAt = Date.now();
-  const timeoutMs = 45_000;
-  const requestTimeoutMs = 1_000;
-
-  while (Date.now() - startedAt < timeoutMs) {
-    if (process.exitCode !== null) {
-      throw new Error(`Perf server exited early with code ${process.exitCode}.`);
-    }
-    try {
-      const response = await fetch(url, {
-        redirect: "manual",
-        signal: AbortSignal.timeout(requestTimeoutMs),
-      });
-      if (response.ok) {
-        return;
-      }
-    } catch {
-      // Ignore connection races during startup.
-    }
-    await new Promise((resolveDelay) => setTimeout(resolveDelay, 200));
-  }
-
-  throw new Error(`Timed out waiting for perf server readiness at ${url}.`);
-}
-
-async function stopChildProcess(process: ChildProcess): Promise<void> {
-  if (process.exitCode !== null) {
-    return;
-  }
-
-  process.kill("SIGTERM");
-  const exited = await new Promise<boolean>((resolveExited) => {
-    const timer = setTimeout(() => resolveExited(false), 5_000);
-    process.once("exit", () => {
-      clearTimeout(timer);
-      resolveExited(true);
-    });
-  });
-
-  if (!exited && process.exitCode === null) {
-    process.kill("SIGKILL");
-    await new Promise<void>((resolveExited) => {
-      process.once("exit", () => resolveExited());
-    });
-  }
-}
-
-async function cleanupPerfRunDir(runParentDir: string): Promise<void> {
-  await rm(runParentDir, { recursive: true, force: true });
-}
-
 function openUrl(url: string): void {
   const command: [string, ...string[]] =
     process.platform === "darwin"
@@ -320,7 +232,7 @@
 
 async function main(): Promise<void> {
   const options = parseArgs(process.argv.slice(2));
-  await verifyBuiltArtifacts();
+  await verifyBuiltArtifacts([serverBinPath, serverClientIndexPath]);
   const seededState = await seedPerfState(options.scenarioId);
   const port = options.port === 0 ? await pickFreePort() : options.port;
 

diff --git a/scripts/tsconfig.json b/scripts/tsconfig.json
--- a/scripts/tsconfig.json
+++ b/scripts/tsconfig.json
@@ -12,5 +12,5 @@
       }
     ]
   },
-  "include": ["**/*.ts"]
+  "include": ["**/*.ts", "../test/perf/support/perfProcess.ts"]
 }

diff --git a/test/perf/support/perfProcess.ts b/test/perf/support/perfProcess.ts
new file mode 100644
--- /dev/null
+++ b/test/perf/support/perfProcess.ts
@@ -1,0 +1,100 @@
+import { type ChildProcess } from "node:child_process";
+import { access, rm } from "node:fs/promises";
+import { createServer } from "node:net";
+
+const PERF_SEED_JSON_START = "__T3_PERF_SEED_JSON_START__";
+const PERF_SEED_JSON_END = "__T3_PERF_SEED_JSON_END__";
+
+export async function pickFreePort(): Promise<number> {
+  return await new Promise<number>((resolvePort, reject) => {
+    const server = createServer();
+    server.on("error", reject);
+    server.listen(0, "127.0.0.1", () => {
+      const address = server.address();
+      if (!address || typeof address === "string") {
+        reject(new Error("Unable to resolve a free localhost port."));
+        return;
+      }
+      const { port } = address;
+      server.close((closeError) => {
+        if (closeError) {
+          reject(closeError);
+          return;
+        }
+        resolvePort(port);
+      });
+    });
+  });
+}
+
+export async function waitForServerReady(url: string, process: ChildProcess): Promise<void> {
+  const startedAt = Date.now();
+  const timeoutMs = 45_000;
+  const requestTimeoutMs = 1_000;
+
+  while (Date.now() - startedAt < timeoutMs) {
+    if (process.exitCode !== null) {
+      throw new Error(`Perf server exited early with code ${process.exitCode}.`);
+    }
+    try {
+      const response = await fetch(url, {
+        redirect: "manual",
+        signal: AbortSignal.timeout(requestTimeoutMs),
+      });
+      if (response.ok) {
+        return;
+      }
+    } catch {
+      // Ignore connection races while the server is still starting.
+    }
+    await new Promise((resolveDelay) => setTimeout(resolveDelay, 200));
+  }
+
+  throw new Error(`Timed out waiting for perf server readiness at ${url}.`);
+}
+
+export async function stopChildProcess(process: ChildProcess): Promise<void> {
+  if (process.exitCode !== null) {
+    return;
+  }
+
+  process.kill("SIGTERM");
+  const exited = await new Promise<boolean>((resolveExited) => {
+    const timer = setTimeout(() => resolveExited(false), 5_000);
+    process.once("exit", () => {
+      clearTimeout(timer);
+      resolveExited(true);
+    });
+  });
+
+  if (!exited && process.exitCode === null) {
+    process.kill("SIGKILL");
+    await new Promise<void>((resolveExited) => {
+      process.once("exit", () => resolveExited());
+    });
+  }
+}
+
+export async function cleanupPerfRunDir(runParentDir: string): Promise<void> {
+  await rm(runParentDir, { recursive: true, force: true });
+}
+
+export async function verifyBuiltArtifacts(paths: ReadonlyArray<string>): Promise<void> {
+  await Promise.all(paths.map((p) => access(p))).catch(() => {
+    throw new Error(
+      `Built perf artifacts are missing. Expected ${paths.join(" and ")}. Run bun run test:perf:web or build the app first.`,
+    );
+  });
+}
+
+export function parsePerfSeededState<T>(stdout: string): T {
+  const startIndex = stdout.lastIndexOf(PERF_SEED_JSON_START);
+  const endIndex = stdout.lastIndexOf(PERF_SEED_JSON_END);
+
+  if (startIndex !== -1 && endIndex !== -1 && endIndex > startIndex) {
+    const payload = stdout.slice(startIndex + PERF_SEED_JSON_START.length, endIndex).trim();
+    return JSON.parse(payload) as T;
+  }
+
+  return JSON.parse(stdout) as T;
+}

_{You can send follow-ups to this agent here.}

cursor · 2026-04-02T08:09:45Z

apps/server/integration/perf/seedPerfState.ts

+  const created = createTemplateDir(scenarioId);
+  templateDirPromises.set(scenarioId, created);
+  return created;
+}


Failed template creation cached permanently as rejected promise

Low Severity

getTemplateDir caches the promise returned by createTemplateDir in the module-level templateDirPromises map before it resolves. If createTemplateDir fails (e.g., git not found, disk error), the rejected promise is permanently cached, so every subsequent call for the same scenarioId will immediately return the same rejection instead of retrying. This could cause confusing cascading failures when running multiple perf tests in the same process.

cursor · 2026-04-02T08:09:46Z

scripts/open-perf-app.ts

+      process.once("exit", () => resolveExited());
+    });
+  }
+}


Duplicated utility functions across harness and CLI script

Low Severity

pickFreePort, waitForServerReady, stopChildProcess, cleanupPerfRunDir, verifyBuiltArtifacts, and parsePerfSeededState are duplicated nearly verbatim between scripts/open-perf-app.ts and apps/web/test/perf/appHarness.ts. These could be extracted to a shared module to avoid divergent fixes over time.

Additional Locations (1)

apps/web/test/perf/appHarness.ts#L106-L183

macroscopeapp · 2026-04-02T08:11:54Z

apps/server/integration/perf/seedPerfState.ts

+  );
+  const runtime = ManagedRuntime.make(seedLayer);
+
+  const snapshot = await runtime.runPromise(


🟡 Medium perf/seedPerfState.ts:503

If runtime.runPromise(...) throws (e.g., from eventStore.append or projectionPipeline.projectEvent), runtime.dispose() on line 540 is never called. This leaks the ManagedRuntime resources including the SQLite connection, and because templateDirPromises caches the promise, the leaked runtime persists for the process lifetime. Consider using Effect.tapError or a try/finally pattern to ensure dispose() runs even on failure.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file apps/server/integration/perf/seedPerfState.ts around line 503: If `runtime.runPromise(...)` throws (e.g., from `eventStore.append` or `projectionPipeline.projectEvent`), `runtime.dispose()` on line 540 is never called. This leaks the `ManagedRuntime` resources including the SQLite connection, and because `templateDirPromises` caches the promise, the leaked runtime persists for the process lifetime. Consider using `Effect.tapError` or a `try/finally` pattern to ensure `dispose()` runs even on failure. Evidence trail: apps/server/integration/perf/seedPerfState.ts lines 473-541 (createTemplateDir function): Line 502 creates runtime via `ManagedRuntime.make(seedLayer)`, line 504-522 calls `runtime.runPromise(...)`, line 540 calls `runtime.dispose()`. No try/finally block exists. Line 46 shows module-level cache `templateDirPromises = new Map<PerfSeedScenarioId, Promise<string>>();`. Lines 543-552 show `getTemplateDir` caches the promise before resolution.

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 4 total unresolved issues (including 2 from previous reviews).

Autofix Details

Bugbot Autofix prepared fixes for both issues found in the latest run.

✅ Fixed: Seed process stdout may be incomplete due to 'exit' event
- Changed once(seedProcess, "exit") to once(seedProcess, "close") in both appHarness.ts and open-perf-app.ts to ensure piped stdio buffers are fully drained before parsing stdout.
✅ Fixed: Duplicated buildPerfServerEnv across server and web harnesses
- Extracted buildPerfServerEnv and its env-var constants into a new shared module at @t3tools/shared/perf/serverEnv and updated both consumers to import from it.

Or push these changes by commenting:

@cursor push 82b826072f

Preview (82b826072f)

diff --git a/apps/server/integration/perf/serverPerfHarness.ts b/apps/server/integration/perf/serverPerfHarness.ts
--- a/apps/server/integration/perf/serverPerfHarness.ts
+++ b/apps/server/integration/perf/serverPerfHarness.ts
@@ -33,13 +33,11 @@
   PerfProviderScenarioId,
   PerfSeedScenarioId,
 } from "@t3tools/shared/perf/scenarioCatalog";
+import { buildPerfServerEnv } from "@t3tools/shared/perf/serverEnv";
 import { seedPerfState, type PerfSeededState } from "./seedPerfState.ts";
 
 const repoRoot = fileURLToPath(new URL("../../../../", import.meta.url));
 const PERF_ARTIFACT_DIR_ENV = "T3CODE_PERF_ARTIFACT_DIR";
-const PERF_PROVIDER_ENV = "T3CODE_PERF_PROVIDER";
-const PERF_SCENARIO_ENV = "T3CODE_PERF_SCENARIO";
-const AUTO_BOOTSTRAP_PROJECT_ENV = "T3CODE_AUTO_BOOTSTRAP_PROJECT_FROM_CWD";
 
 const makeWsRpcClient = RpcClient.make(WsRpcGroup);
 type WsRpcClient =
@@ -184,28 +182,6 @@
   ]);
 }
 
-function buildPerfServerEnv(
-  baseEnv: NodeJS.ProcessEnv,
-  providerScenarioId?: PerfProviderScenarioId,
-): NodeJS.ProcessEnv {
-  const env: NodeJS.ProcessEnv = {
-    ...baseEnv,
-    [AUTO_BOOTSTRAP_PROJECT_ENV]: "false",
-  };
-
-  if (!providerScenarioId) {
-    delete env[PERF_PROVIDER_ENV];
-    delete env[PERF_SCENARIO_ENV];
-    return env;
-  }
-
-  return {
-    ...env,
-    [PERF_PROVIDER_ENV]: "1",
-    [PERF_SCENARIO_ENV]: providerScenarioId,
-  };
-}
-
 export class PerfWsRpcClient {
   private readonly runtime: ManagedRuntime.ManagedRuntime<RpcClient.Protocol, never>;
   private readonly clientScope: Scope.Closeable;

diff --git a/apps/web/test/perf/appHarness.ts b/apps/web/test/perf/appHarness.ts
--- a/apps/web/test/perf/appHarness.ts
+++ b/apps/web/test/perf/appHarness.ts
@@ -266,7 +266,7 @@
     seedProcess.stderr?.on("data", (chunk) => {
       stderr += chunk.toString();
     });
-    const [exitCode] = (await once(seedProcess, "exit")) as [number | null];
+    const [exitCode] = (await once(seedProcess, "close")) as [number | null];
     if (exitCode !== 0) {
       throw new Error(`Perf seed command failed with code ${exitCode ?? "unknown"}.\n${stderr}`);
     }

diff --git a/apps/web/test/perf/serverEnv.ts b/apps/web/test/perf/serverEnv.ts
--- a/apps/web/test/perf/serverEnv.ts
+++ b/apps/web/test/perf/serverEnv.ts
@@ -1,27 +1,5 @@
-import type { PerfProviderScenarioId } from "@t3tools/shared/perf/scenarioCatalog";
-
-export const PERF_PROVIDER_ENV = "T3CODE_PERF_PROVIDER";
-export const PERF_SCENARIO_ENV = "T3CODE_PERF_SCENARIO";
-const AUTO_BOOTSTRAP_PROJECT_ENV = "T3CODE_AUTO_BOOTSTRAP_PROJECT_FROM_CWD";
-
-export function buildPerfServerEnv(
-  baseEnv: NodeJS.ProcessEnv,
-  providerScenarioId?: PerfProviderScenarioId,
-): NodeJS.ProcessEnv {
-  const env: NodeJS.ProcessEnv = {
-    ...baseEnv,
-    [AUTO_BOOTSTRAP_PROJECT_ENV]: "false",
-  };
-
-  if (!providerScenarioId) {
-    delete env[PERF_PROVIDER_ENV];
-    delete env[PERF_SCENARIO_ENV];
-    return env;
-  }
-
-  return {
-    ...env,
-    [PERF_PROVIDER_ENV]: "1",
-    [PERF_SCENARIO_ENV]: providerScenarioId,
-  };
-}
+export {
+  buildPerfServerEnv,
+  PERF_PROVIDER_ENV,
+  PERF_SCENARIO_ENV,
+} from "@t3tools/shared/perf/serverEnv";

diff --git a/packages/shared/package.json b/packages/shared/package.json
--- a/packages/shared/package.json
+++ b/packages/shared/package.json
@@ -51,6 +51,10 @@
     "./perf/artifact": {
       "types": "./src/perf/artifact.ts",
       "import": "./src/perf/artifact.ts"
+    },
+    "./perf/serverEnv": {
+      "types": "./src/perf/serverEnv.ts",
+      "import": "./src/perf/serverEnv.ts"
     }
   },
   "scripts": {

diff --git a/packages/shared/src/perf/serverEnv.ts b/packages/shared/src/perf/serverEnv.ts
new file mode 100644
--- /dev/null
+++ b/packages/shared/src/perf/serverEnv.ts
@@ -1,0 +1,27 @@
+import type { PerfProviderScenarioId } from "./scenarioCatalog.ts";
+
+export const PERF_PROVIDER_ENV = "T3CODE_PERF_PROVIDER";
+export const PERF_SCENARIO_ENV = "T3CODE_PERF_SCENARIO";
+const AUTO_BOOTSTRAP_PROJECT_ENV = "T3CODE_AUTO_BOOTSTRAP_PROJECT_FROM_CWD";
+
+export function buildPerfServerEnv(
+  baseEnv: NodeJS.ProcessEnv,
+  providerScenarioId?: PerfProviderScenarioId,
+): NodeJS.ProcessEnv {
+  const env: NodeJS.ProcessEnv = {
+    ...baseEnv,
+    [AUTO_BOOTSTRAP_PROJECT_ENV]: "false",
+  };
+
+  if (!providerScenarioId) {
+    delete env[PERF_PROVIDER_ENV];
+    delete env[PERF_SCENARIO_ENV];
+    return env;
+  }
+
+  return {
+    ...env,
+    [PERF_PROVIDER_ENV]: "1",
+    [PERF_SCENARIO_ENV]: providerScenarioId,
+  };
+}

diff --git a/scripts/open-perf-app.ts b/scripts/open-perf-app.ts
--- a/scripts/open-perf-app.ts
+++ b/scripts/open-perf-app.ts
@@ -208,7 +208,7 @@
     stderr += chunk.toString();
   });
 
-  const [exitCode] = (await once(seedProcess, "exit")) as [number | null];
+  const [exitCode] = (await once(seedProcess, "close")) as [number | null];
   if (exitCode !== 0) {
     throw new Error(`Perf seed command failed with code ${exitCode ?? "unknown"}.\n${stderr}`);
   }

_{You can send follow-ups to this agent here.}

cursor · 2026-04-02T15:28:45Z

apps/web/test/perf/appHarness.ts

+    seedProcess.stderr?.on("data", (chunk) => {
+      stderr += chunk.toString();
+    });
+    const [exitCode] = (await once(seedProcess, "exit")) as [number | null];


Seed process stdout may be incomplete due to 'exit' event

Medium Severity

The code awaits once(seedProcess, "exit") but the child process uses piped stdio. The 'exit' event fires when the process terminates, but pipe buffers may not have fully drained yet — remaining 'data' events can arrive after the 'exit' promise resolves. This means stdout may be incomplete when parsePerfSeededState(stdout) runs, causing JSON parse failures or missing marker errors. Using once(seedProcess, "close") instead ensures all piped data has been consumed before proceeding.

Additional Locations (1)

scripts/open-perf-app.ts#L210-L211

cursor · 2026-04-02T15:28:45Z

apps/server/integration/perf/serverPerfHarness.ts

+    [PERF_PROVIDER_ENV]: "1",
+    [PERF_SCENARIO_ENV]: providerScenarioId,
+  };
+}


Duplicated buildPerfServerEnv across server and web harnesses

Low Severity

buildPerfServerEnv is identically implemented in both serverPerfHarness.ts (private) and serverEnv.ts (exported). This function contains non-trivial env var deletion and conditional provider-enablement logic. Since the shared perf package already exists at @t3tools/shared/perf/, this utility could live there to avoid the two copies diverging when the env var contract changes.

Additional Locations (1)

apps/web/test/perf/serverEnv.ts#L6-L27

- Seed reusable perf fixtures for server scenarios - Add perf provider adapter and web perf tests - Capture artifacts for virtualization and websocket runs

- Refresh virtualization and websocket perf snapshots - Keep perf baselines in sync with latest run

- Expand perf fixtures with multi-thread live stream events and namespaced IDs - Harden stream getters and update web perf tests for the heavier workload

- Keep `bun run test` focused on non-perf suites

- Add README entry for perf benchmarks - Document scenarios, commands, artifacts, and env vars

- Persist and remove the seeded run parent directory after perf runs - Add regression coverage for seed cleanup, percentile clamping, and RAF reset

- Seed large_threads across 5 projects with bounded heavy turns - Add shared perf server env setup and update harness assertions - Co-authored-by: codex <codex@users.noreply.github.com>

Co-authored-by: codex <codex@users.noreply.github.com>

macroscopeapp · 2026-04-13T17:14:57Z

apps/server/integration/perf/seedPerfState.ts

+  const runtime = ManagedRuntime.make(snapshotLayer);
+  const snapshot = await runtime.runPromise(
+    Effect.gen(function* () {
+      const snapshotQuery = yield* ProjectionSnapshotQuery;
+      return yield* snapshotQuery.getSnapshot();
+    }),
+  );
+  await runtime.dispose();


🟡 Medium perf/seedPerfState.ts:573

If runtime.runPromise(...) throws, runtime.dispose() is never reached and ManagedRuntime resources (database connections) leak. Use a try-finally block to ensure disposal runs regardless of success or failure.

- const runtime = ManagedRuntime.make(snapshotLayer); - const snapshot = await runtime.runPromise( - Effect.gen(function* () { - const snapshotQuery = yield* ProjectionSnapshotQuery; - return yield* snapshotQuery.getSnapshot(); - }), - ); - await runtime.dispose(); + const runtime = ManagedRuntime.make(snapshotLayer); + let snapshot: OrchestrationReadModel; + try { + snapshot = await runtime.runPromise( + Effect.gen(function* () { + const snapshotQuery = yield* ProjectionSnapshotQuery; + return yield* snapshotQuery.getSnapshot(); + }), + ); + } finally { + await runtime.dispose(); + }

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file apps/server/integration/perf/seedPerfState.ts around lines 573-580: If `runtime.runPromise(...)` throws, `runtime.dispose()` is never reached and `ManagedRuntime` resources (database connections) leak. Use a try-finally block to ensure disposal runs regardless of success or failure. Evidence trail: apps/server/integration/perf/seedPerfState.ts lines 573-580 at REVIEWED_COMMIT: ManagedRuntime.make creates runtime at line 573, runPromise at lines 574-579, dispose at line 580. No try-finally block wraps the runPromise call - if it rejects, dispose is skipped.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 5 total unresolved issues (including 4 from previous reviews).

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: App harness lacks standalone dispose for failure cleanup
- Added a dispose() method to PerfAppHarness that tears down browser/server/temp files without writing artifacts, matching ServerPerfHarness API, and simplified test finally blocks to use it.

Or push these changes by commenting:

@cursor push 03c098d248

Preview (03c098d248)

diff --git a/apps/web/test/perf/appHarness.ts b/apps/web/test/perf/appHarness.ts
--- a/apps/web/test/perf/appHarness.ts
+++ b/apps/web/test/perf/appHarness.ts
@@ -102,6 +102,7 @@
     readonly browserMetrics: BrowserPerfMetrics;
     readonly serverMetrics: ReadonlyArray<PerfServerMetricSample> | null;
   }>;
+  readonly dispose: () => Promise<void>;
 }
 
 async function pickFreePort(): Promise<number> {
@@ -467,6 +468,10 @@
 
         return await finishPromise;
       },
+      dispose: async () => {
+        await sampler.stop().catch(() => undefined);
+        await teardown();
+      },
     };
   } catch (error) {
     await Promise.allSettled([

diff --git a/apps/web/test/perf/virtualization.perf.test.ts b/apps/web/test/perf/virtualization.perf.test.ts
--- a/apps/web/test/perf/virtualization.perf.test.ts
+++ b/apps/web/test/perf/virtualization.perf.test.ts
@@ -163,19 +163,7 @@
     finished = true;
   } finally {
     if (harness && !finished) {
-      await harness.finishRun({
-        suite: "virtualization",
-        scenarioId: "large_threads",
-        thresholds,
-        metadata: {
-          heavyThreadMessageCount: harness.seededState.threadSummaries.find(
-            (thread) => thread.id === PERF_CATALOG_IDS.largeThreads.heavyAThreadId,
-          )?.messageCount,
-        },
-        actionSummary: {
-          threadSwitchActionPrefix: "thread-switch",
-        },
-      });
+      await harness.dispose();
     }
   }
 });

diff --git a/apps/web/test/perf/websocket-application.perf.test.ts b/apps/web/test/perf/websocket-application.perf.test.ts
--- a/apps/web/test/perf/websocket-application.perf.test.ts
+++ b/apps/web/test/perf/websocket-application.perf.test.ts
@@ -117,24 +117,7 @@
     finished = true;
   } finally {
     if (harness && !finished) {
-      await harness.finishRun({
-        suite: "websocket-application",
-        scenarioId: "dense_assistant_stream",
-        thresholds,
-        metadata: {
-          burstSeedThreadId: PERF_CATALOG_IDS.burstBase.burstThreadId,
-          navigationThreadId: PERF_CATALOG_IDS.burstBase.navigationThreadId,
-          fillerThreadId: PERF_CATALOG_IDS.burstBase.fillerThreadId,
-          navigationLiveAssistantMessageId:
-            PERF_CATALOG_IDS.provider.navigationLiveAssistantMessageId,
-          burstLiveAssistantMessageId: PERF_CATALOG_IDS.provider.burstLiveAssistantMessageId,
-          sentinelText: PERF_PROVIDER_SCENARIOS.dense_assistant_stream.sentinelText,
-        },
-        actionSummary: {
-          threadSwitchActionPrefix: "thread-switch",
-          burstActionName: "burst-completion",
-        },
-      });
+      await harness.dispose();
     }
   }
 });

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit e9fa2b7. Configure here.}

cursor · 2026-04-13T17:15:53Z

apps/web/test/perf/appHarness.ts

+    readonly browserMetrics: BrowserPerfMetrics;
+    readonly serverMetrics: ReadonlyArray<PerfServerMetricSample> | null;
+  }>;
+}


App harness lacks standalone dispose for failure cleanup

Low Severity

PerfAppHarness exposes cleanup only through finishRun, unlike ServerPerfHarness which has a separate dispose method. If finishRun itself throws (e.g. artifact write fails) after teardown already ran, the original test error is masked. And if a test needs to abort without writing an artifact, there's no lightweight disposal path — the finally blocks in test files are forced to call finishRun with full artifact parameters even on early failures.

^{Reviewed by Cursor Bugbot for commit e9fa2b7. Configure here.}

github-actions bot added size:XXL 1,000+ changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. labels Apr 1, 2026

macroscopeapp bot reviewed Apr 1, 2026

View reviewed changes

test/perf/support/browserMetrics.ts Outdated Show resolved Hide resolved

apps/server/integration/perf/seedPerfState.test.ts Outdated Show resolved Hide resolved

cursor bot reviewed Apr 1, 2026

View reviewed changes

test/perf/support/browserMetrics.ts Show resolved Hide resolved

artifacts/perf/virtualization-large_threads-1775077017735/virtualization-large_threads.json Outdated Show resolved Hide resolved

test/perf/support/artifact.ts Outdated Show resolved Hide resolved

macroscopeapp bot previously approved these changes Apr 1, 2026

View reviewed changes

juliusmarminge force-pushed the t3code/performance-regression-tests branch from b9a8795 to 18f19a6 Compare April 2, 2026 01:57

macroscopeapp bot reviewed Apr 2, 2026

View reviewed changes

test/perf/support/artifact.ts Outdated Show resolved Hide resolved

apps/web/test/perf/appHarness.ts Outdated Show resolved Hide resolved

cursor bot reviewed Apr 2, 2026

View reviewed changes

apps/web/test/perf/appHarness.ts Outdated Show resolved Hide resolved

cursor bot reviewed Apr 2, 2026

View reviewed changes

macroscopeapp bot reviewed Apr 2, 2026

View reviewed changes

juliusmarminge mentioned this pull request Apr 2, 2026

Reduce checkpoint reactor stream fanout #1687

Open

cursor bot reviewed Apr 2, 2026

View reviewed changes

juliusmarminge and others added 8 commits April 13, 2026 10:05

Add perf regression test harnesses

eecc2bc

- Seed reusable perf fixtures for server scenarios - Add perf provider adapter and web perf tests - Capture artifacts for virtualization and websocket runs

Record updated performance regression artifacts

971df82

- Refresh virtualization and websocket perf snapshots - Keep perf baselines in sync with latest run

Add perf regression coverage for dense stream workloads

4bf8d7b

- Expand perf fixtures with multi-thread live stream events and namespaced IDs - Harden stream getters and update web perf tests for the heavier workload

Exclude perf tests from Vitest default runs

ad4eb99

- Keep `bun run test` focused on non-perf suites

Document local performance benchmark harness

1f83f78

- Add README entry for perf benchmarks - Document scenarios, commands, artifacts, and env vars

Clean up perf run dirs and cover collector reset

0ea58f2

- Persist and remove the seeded run parent directory after perf runs - Add regression coverage for seed cleanup, percentile clamping, and RAF reset

Broaden perf seed coverage across multiple projects

885a825

- Seed large_threads across 5 projects with bounded heavy turns - Add shared perf server env setup and update harness assertions - Co-authored-by: codex <codex@users.noreply.github.com>

Add server perf latency baselines

e9fa2b7

Co-authored-by: codex <codex@users.noreply.github.com>

juliusmarminge force-pushed the t3code/performance-regression-tests branch from 0be82d3 to e9fa2b7 Compare April 13, 2026 17:11

macroscopeapp bot reviewed Apr 13, 2026

View reviewed changes

cursor bot reviewed Apr 13, 2026

View reviewed changes

Conversation

juliusmarminge commented Apr 1, 2026 • edited by macroscopeapp bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Add performance regression test harnesses for server and web

Uh oh!

coderabbitai bot commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Uh oh!

macroscopeapp bot commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approvability

Uh oh!

cursor bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot Apr 2, 2026

Choose a reason for hiding this comment

Failed template creation cached permanently as rejected promise

Uh oh!

cursor bot Apr 2, 2026

Choose a reason for hiding this comment

Duplicated utility functions across harness and CLI script

Uh oh!

macroscopeapp bot Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

cursor bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot Apr 2, 2026

Choose a reason for hiding this comment

Seed process stdout may be incomplete due to 'exit' event

Uh oh!

cursor bot Apr 2, 2026

Choose a reason for hiding this comment

Duplicated buildPerfServerEnv across server and web harnesses

Uh oh!

macroscopeapp bot Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

cursor bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot Apr 13, 2026

Choose a reason for hiding this comment

App harness lacks standalone dispose for failure cleanup

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

juliusmarminge commented Apr 1, 2026 •

edited by macroscopeapp bot

Loading

coderabbitai bot commented Apr 1, 2026 •

edited

Loading

macroscopeapp bot commented Apr 1, 2026 •

edited

Loading

cursor bot left a comment •

edited

Loading

cursor bot left a comment •

edited

Loading

cursor bot left a comment •

edited

Loading

cursor bot left a comment •

edited

Loading

Duplicated `buildPerfServerEnv` across server and web harnesses

cursor bot left a comment •

edited

Loading