Require measured WebGPU frame completion and reject missing performance data

.github/workflows/PIPELINES.md

@@ -1,26 +1,85 @@
 # Performance Tracking
 
-## Improvement Checklist
-- [ ] Profile GPU memory usage
-- [x] Reduce vertex buffer updates
-- [x] Cache computations on the GPU side
-- [ ] Minimize CPU/GPU synchronization
+The required CI performance job runs a release build of `tests/performance_limit.rs`
+on Ubuntu 24.04 with Rust 1.99.0, Chrome 155.0.8059.79 and SwiftShader.
+It uses deterministic 1,000 / 50,000 / 1,000,000-candle base datasets, an
+800x600 canvas and the production renderer's LOD and 4x MSAA.
 
-### Additional Changes
-- [x] `CandleSeries` switched to `VecDeque`
-- [x] Store `Chart` in state
- - [x] Log vector on `VecDeque`
+Each dataset receives 10 completed warmup frames followed by five samples of
+30 frames. Every frame calls the real renderer and awaits the WebGPU queue
+completion via a four-byte readback fence. This includes fence and mapping
+overhead; wgpu 25 does not implement its queue-completion callback on WebGPU.
+A missing adapter, failed render, empty geometry, missing
+sample, duplicate sample, zero, negative or non-finite value fails the job.
 
-## WebGPU Performance Benchmark
-- [x] Measure FPS for different data volumes
-- [x] Log render pass times through `wgpu::CommandEncoder::finish`
-- [x] Use `Performance.now()` in the browser for frame timing
-- [x] Automate running a fixed-candle test scene
+This measures serialized, cached-chart throughput including CPU submission,
+browser scheduling and GPU completion latency. It does not measure display FPS,
+initial bulk-load cost, live updates, physical GPU speed or memory consumption.
+The million source candles are reduced by the production LOD; they are not a
+million individual GPU draw calls.
 
-The benchmark is run with:
+`scripts/parse_perf_log.py` keeps the three workloads separate. The comparator
+requires five positive measurements for every workload in both the result and
+the reference. The reference is pinned to reviewed commit
+`1b79e9df3d64542d367a1291273a48dff5ce57a6` and measured first in a separate
+checkout on the same runner, with the same compiler, browser and profile as the
+candidate. Any workload median more than 30% below that reference fails. Missing
+reference measurements fail too; a fast workload cannot hide a slower one.
+The candidate and reference measurement harness must match byte for byte before
+either runs. Its hash and the shared browser-profile hash are recorded; changing
+dataset/canvas/warmup/sample parameters requires a reviewed reference update.
+
+An absolute cross-host baseline was unsuitable: identical renderer code measured
+about 160 and 91-109 completed frames/s on different hosted machines. Paired
+measurements reduce that host variance without increasing the threshold. The
+broad guard still cannot establish a 5% guarantee or eliminate within-run load
+changes. `DOCS/perf_baseline.json` retains the original calibration as historical
+evidence; its absolute numbers are not the CI acceptance threshold.
+
+Run a candidate measurement in a disposable checkout with the pinned compiler/browser,
+a matching wasm-bindgen runner and ChromeDriver on PATH. Set `CHROME_BINARY` to
+the browser executable; the preparation replaces only this checkout's profile:
 
 ```bash
-wasm-pack test --chrome --headless
+set -o pipefail
+python3 - <<'PY'
+import json
+import os
+from pathlib import Path
+config = json.loads(Path('webdriver-perf.json').read_text())
+config['goog:chromeOptions']['binary'] = os.environ['CHROME_BINARY']
+Path('webdriver.json').write_text(json.dumps(config))
+PY
+WASM_BINDGEN_TEST_ONLY_WEB=1 WASM_BINDGEN_TEST_TIMEOUT=180 \
+CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_RUNNER=wasm-bindgen-test-runner \
+cargo test --release --locked --features performance --test performance_limit -- --nocapture 2>&1 | tee perf.log
+python3 scripts/parse_perf_log.py perf.log benchmark_result.json
 ```
 
-Workflow `perf.yml` stores the `perf-log` file.
+The reusable workflow performs the complete paired comparison, using
+`reference_result.json` from its same-runner reference measurement. Dispatch that
+workflow to reproduce the CI gate instead of comparing unrelated host results.
+
+`benchmark.yml` is called by CI for every PR and main push; it can also be
+dispatched manually. It always retains raw samples, comparison and environment
+metadata as `completed-frame-performance`. The duplicate unmeasured `perf.yml`
+workflow has been removed. These measurement jobs have read-only permissions
+and no notification credentials.
+
+Changing the pinned reference requires review with raw paired samples, both
+tested SHAs, browser/compiler versions and software backend. Review changes to
+workload or environment before recalibrating; do not advance the reference to
+a regressed implementation just to make CI green.
+
+The dedicated `webdriver-perf.json` selects SwiftShader for both ANGLE and Dawn
+WebGPU/Vulkan. Selecting only ANGLE allowed a browser adapter to be created but
+its device was destroyed before readback. A raw browser preflight now requires
+successful mapping and verifies the SwiftShader adapter before measuring Rust.
+The job writes that profile to `webdriver.json`, the only filename supported by
+the locked wasm-bindgen 0.2.100 runner. Other jobs have independent checkouts.
+
+Provenance records both `tested_sha` (the checked-out workflow SHA, a synthetic
+merge commit for pull requests) and `tested_head_sha` (the PR head; identical to
+`tested_sha` for pushes/manual runs), the pinned reference SHA and CPU details.
+Reference/candidate logs and complete sample sets are retained separately.
+Baseline calibration preserves its original tested workflow SHA and PR head.

.github/workflows/benchmark.yml

@@ -1,73 +1,133 @@
 name: benchmark
 on:
+  workflow_call:
   workflow_dispatch:
-  push:
-    branches: [main]
 
 permissions:
   contents: read
-concurrency:
-  group: benchmark-${{ github.ref }}
-  cancel-in-progress: true
 
 jobs:
-  perf:
-    runs-on: ubuntu-latest
+  measure:
+    runs-on: ubuntu-24.04
+    timeout-minutes: 30
+    env:
+      CARGO_INCREMENTAL: '0'
     steps:
       - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
+      - name: Check out the reviewed measurement reference
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          repository: qqrm/webgpu-candles
+          ref: 1b79e9df3d64542d367a1291273a48dff5ce57a6
+          path: _perf_reference
+          persist-credentials: false
       - uses: dtolnay/rust-toolchain@e2a55d2ffb04f378e9626c28d38b36d230d1e12f # master
         with:
-          toolchain: stable
+          toolchain: '1.99.0'
           targets: wasm32-unknown-unknown
-      - name: Cache cargo
-        uses: actions/cache@caa296126883cff596d87d8935842f9db880ef25 # v5
+      - name: Read the locked WASM runner version
+        id: bindgen
+        run: |
+          python3 - <<'PY'
+          import os
+          import tomllib
+          from pathlib import Path
+          packages = tomllib.loads(Path('Cargo.lock').read_text())['package']
+          versions = {p['version'] for p in packages if p['name'] == 'wasm-bindgen'}
+          assert len(versions) == 1, versions
+          with open(os.environ['GITHUB_OUTPUT'], 'a') as output:
+              output.write(f'version={versions.pop()}\n')
+          PY
+      - uses: taiki-e/install-action@f7e5d7c961414b23f5b25b2da9294395d08513ad # v2.87.26
         with:
-          path: |
-            ~/.cargo/registry
-            ~/.cargo/git
-            target
-          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
-          restore-keys: |
-            ${{ runner.os }}-cargo-
-      - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
+          tool: wasm-bindgen@${{ steps.bindgen.outputs.version }}
+          fallback: none
+      - uses: browser-actions/setup-chrome@48ad923757ca74d66703209fe939badbdf80f2f4 # v2
+        id: chrome
         with:
-          python-version: '3.x'
-      - name: Install wasm-pack
-        run: cargo install wasm-pack --version 0.13.1 --locked
-      - name: Run benchmarks
+          chrome-version: '155.0.8059.79'
+          install-chromedriver: true
+      - name: Select fixed software WebGPU browser and record environment
         env:
-          WASM_BINDGEN_TEST_BROWSER_ARGS: "--no-sandbox --enable-unsafe-webgpu"
-        run: |
-          set -o pipefail
-          wasm-pack test --chrome --headless -- --locked | tee perf.log
-          python scripts/parse_perf_log.py perf.log benchmark_result.json
-      - name: Analyze FPS
+          CHROME_BINARY: ${{ steps.chrome.outputs.chrome-path }}
+          TESTED_SHA: ${{ github.sha }}
+          TESTED_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
+          EVENT_NAME: ${{ github.event_name }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
         run: |
-          python scripts/compare_fps.py benchmark_result.json DOCS/perf_baseline.json DOCS/perf_result.json 5
-      - name: Update perf markdown
+          python3 - <<'PY'
+          import hashlib
+          import json
+          import os
+          import platform
+          import subprocess
+          from pathlib import Path
+          harness = Path('tests/performance_limit.rs').read_bytes()
+          reference_harness = Path('_perf_reference/tests/performance_limit.rs').read_bytes()
+          assert harness == reference_harness, 'Workload changed: review and update the pinned measurement reference'
+          config = json.loads(Path('webdriver-perf.json').read_text())
+          config['goog:chromeOptions']['binary'] = os.environ['CHROME_BINARY']
+          profile = json.dumps(config)
+          # The locked wasm-bindgen 0.2.100 runner reads only webdriver.json.
+          # Materialize the dedicated profile in this isolated job checkout.
+          Path('webdriver.json').write_text(profile)
+          Path('_perf_reference/webdriver.json').write_text(profile)
+          reference_sha = subprocess.check_output(
+              ['git', '-C', '_perf_reference', 'rev-parse', 'HEAD'], text=True).strip()
+          assert reference_sha == '1b79e9df3d64542d367a1291273a48dff5ce57a6'
+          metadata = {
+              'tested_sha': os.environ['TESTED_SHA'],
+              'tested_head_sha': os.environ['TESTED_HEAD_SHA'],
+              'event': os.environ['EVENT_NAME'],
+              'run_url': os.environ['RUN_URL'],
+              'browser': subprocess.check_output([os.environ['CHROME_BINARY'], '--version'], text=True).strip(),
+              'rust': subprocess.check_output(['rustc', '--version'], text=True).strip(),
+              'os': platform.platform(),
+              'cpu': platform.processor(),
+              'backend': 'SwiftShader software WebGPU',
+              'browser_flags': config['goog:chromeOptions']['args'],
+              'cpu_details': json.loads(subprocess.check_output(['lscpu', '--json'], text=True)),
+              'reference_sha': reference_sha,
+              'harness_sha256': hashlib.sha256(harness).hexdigest(),
+              'browser_profile_sha256': hashlib.sha256(profile.encode()).hexdigest(),
+              'comparison': 'candidate versus pinned reference measured on this same runner',
+          }
+          Path('perf-environment.json').write_text(json.dumps(metadata, indent=2) + '\n')
+          PY
+      - name: Measure the pinned reference on this runner
+        env:
+          CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_RUNNER: wasm-bindgen-test-runner
+          CHROMEDRIVER: ${{ steps.chrome.outputs.chromedriver-path }}
+          WASM_BINDGEN_TEST_ONLY_WEB: '1'
+          WASM_BINDGEN_TEST_TIMEOUT: '180'
         run: |
-          python - <<'EOF'
-          import json, pathlib
-          data = json.load(open('DOCS/perf_result.json'))
-          lines = ['# Latest metrics', '', '| Metric | Value |', '| --- | --- |']
-          for k, v in data.items():
-              lines.append(f'| {k} | {v} |')
-          pathlib.Path('DOCS/perf.md').write_text('\n'.join(lines) + '\n')
-          EOF
-      - uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
-        with:
-          name: perf_results
-          path: DOCS/perf_result.json
-      - name: Send benchmark results to bot
-        continue-on-error: true
+          set -o pipefail
+          (cd _perf_reference && cargo test --release --locked --features performance --test performance_limit -- --nocapture) 2>&1 | tee reference-perf.log
+          python3 scripts/parse_perf_log.py reference-perf.log reference_result.json
+      - name: Measure completed frames for each deterministic dataset
         env:
-          TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }}
-          TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }}
+          CARGO_TARGET_WASM32_UNKNOWN_UNKNOWN_RUNNER: wasm-bindgen-test-runner
+          CHROMEDRIVER: ${{ steps.chrome.outputs.chromedriver-path }}
+          WASM_BINDGEN_TEST_ONLY_WEB: '1'
+          WASM_BINDGEN_TEST_TIMEOUT: '180'
         run: |
-          if [[ -z "$TELEGRAM_BOT_TOKEN" || -z "$TELEGRAM_CHAT_ID" ]]; then
-            echo "Telegram secrets are not configured; skipping notification."
-            exit 0
-          fi
-          scripts/send_to_bot.sh DOCS/perf_result.json "Benchmark results"
+          set -o pipefail
+          cargo test --release --locked --features performance --test performance_limit -- --nocapture 2>&1 | tee perf.log
+          python3 scripts/parse_perf_log.py perf.log benchmark_result.json
+      - name: Reject per-workload throughput regressions
+        run: python3 scripts/compare_fps.py benchmark_result.json reference_result.json perf_result.json 30
+      - name: Preserve measurements and environment, including failures
+        if: always()
+        uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
+        with:
+          name: completed-frame-performance
+          path: |
+            perf.log
+            reference-perf.log
+            reference_result.json
+            perf-environment.json
+            benchmark_result.json
+            perf_result.json
+          if-no-files-found: error

.github/workflows/build.yml

@@ -22,7 +22,7 @@ jobs:
           toolchain: stable
           targets: wasm32-unknown-unknown
       - name: Cache cargo
-        uses: actions/cache@caa296126883cff596d87d8935842f9db880ef25 # v5
+        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: |
             ~/.cargo/registry

.github/workflows/ci.yml

@@ -103,7 +103,7 @@ jobs:
         with:
           toolchain: stable
           targets: wasm32-unknown-unknown
-      - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
+      - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
         with:
           node-version: '24'
       - name: Read the locked WASM runner version
@@ -142,7 +142,7 @@ jobs:
         with:
           toolchain: stable
           targets: wasm32-unknown-unknown
-      - uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
+      - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
         with:
           node-version: '24'
           cache: npm
@@ -176,9 +176,11 @@ jobs:
           fallback: none
       - name: Audit the committed lockfile against current RustSec advisories
         run: cargo audit --deny unsound
+  performance:
+    uses: ./.github/workflows/benchmark.yml
   ci-gate:
     if: always()
-    needs: [checks, render-tests, node-tests, browser, dependency-audit]
+    needs: [checks, render-tests, node-tests, browser, dependency-audit, performance]
     runs-on: ubuntu-latest
     timeout-minutes: 2
     steps:
@@ -189,9 +191,11 @@ jobs:
           NODE: ${{ needs.node-tests.result }}
           BROWSER: ${{ needs.browser.result }}
           AUDIT: ${{ needs.dependency-audit.result }}
+          PERFORMANCE: ${{ needs.performance.result }}
         run: |
           test "$CHECKS" = success
           test "$RENDER" = success
           test "$NODE" = success
           test "$BROWSER" = success
           test "$AUDIT" = success
+          test "$PERFORMANCE" = success