Skip to content

Commit c601c41

Browse files
committed
fix(perf)!: unify local and release performance workflows
- Retain local Markdown, CSV, and provenance artifacts while allowing same-version comparisons of tracked changes. - Exclude untracked files explicitly and isolate the narrowed non-exact comparison bundle. - Promote distinct-release artifacts through performance-doc and keep performance-release atomic and rollback-capable. - Render one consolidated table per benchmark suite and reject invalid release publication states. BREAKING CHANGE: rename performance-rerender to performance-doc and performance-local-vs-linalg to performance-local-non-exact, replace --rerender with --promote-artifacts, and remove performance-archive-published.
1 parent a94e698 commit c601c41

14 files changed

Lines changed: 601 additions & 238 deletions

CONTRIBUTING.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -183,9 +183,11 @@ just performance-local
183183
```
184184

185185
It writes `target/bench-reports/performance.md` without changing committed
186-
release documentation. Treat regressions as design feedback. If a slowdown is
187-
intentional, explain the correctness, API clarity, or composability benefit
188-
that justifies it.
186+
release documentation or requiring a version bump. Staged and unstaged changes
187+
to tracked files participate; untracked files are excluded, so stage any new
188+
benchmark-relevant file first. Treat regressions as design feedback. If a
189+
slowdown is intentional, explain the correctness, API clarity, or composability
190+
benefit that justifies it.
189191

190192
Core Rust, Cargo, or public-behavior changes must pass `just ci` before a pull
191193
request is ready. Documentation, configuration, Python, test-only,

docs/BENCHMARKING.md

Lines changed: 72 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,12 @@ the commands measure and where their outputs go.
2525
| Goal | Recipe |
2626
|------|--------|
2727
| Latest-release local audit | `just performance-local` |
28-
| Release-signal check against tags | `just performance-local-vs-linalg v0.4.4 v0.4.3` |
28+
| Non-exact release-signal check against tags | `just performance-local-non-exact v0.4.4 v0.4.3` |
2929
| Fast saved-baseline loop | `just bench-save-baseline <name> <suite>` then `just bench-compare <name> <suite> all-benches` |
3030
| Full crate comparison | `just bench-vs-linalg` |
3131
| README table and plot | `just plot-vs-linalg-readme` |
3232
| Release report | `just performance-release v0.4.4 v0.4.3` |
33-
| Re-render retained release inputs | `just performance-rerender` |
33+
| Build docs from retained release inputs | `just performance-doc` |
3434
| Published-asset comparison | `just performance-github-assets v0.4.4 v0.4.3` |
3535

3636
Rule of thumb:
@@ -40,13 +40,34 @@ Rule of thumb:
4040
- Use `bench-vs-linalg` plus plotting when updating README crate-to-crate
4141
comparisons.
4242
- Use `performance-release` only when preparing committed release artifacts.
43-
- Use `performance-rerender` for report-format changes after a valid release
44-
dataset has already been retained.
43+
- Use `performance-doc` for report-format changes after a valid, promotable
44+
comparison dataset has already been retained.
45+
46+
The three canonical workflows compose around one artifact schema, metric set,
47+
and renderer:
48+
49+
| Recipe | Measure | Retain CSV/JSON | Promote release docs |
50+
|--------|---------|-----------------|----------------------|
51+
| `performance-local` | Yes | Yes | No |
52+
| `performance-doc` | No | Consumes retained inputs | Yes |
53+
| `performance-release` | Yes | Yes | Yes |
54+
55+
For a distinct release pair with no intervening source or configuration changes,
56+
running `performance-local` followed by `performance-doc` produces the same
57+
report and committed documentation as `performance-release`.
58+
`performance-release` exists as the safer one-step release operation: it keeps
59+
fresh measurement, validated artifact publication, and rollback-capable document
60+
promotion in one command.
4561

4662
## Benchmark Suites
4763

4864
`la-stack` has two Criterion benchmark suites.
4965

66+
Newly rendered reports use one table per selected suite. Dimension and
67+
adversarial-input group appear in a `Case` column instead of creating a separate
68+
table for every group. The `vs_linalg` table is the one wider variant because it
69+
adds nalgebra and faer context columns where matching peer measurements exist.
70+
5071
**`vs_linalg`** (`benches/vs_linalg.rs`) compares `la-stack` against
5172
`nalgebra` and `faer` across D=2-64 for LU, solve, determinant, dot, norm, and
5273
SPD factorization operations. Use this suite to answer "why choose la-stack over
@@ -76,16 +97,20 @@ just performance-local
7697

7798
This creates isolated temporary worktrees and runs both library revisions on the
7899
same machine with the current checkout's benchmark sources, manifests, lockfile,
79-
benchmark-input tests, recipes, and Rust toolchain. Only the baseline library
80-
implementation comes from the release tag. Before either timing run, the command
81-
runs `just test-bench-inputs` against that revision under the shared current
82-
fixture harness. This is a prerequisite correctness gate over the deterministic
83-
fixtures and operations, not validation of each timed Criterion sample. It
84-
writes `target/bench-reports/performance.md` and records both
85-
commits, CPU, operating system, Rust toolchain, lockfile and harness digests,
86-
Criterion selection/commands, and both correctness-gate results. The report
87-
reader rejects malformed or mismatched provenance and incomplete selected-suite
88-
coverage.
100+
benchmark-input tests, recipes, and Rust toolchain. Staged and unstaged changes
101+
to tracked files are applied to the current worktree. Untracked files are
102+
excluded; stage a new file before running the command if it must participate in
103+
the comparison. Only the baseline library implementation comes from the release
104+
tag. Before either timing run, the command runs `just test-bench-inputs` against
105+
that revision under the shared current fixture harness. This is a prerequisite
106+
correctness gate over the deterministic fixtures and operations, not validation
107+
of each timed Criterion sample. It writes
108+
`target/bench-reports/performance.md` plus retained `performance.csv` and
109+
`performance.provenance.json` comparison inputs. The report and sidecar embed
110+
both commits, CPU, operating system, Rust toolchain, lockfile and harness
111+
digests, Criterion selection/commands, and both correctness-gate results. The
112+
report reader rejects malformed or mismatched provenance and incomplete
113+
selected-suite coverage.
89114

90115
The shared harness carries an explicit v0.4.3-only API adapter for renamed or
91116
ownership-adjusted calls (`det_sign_exact`, `Tolerance`, and vector dot
@@ -116,24 +141,33 @@ timing, current validation, and current timing, so a long comparison exposes
116141
completed samples and its active phase instead of remaining silent until the
117142
final report is rendered.
118143

119-
If the checkout's package version is identical to the latest published release,
120-
the command now stops before creating worktrees or running benchmarks because a
121-
release report requires two distinct identifiers. For repeated optimization
122-
within one package version, use the named-baseline loop below instead.
144+
The local report may compare a checkout whose package version is identical to
145+
the latest published release. Commit/ref and source-state provenance distinguish
146+
the modified checkout from the tagged baseline even though both display the same
147+
package version. Release artifact publication remains stricter and requires two
148+
distinct release identifiers.
123149

124150
### Compare Current Code With A Specific Release
125151

126152
For a narrower non-exact check against a known release pair, run:
127153

128154
```bash
129-
just performance-local-vs-linalg v0.4.4 v0.4.3
155+
just performance-local-non-exact v0.4.4 v0.4.3
130156
```
131157

132158
This generates a local `v0.4.3` `vs_linalg` baseline, measures the current
133159
la-stack `vs_linalg` rows, and renders a `vs_linalg` report. The report includes
134160
saved baseline nalgebra/faer timings as context where matching peer rows exist,
135161
without rerunning current peer crates.
136162

163+
This narrowed peer-context view uses the same metrics and renderer but writes a
164+
separate `performance-non-exact.*` scratch bundle so it cannot replace the
165+
canonical full comparison inputs accidentally.
166+
167+
When tags are provided explicitly, the current tag must match the package
168+
version in the `HEAD` checkout. A mismatch is rejected before tags are fetched,
169+
worktrees are created, or benchmarks run.
170+
137171
### Iterate Against A Local Saved Baseline
138172

139173
Use local saved baselines when tuning one kernel and comparing several edits
@@ -240,13 +274,18 @@ To reproduce and promote the report without running Cargo or creating Git
240274
worktrees, use:
241275

242276
```bash
243-
just performance-rerender
277+
just performance-doc
244278
```
245279

246280
This command fails closed on a missing, partial, malformed, mismatched, or
247-
unsupported artifact pair. Use it for presentation-only report corrections;
248-
changes to benchmark inputs, code, toolchains, or measurement configuration
249-
require a fresh `performance-release` run.
281+
unsupported artifact pair. It consumes the default CSV/JSON pair retained by a
282+
successful `performance-local` or `performance-release` run, rewrites the
283+
scratch Markdown, promotes it to `docs/PERFORMANCE.md`, and archives the previous
284+
committed report when the release pair changes. Promotion requires distinct
285+
current and baseline package versions, so a same-version local comparison is
286+
retained and reproducible but cannot become release documentation. Use promotion
287+
for presentation-only report corrections; changes to benchmark inputs, code,
288+
toolchains, or measurement configuration require a fresh local or release run.
250289

251290
### Compare Published Release Artifacts
252291

@@ -276,14 +315,15 @@ shared-harness workflow before attributing a difference solely to library code.
276315
| Path | Committed? | Producer | Purpose |
277316
|------|------------|----------|---------|
278317
| `target/criterion/` | No | `cargo bench`, `bench-save-*` | Local Criterion measurements and named baselines. |
279-
| `target/bench-reports/performance.md` | No | `bench-compare`, `performance-local*` | Local comparison report. |
280-
| `target/bench-reports/performance.csv` | No | `performance-local*`, `performance-release` | Validated tabular inputs for the release comparison. |
281-
| `target/bench-reports/performance.provenance.json` | No | `performance-local*`, `performance-release` | Schema, release, source, command, toolchain, host, digest, and harness provenance. |
318+
| `target/bench-reports/performance.md` | No | `bench-compare`, `performance-local`, `performance-release`, `performance-doc` | Canonical local comparison report. |
319+
| `target/bench-reports/performance.csv` | No | `performance-local`, `performance-release` | Validated tabular inputs for the canonical comparison. |
320+
| `target/bench-reports/performance.provenance.json` | No | `performance-local`, `performance-release` | Schema, package identifiers, source, command, toolchain, host, digest, and harness provenance. |
321+
| `target/bench-reports/performance-non-exact.*` | No | `performance-local-non-exact` | Narrowed non-exact report and retained peer-context comparison inputs. |
282322
| `target/bench-reports/github-assets-performance.md` | No | `performance-github-assets` | Local report from published release artifacts. |
283323
| `target/bench-reports/github-assets-performance.csv` | No | `performance-github-assets` | Tabular inputs derived from published native archives. |
284324
| `target/bench-reports/github-assets-performance.provenance.json` | No | `performance-github-assets` | Provenance for the published-asset report inputs. |
285-
| `docs/PERFORMANCE.md` | Yes | `performance-release` | Latest curated release-to-release comparison. |
286-
| `docs/archive/performance/` | Yes | `performance-release` | Older curated release-to-release comparisons. |
325+
| `docs/PERFORMANCE.md` | Yes | `performance-release`, `performance-doc` | Latest curated release-to-release comparison. |
326+
| `docs/archive/performance/` | Yes | `performance-release`, `performance-doc` | Older curated release-to-release comparisons. |
287327
| `docs/assets/bench/` | Yes | `plot-vs-linalg-readme` | README benchmark CSV/SVG assets and JSON provenance. |
288328
| GitHub Release | Remote | `.github/workflows/release-benchmarks.yml` | Criterion baseline archive. |
289329

@@ -293,9 +333,10 @@ Published baseline assets use the filename
293333
Everything under `target/bench-reports/` is reproducible local scratch owned by
294334
the performance-report workflows. It survives temporary-worktree cleanup but
295335
may be removed by `just clean` or `cargo clean`; retain or copy the CSV/JSON pair
296-
while reviewing or rerendering a release PR. The compact CSV is the analysis
297-
and Markdown-reproduction layer. It does not replace the full native Criterion
298-
`.tar.gz` archive attached to each GitHub Release, which remains the durable raw
336+
while reviewing or re-rendering a release PR. The compact CSV is the analysis and
337+
Markdown-reproduction layer for both local and release comparisons. It does not
338+
replace the full native Criterion `.tar.gz` archive attached to each GitHub
339+
Release, which remains the durable raw
299340
baseline for post-release comparisons.
300341

301342
## `vs_linalg` Methodology

docs/PERFORMANCE.md

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -360,15 +360,23 @@ just performance-local
360360
# Release PR: update docs/PERFORMANCE.md and archive the previous report
361361
just performance-release
362362

363+
# Build release docs from retained CSV/JSON inputs (no benchmarks)
364+
just performance-doc
365+
363366
# GitHub Actions release assets
364367
just performance-github-assets
365368

366369
# Explicit repair
367370
just performance-release <current-tag> <previous-tag>
368371
```
369372

370-
`just performance-local` writes `target/bench-reports/performance.md`.
373+
`just performance-local` writes `performance.md` plus retained `performance.csv` and
374+
`performance.provenance.json` comparison inputs under `target/bench-reports/` without promoting documentation.
375+
It applies staged and unstaged tracked changes; untracked files are excluded.
371376
`just performance-github-assets` writes `target/bench-reports/github-assets-performance.md`.
377+
`just performance-release` performs the same measurement and retention work, then promotes distinct-release documentation.
378+
`just performance-doc` consumes the retained pair from either workflow without benchmarking and promotes it when the package versions differ.
379+
For a distinct pair, `performance-local` followed by `performance-doc` is equivalent to the atomic `performance-release` workflow.
372380

373381
Older curated release-to-release reports are archived in `docs/archive/performance/`.
374382

docs/RELEASING.md

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -140,20 +140,36 @@ incomplete.
140140
just performance-release
141141
```
142142

143+
Run this only after the package version has been bumped. The no-argument form
144+
uses the `Cargo.toml` version as the current release and discovers the previous
145+
stable published release. For an explicit repair, the supplied current tag must
146+
match the package version in the `HEAD` checkout; mismatches are rejected before
147+
tag fetching, worktree creation, or benchmarking. Release report publication
148+
also requires distinct current and baseline release identifiers.
149+
143150
Review `docs/PERFORMANCE.md` for the latest release-to-release comparison. Older
144151
committed comparisons are archived under `docs/archive/performance/` with
145152
lexicographically sorted filenames such as `v0.4.2-vs-v0.4.1.md`. Iterative
146153
local reports still live under `target/bench-reports/`.
147154

155+
The temporary current worktree includes staged and unstaged changes to tracked
156+
files. Untracked files are excluded, so stage every new benchmark-relevant file
157+
before running the release comparison.
158+
148159
The release command retains `target/bench-reports/performance.csv` and
149160
`performance.provenance.json` before its temporary worktrees are removed, then
150161
renders and promotes Markdown from a validated reload of that pair. Review the
151162
CSV coverage/timing rows and JSON release, revision, command, toolchain, host,
152163
harness, and digest metadata alongside the Markdown. For a presentation-only
153-
correction, run `just performance-rerender`; it reproduces and promotes the
154-
report from those files without invoking Cargo or creating worktrees. The pair
155-
is local scratch and may be removed by `just clean` or `cargo clean`, so do not
156-
clean `target/` until the release report review is complete.
164+
correction, run `just performance-doc`; it reproduces and promotes the
165+
report from those files without invoking Cargo or creating worktrees. The same
166+
command can consume a pair retained by `performance-local` after the version
167+
bump. It rejects same-version local artifacts because committed performance docs
168+
must compare distinct releases. The command updates `docs/PERFORMANCE.md` and the
169+
performance archive, so review and stage those changes as release artifacts.
170+
The retained pair is local scratch and may be removed by `just clean` or
171+
`cargo clean`, so do not clean `target/` until the release report review is
172+
complete.
157173

158174
For an explicit measurement repair, run
159175
`just performance-release <current-tag> <previous-tag>`. To compare the stored
@@ -165,6 +181,16 @@ and validation provenance. Stored release assets retain their original
165181
per-release harnesses; unavailable historical measurement metadata is labelled
166182
explicitly rather than treated as an isolated library-code comparison.
167183

184+
`just performance-local` is the non-promoting half of the release workflow: it
185+
measures and writes `target/bench-reports/performance.md` plus the adjacent
186+
retained CSV/JSON pair. The local report may compare revisions with the same
187+
package version because commit/ref and source-state provenance distinguish them.
188+
For a distinct pair, following it with `performance-doc` is equivalent to
189+
`performance-release`; the latter performs fresh measurement, retention, and
190+
rollback-capable promotion atomically. `performance-local-non-exact` uses the
191+
same metric and report model for the narrowed peer-context view but writes a
192+
separate `performance-non-exact.*` scratch bundle.
193+
168194
After the GitHub Release is published, the `Release Benchmarks` workflow checks
169195
out the release tag, runs the independent benchmark-input tests, saves a full
170196
Criterion baseline, and attaches

0 commit comments

Comments
 (0)