Skip to content

Commit d31e26a

Browse files
acgetchellacgetchell
committed
feat(bench): automate published performance report archiving
- Track the latest curated release comparison in docs/PERFORMANCE.md and archive older comparisons under docs/archive/performance/ - Let performance-archive-published discover the latest stable GitHub release and previous stable baseline automatically - Generate release comparisons in isolated temporary worktrees, with release-asset restore and local baseline fallback paths - Update benchmark and release docs to use the scripted workflow instead of manual checkout steps
1 parent 2817d01 commit d31e26a

12 files changed

Lines changed: 882 additions & 141 deletions

‎.gitignore‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,6 @@
55
/cobertura.xml
66
.DS_Store
77

8-
# Generated benchmark results (machine-specific)
9-
docs/PERFORMANCE.md
10-
118
# Python / uv
129
**/__pycache__/
1310
*.egg-info/

‎docs/BENCHMARKING.md‎

Lines changed: 19 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -151,40 +151,27 @@ benchmarks on every iteration.
151151
### Workflow
152152

153153
```bash
154-
# 1. Check out the old release and save its full baseline
155-
git checkout v0.2.0
156-
just bench-save-baseline v0.2.0
154+
# Latest published release vs previous stable release
155+
just performance-archive-published
157156

158-
# 2. Switch to current code and run latest la-stack measurements
159-
git checkout main # or your feature branch
160-
just bench-latest # populates target/criterion/*/new/
161-
162-
# 3. Generate a local comparison report
163-
just bench-compare v0.2.0
157+
# Explicit historical repair
158+
just performance-archive-published v0.4.2 v0.4.1
164159
```
165160

166-
You can save multiple baselines and compare against any of them.
161+
These recipes create isolated temporary worktrees, restore release baselines
162+
from GitHub Release assets when available, and fall back to generating a missing
163+
baseline in a second temporary worktree. They do not require changing the current
164+
checkout.
167165

168-
If the release baseline is already present in `target/criterion/`, skip the
169-
checkout step and compare directly. For example, to compare current code against
170-
the saved `v0.4.2` release baseline:
166+
For local scratch comparisons, you can save multiple baselines and compare
167+
against any of them. If the release baseline is already present in
168+
`target/criterion/`, compare directly:
171169

172170
```bash
173171
just bench-latest # gather latest la-stack measurements
174172
just bench-compare v0.4.2 # compare latest measurements against v0.4.2
175173
```
176174

177-
If the release baseline is not present locally, download and restore the release
178-
asset first:
179-
180-
```bash
181-
gh release download v0.4.2 --pattern "la-stack-v0.4.2-criterion-baseline.tar.gz" # fetch archived release baseline
182-
mkdir -p target # ensure Criterion parent directory exists
183-
tar -C target -xzf la-stack-v0.4.2-criterion-baseline.tar.gz # restore target/criterion baseline data
184-
just bench-latest # gather latest la-stack measurements
185-
just bench-compare v0.4.2 # compare latest measurements against v0.4.2
186-
```
187-
188175
### Output
189176

190177
`just bench-compare` writes `target/bench-reports/performance.md` by
@@ -206,13 +193,17 @@ an isolated temporary worktree, copies the finished report to
206193
`v0.4.2-vs-v0.4.1.md`, so the directory and generated index stay
207194
lexicographically sorted.
208195

209-
To regenerate and archive a historical published release comparison without
196+
To regenerate and archive the latest published release comparison without
210197
touching the current checkout:
211198

212199
```bash
213-
just performance-archive-published v0.4.2 v0.4.1
200+
just performance-archive-published
214201
```
215202

203+
The recipe discovers the latest stable published GitHub release and its previous
204+
stable release automatically. For explicit historical repair, pass both tags:
205+
`just performance-archive-published v0.4.2 v0.4.1`.
206+
216207
For exact-arithmetic comparisons against v0.4.2 or older baselines, rows such
217208
as `det_exact_rounded_f64 (vs det_exact_f64)` mean the current rounded API is
218209
being compared to the historical lossy `*_exact_f64` benchmark. Rows such as
@@ -261,4 +252,5 @@ just bench-save-last
261252
When the GitHub Release is published, `.github/workflows/release-benchmarks.yml`
262253
saves a full release baseline and attaches
263254
`la-stack-$TAG-criterion-baseline.tar.gz` to the release as the durable archive.
264-
See `docs/RELEASING.md` step 5 for where this fits in the release process.
255+
See the `just performance-release` step in `docs/RELEASING.md` for where the
256+
curated `docs/PERFORMANCE.md` comparison fits in the release process.

‎docs/PERFORMANCE.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# Exact Arithmetic Performance
2+
3+
**la-stack** v0.4.2 · `7e11f93` (HEAD) · 2026-06-08 20:39:03 UTC
4+
**Statistic**: median
5+
6+
## Benchmark Results
7+
8+
Comparison against baseline **v0.4.1**:
9+
10+
Negative change = faster. Speedup > 1.00x = improvement.
11+
12+
### D=2
13+
14+
| Benchmark | v0.4.1 | Current | Change | Speedup |
15+
|-----------|-------:|--------:|-------:|--------:|
16+
| det | 0.6 ns | 0.9 ns | +61.1% | 0.62x |
17+
| det_direct | 0.7 ns | 1.0 ns | +44.7% | 0.69x |
18+
| det_exact | 315.5 ns | 318.4 ns | +0.9% | 0.99x |
19+
| det_exact_f64 | 555.7 ns | 555.7 ns | -0.0% | 1.00x |
20+
| det_sign_exact | 0.7 ns | 1.5 ns | +128.2% | 0.44x |
21+
| solve_exact | 7.05 µs | 7.06 µs | +0.2% | 1.00x |
22+
| solve_exact_f64 | 7.50 µs | 7.67 µs | +2.3% | 0.98x |
23+
24+
### D=3
25+
26+
| Benchmark | v0.4.1 | Current | Change | Speedup |
27+
|-----------|-------:|--------:|-------:|--------:|
28+
| det | 1.3 ns | 1.8 ns | +36.3% | 0.73x |
29+
| det_direct | 4.7 ns | 2.2 ns | **-51.9%** | 2.08x |
30+
| det_exact | 936.9 ns | 924.3 ns | **-1.3%** | 1.01x |
31+
| det_exact_f64 | 1.18 µs | 1.19 µs | +1.1% | 0.99x |
32+
| det_sign_exact | 2.4 ns | 4.2 ns | +78.1% | 0.56x |
33+
| solve_exact | 27.02 µs | 27.41 µs | +1.5% | 0.99x |
34+
| solve_exact_f64 | 28.06 µs | 27.98 µs | -0.3% | 1.00x |
35+
36+
### D=4
37+
38+
| Benchmark | v0.4.1 | Current | Change | Speedup |
39+
|-----------|-------:|--------:|-------:|--------:|
40+
| det | 2.4 ns | 3.3 ns | +36.8% | 0.73x |
41+
| det_direct | 2.4 ns | 4.1 ns | +70.2% | 0.59x |
42+
| det_exact | 2.33 µs | 2.33 µs | -0.0% | 1.00x |
43+
| det_exact_f64 | 2.59 µs | 2.58 µs | -0.7% | 1.01x |
44+
| det_sign_exact | 5.3 ns | 6.9 ns | +30.5% | 0.77x |
45+
| solve_exact | 67.14 µs | 67.99 µs | +1.3% | 0.99x |
46+
| solve_exact_f64 | 67.86 µs | 68.51 µs | +1.0% | 0.99x |
47+
48+
### D=5
49+
50+
| Benchmark | v0.4.1 | Current | Change | Speedup |
51+
|-----------|-------:|--------:|-------:|--------:|
52+
| det | 21.6 ns | 24.5 ns | +13.7% | 0.88x |
53+
| det_direct | 2.3 ns | 4.7 ns | +104.8% | 0.49x |
54+
| det_exact | 5.04 µs | 4.99 µs | -1.0% | 1.01x |
55+
| det_exact_f64 | 5.32 µs | 5.31 µs | -0.1% | 1.00x |
56+
| det_sign_exact | 4.97 µs | 4.99 µs | +0.3% | 1.00x |
57+
| solve_exact | 134.99 µs | 136.04 µs | +0.8% | 0.99x |
58+
| solve_exact_f64 | 137.11 µs | 138.97 µs | +1.4% | 0.99x |
59+
60+
### Near-singular 3x3
61+
62+
| Benchmark | v0.4.1 | Current | Change | Speedup |
63+
|-----------|-------:|--------:|-------:|--------:|
64+
| det_sign_exact | 871.8 ns | 877.6 ns | +0.7% | 0.99x |
65+
| det_exact | 907.3 ns | 904.4 ns | -0.3% | 1.00x |
66+
| solve_exact | 4.31 µs | 4.25 µs | **-1.5%** | 1.02x |
67+
| solve_exact_f64 | 4.29 µs | 4.32 µs | +0.7% | 0.99x |
68+
69+
### Large entries 3x3
70+
71+
| Benchmark | v0.4.1 | Current | Change | Speedup |
72+
|-----------|-------:|--------:|-------:|--------:|
73+
| det_sign_exact | 3.14 µs | 3.09 µs | **-1.3%** | 1.01x |
74+
| det_exact | 3.19 µs | 3.11 µs | **-2.3%** | 1.02x |
75+
| solve_exact | 84.77 µs | 83.89 µs | **-1.0%** | 1.01x |
76+
| solve_exact_f64 | 84.62 µs | 83.92 µs | -0.8% | 1.01x |
77+
78+
### Hilbert 4x4
79+
80+
| Benchmark | v0.4.1 | Current | Change | Speedup |
81+
|-----------|-------:|--------:|-------:|--------:|
82+
| det_sign_exact | 5.3 ns | 6.9 ns | +30.4% | 0.77x |
83+
| det_exact | 2.39 µs | 2.31 µs | **-3.2%** | 1.03x |
84+
| solve_exact | 51.69 µs | 52.27 µs | +1.1% | 0.99x |
85+
| solve_exact_f64 | 52.90 µs | 53.26 µs | +0.7% | 0.99x |
86+
87+
### Hilbert 5x5
88+
89+
| Benchmark | v0.4.1 | Current | Change | Speedup |
90+
|-----------|-------:|--------:|-------:|--------:|
91+
| det_sign_exact | 5.03 µs | 4.88 µs | **-2.9%** | 1.03x |
92+
| det_exact | 5.07 µs | 4.96 µs | **-2.1%** | 1.02x |
93+
| solve_exact | 105.35 µs | 102.72 µs | **-2.5%** | 1.03x |
94+
| solve_exact_f64 | 104.99 µs | 103.94 µs | -1.0% | 1.01x |
95+
96+
## How to Update
97+
98+
Release performance docs are generated in isolated temporary worktrees:
99+
100+
```bash
101+
# Release PR: update docs/PERFORMANCE.md and archive the previous report
102+
just performance-release <current-tag> <previous-tag>
103+
104+
# Historical published comparison
105+
just performance-archive-published
106+
107+
# Explicit historical repair
108+
just performance-archive-published <current-tag> <previous-tag>
109+
```
110+
111+
For local scratch comparisons, use `just bench-latest` and `just bench-compare`.
112+
Those write `target/bench-reports/performance.md`.
113+
114+
See `docs/BENCHMARKING.md` for the full comparison workflow.

‎docs/RELEASING.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -113,10 +113,11 @@ just performance-release "$TAG" "$PREVIOUS_TAG"
113113
Review `docs/PERFORMANCE.md` for the latest release-to-release comparison. Older
114114
committed comparisons are archived under `docs/archive/performance/` with
115115
lexicographically sorted filenames such as `v0.4.2-vs-v0.4.1.md`. Iterative
116-
local reports still live under `target/bench-reports/`. To regenerate a
117-
historical published release comparison, use
118-
`just performance-archive-published <current-tag> <baseline-tag>`; it also runs
119-
inside a temporary worktree.
116+
local reports still live under `target/bench-reports/`. To regenerate the
117+
latest published release comparison, use `just performance-archive-published`;
118+
it discovers the latest stable published GitHub release and its previous stable
119+
release automatically, then runs inside a temporary worktree. For explicit
120+
historical repair, pass both tags.
120121

121122
6. Save benchmark baselines for this release
122123

‎docs/archive/performance/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# Archived Performance Reports
2+
3+
Older release-to-release benchmark comparisons are archived here.
4+
`docs/PERFORMANCE.md` contains the latest curated comparison.
5+
6+
- [v0.4.1-vs-v0.4.0](v0.4.1-vs-v0.4.0.md)

‎docs/archive/performance/v0.4.1-vs-v0.4.0.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Exact Arithmetic Performance
2+
3+
**la-stack** v0.4.1 · `c6e04fd` (main) · 2026-04-21 22:30:49 UTC
4+
**Statistic**: median
5+
6+
## Benchmark Results
7+
8+
Comparison against baseline **v0.4.0**:
9+
10+
Negative change = faster. Speedup > 1.00x = improvement.
11+
12+
### D=2
13+
14+
| Benchmark | v0.4.0 | Current | Change | Speedup |
15+
|-----------|-------:|--------:|-------:|--------:|
16+
| det | 0.6 ns | 0.6 ns | +0.9% | 0.99x |
17+
| det_direct | 0.7 ns | 0.7 ns | +0.6% | 0.99x |
18+
| det_exact | 250.3 ns | 250.0 ns | -0.1% | 1.00x |
19+
| det_exact_f64 | 434.2 ns | 421.7 ns | **-2.9%** | 1.03x |
20+
| det_sign_exact | 1.1 ns | 0.7 ns | **-39.6%** | 1.66x |
21+
| solve_exact | 15.74 µs | 6.51 µs | **-58.7%** | 2.42x |
22+
| solve_exact_f64 | 16.76 µs | 7.06 µs | **-57.9%** | 2.38x |
23+
24+
### D=3
25+
26+
| Benchmark | v0.4.0 | Current | Change | Speedup |
27+
|-----------|-------:|--------:|-------:|--------:|
28+
| det | 1.4 ns | 1.3 ns | **-3.0%** | 1.03x |
29+
| det_direct | 4.7 ns | 4.6 ns | **-2.3%** | 1.02x |
30+
| det_exact | 719.2 ns | 741.2 ns | +3.1% | 0.97x |
31+
| det_exact_f64 | 942.0 ns | 933.1 ns | -0.9% | 1.01x |
32+
| det_sign_exact | 4.1 ns | 2.3 ns | **-43.3%** | 1.76x |
33+
| solve_exact | 51.03 µs | 25.19 µs | **-50.6%** | 2.03x |
34+
| solve_exact_f64 | 53.31 µs | 26.64 µs | **-50.0%** | 2.00x |
35+
36+
### D=4
37+
38+
| Benchmark | v0.4.0 | Current | Change | Speedup |
39+
|-----------|-------:|--------:|-------:|--------:|
40+
| det | 2.4 ns | 2.4 ns | **-2.6%** | 1.03x |
41+
| det_direct | 2.4 ns | 2.4 ns | -0.6% | 1.01x |
42+
| det_exact | 1.89 µs | 1.88 µs | -0.3% | 1.00x |
43+
| det_exact_f64 | 2.09 µs | 2.09 µs | +0.2% | 1.00x |
44+
| det_sign_exact | 6.4 ns | 5.4 ns | **-15.7%** | 1.19x |
45+
| solve_exact | 147.18 µs | 64.62 µs | **-56.1%** | 2.28x |
46+
| solve_exact_f64 | 147.75 µs | 64.03 µs | **-56.7%** | 2.31x |
47+
48+
### D=5
49+
50+
| Benchmark | v0.4.0 | Current | Change | Speedup |
51+
|-----------|-------:|--------:|-------:|--------:|
52+
| det | 26.3 ns | 24.4 ns | **-7.5%** | 1.08x |
53+
| det_direct | 2.3 ns | 2.3 ns | **-1.7%** | 1.02x |
54+
| det_exact | 4.24 µs | 4.13 µs | **-2.6%** | 1.03x |
55+
| det_exact_f64 | 4.40 µs | 4.48 µs | +1.9% | 0.98x |
56+
| det_sign_exact | 4.19 µs | 4.08 µs | **-2.7%** | 1.03x |
57+
| solve_exact | 339.25 µs | 132.55 µs | **-60.9%** | 2.56x |
58+
| solve_exact_f64 | 340.14 µs | 135.75 µs | **-60.1%** | 2.51x |
59+
60+
### Near-singular 3x3
61+
62+
| Benchmark | v0.4.0 | Current | Change | Speedup |
63+
|-----------|-------:|--------:|-------:|--------:|
64+
| det_sign_exact | 909.2 ns | 705.9 ns | **-22.4%** | 1.29x |
65+
| det_exact | 929.6 ns | 729.8 ns | **-21.5%** | 1.27x |
66+
67+
## How to Update
68+
69+
Release performance docs are generated in isolated temporary worktrees:
70+
71+
```bash
72+
# Release PR: update docs/PERFORMANCE.md and archive the previous report
73+
just performance-release <current-tag> <previous-tag>
74+
75+
# Historical published comparison
76+
just performance-archive-published
77+
78+
# Explicit historical repair
79+
just performance-archive-published <current-tag> <previous-tag>
80+
```
81+
82+
For local scratch comparisons, use `just bench-latest` and `just bench-compare`.
83+
Those write `target/bench-reports/performance.md`.
84+
85+
See `docs/BENCHMARKING.md` for the full comparison workflow.

‎justfile‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -191,8 +191,20 @@ performance-release current_tag baseline_tag: python-sync
191191
uv run archive-performance "{{current_tag}}" "{{baseline_tag}}" --generate-in-temp-worktree --worktree-ref HEAD
192192

193193
# Generate a published-tag comparison in a temp worktree, then promote/archive docs.
194-
performance-archive-published current_tag baseline_tag: python-sync
195-
uv run archive-performance "{{current_tag}}" "{{baseline_tag}}" --generate-in-temp-worktree --worktree-ref "{{current_tag}}" --no-apply-current-diff
194+
performance-archive-published current_tag="" baseline_tag="": python-sync
195+
#!/usr/bin/env bash
196+
set -euo pipefail
197+
current_tag="{{current_tag}}"
198+
baseline_tag="{{baseline_tag}}"
199+
if [[ -n "$current_tag" || -n "$baseline_tag" ]]; then
200+
if [[ -z "$current_tag" || -z "$baseline_tag" ]]; then
201+
echo "current_tag and baseline_tag must be provided together" >&2
202+
exit 2
203+
fi
204+
uv run archive-performance "$current_tag" "$baseline_tag" --generate-in-temp-worktree --worktree-ref "$current_tag" --no-apply-current-diff
205+
else
206+
uv run archive-performance --published-latest --generate-in-temp-worktree --no-apply-current-diff
207+
fi
196208
197209
# Run the exact-arithmetic benchmark suite.
198210
bench-exact:

‎scripts/README.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -41,13 +41,16 @@ runs in a temporary worktree:
4141
just performance-release v0.4.3 v0.4.2
4242
```
4343

44-
To regenerate a historical published-tag comparison without changing the current
44+
To regenerate the latest published-tag comparison without changing the current
4545
checkout:
4646

4747
```bash
48-
just performance-archive-published v0.4.2 v0.4.1
48+
just performance-archive-published
4949
```
5050

51+
For explicit historical repair, pass both tags:
52+
`just performance-archive-published v0.4.2 v0.4.1`.
53+
5154
### Plotting Criterion benchmarks (la-stack vs nalgebra/faer)
5255

5356
The plotter reads Criterion output under:

0 commit comments

Comments
 (0)