perf: adopt fat-LTO release profile and a release-fast test profile (#543)

## 🗒️ Description / Motivation

We had no `[profile]` section at all, so release builds ran on cargo's
defaults:
**16 codegen units, no LTO**. The optimizer never inlined across crate
boundaries, and our hot paths are spread across workspace crates and
their
dependencies (SSZ encode/decode, `hash_tree_root`, fork choice
traversal) — the
shape that loses most to per-crate codegen units.

```toml
[profile.release]          # ships the binary
opt-level = 3
lto = "fat"
codegen-units = 1

[profile.release-fast]     # runs the tests
inherits = "release"
lto = false
codegen-units = 16
debug = "line-tables-only"
incremental = true
```

Two deliberate divergences from
[ethrex](https://github.com/lambdaclass/ethrex),
whose profiles this started as a copy of:

1. **`lto = "fat"`, not thin.** Measured, see below.
2. **`release-fast` runs the whole test suite**, where ethrex applies it
only to
its `ef_tests` Makefiles and runs everything else on the dev profile. We
   can't: signature verification/aggregation stack-overflows without
release-grade opt-level, which is why `make test` used `--release` to
begin
   with.

ethrex's other profiles are not ported: `[profile.dev] debug = 2` is
already
cargo's default, and `release-with-debug` /
`release-with-debug-assertions` have
no user here.

## What Changed

| File | Change |
|---|---|
| `Cargo.toml` | added `[profile.release]` (fat LTO, 1 CGU) and
`[profile.release-fast]` |
| `Makefile` | `test` target: `--release` → `--profile release-fast` |
| `CLAUDE.md` | documented the test profile and where its artifacts land
|
| `.github/PULL_REQUEST_TEMPLATE.md` | checklist now names `make test` |

No source changes.

## Correctness / Behavior Guarantees

- No behavioral change beyond optimization and code layout.
- `debug-assertions` / `overflow-checks` stay at release defaults (off)
in both
profiles, so tests keep the assertion semantics they had under
`--release`.
`release-fast` inherits `opt-level = 3`, the property the stack-overflow
  avoidance depends on.
- The fat-LTO release build was exercised beyond compiling: it ran 8
nodes on the
  eth-4 devnet for over 30 minutes, finalizing normally, heads in sync.

## Measured: latency, on an 8-node devnet

Method: images built from the same commit differing only in
`[profile.release]`,
then run **simultaneously** on 4 nodes each so both arms see identical
chain
conditions (a sequential before/after would confound the profile with
chain
load — an earlier crossover attempt of mine did exactly that and had to
be
thrown out). Aggregator node excluded; its extra load isn't what's being
compared. Negative = lower latency.

| metric | default | thin | fat | thin vs default | fat vs thin | **fat
vs default** |
|---|---|---|---|---|---|---|
| state transition | 8.998 ms | 8.961 ms | 8.771 ms | −0.41% |
**−2.12%** | **−2.52%** |
| block processing | 0.404 ms | 0.355 ms | 0.361 ms | **−12.13%** |
+1.59% | **−10.73%** |
| attestation processing | 16.06 µs | 13.32 µs | 13.48 µs | **−17.04%**
| +1.20% | **−16.05%** |
| block build | 946.9 ms | 941.1 ms | 931.9 ms | −0.61% | −0.98% |
−1.58% |
| payload aggregation | 8.930 ms | 8.927 ms | 8.822 ms | −0.03% | −1.18%
| −1.21% |

The two LTO steps buy different things:

- **default → thin** buys cross-crate inlining: block processing −12%,
attestation processing −17%. Total state transition barely moves, since
those
  sub-millisecond stages are a small slice of it.
- **thin → fat** buys total state transition (−2.12%) and block build
(−0.98%),
while block processing and attestation processing land *slightly* the
wrong way
  (+1.6%, +1.2%) — both inside noise at 0.2σ and 0.5σ.

On the fat-vs-thin window, every fat node beat every thin node, no
overlap:

```
state transition   thin: 8.997  9.026  9.036  9.018   fat: 8.791  8.822  8.855  8.875
block build        thin: 945.1  946.1  943.9  950.0   fat: 939.0  936.4  938.8  935.3
```

Confidence differs sharply by leg, so both are worth stating:

| leg | separates cleanly | inside noise |
|---|---|---|
| thin vs default (30 min) | block processing 3.7σ, attestation ~8σ,
block build 5.5σ | state transition 0.9σ |
| fat vs thin (30 min) | state transition 7.1σ, block build 3.6σ | block
processing 0.2σ, attestation 0.5σ, payload agg 1.2σ |

`fat vs default` is **composed** from the two 30-minute measurements,
not measured
directly. A direct 5-minute fat-vs-default window gave larger numbers
(−5.30% / −21.40% / −19.10%) but its baselines ran 1–5% high, so I trust
the
composed column and report the short one only as a sanity check.

### Heavy crypto specifically

leanVM's XMSS aggregation benchmark at the rev we pin, its profiles
replaced by
ours, paired over 5 alternating rounds on a quiesced eth-4 (devnet
paused):

| profile | proof time | 95% CI |
|---|---|---|
| default | 5.0905 s | ±0.0171 |
| thin LTO | 5.0642 s | ±0.0104 |

−0.52%, significant (all 5 rounds agree). Small, as expected: that code
is
intra-crate, heavily monomorphized numeric work with little cross-crate
inlining
for LTO to find. It rules out a regression on the proving path, which is
what it
was run for.

## Measured: build cost

Whole-workspace binary, clean builds, isolated `CARGO_TARGET_DIR`,
11-core
M-series (thin measured; the default→LTO delta is the point):

| | wall | CPU (user+sys) | binary |
|---|---|---|---|
| default release | 315.9 s | 1714 s | 30.6 MiB |
| thin LTO, 1 CGU | 377.8 s | 1516 s | 26.6 MiB |
| Δ | +20% | −12% | −13% |

Wall grows because one codegen unit serializes per-crate codegen even as
total
CPU drops. Docker builds on the 16-core eth-4 host: **fat 284 s / 205 MB
against
thin 290 s / 209 MB**, i.e. fat costs nothing measurable relative to
thin.

Test builds, which is why `release-fast` exists — rebuild after one real
one-file
edit, `cargo test --workspace --no-run`, both profile dirs pre-warmed:

| | wall |
|---|---|
| `--release` | 88.0 s |
| `--profile release-fast` | **4.6 s** |

Without `release-fast`, this PR would make every test rebuild pay a full
LTO
relink. Costs: cold build of that profile dir is 461.9 s and
`target/release-fast`
holds ~5.3 GiB against `target/release`'s 1.6 GiB, so local disk grows
for anyone
who both builds `--release` and runs tests.

**CI note:** the test job's `Swatinem/rust-cache` sets
`CARGO_INCREMENTAL=0`,
which overrides `incremental = true` (verified — no `-C incremental=`
reaches
rustc under that env var, and the action strips incremental artifacts
before
caching). In CI the `release-fast` win is `lto = false` plus parallel
codegen.

## Risks / open items

- Fat LTO's **peak link memory** is untested on 16 GB CI runners. CI
never builds
this profile today (`cargo check` for lint, `release-fast` for tests),
but the
  Docker publish workflow does.
- Build-time parity between fat and thin is one measurement per arm on
one host.
- Branch is still named `...thin-lto` from before the fat switch; the
profile is
  fat.

## Tests Added / Run

No tests added (build-configuration change).

- `make fmt` — clean
- `make lint` (`cargo clippy --workspace --all-targets -- -D warnings`)
— clean
- `make test` under `release-fast` — all passing: forkchoice spectests
122, stf
spectests 46, ssz spectests 119, signature spectests 3, plus all unit
tests;
  0 failures. Also green under `--release` before the profile switch.
- Fat release build validated by `docker build` on eth-4 and by running
the
  resulting image on 8 devnet nodes for 30+ minutes.

## Related Issues / PRs

- None.

## ✅ Verification Checklist

- [x] Ran `make fmt` — clean
- [x] Ran `make lint` (clippy with `-D warnings`) — clean
- [x] Ran `make test` (`cargo test --workspace --profile release-fast`)
— all passing

---------

Co-authored-by: Pablo Deymonnaz <pdeymon@fi.uba.ar>

Author: MegaRedHand

.github/PULL_REQUEST_TEMPLATE.md

@@ -23,4 +23,4 @@
 
 - [ ] Ran `make fmt` — clean
 - [ ] Ran `make lint` (clippy with `-D warnings`) — clean
-- [ ] Ran `cargo test --workspace --release` — all passing
+- [ ] Ran `make test` (`cargo test --workspace --profile release-fast`) — all passing

CLAUDE.md

@@ -316,11 +316,17 @@ GENESIS_VALIDATORS:
 
 ### Running Tests
 ```bash
-cargo test --workspace --release                                    # All workspace tests
+cargo test --workspace --profile release-fast                       # All workspace tests
 cargo test -p ethlambda-blockchain --test forkchoice_spectests
 cargo test -p ethlambda-blockchain --test forkchoice_spectests -- --test-threads=1  # Sequential
 ```
 
+Tests run under `release-fast`: release-grade opt-level (needed to avoid stack
+overflows in signature verification/aggregation) but no LTO, parallel codegen,
+incremental, and line-tables-only debuginfo, so rebuilds are much faster than
+`--release`. Artifacts land in `target/release-fast/`, separate from
+`cargo build --release`.
+
 ## Common Gotchas
 
 ### Aggregator Flag Required for Finalization

Cargo.toml

@@ -26,6 +26,32 @@ repository = "https://github.com/lambdaclass/ethlambda"
 rust-version = "1.97.1"
 version = "0.1.0"
 
+# Cross-crate inlining matters here: the hot paths (SSZ encode/decode, hashing,
+# fork choice traversal) are split across workspace crates and their
+# dependencies, so keeping every crate in its own codegen unit leaves inlining
+# opportunities on the table. Fat LTO plus a single codegen unit trades build
+# time for that.
+#
+# Fat rather than thin: on an 8-node devnet, fat cut mean state transition time
+# a further 2.1% over thin (every fat node beat every thin node) at no measurable
+# build-time or image-size cost. ethrex uses thin here; we diverge deliberately.
+[profile.release]
+opt-level = 3
+lto = "fat"
+codegen-units = 1
+
+# Test profile (see the `test` target in the Makefile). Tests need release-grade
+# opt-level: signature verification/aggregation stack-overflows without it. They
+# do not need a whole-program-optimized binary, so this drops LTO, restores
+# parallel codegen, and keeps line tables for backtraces, which makes test
+# rebuilds finish in a fraction of the time of a `release` build.
+[profile.release-fast]
+inherits = "release"
+lto = false
+codegen-units = 16
+debug = "line-tables-only"
+incremental = true
+
 [workspace.dependencies]
 ethlambda-blockchain = { path = "crates/blockchain" }
 ethlambda-fork-choice = { path = "crates/blockchain/fork_choice" }

Makefile

@@ -10,8 +10,9 @@ lint: ## 🔍 Run clippy on all workspace crates
 	cargo clippy --workspace --all-targets -- -D warnings
 
 test: leanSpec/fixtures ## 🧪 Run all tests
-	# Tests need to be run on release to avoid stack overflows during signature verification/aggregation
-	cargo test --workspace --release
+	# release-fast: release-grade opt-level to avoid stack overflows during
+	# signature verification/aggregation, without paying for LTO on every rebuild
+	cargo test --workspace --profile release-fast
 
 GIT_COMMIT=$(shell git rev-parse HEAD)
 GIT_BRANCH=$(shell git rev-parse --abbrev-ref HEAD)