Skip to content

Reuse normalization scratch buffer - #17

Merged
vicotrbb merged 1 commit into
mainfrom
codex/reusable-normalization-scratch
Jul 17, 2026
Merged

Reuse normalization scratch buffer#17
vicotrbb merged 1 commit into
mainfrom
codex/reusable-normalization-scratch

Conversation

@vicotrbb

Copy link
Copy Markdown
Owner

What changed

  • reuse one hidden-size normalization scratch vector across attention, feed-forward, and output normalization in a single-stream token step
  • preserve the public allocation-returning normalization API while adding an internal refill path
  • add an allocation-reuse unit test, changelog entry, and reproducible benchmark note

Why

A single-stream token step previously allocated a fresh normalization vector twice per transformer layer, plus once for the output projection. These stages do not overlap, so one vector can be safely refilled without changing arithmetic order or model output.

Impact

For a model with L layers, normalization scratch allocations per generated token fall from 2L + 1 to 1. Five interleaved 128-token benchmark pairs produced identical complete token traces. The timing median improved by 0.57 percent on a contaminated host, so the timing is documented as diagnostic only.

Validation

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • cargo test --workspace --all-targets --locked
  • cargo test --workspace --all-targets --all-features --locked
  • RUSTDOCFLAGS="-D warnings" cargo doc --workspace --all-features --no-deps --locked
  • cargo test --workspace --all-features --doc --locked
  • 68 available release-mode real-model tests across five GGUF artifacts
  • Python eval and release-tool tests
  • documentation and repository policy checks
  • cargo audit, cargo deny, cargo machete, and cargo tree --duplicates
  • package verification for ferrite-fixtures and ferrite-model, plus ferrite-inference archive listing

@vicotrbb
vicotrbb marked this pull request as ready for review July 17, 2026 21:53
@vicotrbb
vicotrbb merged commit c807ed5 into main Jul 17, 2026
9 checks passed
@vicotrbb
vicotrbb deleted the codex/reusable-normalization-scratch branch July 17, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant