Skip to content

perf: apply RoPE in place - #18

Merged
vicotrbb merged 1 commit into
mainfrom
codex/in-place-rope
Jul 21, 2026
Merged

perf: apply RoPE in place#18
vicotrbb merged 1 commit into
mainfrom
codex/in-place-rope

Conversation

@vicotrbb

Copy link
Copy Markdown
Owner

What changed

  • Apply rotary position encoding inside existing query and key projection buffers for single and batched inference.
  • Preserve the public allocating API and add independent bit-identical parity coverage that verifies storage is not replaced.
  • Correct the published crate documentation, record the allocation diagnostic, and document the required next pre-1.0 compatibility boundary.

Why

The prior internal path allocated a replacement vector for every query and key head, then copied those results into two more replacement vectors per layer. Reusing buffers already owned by the inference step removes deterministic allocator overhead without changing observable token output.

Validation

  • Rust 1.96.1 formatting and strict all-target, all-feature Clippy
  • Full default-feature and all-feature workspace test matrices
  • Strict rustdoc and workspace doctests
  • x86_64 macOS all-target, all-feature workspace check and batched-decode tests
  • 16 optimized real-model Tier 1 HTTP and async OpenAI client tests with local Qwen2.5 0.5B Q4_K_M
  • Exact baseline and candidate token-trace parity
  • Documentation, repository layout, eval harness, release tooling, dependency audit, license, source, and unused-dependency gates
  • Fixture and model crate package verification, plus inference package-content inspection

The benchmark host failed the clean-load preflight, so the recorded timing remains diagnostic and this PR makes no throughput claim.

@vicotrbb
vicotrbb merged commit ced523b into main Jul 21, 2026
9 checks passed
@vicotrbb
vicotrbb deleted the codex/in-place-rope branch July 21, 2026 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant