test(gemma4): long context waypoints answer to the reference - #941
Conversation
Signed-off-by: Feathbow <feathbow@gmail.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d0d0d7c723
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| assert_eq!( | ||
| manifest["revision"], golden["revision"], | ||
| "the longctx fixture was dumped from a different revision than the golden one" |
There was a problem hiding this comment.
The checked-in long-context fixture records transformers: 5.14.1, while both the base and donor window fixtures record the documented pinned version 5.11.0, but this validation checks only the checkpoint revision. Consequently, the test calibrates its tolerance and top-1 floor from 5.11.0 outputs and applies them to 5.14.1 outputs, so a modeling change between those releases can be mistaken for an engine error or absorbed by the borrowed floor. Regenerate the fixture with 5.11.0 or require the manifests' Transformers versions to match.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Ok regenerate with 5.11.0
| const LONGCTX_FIXTURE: &str = concat!( | ||
| env!("CARGO_MANIFEST_DIR"), | ||
| "/../test_data/gemma4-12b-hf-longctx-golden.safetensors" |
There was a problem hiding this comment.
Adding this third Gemma 4 reference leaves docs/models/gemma4/hf-golden.md claiming there are only two fixtures and listing regeneration commands, hashes, and provenance only for the base and window fixtures; docs/index.md repeats the now-misleading two-fixture summary. This also hides the new fixture's actual 5.14.1 provenance, which already contradicts the documented 5.11.0 reference environment. Update the model document and its index summary with the new cases, calibration strategy, regeneration command, hash, and pinned environment.
AGENTS.md reference: AGENTS.md:L168-L170
Useful? React with 👍 / 👎.
Signed-off-by: Feathbow <feathbow@gmail.com>
1 participant
Description
Closes #940
The window fixture's deepest teacher-forced comparison sat at 4096 tokens while the serving contract now reaches the checkpoint's 262144 — proportional rope at high positions and global attention at depth ran without a reference ever having looked at them.
Two waypoints far past that (16384 and 32768) are now teacher-forced against the Hugging Face reference, with the window fixture's top-64 design: eight continuation positions each, recorded under both attention backends where they fit. Neither waypoint fits eager next to the tower on the dump device, so both record sdpa-only and the manifest says so; the gate borrows the window fixture's deepest dual-backend floor (4096) as its tolerance — the widest measured agreement bound available — and a depth-grown gap past it fails loud rather than being absorbed. The widest waypoint runs again in 2048-token chunks, the raised ceiling's production prefill shape, and every case keeps the window gate's top-1 bar and exact page-residency assertions.
Test Env
Verification
-D warnings,--all-targets) green.