Summary
With the gemma-4 geometry/tensor-map/BOS fixes applied (companion PR from layer5one:gemma4-gguf-geometry), the LQL INFER path, which re-forwards the full prompt each step, is byte-faithful to the HF reference on the gemma-4 12B: top-1 parity at 100.00% on a templated prompt, and an autoregressive INFER loop generated "The capital of France is Paris." with per-step probabilities matching the reference. The same vindex, same tokenization, same prompt through larql run / chat (the incremental KV-cache decode path) produces token salad.
Because the two engines share weights, tokenizer, and prompt handling, the delta isolates the bug to the cached-decode attention path.
Repro
# faithful (INFER, full re-forward): top-1 'The' @ ~100%
larql lql "USE \"gemma-4-12b.vindex\"; INFER \"<templated prompt>\";"
# broken (cached decode): salad
larql run gemma-4-12b.vindex "What is the capital of France?" -n 48 -v
Templated prompt (gemma-4 dialect; note the thought channel, the 12B is a reasoning model):
<bos><|turn>user\nWhat is the capital of France?<turn|>\n<|turn>model\n<|channel>thought\n<channel|>
Suspected mechanism
Gemma-4 dense KV geometry is heterogeneous per layer:
- sliding layers (5 of every 6): 8 KV heads x head_dim 256, RoPE theta=10k, fully rotary
- global layers (5, 11, 17, 23, 29, 35, 41, 47): ONE KV head x head_dim 512, K identical to V (
attention_k_eq_v, no V tensor exists), RoPE theta=1M, rotary fraction 0.25 (first 128 dims), QK-norm [512] vs [256]
A KV cache sized and indexed on a single global (num_kv_heads, head_dim) will read garbage on one class of layers, which matches the salad-not-crash signature observed. The full-reforward INFER path computes attention per layer from Gemma4Arch's per-layer getters and is provably correct, so those getters are the reference for what the cache must store per layer.
Evidence available on request
- HF reference hidden states (49 layers) + logits for the test prompts
- chain logs: INFER parity runs,
run -v salad output on identical input
Summary
With the gemma-4 geometry/tensor-map/BOS fixes applied (companion PR from layer5one:gemma4-gguf-geometry), the LQL
INFERpath, which re-forwards the full prompt each step, is byte-faithful to the HF reference on the gemma-4 12B: top-1 parity at 100.00% on a templated prompt, and an autoregressive INFER loop generated "The capital of France is Paris." with per-step probabilities matching the reference. The same vindex, same tokenization, same prompt throughlarql run/chat(the incremental KV-cache decode path) produces token salad.Because the two engines share weights, tokenizer, and prompt handling, the delta isolates the bug to the cached-decode attention path.
Repro
Templated prompt (gemma-4 dialect; note the thought channel, the 12B is a reasoning model):
Suspected mechanism
Gemma-4 dense KV geometry is heterogeneous per layer:
attention_k_eq_v, no V tensor exists), RoPE theta=1M, rotary fraction 0.25 (first 128 dims), QK-norm [512] vs [256]A KV cache sized and indexed on a single global (num_kv_heads, head_dim) will read garbage on one class of layers, which matches the salad-not-crash signature observed. The full-reforward INFER path computes attention per layer from
Gemma4Arch's per-layer getters and is provably correct, so those getters are the reference for what the cache must store per layer.Evidence available on request
run -vsalad output on identical input