Skip to content

larql run/chat KV-cache decode emits garbage on gemma-4 dense (INFER path is correct on identical input) #177

Description

@layer5one

Summary

With the gemma-4 geometry/tensor-map/BOS fixes applied (companion PR from layer5one:gemma4-gguf-geometry), the LQL INFER path, which re-forwards the full prompt each step, is byte-faithful to the HF reference on the gemma-4 12B: top-1 parity at 100.00% on a templated prompt, and an autoregressive INFER loop generated "The capital of France is Paris." with per-step probabilities matching the reference. The same vindex, same tokenization, same prompt through larql run / chat (the incremental KV-cache decode path) produces token salad.

Because the two engines share weights, tokenizer, and prompt handling, the delta isolates the bug to the cached-decode attention path.

Repro

# faithful (INFER, full re-forward): top-1 'The' @ ~100%
larql lql "USE \"gemma-4-12b.vindex\"; INFER \"<templated prompt>\";"

# broken (cached decode): salad
larql run gemma-4-12b.vindex "What is the capital of France?" -n 48 -v

Templated prompt (gemma-4 dialect; note the thought channel, the 12B is a reasoning model):

<bos><|turn>user\nWhat is the capital of France?<turn|>\n<|turn>model\n<|channel>thought\n<channel|>

Suspected mechanism

Gemma-4 dense KV geometry is heterogeneous per layer:

  • sliding layers (5 of every 6): 8 KV heads x head_dim 256, RoPE theta=10k, fully rotary
  • global layers (5, 11, 17, 23, 29, 35, 41, 47): ONE KV head x head_dim 512, K identical to V (attention_k_eq_v, no V tensor exists), RoPE theta=1M, rotary fraction 0.25 (first 128 dims), QK-norm [512] vs [256]

A KV cache sized and indexed on a single global (num_kv_heads, head_dim) will read garbage on one class of layers, which matches the salad-not-crash signature observed. The full-reforward INFER path computes attention per layer from Gemma4Arch's per-layer getters and is provably correct, so those getters are the reference for what the cache must store per layer.

Evidence available on request

  • HF reference hidden states (49 layers) + logits for the test prompts
  • chain logs: INFER parity runs, run -v salad output on identical input

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions