Non-record: SLOT eval-time delta optimization + QK-Gain (val_bpb=1.1179)

[ibarrajo]

Summary

• SLOT (Stochastic Logit Offset Tuning): eval-time delta optimization that adjusts logit biases per-token, achieving -0.009 BPB improvement over base sliding window eval
• QK-Gain: scaling Q/K projections by 4.0x — hurt slightly, included for documentation
• Based on Approach B architecture (d=576, 11L, int5 GPTQ)
• TTT disabled; improvement comes entirely from SLOT

Results

Metric	Value
val_bpb (SLOT)	1.1179
val_bpb (base sliding window)	1.1267
SLOT improvement	-0.009 BPB
Artifact size	15.2 MB (762 KB headroom)
Eval time	419s
Current SOTA	1.1147

Key Findings

• SLOT works: eval-time logit delta optimization gives a reliable -0.009 BPB without any training changes
• QK-Gain 4.0 hurts slightly: scaling Q/K projections didn't improve quality
• Non-record: 1.1179 does not beat SOTA of 1.1147

Rule Compliance

• Training time < 600s
• Eval time < 600s (419s)
• Artifact < 16MB (15.2MB)
• No val tokens in artifact
• No pre-eval TTT (TTT disabled, SLOT-only)
• Single-pass evaluation

🤖 Generated with Claude Code