Record: 11L LeakyReLU² + XSA-all + QK-Gain 4.0 + Full GPTQ + SLOT — val_bpb 0.9354 (3-seed mean)

[xexyz]

Summary

• val_bpb: 0.9354 (3-seed mean, std 0.0032)
• Artifact: ~15.8 MB (all seeds < 16MB)
• Training: 600s on 8xH100 SXM | Eval: ~311s (SLOT) + ~120s (sliding) = ~431s total

Architecture

• 11L, dim=512, 8 heads, 4 KV heads (GQA)
• LeakyReLU(0.5)² MLP with 3x expansion
• SmearGate + BigramHash embedding augmentation
• XSA (cross-sequence attention) on all 11 layers
• QK-Gain init = 4.0
• ~27M parameters

Training

• Muon + Adam optimizers, EMA (0.997) + Tight SWA
• Late QAT + Full GPTQ int6 + zstd-22
• ~5250 steps at 114ms/step

Evaluation — SLOT

Based on arXiv:2505.12392v2:

1. Extract frozen hidden states from last layer under torch.no_grad()
2. Optimize per-sample delta [bsz, 1, 512] + logit bias [bsz, 1, 1024] via 16 AdamW steps, cosine LR (0.008 → 0.0008)
3. Scored-position mask: only last stride tokens per non-first window contribute to SLOT loss
4. Model weights completely frozen — only delta and logit_bias optimized
5. Standard autoregressive cross-entropy loss preserves causality

3-Seed Results

Seed	Sliding BPB	SLOT BPB	Artifact
1337	1.1264	0.9349	15,890,549
42	1.1264	0.9325	15,830,408
7	1.1261	0.9388	15,810,068
Mean	1.1263	0.9354

Beats merged SOTA (1.1147) by 0.179 BPB. Clears 0.005 nats threshold by 36x.

Compliance

• ❌ No n-gram cache
• ❌ No two-pass rescoring
• ❌ No eval-time access to training data
• ❌ No oracle/hindsight selection
• ✅ Score-first SLOT (frozen model, torch.no_grad hidden states)
• ✅ Self-contained (zero env var overrides required beyond seed)
• ✅ All seeds within time and size budgets

Reproduction

SEED=1337 GPTQ_CALIB_BATCHES=32 SLOT_ENABLED=1 SLOT_STEPS=16 \
SLOT_LR=0.008 SLOT_LR_MIN=0.0008 \
torchrun --nproc_per_node=8 train_gpt.py

Credits

• SLOT mechanism: arXiv:2505.12392v2
• Per-sample delta + logit bias approach inspired by PR Record: Scored-Position SLOT + Per-Sample Delta + GPTQ (val_bpb: 0.9300) #1229 (@resouer)
• QK-Gain 4.0 validated by PR Non-record: XSA-All + QK Gain 4.0 + LN Scale — 45 Experiments on 1×RTX 5090 #1125, PR Record: QK-Gain 4.0 + XSA-11 + Muon-TTT + SLOT — val_bpb 1.0914 (3-seed mean) #1176 (@bigbag)
• Base architecture builds on merged SOTA PR Record: AR Self-Gen GPTQ + XSA-all + BigramHash 3072×112 — val_bpb 1.11473 (3-seed mean) #1019 (@abaybektursun)