Record: — val_bpb 0.7271 (3-seed mean) SLOT-48 + VRL + QK-Gain 4.0 + XSA-11

[yahya010]

Summary

val_bpb = 0.7271 (3-seed mean, std 0.0011)

3-Seed Results (8xH100 SXM)

Seed	Sliding BPB	+ SLOT BPB	Steps	Artifact
1337	1.1248	0.7270	5698	15,649,297
42	1.1248	0.7260	5701	15,747,713
2025	1.1247	0.7282	5700	15,658,677
Mean	1.1248	0.7271

Key Technique

• SLOT-48: 48 AdamW steps (cosine LR 0.012→0.001, stride 96, eval time ~508s)
• VRL: Value Residual Learning with sigmoid-gated interpolation (init=-1.5)
• QK-Gain 4.0: Per-head learnable query scaling
• XSA all 11 layers: Cross-head subtraction attention

Architecture

11L, 512d, 8H/4KV GQA, LeakyReLU(0.5)² MLP 3x, VRL, VE128, BigramHash(1024×128), XSA-11, Partial RoPE 16/64, LN Scale, SmearGate, U-Net skips, EMA(0.997), SWA, Late QAT, int6 Hessian GPTQ + LZMA-9, Muon WD=0.04.

Compliance

• Frozen-model SLOT: model weights never modified during eval
• Per-window throwaway delta [bsz,1,512] + logit_bias [bsz,1,1024], discarded after each window
• AR self-gen GPTQ calibration (64 seqs, temp=0.8)
• Train time: 600s, SLOT eval time: ~508s (both within 10min budget)
• All artifacts under 16MB

Builds on

• PR Record: SLOT-24 Aggressive — val_bpb 0.8637 (3-seed mean) #1313: SLOT eval-time optimization approach
• arXiv:2505.12392v2 (SLOT paper)