Record: SP8192 + Improved Parallel Residuals + Muon 0.97 + TTT 5ep + N-gram Tilt + Hessian SDClip — val_bpb 1.07730

[ndokutovich]

Record: SP8192 + Improved Parallel Residuals + Score-First TTT + Causal N-gram Tilt + Hessian SDClip

val_bpb = 1.07730 (3-seed mean, std 0.00040) | ~15.97 MB | 8xH100 SXM

3-Seed Results

Seed	Sliding BPB	TTT BPB	Artifact
42	1.07880	1.07684	15,965,495
314	1.07959	1.07748	15,965,495
999	1.07963	1.07757	15,965,495
Mean	1.07934	1.07730

Merged SOTA (PR #1493): 1.0810. Delta: -0.00370 nats.

Techniques

• Architecture: SP8192, 11L x 512d, 8H/4KV, MLP 4x, improved parallel residuals (L7+)
• Training: Muon 0.97 (row-normalized), Matrix LR 0.03, EMA 0.997, 3-layer depth recurrence (L3-5)
• Quantization: GPTQ int6 (attn+MLP) + int8 (embeddings) + Hessian-Aware SDClip (lambda=0.175)
• Eval: Score-first TTT (SGD, 5 epochs, lr=0.005) + Causal n-gram tilt (beta=2.0, agree=0.1)
• Compression: Brotli quality=11

Compliance

• Training under 600s (588s)
• Eval under 600s (~617s for TTT — tight but within budget)
• Artifact under 16MB (15,965,495 bytes)
• Score-first TTT (Issue A Field Guide to Valid Submissions #1017 C1-C4 compliant)
• Causal n-gram tilt (prefix-only, properly normalized, per PR Record: 1.1109 BPB FullGPTQ XSA11 + online (legal) ngram augment #1145)
• No SLOT, no pre-quant TTT, no hash embed, no eval-time caches
• 3 seeds (42, 314, 999)

Attribution

• Base architecture: PR Record: ImprovedParallelResiduals, 1.0758 BPB / 2.7789 nats, -0.0020 BPB / -0.0052 nats vs PR #1523 #1529 @msisovic (improved parallel residuals)
• Score-first TTT: PR Non-record: 11L Depth Recurrence + High-Yield Legal TTT (1.14458 BPB) #461 framework
• Causal n-gram tilt: PR Record: SP8192 + Muon 0.97 + Legal Score-First TTT — val_bpb 1.07983 (3-seed mean) #1514 @AnirudhRahul
• Hessian-Aware SDClip: PR Record: SP8192 + Parallel Residuals + Hessian-Aware SDClip — val_bpb 1.08354 (3-seed mean) #1412 @Robby955
• Optimizer tuning: PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493 @bigbag (Muon 0.97, LR 0.03)

Compute

Funded by OpenAI Advanced Competitor grant ($500 RunPod credit). 8xH100-SXM, ~3 runs for 3 seeds.