Record: MuonEq-R + 3-Layer Recurrence + WD=0.095 + MLR=0.022 + All-Int6 — val_bpb 1.0900 (3-seed mean)

[dexhunter]

Summary

• val_bpb = 1.0900 (3-seed mean, std 0.0005) | 2.5077 nats | ~15.96 MB | 8xH100 SXM, 590s | No TTT
• 3-layer depth recurrence (layers 3,4,5) with WD-LR synergy: WD=0.095 compresses for headroom, MLR=0.022 recovers quality
• All seeds under 16MB with 36K+ margins
• No SLOT, no TTT, no eval-time adaptation

Key Innovation: 3-Layer Recurrence + WD-LR Synergy

Extends 2-layer recurrence (PR #1285) to 3 layers. The extra virtual layer needs more artifact budget, compensated by:

• Higher WD (0.095 vs 0.090) → better compression → headroom for 3-layer recurrence
• Higher MLR (0.022 vs 0.020) → recovers quality lost from WD increase

Results

Seed	Sliding BPB	val_loss (nats)	Artifact
42	1.0898	2.50733	15,961,029
0	1.0895	2.50672	15,955,962
7	1.0905	2.50901	15,964,018
Mean	1.0900	2.50769	15,960,336

Changes from PR #1285 (1.0912)

	PR #1285	This
val_bpb	1.09124	1.08995 (-0.00129)
Recurrence	2-layer (4,5)	3-layer (3,4,5)
WD	0.090	0.095
Matrix LR	0.020	0.022

Credits

• @clarkkev for PR Record: 4096-Vocab + 4.0-MLP-mult + 0.085-WD + Simplifications — val_bpb 1.09785 (3-seed mean) #1218
• @abaybektursun for PR Record: AR Self-Gen GPTQ + XSA-all + BigramHash 3072×112 — val_bpb 1.11473 (3-seed mean) #1019
• @msisovic for PR Record: ParallelResiduals + MiniDepthRecurrence, 1.1063 BPB / 1.8679 nats, -0.0072 vs PR #1179, -0.0143 vs merged SOTA #1204 (depth recurrence)
• @dexhunter for PR Record: MuonEq-R + Depth Recurrence + WD=0.090 + All-Int6 GPTQ — val_bpb 1.0912 (3-seed mean) #1285 (WD-quantization synergy)