Record: 11L Muon Legal TTT + Entropy-Adaptive Epochs (8×H100) — val_bpb 1.1179 (3-seed mean)

[aamodbhatt]

Summary

Two novel TTT innovations on the SOTA base stack (PR #399 + PR #414 + PR #461): Muon-style Newton-Schulz orthogonalized updates replace SGD in the TTT loop, and entropy-adaptive epoch selection concentrates adaptation budget on harder content. Beats current SOTA (1.1194) with a 3-seed mean of 1.1179.

Run Results (3 seeds)

Seed	legal_ttt_exact val_bpb	legal_ttt_exact val_loss	pre-quant val_bpb	train time	eval time (TTT)	artifact size
1337	1.11765030	1.88710072	1.1366	599.1s	477.9s	15,944,410 bytes
42	1.11812929	1.88790947	1.1371	599.1s	485.3s	15,873,826 bytes
2025	1.11789934	1.88752121	1.1367	599.1s	479.2s	15,879,042 bytes
mean	1.11789

Method Notes

• NUM_LAYERS=11, BIGRAM_VOCAB_SIZE=1536, XSA_LAST_N=4
• TTT_ENABLED=1, score-first path
• TTT_MUON=1 — Newton-Schulz orthogonalized updates in TTT loop (NS steps=3)
• TTT_ENTROPY_ADAPT=1 — entropy-adaptive 2/3/4 epochs per chunk (H_HIGH=2.1, H_LOW=1.75)
• TTT_LR=0.002, TTT_EPOCHS=3, TTT_CHUNK_TOKENS=32768
• NGRAM_EVAL_ENABLED=0
• NGRAM_TWO_PASS_ENABLED=0
• NGRAM_FULL_RESCORE=0
• EMA_ENABLED=1, SWA_ENABLED=1, LATE_QAT=1, VE_ENABLED=1
• WARMDOWN_ITERS=3500, MAX_WALLCLOCK_SECONDS=599

Submission Checklist

• One folder under records/track_10min_16mb/
• Included README.md, submission.json, train_gpt.py, and train logs (3 seeds)
• Training <= 600s
• Eval <= 600s
• Artifact <= 16,000,000 bytes
• No tokenizer/dataset modifications
• Score-first TTT (SCORE under inference_mode before TRAIN on same chunk)
• No n-gram, no two-pass, no external data lookup

[MatoTeziTanka]

Community Review — Record: 11L Muon Legal TTT + Entropy-Adaptive Epochs (8×H100) — val_bpb 1.1179 (3-seed mean)

BPB: 1.1179 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA f21923533c70, file records/track_10min_16mb/2026-03-28_MuonTTT_EntropyAdaptive_11L_8xH100/train_gpt.py):

The TTT path at line 1079 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.05s, dim=512, layers=11, vocab=1024, code=93038 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.05s, dim=512, layers=11, vocab=1024, code=93038 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.