Record: 11L LatentMask TTT + GPTQ + Product-Key Bigram + Brotli — val_bpb 1.1124 (3-seed mean)

[izlley]

Summary

11L GPT with two novel techniques — LatentMask TTT and Product-Key Bigram — plus Brotli-11 compression and Flash Attention 3, achieving a 3-seed mean val_bpb of 1.1124 on 8xH100.

What's New in This Submission

• LatentMask TTT (novel): Per-channel sigmoid masks + biases on MLP/attention outputs, trained per-chunk at eval time with sign-based Muon-lite optimizer. Score-first (legal) evaluation. Provides ~−0.002 bpb improvement over sliding window eval.
• Product-Key Bigram (novel): Factored embed_prev(1024,512) * embed_cur(1024,512) — zero hash collision, no projection layer. Cleaner and more parameter-efficient than hash-based BigramHash.
• Alternating GatedAttention: GatedAttention on layers [0,2,4,6,8,10] (every other layer), standard attention on the rest — reduces parameters while improving bpb.
• Brotli-11 + uint8 log-scale: Custom binary serialization replacing LZMA/zstd, squeezing more into the 16MB budget.
• Flash Attention 3: Hopper-optimized FA3 kernels, reducing step_avg from ~103ms to ~91.5ms (+725 more training steps in 10 minutes).

Results (8xH100, 3-seed)

Seed	val_bpb	Steps	Step Avg (ms)	Artifact (bytes)
777	1.11195	6,555	91.54	15,985,742
999	1.11218	6,555	91.54	15,994,891
1337	1.11297	6,556	91.52	15,988,042
Mean	1.11237	6,555	91.53

Test plan

• torchrun --standalone --nproc_per_node=8 train_gpt.py on 8xH100
• Reproduce 3-seed results (seeds: 777, 999, 1337)
• Verify artifact size < 16,000,000 bytes

[izlley]

Updated results with Flash Attention 3. FA3 Hopper kernels brought step_avg down from ~103ms to ~91.5ms, gaining ~725 extra steps and improving val_bpb from 1.1158 to 1.1124 (3-seed mean) :)

[MatoTeziTanka]

Community Review — Record: 11L LatentMask TTT + GPTQ + Product-Key Bigram + Brotli — val_bpb 1.1124 (3-seed mean)

BPB: 1.1124 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA 6730fb5685e4, file records/track_10min_16mb/2026-04-05_11L_LatentMaskTTT_GPTQ_ProductKey_Brotli/train_gpt.py):

The TTT path at line 708 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=11, vocab=1024, code=61917 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=11, vocab=1024, code=61917 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.