Non-record: 11L GEPA + 30k Steps + Pure Int6 + Legal TTT (val_bpb=1.0920)

[Christopher-Lee-McClendon]

Summary

• val_bpb = 1.0920 — new personal best with legal score-first TTT
• 11L GEPA architecture (27M params) trained for 30000 steps (12000 peak-LR + 18000 warmdown)
• Pure int6 per-row quantization with 15-candidate GPTQ-lite + zstd-22 compression
• Legal score-first TTT (SGD, momentum 0.9, 10 epochs): −0.035 BPP gain
• Artifact: 13.40 MB (14,136,140 bytes total) — smallest in our series
• Includes model artifact (final_model.int6.ptz) for reproducibility

Key Result

Metric	Value
Float base (30k steps)	1.1043
Int6 quantized	1.1267
After legal TTT	1.0920
Quant gap	0.022 BPP
TTT gain	−0.035 BPP
Eval time	2,064s on 4×A100-40GB
Training wallclock	14,998s (~4.2 hours)

Scaling Law (6 data points)

Steps	Peak-LR	Warmdown	Float Base	TTT BPP	Artifact
9,000	5,000	4,000	1.135	1.116	14.94 MB
12,000	7,000	5,000	1.127	1.108	14.79 MB
15,000	9,000	6,000	1.122	1.104	14.52 MB
20,000	12,000	8,000	1.115	1.098	14.22 MB
25,000	12,000	13,000	1.109	1.094	13.75 MB
30,000	12,000	18,000	1.104	1.092	13.40 MB

All three metrics improve monotonically across all 6 experiments.

Key Insights

1. 60% warmdown ratio (18k of 30k steps) reduces quantization gap from 0.027 → 0.022
2. Warmdown acceleration: Final 5000 steps produce −0.052 BPP decline (25k→30k)
3. Diminishing returns: Δ from 25k→30k is only −0.002 BPP (vs −0.004 for 20k→25k)

Non-record unlimited-compute submission (4×A100-40GB, ~4.2 hours).

Prior Submissions in This Series

• PR Non-record: 11L GEPA + 25k Steps + Pure Int6 + Legal TTT (val_bpb=1.0944) - unlimited compute category #644: 25k steps → 1.0944 BPP
• PR Non-record: 11L GEPA + 20k Steps + Pure Int6 + Legal TTT (val_bpb=1.0983): unlimited compute: 4×A100-40GB, ~2.8 hours #628: 20k steps → 1.0983 BPP
• PR Non-record: 11L GEPA + 12k Steps + Pure Int6 + Legal TTT (val_bpb=1.1079) #612: 12k steps → 1.1079 BPP
• PR Non-Record: BPB 1.1334 — 7000-Step Training + Mixed Int6/Int8 Quantization + Legal TTT #598: 7k steps → 1.1334 BPP

Acknowledgments

Builds on techniques from: @signalrush (PR #414, GPTQ-lite/EMA), @jfprincz (PRs #287/#315, XSA/Partial RoPE/LN Scale), @unnir (PR #265, Efficient XSA), raahilshah (PR #162, SmearGate/BigramHash), @aruniyer (PR #86, Int6 QAT), samacqua (LoRA TTT), @abaybektursun (PR #549, LeakyReLU²), and the OpenAI baseline.

[MatoTeziTanka]

Community Review — Non-record: 11L GEPA + 30k Steps + Pure Int6 + Legal TTT (val_bpb=1.0920)

BPB: 1.0920 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA cbbf3d6347c1, file records/track_non_record_16mb/2026-03-24_11L_GEPA_30kSteps_PureInt6_LegalTTT/train_gpt.py):

The TTT path at line 399 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.16s, dim=512, layers=11, vocab=1024, code=78689 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.16s, dim=512, layers=11, vocab=1024, code=78689 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.