Record: GatedAttn + ValueResidual + Full QAT + lzma-9 + BigramHash(2048)

[FlashyFlash3011]

Submission

Experiment: `records/track_10min_16mb/2026-03-27_GPTQLite_QAT_MaxLZMA_LegalTTT/`

---

Strategy: Pure Velocity & TTT Preservation

Initial attempts tried to maximize model capacity (GatedAttention, ValueResidual, BigramHash=2048). Ablations showed these features add ~1.5ms/step overhead and destabilize TTT, costing more in training steps than they gain in quality under the 10min/16MB constraint. The winning strategy strips the model to its leanest form.

Results (8×H100 80GB SXM)

Seed	step_avg	Steps	Pre-TTT BPB	Post-TTT BPB	TTT Gain	TTT Time	Artifact
1337	83.87ms	7155	1.12163921	1.11901233	-0.00262688	421.9s	15.851MB
42	83.86ms	7156	1.12228806	1.11960558	-0.00268248	423.2s	15.858MB
2025	83.89ms	7154	1.12197720	1.11920302	-0.00277418	423.4s	15.888MB
Mean	83.87ms	7155	1.12196816	1.11927364	-0.00269451	422.8s	15.866MB

Key Changes

Change	Why
GATED_ATTENTION=0, VALUE_RESIDUAL=0	+1.5ms/step overhead → 130+ lost training steps in 600s
SWA_ENABLED=0	Was copying hundreds of MB GPU→CPU every 50 steps — EMA is used at the end, not SWA
BANK_QAT_THRESHOLD=0	Was snapping FP32 TTT weights back to Int6 mid-evaluation, causing catastrophic forgetting
LATE_QAT_THRESHOLD=0.15	QAT only in final 15% of warmdown — no overhead during main training
TRAIN_SEQ_LEN=2048	Allows full warmdown (7155 steps vs ~5776 at 4096 ctx)

Features Explored but Disabled

These were implemented and tested but hurt under the 10min/16MB constraint. They remain in the codebase and are expected to help significantly with more budget:

Feature	Why disabled	Why it helps with more budget
GatedAttention, ValueResidual	+1.5ms/step → 130+ lost steps	Legitimate architectural gains with 30min+ training
BigramHash=2048	Pushed artifact over 16MB	Better subword context modeling
QAT from step 1	Overhead throughout training	Full-run quant adaptation reduces post-quant degradation
BANK_QAT_THRESHOLD > 0	Corrupts TTT weights	Enables aggressive compression of larger models

Headroom & Scaling Evidence

Submission sits at 15.851–15.888MB across seeds (mean 15.866MB) — ~134KB under the 16MB limit. Attempts to fill headroom (BigramHash=1664, 2048) produced worse BPB and exceeded the size limit. In an uncapped scenario, all disabled levers can be opened simultaneously for significantly better BPB.

[MatoTeziTanka]

Community Review — Record: GatedAttn + ValueResidual + Full QAT + lzma-9 + BigramHash(2048)

BPB: (not parsed — see PR title) | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA 92e02e056713, file records/track_10min_16mb/2026-03-27_GPTQLite_QAT_MaxLZMA_LegalTTT/train_gpt.py):

The TTT path at line 1105 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.05s, dim=512, layers=11, vocab=1024, code=93765 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.05s, dim=512, layers=11, vocab=1024, code=93765 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.