Record: 11L Int6 QAT + SmearGate + OrthoInit + SWA + TTT (val_bpb=1.1478)

[yahya010]

Summary

Full-stack submission: val_bpb = 1.1478 (seed 1337, sliding window stride=64)

12 techniques stacked:

• 11 transformer layers (MLP 3x = 1536 hidden)
• STE int6 QAT — zero quantization gap
• SmearGate — learned token blending
• BigramHash (2048 buckets, dim=128)
• OrthoInit + muP scaling for output projections
• SWA — 8 checkpoint average during warmdown
• TTT — full-weight SGD on val data (lr=0.002, 3 epochs, freeze first 2 blocks)
• NTK-RoPE base=50000
• Muon WD=0.04, momentum=0.99, LR=0.025
• zstd-22 compression, FP16 tied embeddings
• Sliding window eval stride=64

Results

Seed	Steps	Sliding BPB	Artifact
1337	5,166	1.1478	15.76MB

Timing Budget

Phase	Time
Training	600s
TTT	73s
Sliding eval	~370s
Total eval	~443s (< 600s)

Requires: pip install zstandard

[MatoTeziTanka]

[RETRACTED 2026-04-11] — This IMPORT_FAIL was a false positive. Root cause: sibling module exists in same records/ folder; runner sys.path bug. Your code is not broken. See correction below: #150 (comment)

---

Community Review — Record: 11L Int6 QAT + SmearGate + OrthoInit + SWA + TTT (val_bpb=1.1478)

Compliance: NEEDS AUTHOR ACTION — train_gpt.py fails to import on CT2038 (Python 3.10 / torch 2.10.0+cpu)

What I found: The CPU smoke test on CT2038 (proteus-engine, 128 GB RAM, Triton 3.6.0, flash_attn stub, cutlass_evt_fusion stub) failed at the import step with:

ModuleNotFoundError: No module named 'flash_attn'

A few of the common patterns I've seen for this class of error in the 2026-04-11 sweep:

• PEP 701 f-string nesting — e.g. log(f" {cat}: {", ".join(...)}") is valid Python 3.12+ but invalid Python 3.10 because the inner ", " re-enters the outer double-quote context. One-character fix: ', ' instead of ", ". See PR Record: SP8192 + Improved Parallel Residuals + Muon 0.97 + LR 0.03 + Legal TTT — val_bpb 1.07785 (3-seed mean) #1541 / Record: SP8192 + Triple Recurrence + Banking + Fused MLP + Muon 0.97 — val_bpb 1.0778 (3-seed mean) #1523 for reference.
• Missing flash_attn variants — e.g. from flash_attn_interface import flash_attn_varlen_func when the wrapper script only stubs flash_attn_func. Not a PR defect on H100s, but the eval image / CPU preflight path needs a guarded import.
• Local compiled extension — e.g. import cutlass_evt_fusion from a records/*/cutlass_evt_fusion/ subfolder that isn't on the import path at smoke time. Usually an import-order issue inside the script.
• Actual syntax error — typo, missing bracket, etc.

Recommendation: Could you run python3 -c "import py_compile; py_compile.compile('train_gpt.py')" on your records-folder train_gpt.py under Python 3.10 specifically? The eval image is Python 3.10 per Issue #17 / the README, so any parse error on 3.10 blocks the submission at import time before any of the scored-eval logic runs.

Once the parse/import issue is fixed, I'll re-run the compliance audit through the normal pipeline. No other flags identified yet because the audit halts at the import step.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): IMPORT_FAIL — ModuleNotFoundError: No module named 'flash_attn'. Classification via classify_prs.py AST-based classifier; full compliance audit deferred until the import issue is resolved. Auto-drafted from a template and spot-checked before posting.

[MatoTeziTanka]

Retraction — this IMPORT_FAIL was a flash_attn stub gap in my runner

Sorry @yahya010, this one's on me. My CPU smoke runner already ships a stub for flash_attn_interface (so imports like from flash_attn_interface import causal_attention resolve), but it does not stub the bare flash_attn top-level package. Your records/track_10min_16mb/2026-03-20_11L_SmearGate_OrthoInit_SWA_TTT/train_gpt.py imports flash_attn directly, hit the missing stub, and the runner reported ModuleNotFoundError: No module named 'flash_attn'.

On the real eval image (8×H100 SXM Python 3.10), flash_attn is present and your import resolves correctly. The error was a CPU-preflight path gap, not a submission defect.

Your PR is not broken. I'm retracting the IMPORT_FAIL classification and adding a flash_attn stub to the runner so this doesn't hit other PRs. I'll re-queue the full compliance audit and post findings separately.

Again — sorry for the noise.

[MatoTeziTanka]

Community Review — Record: 11L Int6 QAT + SmearGate + OrthoInit + SWA + TTT (val_bpb=1.1478)

BPB: 1.1478 | Compliance: FLAG — Pre-Quant TTT runs multi-epoch on val_tokens with no score-first discipline

What I found in the code (head SHA 1a89f0027bdd, file records/track_10min_16mb/2026-03-20_11L_SmearGate_OrthoInit_SWA_TTT/train_gpt.py):

At line 388 the pre-quant TTT function takes val_tokens as an input argument and runs an epoch loop over it with loss.backward()/optimizer.step(), with no prior torch.no_grad() scoring pass over the same tokens:

ttt_adapt(args, base_model, device, val_tokens, rank, world_size) — for epoch in range(args.ttt_epochs), loss.backward() without prior no_grad score pass

Per Issue #402 and Issue #677 (@valerio-oai, 2026-03-27), TTT is valid only if each token is scored BEFORE the adapter trains on it; multi-epoch TTT that scores only on the final pass is explicitly called out as invalid. This implementation matches the pattern that closed PR #1376 (stukenov) and was subsequently confirmed in #1485/#1487/#1488/#1489/#1517/#1539 — see Issue #677 meta-comment from 2026-04-11 which lists the 6+ PRs in the cluster.

Contrast with the legal Pre-Quant TTT pattern (e.g. PR #1416 / PR #1423 lineage): those train the adapter on a held-out slice of training data (not val_tokens) with score-first-per-chunk discipline. The distinction is on the function signature itself — the argument tensor passed in.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.05s, dim=512, layers=11, vocab=1024, code=64336 B, SMOKE_TEST_PASS

Verdict: COMPLIANCE FLAG — same pattern as the closed Pre-Quant TTT cluster.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: CLOSE under the same ruling as #1376 and the rest of the cluster. A resubmission with the TTT function taking a training-data slice instead of val_tokens (per #1416/#1423 reference implementation) would be welcomed.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.05s, dim=512, layers=11, vocab=1024, code=64336 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.

[MatoTeziTanka]

Correction to the review above — I cited "PR #1416 / PR #1423 lineage" as the "legal Pre-Quant TTT pattern" that trains on a held-out slice of training data with score-first-per-chunk discipline. That citation is wrong. The CLOSE verdict on your PR still stands (your ttt_adapt at line 388 takes val_tokens as an argument and loops args.ttt_epochs over it without per-chunk torch.no_grad() scoring — that's the pattern Issue #677 was opened to rule out, and PR #1376 is the binding precedent).

What I got wrong: I pointed at #1416 and #1423 as the legal contrast pattern. At their current heads, both actually have the same pattern your PR has — ttt_adapt_adamw(args, base_model, device, val_tokens, ...) at line 1132, called with val_tokens, no per-chunk score-first discipline. They belong in the illegal cluster, not as its legal contrast.

The actual legal reference is PR #1413 (dexhunter) — the current leaderboard entry at val_bpb 1.0828 (SP8192 + QK-Gain 5 + Legal Score-First TTT). I decompressed its lzma shim and verified the per-chunk pattern: for each chunk ci, the eval NLL is accumulated into a sliding-BPB loss_sum before the is_last_chunk guard decides whether to let the optimizer touch the weights on that chunk. No token is seen by the optimizer before it has been scored.

Practical implication for a resubmission: to match #1413's legal shape, the TTT function would:

1. Iterate over chunks of val_tokens (or a training-data slice) in eval order
2. For each chunk, compute the NLL under torch.no_grad() and accumulate into the sliding BPB
3. Only then call base_model.train() and optimizer.step() on that chunk's loss
4. Guard the last chunk with is_last_chunk so it never gets adapted before its own scoring pass

Apologies for the wrong citation. The verdict is unchanged — CLOSE under the #1376 ruling — only the legal reference I pointed you at needed fixing.

---

Correction by @MatoTeziTanka — The Agora. The CLOSE verdict stands; only the legal reference citation was wrong.