SP8192 + Depth Recurrence + Parallel Residuals (14.09MB)

[dippatel1994]

Summary

• SP8192 tokenizer (from kevclark/parameter-golf) — 8192 vocab BPE for lower tokens-per-byte
• 3-layer depth recurrence — layers 3-5 looped 2x (13 effective layers from 10 physical, +5.2% BPB in ablation)
• Parallel residuals on layers 7+ — attention + MLP in parallel from same normalized input
• U-Net skip connections — 5 encoder + 5 decoder with learned skip weights
• Full-Hessian GPTQ int6/int5 with percentile-search optimal scales
• 14.09MB artifact (under 16MB limit)

Stacked with: BigramHash(10240), SmearGate, EMA(0.997), LeakyReLU squared, GQA(8q/4kv), partial RoPE(16), value residual, XSA(last 4), orthogonal init, Muon+AdamW, late QAT.

Results

• val_bpb: 1.6323 (1xH100 SXM, seed=42, 600s wallclock)
• pre-quant val_bpb: 1.2956
• 2855 steps at 210ms/step
• Note: tested on 1xH100 (1/8 competition compute). BPB will improve significantly on 8xH100

Reproduction

rm -f data/manifest.json
MATCHED_FINEWEB_REPO_ID=kevclark/parameter-golf python3 data/cached_challenge_fineweb.py --variant sp8192
torchrun --standalone --nproc_per_node=8 train_gpt.py

Ablation (sp1024, 2-min, 1xH100)

Technique	final_bpb	Delta
Baseline	3.574	--
+ Depth Recurrence	3.387	-5.2%
+ Big Batch (262K)	3.448	-3.5%
+ QK-Gain 5.0	3.758	+5.1% (GPTQ degrades without SDClip)

Test plan

• Validate on 8xH100 with TRAIN_BATCH_TOKENS=786432
• Run 3 seeds for statistical significance
• Enable sliding window + TTT eval for final BPB

[MatoTeziTanka]

Community Review — SP8192 + Depth Recurrence + Parallel Residuals (14.09MB)

BPB: 1.6323 | Compliance: FLAG — Pre-Quant TTT runs multi-epoch on val_tokens with no score-first discipline

What I found in the code (head SHA e273ecdd20f9, file records/track_10min_16mb/2026-04-09_SP8192_DepthRecur_ParResid/train_gpt.py):

At line 996 the pre-quant TTT function takes val_tokens as an input argument and runs an epoch loop over it with loss.backward()/optimizer.step(), with no prior torch.no_grad() scoring pass over the same tokens:

prequant_ttt(args, base_model, rank, world_size, device, val_tokens) — for epoch in range(pq_epochs), loss.backward() without prior no_grad score pass

Per Issue #402 and Issue #677 (@valerio-oai, 2026-03-27), TTT is valid only if each token is scored BEFORE the adapter trains on it; multi-epoch TTT that scores only on the final pass is explicitly called out as invalid. This implementation matches the pattern that closed PR #1376 (stukenov) and was subsequently confirmed in #1485/#1487/#1488/#1489/#1517/#1539 — see Issue #677 meta-comment from 2026-04-11 which lists the 6+ PRs in the cluster.

Contrast with the legal Pre-Quant TTT pattern (e.g. PR #1416 / PR #1423 lineage): those train the adapter on a held-out slice of training data (not val_tokens) with score-first-per-chunk discipline. The distinction is on the function signature itself — the argument tensor passed in.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=11, vocab=8192, code=74713 B, SMOKE_TEST_PASS

Verdict: COMPLIANCE FLAG — same pattern as the closed Pre-Quant TTT cluster.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: CLOSE under the same ruling as #1376 and the rest of the cluster. A resubmission with the TTT function taking a training-data slice instead of val_tokens (per #1416/#1423 reference implementation) would be welcomed.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.04s, dim=512, layers=11, vocab=8192, code=74713 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.

[dippatel1994]

Thanks @MatoTeziTanka for the detailed review and catching the compliance issue.

Fixed in commit fd9bde7: Pre-quant TTT is now disabled by default (PREQUANT_TTT_ENABLED=0). The function remains in code but is not called.

The confirmed val_bpb=1.4794 was already measured without pre-quant TTT (the 1xH100 test run had PREQUANT_TTT_ENABLED=0), so the reported score is unaffected.

All other techniques are compliant:

• SDClip GPTQ (k=12.85) — quantization method, no val data dependency
• MuonEq-R — optimizer modification, training-phase only
• Depth recurrence — architecture, no eval dependency
• Parallel residuals — architecture
• Brotli + byte shuffle — compression method
• QK-Gain 5.25 — learnable parameter

Happy to make further adjustments if needed.

[MatoTeziTanka]

Re-audited at head SHA 0215c2e1. Full code read (1,375 lines) + CPU gauntlet on CT2038.

Gauntlet result (CT2038, Python 3.10, torch 2.10.0+cpu):

Import: PASS (0.1s)
Hyperparameters: dim=512, layers=11, heads=8, vocab=8192
Model: PASS (37,338,221 params)
Forward pass: PASS (loss=9.0132)
Code size: 75,005 bytes
Artifact: FAIL — expected on CPU (GPTQ int6 quantization requires GPU)

Pre-quant TTT fix confirmed. Line 1300: PREQUANT_TTT_ENABLED defaults to "0". The multi-epoch AdamW function (lines 997-1061) is never called. The reported val_bpb=1.4794 was measured with standard eval (EVAL_STRIDE=0, per submission.json) on 1×H100, so neither sliding-window TTT nor the n-gram cache were active for that number.

What the code does at default competition settings (eval_stride=64, ttt_enabled=1, ngram_alpha=0.20):

1. Sliding-window TTT with LoRA (lines 908-994): Scores each batch under torch.inference_mode() (lines 942-964), records loss, THEN adapts LoRA layers via AdamW (lines 966-971). Next batch sees updated weights. This is the legal score-first-per-chunk pattern — each token is scored before the adapter trains on it. Reset every 8 chunks (line 973). Structurally matches PR Record: SP8192 + QK-Gain 5 + Legal Score-First TTT — val_bpb 1.08279 (3-seed mean) #1413 (dexhunter).

2. 7-gram cache (lines 775-800): Context-only key construction (ctx = tuple(val_np[ctx_end - order:ctx_end].tolist()) at line 789) — no target token in the key. Pre-trained on 2M training tokens (line 820, legal), updated with scored tokens AFTER scoring (line 855/964). Same pattern as merged PR Record: 0.4416 BPB -- Complementary Training + Backoff N-gram Mixer #803 (pentxayc). Not the target-in-key family bug from PR Record: BackoffNgramMixer + Drift-Free TTT (3-seed mean val_bpb=0.6683) #779.

Updated verdict: LOOKS CLEAN. The pre-quant TTT that was flagged is disabled. The two active eval-time mechanisms (sliding TTT + n-gram cache) are both legal — score-first TTT matches #1413, and the n-gram uses context-only keys matching #803.

Citation correction: My original review cited #1416/#1423 as "the legal Pre-Quant TTT pattern." That was wrong — both have the illegal flat-epoch pattern. The correct legal TTT reference is PR #1413 (dexhunter).

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending usual record-track checks. Note that the reported 1.4794 BPB is a conservative 1×H100 standard-eval baseline — the 8×H100 score with sliding TTT + n-gram active will be lower.

Thanks for the fast turnaround @dippatel1994.

---

Re-audit by @MatoTeziTanka. CPU gauntlet on CT2038 (Python 3.10, torch 2.10.0+cpu): IMPORT_OK, MODEL_OK, FORWARD_OK. Full code review: pre-quant TTT disabled (line 1300), sliding TTT is score-first (lines 942-971), n-gram cache uses context-only keys (line 789).