Record: Score-First TTT + N-gram Backoff (3-seed mean val_bpb=0.9581)

[Asukabot0]

Record: Score-First TTT + Multi-Order N-gram Backoff (val_bpb=0.9581)

3-seed mean val_bpb: 0.9581 (std=0.0005) | ~15.7 MB artifact | 8xH100 SXM

Results

Seed	Sliding BPB (s64)	Artifact	Steps	ms/step	TTT time	Total eval
1337	0.9576	15,721,728	6409	93.63	107.0s	~303s
42	0.9581	15,702,393	6403	93.73	107.9s	~255s
7	0.9585	15,768,158	6407	93.65	105.2s	~251s
Mean	0.9581		~6406	~93.67	~106.7s	~270s

Architecture

• 11L, 512d, GQA (8H/4KV), MLP 3x, U-Net skip connections
• LeakyReLU(0.5)^2: preserves negative gradient flow
• XSA on all 11 layers: removes self-position bias
• Value Residual (VR): layer 0 V output mixed via sigmoid gates
• Gated Attention (GA): per-head sigmoid gates
• SmearGate + OrthoInit, BigramHash(4096), Partial RoPE (16/64), LN Scale
• EMA(0.997), warmdown=3000, int6 per-row + zstd-16

Eval-Time Techniques

Score-First TTT (compliant with Issue #677)

• Process val data in sequential 131K-token chunks
• Phase 1: Score chunk under inference_mode (forward only)
• Phase 2: Train on scored tokens with AdamW (lr=0.0001, 4 epochs)
• Freeze first 2 blocks, grad clip 1.0
• Each token scored BEFORE model trains on it

Multi-Order N-gram Backoff + Entropy-Adaptive Alpha

• Orders 2-7: highest order first, cascade on miss
• Entropy-adaptive: alpha = 0.05 + 0.55 * sigmoid(2 * (H - 4.0))
• Fixed formula, no oracle selection, no target-aware gating
• Backward-looking: cache built from already-scored tokens only

Compliance

• Score-first TTT: tokens scored under inference_mode before training
• N-gram cache: backward-looking, entropy-based mixing (not target-aware)
• GPTQ: not used (naive int6 per-row quantization)
• All training within 600s, all eval within 600s
• No training data accessed at eval time

Reproduction

python3 data/cached_challenge_fineweb.py --variant sp1024
SEED=1337 TTT_ENABLED=1 NGRAM_CACHE=1 \
  torchrun --standalone --nproc_per_node=8 train_gpt.py

Credits

• Base: modded-nanogpt, PR Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248) #315, Non-record: 11L XSA-all + Full GPTQ + Selective Pruning (val_bpb=1.1154, 3-seed) #609
• LeakyReLU^2: PR Record: 11L EMA + Int6 + XSA + LeakyReLU² + Partial RoPE (val_bpb: 1.1309) #493, Record: 11L XSA4 + LeakyReLU(0.5)² + Cosine TTT 50ep (val_bpb=1.0622) #518
• Value Residual: PR Non-record: Value Residual (-0.015 BPB) + Gated Attention (-0.003 BPB) with ablations #413 (arXiv:2410.17897)
• Gated Attention: NeurIPS 2025 (arXiv:2505.06708)
• N-gram cache concept: PR Record: 5-gram Eval Cache + LeakyReLU² + Parallel Muon val_bpb: 1.0920 (3-seed mean, std 0.0007) | ~15.9 MB | 8×H100 SXM #659, Record: 1.0240 BPB — Multi-Order N-gram Backoff + Entropy-Adaptive Alpha (100% autonomous research via goldfish) #702
• Score-first TTT: PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549

[MatoTeziTanka]

Community Review — Score-First TTT + Multi-Order N-gram Backoff

BPB: 0.9581 (3-seed) | Seeds: 3 | Artifact: 15.7MB (98.1% of 16MB budget) | Compliance: FLAG (n-gram cache)

What this does: 11L/512d GQA model with XSA, Value Residual, Gated Attention, LeakyReLU^2, trained ~6,400 steps, then score-first TTT (4 epochs, lr=1e-4, freeze 2 blocks) and evaluated with a sliding-window (stride 64) eval that mixes model probabilities with an entropy-adaptive multi-order (2–7) n-gram cache built online from already-scored tokens.

What I found in the code (records/track_10min_16mb/2026-03-26_ScoreFirst_TTT_Ngram_Backoff/train_gpt.py @ SHA 6827973):

1. TTT ordering is correct (line ~1229 docstring; ordering in eval_val_sliding): chunk is scored under inference_mode in Phase 1 and only trained on in Phase 2. The n-gram table updates (lines 1193–1198) are also deferred until after the segment is scored. "Score-first" is accurately implemented and this is orthogonal to the compliance concern below.

2. N-gram lookup key is target-dependent — same pattern flagged on PRs Record: 11L + Multi-Order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.6672) #770, Record: BackoffNgramMixer + Drift-Free TTT (3-seed mean val_bpb=0.6683) #779, 0.8128 BPB: Classical Compression Eval + N-gram Backoff on PR #549 Base #786, Record: 7-gram N-gram Cache (0.8960 bpb) #797, Record: Order-Adaptive Entropy Gating + BackoffNgramMixer (val_bpb=0.5466) #798, Record: 0.6360 BPB - Depth Recurrence + Multi-Order N-gram Backoff #808, Record: Order-Adaptive BackoffMixer (mean val_bpb=0.5440) #825, Record: 11-gram Eval Cache + Hedge Mixer (val_bpb: 0.8609) #909:

   # line 1163-1164
   tgt_np = val_np[jv].astype(np.uint64)
   full_key = ((ctx_hash ^ (tgt_np * ng_primes[ctx_w % len(ng_primes)])) & ng_mask).astype(np.int64)

   full_key is a function of the target token tgt_np. The cache is read at line 1174 (full_tables[oi][full_key]) — i.e., the shared hash table is probed at a slot that already depends on what the target token is. Because full_tables and ctx_tables are shared across (context, target) pairs and updated incrementally from previously-scored tokens, the count returned at slot full_key is correlated with the target token's identity through the hash, even before any update for the current token has run.

3. Score-first does not fix this. The deferral at lines 1193–1198 correctly prevents self-update leakage, but the lookup key itself — not the update — is where tgt_np enters. Score-first ordering of update-vs-lookup is necessary but not sufficient; the bug is that the lookup KEY is a function of the target token, regardless of when the update runs.

Per @valerio-oai's ruling on PR #779 (comment 4145781641, 2026-03-27), hashed n-gram caches that hash the target token into the lookup key are disallowed. Mechanism detail is in comment 4146407380. Per Issue #1017 condition 1, "p_t may depend only on the artifact and x_1...x_{t-1}."

Logs (seed 42, logs/p23_s42.txt):

• step:6403 val_bpb:1.1434 — training end
• final_int6_roundtrip val_bpb:1.1365 — post-quant, no TTT, no sliding, no n-gram
• final_int6_sliding_window_s64 val_bpb:0.9581 — post-TTT, sliding, n-gram enabled

Total gain from TTT + sliding-window + n-gram is ~0.178 BPB; the logs do not ablate n-gram alone, so the attributable slice is not measurable from the attached artifacts. @Asukabot0, could you attach a seed-42 run with NGRAM_CACHE=0 so the TTT-only contribution vs n-gram contribution can be separated?

Gauntlet (CPU pre-flight): PASS across the board. 27.1M params, 4.56MB int6+lzma artifact in the CPU simulator (reported 15.7MB in the 8xH100 logs — the difference is the sliding-window eval buffer or the submitted model uses different tensor shapes than the CPU stub instantiates; the submitted size is still comfortably under 16MB).

Cluster lineage: The README credits "N-gram cache concept: PR #659, #702" — @lukacf's PRs — not @Asukabot0's own #727 as previously suspected. So PR #761 appears to be a downstream adopter of the n-gram cache pattern, not the originator. The credit chain in this family points upstream to #659/#702; those PRs should be audited for whether they introduced the ctx_hash ^ (target * prime) construction or whether it was added in this branch. I did not audit #659/#702 in this review.

Questions / flags:

• Primary: Per @valerio-oai's ruling on Record: BackoffNgramMixer + Drift-Free TTT (3-seed mean val_bpb=0.6683) #779, is any keyed eval cache where the key depends on the target token disallowed? The lookup at line 1174 reads a shared hash table at a target-indexed slot, which appears to match the flagged pattern.
• Ablation: Can you post a NGRAM_CACHE=0 seed-42 run so TTT-only vs TTT+ngram can be separated?
• Upstream: @lukacf, does the construction at line 1164 of this PR match what you had in PR Record: 1.0240 BPB — Multi-Order N-gram Backoff + Entropy-Adaptive Alpha (100% autonomous research via goldfish) #702? The credits point to your PR — worth confirming where the target-dependent hashing first entered this family.

Verdict: Score-first TTT implementation is correct and careful. Architecture (XSA, VR, GA, LeakyReLU^2) is solid, gauntlet passes, artifact is within budget, 3-seed reproduction is tight (std=0.0005). The n-gram cache at lines 1158–1198 shares the full_key = ((ctx_hash ^ (target * prime)) & mask) construction already flagged as non-compliant on eight sibling PRs.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: CLOSE or NEEDS AUTHOR ACTION — n-gram cache uses the target-dependent lookup-key pattern ruled non-compliant on #779. Author has a clear path forward: remove the ngram_cache eval path and resubmit with the TTT contribution cleanly isolated. Given the tight std, the architecture quality, and the correctness of score-first TTT itself, the non-ngram BPB number would still be a valuable datapoint in the record track.

---

Reviewed by @MatoTeziTanka — The Agora. CPU gauntlet PASS (import/model/forward/artifact/step-time). AI tooling: review drafted with Claude Code (Sonnet/Opus) using an internal review template; all citations, file paths, and compliance audits were verified against the PR's actual code at SHA 682797376f06e5c2297f4ffcc6fe45aaeba5c108.