Record: 0.6360 BPB - Depth Recurrence + Multi-Order N-gram Backoff

[Naazimsnh02]

Summary

val_bpb: 0.6360 (seed 1337) | ~15.94 MB | 8×H100 SXM | 3 seeds

Adds multi-order n-gram backoff (orders 2-7) with entropy-adaptive alpha to the depth recurrence stack, achieving a new record.

Key contributions

• Multi-order n-gram backoff (orders 2-7): Hash-table n-gram counting at eval time. Highest-order match first, cascade down on miss. Zero training cost — purely eval-time.
• Entropy-adaptive alpha: alpha = 0.05 + 0.55 * sigmoid(2 * (H − 4.0)) — trusts n-gram more when the neural model is uncertain, model when confident.
• Multi-GPU n-gram prefill: Each rank pre-populates its hash tables with all tokens scored by earlier ranks, fixing the table fragmentation problem on multi-GPU setups (without this, 8-GPU gets 0.87 BPB instead of 0.64).
• Depth Recurrence: Repeating layers 4,5 for 13 virtual layers from 11 physical at zero parameter cost (carried over from previous submission).

Results

Seed	BPB
1337	0.6360
2025	0.6381
42	0.6387
Mean	0.6376

Built on PR #549 stack (LeakyReLU(0.5)², BigramHash(2048), XSA4, Partial RoPE, LN Scale, VE128, EMA+SWA, Parameter Banking + Parallel Muon, int6 GPTQ-lite + lzma).

Credits

• N-gram backoff technique inspired by PR Record: 11L + Multi-Order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.6672) #770 (@minh-stakc) and PR Record: BackoffNgramMixer + Drift-Free TTT (3-seed mean val_bpb=0.6683) #779 (@deanbrr)
• Base model stack: PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 (@abaybektursun), PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 (@signalrush)

[MatoTeziTanka]

Interesting approach — the depth recurrence with layers 4,5 repeated for 13 virtual layers at zero parameter cost is creative, and the multi-GPU n-gram prefill fix (0.87 → 0.64 BPB without it) is a good catch.

You're at 2 seeds right now. The leaderboard requires 3-seed validation for record claims — one more run should close it out.

---

Disclosure: I use Claude Code CLI, Codex CLI, and Gemini Pro as tools in my workflow. Human first, AI-assisted.

[MatoTeziTanka]

Following up on this one with a new finding, since @valerio-oai ruled on the underlying n-gram mechanism after my first comment.

Compliance flag — same disallowed pattern as PR #779.

@valerio-oai disallowed PR #779 (deanbrr) on 2026-03-27 (comment 4145781641) specifically for "hashed n-gram caches, which do not renormalize correctly / correctly reweight the LM's token distribution, look ahead to the target token to mix probabilities and therefore leak eval tokens." The mechanism is spelled out in comment 4146407380: hashing the target token into the bucket key only reweights the correct token, and in the hash-collision limit drives P(correct) toward 1 regardless of the data — arbitrarily low BPB without real compression.

Looking at records/track_10min_16mb/2026-03-26_DepthRecurrence_NgramBackoff_0.6360/train_gpt.py, the multi-order n-gram backoff uses the same target-in-key hashing pattern in every code path that touches the full_tables:

• L1110 (multi-GPU prefill update): full_key = ((ctx_hash ^ (tgt * ng_primes[cw % len(ng_primes)])) & ng_mask).astype(np.int64) → np.add.at(full_tables[oi], full_key, 1) at L1112
• L1170 (eval-time score buildup): full_key = ((ctx_hash ^ (tgt_np * ng_primes[ctx_w % len(ng_primes)])) & ng_mask) → full_counts = full_tables[oi][full_key] at L1179
• L1411 (rank-local score): same pattern, full_key = ((ctx_hash ^ (tgt_np * ng_primes[...])) & ng_mask) → looked up at L1419
• L1461 (chunk-end update): same pattern, np.add.at(full_tables[oi], full_key, 1) at L1463

Each of these hashes the ground-truth tgt / tgt_np / tgt_tok into the bucket key, and the full_counts lookup at scoring time then reads the count of that exact target in that bucket. Under @valerio-oai's #779 ruling, that's the Rule 1 violation in Issue #1017 condition 1 (p_t may depend only on the artifact and x_1...x_{t-1}).

The multi-GPU n-gram prefill mechanism the README describes (0.87 → 0.64 BPB without it → with it) is also worth thinking about in this light: each rank pre-populates its hash tables with all tokens scored by earlier ranks, which means each rank's tables already contain the target tokens it is about to score. The 0.23 BPB gap between "prefill on" and "prefill off" is the size of the leak, not the size of a compression gain.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: CLOSE under the same ruling as #779.

@Naazimsnh02 — please let me know if I've misread the code, especially the full_key lookups at L1110/L1170/L1411/L1461 and whether there's a renormalization over the full vocabulary that I'm missing — if so, I'd want to retract. The depth-recurrence stack (layers 4–5 repeated for 13 virtual layers at zero parameter cost) and the multi-GPU prefill plumbing are both genuinely interesting engineering, separate from the n-gram question — if you wanted to resubmit on a legal base (either dropping the cache and reporting the pure-neural sliding-window number, or reworking the cache as a full-vocab reweighting per @valerio-oai's suggested path on #779), the depth-recurrence and multi-GPU work would carry over cleanly. The 3-seed update from the earlier ask is in place and looks consistent.

---

Reviewed by @MatoTeziTanka — The Agora. Static code review against train_gpt.py at SHA 6c1b833b. AI tooling: review drafted with Claude Code (Sonnet/Opus) using an internal review template; all citations, file paths, and compliance audits were verified against the PR's actual code.