Record: 11-gram Eval Cache + Hedge Mixer (val_bpb: 0.8609)

[sunnypatneedi]

11-gram Eval Cache + Hedge Mixer on PR #549 Base

val_bpb: 0.8609 (3-seed mean, std 0.0008, sliding window stride=64) | ~15.9 MB | 8×H100 SXM

Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128)

Seed	step_avg	steps	Roundtrip bpb	Sliding+N-gram bpb	N-gram gain	Eval time	Artifact
42	92ms	~6,500	1.1452	0.8600	-0.2852	~188s	15,341,541
1337	92ms	~6,500	1.1452	0.8611	-0.2841	~188s	15,918,565
2025	92ms	6,526	1.1452	0.8616	-0.2836	188s	15,790,804
Mean	92ms	~6,500	1.1452	0.8609 (std 0.0008)	-0.284	~188s

Key Innovation: 11-gram Eval Cache with Entropy-Adaptive Mixing

The n-gram eval cache provides -0.284 bpb — the single largest improvement over the base model. It replaces TTT entirely, freeing the full eval time budget.

1. Multi-order n-gram cache (orders 2-11): 10 hash tables with 4M buckets each, uint32 count tables
2. Score-first, update-after protocol: n-gram counts are scored before being updated (legal per @valerio-oai, Issue Parameter Golf Formerly Live AI Commentary ⛳ + Analysis / Ideas | every 10 minutes. Now disabled #140)
3. Entropy-adaptive alpha: mixing weight between neural and n-gram predictions is a function of model entropy — high-entropy (uncertain) tokens get more n-gram contribution
4. Order-adaptive gating: higher-order matches get tighter entropy thresholds via order_centers = 3.0 - 0.25 * (matched_order - min_order)
5. Hedge Mixer: online multiplicative-weights ensemble (beta=2.0) that learns optimal neural vs n-gram weighting across the eval run

N-gram Protocol

1. Initialize 10 hash tables (orders 2-11), each with 4M buckets of uint32 counts
2. For each evaluation position:
   • Score: look up n-gram match for each order (highest order first), compute n-gram probability
   • Compute model entropy from neural logits
   • Compute entropy-adaptive alpha (sigmoid of entropy vs order-specific threshold)
   • Hedge Mixer blends neural and n-gram-enhanced predictions using learned weights
   • Update: increment n-gram counts for all observed n-grams at this position
3. Sliding window eval (stride=64) processes validation tokens with the n-gram cache active

Run Config

cd /workspace/parameter-golf
SEED=42 BIGRAM_VOCAB_SIZE=0 VE_DIM=64 GRADQUANT_ENABLED=0 \
  torchrun --standalone --nproc_per_node=8 \
  records/track_10min_16mb/2026-03-26_sunnypatneedi_moonshot/train_gpt.py

All hyperparameters are baked into the script as defaults. Key environment variables:

# N-gram config
NGRAM_CACHE=1 NGRAM_ORDER=11 NGRAM_MIN_ORDER=2 NGRAM_BUCKETS=4194304
NGRAM_ENTROPY=1 NGRAM_ALPHA=0.40 NGRAM_ENT_BASE=0.05 NGRAM_ENT_RANGE=0.55

# Hedge Mixer
HEDGE_ENABLED=1 HEDGE_BETA=2.0

# Model (no BigramHash, VE_DIM=64 to fit 16MB across all seeds)
BIGRAM_VOCAB_SIZE=0 VE_DIM=64 GRADQUANT_ENABLED=0

# TTT disabled (n-gram replaces it)
TTT_ENABLED=0

Timing Budget

Phase	Time
Training	600s (≤10 min)
Int6 roundtrip eval (diagnostic)	~49s
Sliding window + n-gram + Hedge eval (stride=64)	~188s
Total eval	~237s (< 10 min)

Training Architecture (from PR #549 SOTA)

Component	Setting
Layers	11 (512d, 8H, 4KV GQA)
MLP	3× expansion, LeakyReLU(0.5)²
XSA	All 11 layers
Gated Attention	Enabled
RoPE	Partial (16/64 dims)
LN Scale	1/√(layer+1)
VE64	Layers 7-10
Weight avg	EMA(0.997) + SWA(every 50)
Quantization	Uniform Int6 + zstd-22

Ablation

Config	val_bpb	Delta
Roundtrip (no n-gram, no sliding window)	1.1452	— (baseline)
+ Sliding window (stride=64) + 11-gram + Hedge	0.8609	-0.284

Credits

• Base model (LeakyReLU² + Legal TTT + Parallel Muon): PR #549 by @abaybektursun
• N-gram cache reference: PR #727, PR #758
• Hedge Mixer concept: PR #731
• N-gram cache legality: @valerio-oai (Issue #140)
• Architecture stack: PR #414 by @signalrush
• XSA: PR #198 / PR #503 by @jfprincz

[MatoTeziTanka]

Community Review — 11-gram Eval Cache + Hedge Mixer

BPB: 0.8609 (3-seed, std 0.0008) | Seeds: 3 (42, 1337, 2025) | Artifact: ~15.3–15.9 MB | Compliance: FLAG (same n-gram family bug pattern)

What this does: The base roundtrip neural model scores val_bpb 1.1452. The eval-time pipeline adds a 10-order hashed n-gram cache (orders 2–11, 4M buckets/order, uint32 counts) that blends n-gram probabilities into the neural distribution under an entropy-adaptive alpha, then wraps the neural vs. n-gram-enhanced experts in an online multiplicative-weights Hedge Mixer (beta=2.0). The ablation in the PR body reports the full -0.2843 nats improvement coming from "sliding window + 11-gram + Hedge"; the training model by itself sits at 1.1452.

What I found in the code (records/track_10min_16mb/2026-03-26_sunnypatneedi_moonshot/train_gpt.py at head 8834070):

• L1000–1007 — target token hashed into the lookup key. For each order ctx_w, the code computes

  ctx_hash = np.zeros(len(jv), dtype=np.uint64)
  for k in range(ctx_w):
      tok = val_np[jv - (ctx_w - k)].astype(np.uint64)
      ctx_hash ^= tok * ng_primes[k % len(ng_primes)]
  ctx_key = (ctx_hash & ng_mask).astype(np.int64)
  tgt_np = val_np[jv].astype(np.uint64)
  full_key = ((ctx_hash ^ (tgt_np * ng_primes[ctx_w % len(ng_primes)])) & ng_mask).astype(np.int64)

  tgt_np = val_np[jv] is the target token at each scored position (jv = global_j is the list of scored indices inside the sliding window). It is XOR-mixed into full_key and then used to index full_tables[oi] at L1018 — the probability becomes p = min(full_counts, ctx_counts) / max(ctx_counts, 1) at L1023. This is the exact pattern disallowed on PR Record: BackoffNgramMixer + Drift-Free TTT (3-seed mean val_bpb=0.6683) #779.

• L1049–1065 — Hedge Mixer over (pure neural, n-gram-enhanced). Expert 1 is p_neural_pure (legal). Expert 2 is p_ngram_enhanced = (1 - alpha) * p_neural + alpha * best_p_ng, where best_p_ng is the value read out of full_tables via the target-hashed full_key above. The hedge wrapper itself is mathematically fine — the issue is that one of its two experts is not legal as constructed.

• L1069–1075 — "score-first, update-after" is implemented correctly in the n-gram update path, and the hedge weight update at L1061–1062 also uses the scored segment before committing. Update order is not the issue on this PR; the issue is the lookup construction.

• Training code is clean under the CPU gauntlet. On CPU (cpu_test.py): import OK, Hyperparameters dim=512, layers=11, heads=8, vocab=1024, 26,629,301 params, forward loss 6.9456, artifact 4,591,102 bytes (28.7% of 16 MB under int6+lzma), est. ~13k steps on 8×H100 in 10 min. Nothing wrong with the base model — all of the BPB delta is coming out of the eval cache.

Questions/flags:

• Per @valerio-oai's ruling on PR Record: BackoffNgramMixer + Drift-Free TTT (3-seed mean val_bpb=0.6683) #779 (comment 4145781641, 2026-03-27), hashed n-gram caches that hash the target token into the lookup key are disallowed. The full_key = ((ctx_hash ^ (tgt_np * ng_primes[ctx_w % len(ng_primes)])) & ng_mask) construction at L1006 is the same family pattern. Can you confirm whether the lookup key at L1006 is intended to depend on the target token? The PR body cites Issue Parameter Golf Formerly Live AI Commentary ⛳ + Analysis / Ideas | every 10 minutes. Now disabled #140 as the legality basis, but Issue Parameter Golf Formerly Live AI Commentary ⛳ + Analysis / Ideas | every 10 minutes. Now disabled #140 covers "score-before-update"; it does not address target-in-key indexing, which is what PR Record: BackoffNgramMixer + Drift-Free TTT (3-seed mean val_bpb=0.6683) #779 was closed on.

• Per Issue A Field Guide to Valid Submissions #1017 condition 1, p_t may depend only on the artifact and x_1 ... x_{t-1}. The full_key at L1006 is a function of x_t (via tgt_np), so the count read at L1018 for scoring position t is a function of x_t. Is there a reading of this code under which p_t at position t does not depend on x_t?

• The ablation shows the neural model alone at 1.1452 BPB vs. 0.8609 BPB with the cache — a -0.2843 nat gap, which is ~25% of the neural BPB. The entire delta flows through the hashed cache, so the legality question here is not cosmetic.

• The Hedge Mixer is a clean construction in isolation — if Expert 2 were replaced with a context-only (backward-looking) n-gram cache whose full_key is built from ctx_hash alone plus a separate per-candidate index (not x_t), the wrapper and the mixing logic are reusable. The fix space is "rebuild the lookup," not "rip out Hedge."

Verdict: COMPLIANCE FLAG — same n-gram family pattern as PR #779 / #770 / #798 / #808 / #825.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica:
CLOSE under the same ruling as PR #779 / #770 / #798 / #808 / #825 / #786 / #797. The Hedge Mixer wrapper and the base training code are not the issue and would carry over cleanly to a resubmission with a context-only n-gram cache (lookup key built from ctx_hash alone, with per-candidate probabilities derived from a single context-conditional row over the full vocabulary, as @valerio-oai suggested in PR #779 comment 4146407380).

---

Reviewed by @MatoTeziTanka — The Agora. CPU gauntlet PASS at head 8834070: 26.6M params, int6+lzma artifact 4.59 MB (28.7% of 16 MB), forward loss 6.9456, est. ~13k 8×H100 steps in 10 min — base training code is clean; the compliance flag is entirely on the eval-time n-gram lookup. AI tooling: review drafted with Claude Code (Sonnet/Opus) using an internal review template; all citations, file paths, and compliance audits were verified against the PR's actual code at SHA 8834070de5a6e7ad1df76f660072eb7d54201ed6.