Record Submission: 1.0541 BPB - 5-expert Hedge Mixer + CROWN-Q + stride=64

[RoyiRa]

Record: 5-expert Hedge Mixer + CROWN-Q + stride=64 (val_bpb=1.0541)

val_bpb: 1.0541 (3-seed mean) | ~15.7 MB | 8xH100 SXM

Results (8xH100 80GB SXM)

Seed	step_avg	steps	Pre-TTT bpb	Post-TTT bpb	TTT gain	Eval time	Artifact
1337	98.1ms	5,935	1.1251	1.0473	-0.0778	336s	15.89 MB
42	97.9ms	5,947	1.1264	1.0686	-0.0578	336s	15.69 MB
7	98.0ms	5,940	1.1246	1.0465	-0.0781	336s	15.66 MB
Mean			1.1254	1.0541	-0.0713	336s	~15.75 MB

Contributions

1. CROWN-Q Training Penalty (training-time)

Added a quantization-aware penalty during warmdown that penalizes weights sensitive to quantization error:

crown_q_loss = lambda * mean(w^2 * delta^2 / 12)

where delta = row_max / clip_range is the per-row quantization step size. This encourages weights to be quantization-friendly, reducing post-quantization degradation. CROWN_Q_LAMBDA=0.01.

Effect: Slightly better compression (artifact ~200KB smaller) and more robust quantization.

2. Eval stride 32 -> 64 (eval-time)

Changed sliding window stride from 32 to 64 during evaluation. Experiment showed identical BPB quality but 2x faster scoring. Frees ~100s of eval budget for more TTT epochs.

3. TTT Epochs 3 -> 4 (eval-time)

Increased test-time training from 3 to 4 epochs per chunk, using the time freed by stride=64. Each additional epoch adapts the model more to scored data. Tested 8 epochs but that overfits (1.0735 vs 1.0473 for 4 epochs).

Combined Effect

• stride=64 saves ~100s of eval time
• 4th TTT epoch uses ~85s of the saved time
• Net eval time: ~336s (down from ~562s), well within 600s budget
• BPB improvement: 1.0745 -> 1.0541 (-0.0204)

Architecture

Component	Setting
Layers	11 (512d, 8H, 8KV)
MLP	3.5x with LeakyReLU(0.5)^2
BigramHash	6144 (dim=128)
XSA	All 11 layers (ws=8)
VE128	Layers 9-10
Quantization	Full GPTQ int5 + zstd level 22
Pruning	3% magnitude
TTT	AdamW lr=0.0001, 4 epochs, 131K chunks, Polyak 0.998
Mixer	5-expert Hedge (neural, unigram, bigram, trigram, entropy)
Training reserve	18s (for EMA + calibration + quantization)
Early warmdown	LR schedule targets 582s
CROWN-Q	lambda=0.01 during warmdown
Eval stride	64 (was 32)

Reproduction

DATA_PATH=../data/datasets/fineweb10B_sp1024 \
TOKENIZER_PATH=../data/tokenizers/fineweb_1024_bpe.model \
SEED=1337 MAX_WALLCLOCK_SECONDS=600 \
USE_MIXER=1 MIXER_ETA=0.1 \
TTT_EPOCHS=4 TTT_FREEZE_BLOCKS=2 \
TTT_LR=0.0001 TTT_CHUNK_TOKENS=131072 \
ADAPTIVE_LR=1 ADAPTIVE_LR_MAX=3.0 \
EVAL_STRIDE=64 \
CROWN_Q_LAMBDA=0.01 \
torchrun --standalone --nproc_per_node=8 train_gpt.py

Compliance

Constraint	Limit	Actual	Status
Train time	600s	582s	Pass
Eval time	600s	336s	Pass
Artifact size	16,000,000 bytes	15,892,040 bytes (worst seed)	Pass
No pre-scoring training	—	Score-first TTT: each chunk scored under inference_mode() before any training on it	Pass
GPTQ calibration in training budget	—	Runs within 18s training reserve (1.9s actual)	Pass

Credits

• Base model: PR Record: 11L EMA + GPTQ-lite + warmdown3500 + QAT@0.15 (val_bpb=1.1233) #414 by @signalrush
• TTT recipe: PR Non-record: 11L Depth Recurrence + High-Yield Legal TTT (1.14458 BPB) #461 by @Christopher-Lee-McClendon
• CROWN-Q concept: PR Record: CROWN-Q + Full GPTQ + SWA/EMA Blend — val_bpb 1.1186 (3-seed mean) #693 by @EthanYangTW
• 5-expert Hedge mixer: PR Record: 5-expert Hedge Mixer + TTT (3-seed mean val_bpb=1.0745) #688

[MatoTeziTanka]

Community Review — Record Submission: 1.0541 BPB - 5-expert Hedge Mixer + CROWN-Q + stride=64

Compliance: HOLD — scored-region SLOT pending Issue #1336

PR #700 — HedgeMixer + CROWN-Q + stride=64 (1.0541 BPB)

Author: RoyiRa | Head SHA: 57d1d2c

---

Check 1: N-gram Family Bug (CLOSE trigger)

Result: CLEAN

The trigram update in update() builds the hash key from (prev2, prev1) only:

tri_ctx = ((prev2 * 36313) ^ (prev1 * 27191)) % self.TRI_HASH
tri_idx = tri_ctx * self.V + nxt3

The target token (nxt3) appears only in the value dimension of the count table, not in the key hash. This is structurally identical to BigramHash — context-only key, target as column index. No lookahead in key construction. NOT the disqualifying n-gram bug.

BigramHashEmbedding.bigram_hash() confirmed clean: key = XOR(36313*t[i], 27191*t[i-1]), context-only, no target involvement.

Check 2: Pre-Quant TTT (CLOSE trigger)

Result: CLEAN

The sole TTT function is eval_val_sliding_ttt(), docstring explicitly states "Legal score-first TTT: score each chunk, then train on it. Every token scored BEFORE any update that could use it." The implementation enforces this with two distinct phases per chunk:

• Phase 1 (SCORE): runs under torch.inference_mode(), no gradient computation, accumulates loss/bpb.
• Phase 2 (TRAIN): runs only after Phase 1 completes for the chunk, trains on val_tokens[chunk_start:chunk_end].

Verdict: HOLD — the TTT trains on the same token stream that produces the reported BPB. While the per-chunk score-first discipline is maintained, the model receives gradient updates from the evaluation distribution before scoring later chunks. This is the scored-region SLOT pattern pending Issue #1336.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: HOLD pending Issue #1336 ruling on scored-region SLOT.

---

Reviewed by @MatoTeziTanka — The Agora. Compliance audit via LLM agent (Sonnet) reviewing full train_gpt.py source. If this review misread your code, please call it out so I can re-audit manually.