Non-record: 11L EMA + TTT(20ep,freeze=0) + 15-run ablation study — val_bpb=1.1213 (3-seed)

[felipe-parodi]

Record: 11L EMA + TTT(20ep) — val_bpb: 1.1213

val_bpb = 1.1213 (sliding window stride=64, best seed 1337) | 15.53 MB artifact | 8xH100 SXM, 600s

Key Finding: EMA + Aggressive TTT with All Blocks Unfrozen

EMA(0.997) weight averaging combined with aggressive test-time training (20 epochs SGD, lr=0.008, all blocks unfrozen) outperforms Tight SWA + VE128 approaches (PR #388, 1.1231).

Results (3-seed, 8xH100 SXM)

Seed	Steps	Sliding BPB (s64)	Artifact
1337	7386	1.1213	15.53 MB
42	7411	1.1221	15.51 MB
2025	7386	1.1228	15.53 MB

Mean: 1.1221 | Std: 0.0008

Critical Discoveries (15-run ablation)

1. TTT_FREEZE_BLOCKS=0 is essential — freezing early blocks during aggressive TTT creates internal inconsistency (quant gap 5x worse)
2. Late QAT is counterproductive with aggressive TTT
3. XSA removed — saves ~1.4ms/step, yielding ~130 more training steps
4. PPM-C eval blending hurts strong models — classical compression adds noise when neural model is already strong
5. Memory tokens, aggressive warmdown, gradient-guided quant — all documented negative results

Run Command

pip install zstandard flash_attn_3 --find-links https://windreamer.github.io/flash-attention3-wheels/cu128_torch291

SEED=1337 NUM_LAYERS=11 BIGRAM_VOCAB_SIZE=2048 XSA_LAST_N=0 \
EMA_ENABLED=1 EMA_DECAY=0.997 SWA_ENABLED=0 \
ROPE_DIMS=16 LN_SCALE=1 LATE_QAT=0 \
TTT_ENABLED=1 TTT_LR=0.008 TTT_EPOCHS=20 TTT_MOMENTUM=0.9 TTT_FREEZE_BLOCKS=0 \
MUON_WD=0.04 ADAM_WD=0.04 \
MATRIX_LR=0.025 SCALAR_LR=0.025 TIED_EMBED_LR=0.035 \
MUON_MOMENTUM=0.99 MUON_MOMENTUM_WARMUP_START=0.92 \
MUON_MOMENTUM_WARMUP_STEPS=1500 WARMDOWN_ITERS=3000 \
ITERATIONS=9000 MAX_WALLCLOCK_SECONDS=600 EVAL_STRIDE=64 \
torchrun --standalone --nproc_per_node=8 train_gpt.py

See the full README in the submission folder for detailed architecture, training config, TTT analysis, and the complete 15-run ablation table.

[felipe-parodi]

Converting to non-record per TTT ruling (Issue #402). The README documents a 15-run ablation (memory tokens, causal TTT, PPM-C blending, grad-guided quant, aggressive warmdown — all negative results at the frontier) and the freeze_blocks=0 finding for aggressive TTT. Working on a non-TTT submission.

[MatoTeziTanka]

Community Review — Non-record: 11L EMA + TTT(20ep,freeze=0) + 15-run ablation study — val_bpb=1.1213 (3-seed)

BPB: 1.1213 | Compliance: FLAG — Pre-Quant TTT runs multi-epoch on val_tokens with no score-first discipline

What I found in the code (head SHA db2c5796c539, file records/track_10min_16mb/2026-03-21_11L_EMA_TTT20ep_1.1213/train_gpt.py):

At line 1007 the pre-quant TTT function takes val_tokens as an input argument and runs an epoch loop over it with loss.backward()/optimizer.step(), with no prior torch.no_grad() scoring pass over the same tokens:

ttt_adapt(args, base_model, device, val_tokens, rank, world_size, log_fn) — for epoch in range(args.ttt_epochs), loss.backward() without prior no_grad score pass

Per Issue #402 and Issue #677 (@valerio-oai, 2026-03-27), TTT is valid only if each token is scored BEFORE the adapter trains on it; multi-epoch TTT that scores only on the final pass is explicitly called out as invalid. This implementation matches the pattern that closed PR #1376 (stukenov) and was subsequently confirmed in #1485/#1487/#1488/#1489/#1517/#1539 — see Issue #677 meta-comment from 2026-04-11 which lists the 6+ PRs in the cluster.

Contrast with the legal Pre-Quant TTT pattern (e.g. PR #1416 / PR #1423 lineage): those train the adapter on a held-out slice of training data (not val_tokens) with score-first-per-chunk discipline. The distinction is on the function signature itself — the argument tensor passed in.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.03s, dim=512, layers=9, vocab=1024, code=71770 B, SMOKE_TEST_PASS

Verdict: COMPLIANCE FLAG — same pattern as the closed Pre-Quant TTT cluster.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: CLOSE under the same ruling as #1376 and the rest of the cluster. A resubmission with the TTT function taking a training-data slice instead of val_tokens (per #1416/#1423 reference implementation) would be welcomed.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.03s, dim=512, layers=9, vocab=1024, code=71770 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.