Self-assessment: Causal SLOT legality

I want to be transparent about the legality of this submission and invite community review.

This submission uses Causal SLOT — a context-only variant of SLOT where the delta vector is optimized on already-scored positions only. Standard SLOT was proven to violate Condition 1 by PR #1240 (100% violation rate). Causal SLOT restricts optimization to context-only positions, which should fix the causal violation.

Why I think it's legal

The delta at position t depends only on tokens x_1,...,x_{t-64} (all scored in prior windows). The gradient of the context-only loss w.r.t. delta flows only through context positions — new positions contribute zero gradient because of the context mask. compute_logits is position-independent (RMSNorm normalizes per-feature, linear projection is per-position), so no cross-position leakage in the gradient.

This is the same causal guarantee as score-first TTT: adapt on scored tokens, apply to future predictions. The only difference is delta optimization (512 dims) vs weight updates (34M params).

Why it might not be legal

1. No explicit ruling from @0hq or @valerio-oai on causal SLOT specifically
2. PR Non-record: Does SLOT violate causal dependence? (empirical test + question) #1240 raised concerns about SLOT broadly, though specifically about the non-causal variant
3. The delta is a "free parameter" optimized per-batch on val data — purists may object even if it's backward-looking
4. The -0.013 BPB gain is large enough to look suspicious

Request

@0hq @valerio-oai — could you weigh in on whether context-only SLOT (where optimization loss uses ONLY already-scored positions) satisfies the four conditions from Issue #1017? Several submissions use this approach (PRs #1306, #1322, #1324, and now this one).

I also have a fully legal Track A submission at PR #1334 (1.0897 BPB, no SLOT, no TTT, no eval-time adaptation) as a fallback.

> I also have a fully legal Track A submission at PR #1334 (1.0897 BPB, no SLOT, no TTT, no eval-time adaptation) as a fallback.

Did the 4096 vocab get approved? I remember custom token sets needed approval before adoption, was there any official movement on it, or did everyone just adopt it right away?

@newjordan Good question. The sp4096 tokenizer wasn't individually "approved" — it was introduced by @clarkkev in PR #1218 with data hosted on their HF repo (kevclark/parameter-golf). The README rule (criterion 2) says: "If changes are made to the tokenizer or dataset, prove with certainty that the val_bpb is correctly calculated. Submissions that edit the tokenizer will be examined much more carefully."

I didn't create a new tokenizer — I'm using the same sp4096 SentencePiece BPE model from @clarkkev's export, same as PRs #1218, #1285, #1287, #1291, and several others. The byte-accounting uses the standard build_sentencepiece_luts() function from the codebase, which correctly handles leading-space bytes, byte-fallback tokens, and boundary tokens for any SentencePiece vocab. It's not a custom tokenizer like Scylla (TokenMonster) — it's just a larger standard SentencePiece BPE trained on the same FineWeb docs.

That said, I don't think there's been an explicit "sp4096 is approved" statement from the maintainers. It's been adopted organically by ~8 PRs at this point. The competition description does say "tokenizer-agnostic" and encourages "novel tokenizers" as a valid approach.

@newjordan Good question. The sp4096 tokenizer wasn't individually "approved" — it was introduced by @clarkkev in PR #1218 with data hosted on their HF repo (kevclark/parameter-golf). The README rule (criterion 2) says: "If changes are made to the tokenizer or dataset, prove with certainty that the val_bpb is correctly calculated. Submissions that edit the tokenizer will be examined much more carefully."

I didn't create a new tokenizer — I'm using the same sp4096 SentencePiece BPE model from @clarkkev's export, same as PRs #1218, #1285, #1287, #1291, and several others. The byte-accounting uses the standard build_sentencepiece_luts() function from the codebase, which correctly handles leading-space bytes, byte-fallback tokens, and boundary tokens for any SentencePiece vocab. It's not a custom tokenizer like Scylla (TokenMonster) — it's just a larger standard SentencePiece BPE trained on the same FineWeb docs.

That said, I don't think there's been an explicit "sp4096 is approved" statement from the maintainers. It's been adopted organically by ~8 PRs at this point. The competition description does say "tokenizer-agnostic" and encourages "novel tokenizers" as a valid approach.

Hope it goes through! Looks great and is looking to be strong. I've got a couple tricks to pull on the 11x but it's a hell of a time trying to keep up with the bob on the old engine.