Invalid submissions due to information leakage during TTT

[leloykun]

Don't train on eval tokens your model hasn't scored yet!

---

"Proper" TTT goes like this:

1. For each 1 <= t <= T:
   1.1. Score on eval token t
   1.2. Adapt weights based on eval token <= t

---

What y'all are doing is something like this:

1. For each 1 <= t <= T:
   1.1. Adapt weights based on eval token <= t
2. For each 1 <= t <= T:
   2.1. Score on eval token t

But this is equivalent to appending the eval tokens to the training tokens and switching training strategies before eval! Also see: #152 (comment)

---

Potentially invalid submissions:

PR	comment	status
#136	TTT on half of the batch; eval on full batch	[ ] open
#152	TTT on all eval tokens before evaluation	[x] closed
#254	TTT on multiple parts of eval sequence for multiple epochs	[ ] open
#264	TTT before eval	[ ] open
#338	TTT on multiple parts of eval sequence for multiple epochs	[ ] open
#398	TTT on all eval tokens before evaluation	[ ] Open
#421	TTT before evals	[ ] open
#417	TTT for multiple epochs before evals	[ ] open
#442	TTT before evals	[ ] open

cc @0hq

---

Please feel free to correct me if I'm wrong.

[leloykun]

I've also noticed that people are doing TTT on multiple parts of the evaluation sequence in parallel (via DDP rank), justifying it by only assigning each "doc" to one rank.

I'm not sure if this is comparable to the default eval strategy where we just feed all eval tokens to the model... wdyt @0hq ?

[himanalot]

this is clearly happening bc people are using agents to monitor/use others' prs to improve their strategies. it's had an exponentially strong trickledown effect ever since a few of them made submissions that appear good at first glance with this method of basically cheating. it's a bit odd to watch.

[chris-lee-mc]

For those using AI coding tools, I might suggest referencing the competition requirements in CLAUDE.md or AGENTS.md so they are in every prompt submission. Good hunting!

[hegdeadithyak]

How does TTT even make sense? I don’t think it should be allowed at all.

[sp00mm]

Yes! So far every sub 1.1 ish PR I've looked at is "cheating" like this. Very irritating when I'm trying to understand the true sota techniques.

[lukacf]

this is clearly happening bc people are using agents to monitor/use others' prs to improve their strategies. it's had an exponentially strong trickledown effect ever since a few of them made submissions that appear good at first glance with this method of basically cheating. it's a bit odd to watch.

This is exactly what is happening. I had a fully automated run get to 0.979 val BPB in a pretty convincing way (#517) only to later on closer inspection see that it really had bet all on on TTT annealing (and no, not the proper variety).
Still, kind of informative to see the failure modes of the models. You can get a lot of experiments done quickly, but if they veer off in the wrong direction.. well it's one way to burn gpu hours. On the plus side, the same agent that did the solution also caught on that it might not be fully legit and it did that by reading this thread. I'm pretty sure I would have not noticed it had it not flagged it as I've been busy exploring the next thing for it to try...