Non-record: KNN Hidden State Retrieval — Scale Deception from Weak to Strong Models (8xH100)

[himanshudongre]

Summary

Novel eval-time technique: KNN Hidden State Retrieval. Zero artifact cost, score-first protocol.

Key finding: helps weak models (-2 to -4%), HURTS strong competition-quality models (+1.5%).

Model Quality	KNN Effect
Weak (1xH100, 2K steps)	-1.57% (helps)
Strong (8xH100, 5665 steps, SOTA)	+1.47% (hurts)

Second confirmed case of scale deception in this competition (first: SSM in PR #1013/PR #1227).

8xH100 Results

• Neural roundtrip BPB: 1.1533
• KNN BPB: 1.1702 (+1.47% worse)
• KNN eval time: 168s (fits in 600s)
• Artifact: 15.8MB (KNN adds 0 bytes)

Implication

Eval-time prediction mixing has negative returns on strong models. Techniques that adapt the model (TTT) may work better than mixing external distributions.

Full scaling analysis in README.