How do we prevent these models to train on our benchmark dataset?

[suqi]

Maybe we can use AI to generate new dataset every new month?

[birdybro]

Benchmaxxing is inevitable over time if your dataset is public, there's no real way around it. Keep the new dataset and results for a new quarterly update private until revealing results. Then make a new dataset and prep for test results again. Always keep them guessing.

You really should have a private invite-only repo with the new dataset branch being worked on in the background and then merge into this and get results and post results quickly.

[Khalilbz]

Guys, this benchmark is not only about AI nonsense (as everyone explaining it), it so crucial for long Agentic work, an Agent WILL hit cases when it should make strategic decisions on where to keep going, the more non sense it can accept, the more messed up the long agent loop will be (maybe not the right place for this comment, i loved to see such a hint in the project, even Agentic behviour is ONE of many things this project reveals, I never seen something like this before, thank you a lot)

[petergpt]

It's not super hard to come up with more nonsense questions, with they start over-training on these specific questions but fail on new ones, it would be good content to embarrass them with