Skip to content

Add SciΠvot to Datasets and Benchmarks - #44

Merged
yzhao062 merged 1 commit into
yzhao062:mainfrom
SelfAdajoint:add-scipivot
Oct 28, 2026
Merged

yzhao062 merged 1 commit into
yzhao062:mainfrom
SelfAdajoint:add-scipivot

Conversation

@WhymustIhaveaname

Copy link
Copy Markdown
Contributor

Adds one paper row, SciΠvot, to Datasets and Benchmarks (end of the main table, after AgentLeak).

Why it fits the scope: it checks whether an agent's decision at a key point of a research run is right. The 248 cases come from six months of real research, at points where senior researchers corrected an LLM agent (26 projects, six fields). Each case asks the model to pick the next step. Four of five frontier models score below random, and the models tend to keep the original LLM plan.

I added the row and then ran tools/lintfix.py (it re-padded this one table) and tools/recount.py, which updated the derived figures in README.md and assets/social-card.html. recount.py --check passes. I did not re-render social-card.png.

@yzhao062
yzhao062 merged commit 3bea8cf into yzhao062:main Oct 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants