What happens when you give the agent access to 2M research papers? (+3.2% at 2hr)

[paperlantern-ai]

I connected a research paper index (Paper Lantern — an MCP server over 2M+ CS papers) to the autoresearch agent and ran it head-to-head against the same agent without it.

Setup

• Fork: autoresearch-macos (M4 Pro, 48GB)
• Two identical runs, ~100 experiments each
• Same Claude Code agent, same GPU, same baseline
• Only difference: one had Paper Lantern connected as an MCP tool

Results

	Without PL (43 campaigns)	With PL (50 campaigns)
5-min best val_bpb	0.5563 (-3.67%)	0.5532 (-4.05%)
2hr val_bpb	0.4624	0.4475 (3.2% better)

Gap was still widening at 2hr (2.1% at 1hr → 2.7% at 90min → 3.2% at 2hr).

What the agent found with paper access

AdaGC (arxiv:2502.11034), sqrt batch scaling rule (arxiv:2205.10287), WSD cooldown (arxiv:2508.01483), REX schedule (arxiv:2107.04197), embedding LR scaling (arxiv:2502.19002), and more. 520 papers considered, 100 cited, 25 directly tried. 10 published in 2025+.

Not every idea worked — DyT and SeeDNorm were incompatible with the architecture. But the ones that did were unreachable without the research access.

Full writeup

All 15 paper citations and technique tables: paperlantern.ai/blog/auto-research-case-study

Autoresearch Runs

[paperlantern-ai]

2 Hour run of Best Configs found

[morozow]

I may be mistaken, but this approach is close to the following: if an agent needs to change letter A to letter B, then you teach the agent to recognize letter A in order to change it. I mean, the repo’s goal is a broad analysis of agents’ self-learning, rather than giving the agent more information to solve a specific task.

That said, the idea is still good in sense that extending an agent with additional capabilities is a valid and useful practice. In principle, ACP, MCP, and similar features can also be connected here, as already suggested, and through them it is entirely possible to build a broader agent system. For example, one agent could handle the main task while other agents run additional analysis in parallel on the side. So overall, this direction is definitely expandable and has real practical potential.

[paperlantern-ai]

Yes - the agent here is primarily demonstrating the self-learning ability. Our work above shows that imbibing ideas from our work, gives a large boost to the agent's output.

[connerlambden]

Really interesting results — the 3.2% improvement at 2hr with Paper Lantern is compelling evidence that agents meaningfully benefit from paper access during research.

We've been building something complementary in this space: BGPT MCP, which focuses on structured experimental data extracted from full-text studies rather than abstracts/metadata. Each result returns 25+ structured fields (methods, results, sample sizes, statistical analyses, etc.), which we've found helps agents make more grounded experimental decisions.

It'd be interesting to see how structured experimental data compares to abstract-level paper search for autoresearch runs — whether having concrete methodological details (e.g. exact hyperparameters, training procedures from related work) gives the agent even more actionable information to work with.

Config is straightforward if anyone wants to try it:

{"mcpServers": {"bgpt": {"command": "npx", "args": ["-y", "bgpt-mcp"]}}}

50 free searches, no API key needed to start.

[morozow]

@connerlambden If I'm right, the commercial ads are in the "next branch" (joke)