Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,20 @@ Steel is an open-source browser API purpose-built for AI agents.
| Rank | Agent | Organization | WebVoyager Score | Source | Open Source | New | SOTA |
| ---- | --------------- | -------------- | ---------------- | ------------------------------------------------------------------------------------------------- | ----------- | --- | ---- |
| 1 | Surfer 2 | H Company | 97.1% | [Source](https://hcompany.ai/surfer-2) | No | Yes | Yes |
| 2 | Magnitude | Magnitude | 93.9% | [Source](https://magnitude.run/webvoyager) | Yes | Yes | |
| 2 | Magnitude | Magnitude | 93.9% | [Source](https://magnitude.run/webvoyager) | Yes | No | |
| 3 | AIME Browser-Use | Aime | 92.34% | [Source](https://aime-browser-use.github.io/) | No | Yes | |
| 4 | Surfer-H + Holo1 | H Company | 92.2% | [Source](https://arxiv.org/pdf/2506.02865) | No | Yes | |
| 5 | Browserable | Browserable | 90.4% | [Source](https://www.browserable.ai/blog/web-voyager-benchmark) | Yes | Yes | |
| 6 | Browser Use | Browser Use | 89.1% | [Source](https://browser-use.com/posts/sota-technical-report) | Yes | Yes | |
| 7 | Operator | OpenAI | 87% | [Source](https://openai.com/index/introducing-operator/) | No | Yes | |
| 8 | Skyvern 2.0 | Skyvern | 85.85% | [Source](https://blog.skyvern.com/skyvern-2-0-state-of-the-art-web-navigation-with-85-8-on-webvoyager-eval/) | Yes | Yes | |
| 9 | Project Mariner | Google | 83.5% | [Source](https://deepmind.google/technologies/project-mariner/) | No | | |
| 10 | Notte | Notte | 73.1% | [Source](https://github.com/nottelabs/open-operator-evals#opensource-operators-evals) | Yes | | |
| 11 | Agent-E | Emergence AI | 73.1% | [Source](https://www.emergence.ai/blog/agent-e-sota) | No | | |
| 12 | WebSight | Academic Research | 68% | [Source](https://arxiv.org/abs/2508.16987) | No | | |
| 13 | Runner H 0.1 | H Company | 67% | [Source](https://www.hcompany.ai/blog/a-research-update) | No | | |
| 14 | WebVoyager | Academic Research | 59.1% | [Source](https://arxiv.org/abs/2401.13919) | Yes | | |
| 15 | WILBUR | Academic Research | 53% | [Source](https://arxiv.org/abs/2404.05902) | No | | |
| 6 | Browser Use | Browser Use | 89.1% | [Source](https://browser-use.com/posts/sota-technical-report) | Yes | No | |
| 7 | Operator | OpenAI | 87% | [Source](https://openai.com/index/introducing-operator/) | No | No | |
| 8 | Skyvern 2.0 | Skyvern | 85.85% | [Source](https://blog.skyvern.com/skyvern-2-0-state-of-the-art-web-navigation-with-85-8-on-webvoyager-eval/) | Yes | No | |
| 9 | Project Mariner | Google | 83.5% | [Source](https://deepmind.google/technologies/project-mariner/) | No | No | |
| 10 | Notte | Notte | 73.1% | [Source](https://github.com/nottelabs/open-operator-evals#opensource-operators-evals) | Yes | Yes | |
| 11 | Agent-E | Emergence AI | 73.1% | [Source](https://www.emergence.ai/blog/agent-e-sota) | No | No | |
| 12 | WebSight | Academic Research | 68% | [Source](https://arxiv.org/abs/2508.16987) | No | No | |
| 13 | Runner H 0.1 | H Company | 67% | [Source](https://www.hcompany.ai/blog/a-research-update) | No | No | |
| 14 | WebVoyager | Academic Research | 59.1% | [Source](https://arxiv.org/abs/2401.13919) | Yes | No | |
| 15 | WILBUR | Academic Research | 53% | [Source](https://arxiv.org/abs/2404.05902) | No | No | |

**Notes:**

Expand Down
22 changes: 22 additions & 0 deletions docs/2026-03-13-bench-index-updates.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Benchmark Index Updates - 2026-03-13

- Added new `/results` page with a unified table of all agent benchmark results across all categories. (To give users a single place to compare results across benchmarks without navigating between pages.)
- Created `src/lib/index-data.ts` with a typed `IndexEntry` interface including `agent`, `organization`, `benchmark`, `benchmarkCategory`, `score`, `scoreNum`, `source`, `selfReported`, `openSource`, and `isNew` fields. (To keep index data structured, typed, and easy to extend as new results come in.)
- Added 75+ entries across 15 benchmarks: WebVoyager, WebArena, Online-Mind2Web, BrowseComp, GAIA, OSWorld, AndroidWorld, SWE-bench Verified, MLE-bench, GPQA Diamond, ARC-AGI-2, ToolBench, AgentBench, and more. (To give the index comprehensive coverage of the current agent benchmark landscape.)
- Populated WebArena entries from the community-maintained Google Sheets leaderboard, including DeepSeek v3.2 (74.3%), OpAgent (71.6%), ColorBrowserAgent (71.2%), Claude Code + GBOX (68%), DeepSky (66.9%), and 15+ more entries down to GPT-4 baseline. (To reflect the most complete and up-to-date WebArena standings.)
- Populated OSWorld Verified entries from the XLANG Lab blog post, including Agent S3 (69.9%), CoACT-1 (60.76%), Agent S2.5 w/ o3 (56.0%), GTA1 w/ o3 (53.1%), OpenAI CUA o3 (42.9%), UI-TARS-1.5 (42.5%), and Claude 3.7 CU (28%). (To reflect the July 2025 OSWorld-Verified re-evaluation under unified AWS infrastructure.)
- Populated GAIA entries including h2oGPTe (75%), Manus (~75%), Deep Research (67.36%), MS Research o1 (38%), and HF Agents (33%). (To surface one of the most competitive general-reasoning leaderboards.)
- Added `scoreNum: number` field alongside the display `score: string` field on every entry. (To enable accurate numeric sorting without parsing percentage strings at runtime.)
- Implemented live text search filtering across agent name, organization, and benchmark name simultaneously. (To let users quickly find specific results without scrolling through the full table.)
- Implemented category filter pills (ALL, WEB NAV, RESEARCH, DESKTOP, CODING, TOOL USE, GENERAL, SPECIALIZED) that sync with inline category badges on each row. (To let users narrow results by domain with a single click.)
- Implemented per-benchmark filter pills below the category row. (To let users isolate a single benchmark's full result set.)
- Added clickable score column header that toggles ascending/descending sort on `scoreNum`. (To let users quickly find the top or bottom performers across any filtered view.)
- Added inline category badge click on each row that filters the table to that category, toggling off on second click. (To make filtering feel fast and contextual without reaching for the filter bar.)
- Added cycling behavior to the CATEGORY / BENCHMARK column header — clicking steps through all categories in order. (An alternative way to cycle filters without using the pill buttons.)
- Added SRC column (SELF / 3RD) indicating whether a result is self-reported or independently verified. (To surface data quality signal directly in the table without requiring users to follow source links.)
- Added OSS column indicating open-source status of each agent. (To help users quickly identify reproducible and commercially-usable systems.)
- Added NEW badges on recently added entries across all benchmarks. (To surface fresh data at a glance without sorting by date.)
- Added empty state message when active filters return zero results. (To give clear feedback instead of a blank table.)
- Added live result count in the header bar that updates as filters change. (To give users an immediate sense of how many results match their current filter combination.)
- Added `[INDEX]` nav link to the site header. (To make the new page discoverable from every other page.)
- Added footer legend explaining SRC, OSS, and score sort behavior. (To make the table self-documenting for first-time visitors.)
18 changes: 18 additions & 0 deletions docs/2026-03-13-registry-updates.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Registry Updates - 2026-03-13

- Updated the registry page title, `h1`, and intro copy. (To align the page with the AI agent benchmark registry positioning and improve search relevance.)
- Added URL-backed category pages like `/registry/web-navigation`, `/registry/coding`, and related routes. (To make each category indexable, shareable, and SEO-friendly instead of relying only on client-side filters.)
- Added category-specific explanatory sections for registry categories. (To add crawlable context and help category pages rank for mid-tail queries.)
- Added a "What is an AI agent benchmark?" explainer section below the registry. (To capture top-of-funnel informational search intent and explain the page to new visitors.)
- Broadened registry copy to include `eval`, `evaluation`, `suite`, `framework`, and `leaderboard` terminology. (To match how users actually search, not just the word "benchmark".)
- Added internal links from FAQ and category copy to category routes and benchmark anchors. (To strengthen internal linking, help crawlers discover pages, and improve navigation.)
- Added a "How to read this registry" guidance strip. (To clarify how to compare benchmarks and make the page feel more editorial and trustworthy.)
- Added breadcrumbs for category routes. (To improve hierarchy, UX, and crawl understanding on deeper registry pages.)
- Added real canonical tags and page-specific meta descriptions. (To improve SEO hygiene and avoid weak default metadata across pages.)
- Generated benchmark detail pages from `benchmarks.ts` under `/registry/benchmarks/...`. (To create long-tail landing pages for individual benchmarks without manual page maintenance.)
- Improved benchmark cards with clearer taxonomy labels, clearer stats labels, benchmark-name detail links, and better button placement. (To make the registry easier to scan and cleaner visually.)
- Replaced generic `[PAPER]` / `[GITHUB]` labels with descriptive link text. (To make source links more meaningful for users and search engines.)
- Fixed spacing and underline issues in inline linked copy. (To clean up visual polish and avoid awkward underlined whitespace.)
- Replaced the footer `WEBVOYAGER` link with `STEEL.DEV`. (To point the footer CTA at the company site instead of an unrelated external repo.)
- Added `benchmarkDomainLabels`, `categorySlugs`, `getCategoryPath`, `getBenchmarkPath`, and `getBenchmarkAnchorId` exports to `benchmarks.ts`. (To support URL generation, anchor linking, and human-readable labels across registry pages and the benchmark index without duplicating logic.)
- Expanded registry to 40+ benchmarks across 7 categories including new entries: WebChoreArena, BrowserGym, AssistantBench, WorkArena, WebShop, Vending-Bench, CharacterEval, FORTRESS, MultiAgentBench, AgentHarm, LiveBench, SimpleQA, AgentBoard, OdysseyBench, and AppWorld. (To give the registry comprehensive coverage beyond the most well-known benchmarks.)
Loading
Loading