Skip to content
This repository was archived by the owner on Jul 6, 2026. It is now read-only.

Commit f97df0b

Browse files
docs: add minimal benchmarks doc
1 parent 7b93c45 commit f97df0b

1 file changed

Lines changed: 44 additions & 0 deletions

File tree

docs/benchmarks.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# Benchmarks
2+
3+
Each scenario gives Claude a broken codebase and asks it to fix it. We
4+
run the same task twice — once with Claude's native tools, once with 8v
5+
— and compare input tokens, output tokens, tool calls, and turns. Tests
6+
must pass in both arms.
7+
8+
## Results
9+
10+
| Scenario | Input tokens vs native |
11+
|---|---|
12+
| fix-failing-test (Rust) | −14% |
13+
| fix-go | −22% |
14+
| fix-python | −66% |
15+
| fix-typescript | −12% |
16+
17+
N=6 per condition. Tests pass 6/6 in every scenario, both arms.
18+
19+
## Run
20+
21+
```sh
22+
cargo test -p o8v --test demo_agent_benchmark -- --ignored --nocapture --test-threads=1
23+
```
24+
25+
Requires a working `claude` CLI authenticated locally and the `8v`
26+
binary on `PATH`. Each scenario takes a few minutes; the full suite
27+
takes about an hour.
28+
29+
## Scenarios
30+
31+
Defined in `o8v/tests/demo_agent_benchmark.rs`. Each pair of tests
32+
(`*_baseline` and `*_8v`) shares one fixture: same broken code, same
33+
task prompt, only the tool surface changes.
34+
35+
- `fix_test_baseline` / `fix_test_8v` — Rust, failing unit test
36+
- `fix_python_baseline` / `fix_python_8v` — Python, broken module
37+
- `fix_go_baseline` / `fix_go_8v` — Go, failing test
38+
- `fix_typescript_baseline` / `fix_typescript_8v` — TypeScript, type error
39+
- `diagnose_baseline` / `diagnose_8v` — read-only investigation
40+
41+
## Output
42+
43+
Per-run JSON lands in `~/.8v/bench/`. Aggregated CSVs are written next
44+
to the test run.

0 commit comments

Comments
 (0)