Skip to content

COMPARISON.md

Latest commit

 

History

History
77 lines (65 loc) · 4.25 KB

COMPARISON.md

File metadata and controls

77 lines (65 loc) · 4.25 KB
Raw

How cap-evolve compares

Positioning. Most agent-optimization tools tune prompts against a metric. cap-evolve optimizes prompts and executable tools, MCP surfaces, and whole skill packages against your eval, keeps the evaluation honest in code (sealed test + val-only significance gate), versions every iteration in git, and stays host- and agent-agnostic. This page defines the comparison criteria and places cap-evolve next to adjacent work — including external results as context, not a leaderboard.

Feature comparison

Each criterion is defined below the table. ✅ = first-class; ➖ = partial / possible with effort; ❌ = not a goal.

Criterion cap-evolve DSPy GEPA promptfoo
Optimizes prompts ✅ ✅ ✅ ❌ (eval only)
Optimizes tools/MCP and skill packages ✅ ➖ ➖ ❌
Sealed test + significance gate enforced in code ✅ ➖ ➖ ➖
Host- & agent-agnostic (no framework lock-in) ✅ ❌ ❌ ➖
Onboard a benchmark from a single prompt ✅ ❌ ❌ ➖
Git-versioned iterations + optimizer memory ✅ ❌ ❌ ❌
Live cost-aware dashboard ✅ ❌ ❌ ➖
Zero runtime dependencies ✅ ❌ ❌ ❌

Criterion definitions

  • Optimizes prompts — improves system-prompt / policy text against a metric.
  • Optimizes tools/MCP and skill packages — edits executable tool code (add / wrap / swap tools), MCP tool surfaces, and Agent Skill packages (SKILL.md + references + scripts), not just prose.
  • Sealed test + significance gate in code — the held-out split is scored exactly once and acceptance is a paired val-only significance gate (Δ > k·SE); both live in the core, not in editable docs. See HONEST_EVAL.md.
  • Host- & agent-agnostic — the optimizer is any coding-agent CLI resolved by one registry row; no framework lock-in.
  • Onboard from a single prompt — one intake brief installs the benchmark, wires the adapter, and passes cap-evolve check before any budget is spent.
  • Git-versioned iterations + memory — every candidate is a commit; rejected approaches are remembered and never re-proposed.
  • Live cost-aware dashboard — per-iteration optimizer & runner cost + time, lineage, diffs, and a tasks × iterations heatmap.
  • Zero runtime dependencies — the core is pure Python stdlib.

Primary sources: DSPy, GEPA (arXiv:2507.19457), promptfoo. Roadmap positioning: ROADMAP.md.

External context — tool-optimization results (NOT apples-to-apples)

These numbers come from other papers on different benchmark versions, models, task splits, simulators, trial protocols, metrics, and budgets. They are contextual evidence, not a controlled comparison. We deliberately avoid "beats" / "state-of-the-art" claims.

Work Benchmark Result Relative
EvoTool (arXiv:2603.04900) original τ-Bench airline, GPT-4.1 ReAct 35.9 → 39.1 (+3.2) ~+8.9%
EvoTool original τ-Bench airline, Qwen3-8B ReAct 14.4 → 15.7 (+1.3) ~+9.0%
Evolutionary Context Search τ²-Bench reported +23.3% +23.3%
cap-evolve (this repo) τ²-Bench airline, held-out 30(=val)/20 sealed test 30.0 → 47.5 (+17.5 pp) +58.3%

Notes and honest caveats:

  • EvoTool evaluates the original τ-Bench (Yao et al., 2024), while cap-evolve and Evolutionary Context Search use τ²-Bench — a different, harder benchmark. The rows are therefore not directly comparable.
  • EvoTool figures are quoted from its Table 1 (verified against arXiv:2603.04900).
  • The Evolutionary Context Search +23.3% figure is referenced in cap-evolve issue #46; its primary-source citation is still to be confirmed — treat it as unverified until a reference is added here.
  • cap-evolve's +58.3% is a within-run relative improvement on its own held-out split (RESULTS.md); a different split, model, and simulator than the rows above.

Last reviewed: 2026-07-13. Update this date and re-verify the external rows whenever a number changes.