Add scripts/export_openeval.py: optional EvalPort ResultSet export

[adhabnr-ux]

Resolves #322.

What this adds

An optional, additive script (scripts/export_openeval.py, + a thin scripts/export_openeval.sh wrapper matching scripts/rescore.sh's shape) that converts a batch's rescore-summary.json (enriched with each run's own run-meta.json) into an EvalPort ResultSet — a small open interchange format for portable LLM/agent evaluation results.

Per the discussion on #322: @Perry2004 confirmed this is the results side (complementing clawbench-harbor-adapt/clawbench-edgebench-adapt, which import test-case definitions the other direction) and suggested it land as "a simple script inside script/" rather than a new adapters/-style package or a core dependency — this PR follows that shape exactly. There's a longer story behind why this took a couple of tries to land in the right place, laid out honestly below.

No new dependency

The script does not require evalport-sdk to run — the conversion itself is pure stdlib (argparse/json/pathlib). If evalport-sdk happens to be installed, the script uses it to validate the ResultSet it produces against the real EvalPort JSON Schema before writing it out (and prints a clear error if validation fails); if it isn't installed, the script says so and skips that step. pyproject.toml's dependency list is untouched either way.

Grounded in the real, current source — not just docs/scoring.md

docs/scoring.md describes an idealized/older shape (judge_match/final_pass merged into one per-run record). I read the actual current code instead:

• run-meta.json (make_run_meta() in src/clawbench/runner/run_support/metadata.py) carries test_case, instruction, model, harness, intercepted, result_category, failure_category, adjusted_eligible, duration_seconds — but not judge_match or final_pass.
• The per-run judge verdict (judge_llm.json for the default "lenient" rubric, judge.json for "strict" — JUDGE_FILE in src/clawbench/eval/rescore.py) carries match/reason (src/clawbench/runner/judge_llm.py's judge_request()).
• rescore-summary.json (aggregate_batch()) rolls a batch into n_total, n_intercepted, judge_model, rubrics, and a tasks[] list shaped {"task_id", "test_case", "intercepted", "match_<rubric>", "reason_<rubric>"}. tasks[] rows don't carry instruction/model/harness — only each run's own run-meta.json does, so the script walks the batch dir for those to enrich each Result.

Result.passed is intercepted AND judge_match is True, matching docs/scoring.md's final_pass rule. The interception and judge stages each become their own GraderResult (gr_interception, gr_judge_match) rather than being collapsed into one opaque score, so the two-stage pipeline is still visible in the exported result.

Testing

tests/test_export_openeval.py — 11 tests, all passing locally: direct unit tests of run_to_result()/to_openeval() (interception/judge grader shapes, the never-intercepted case, an inconclusive judge verdict, the task_id-fallback path, rejecting an unknown --rubric, and — importantly — that a stale match_<rubric> key on a task row that was never actually intercepted doesn't produce a fabricated judge grader), plus CLI smoke tests via subprocess (matching the pattern tests/test_harbor_adapter.py::test_harbor_adapter_cli_smoke already uses) covering a full run, a clean error when rescore-summary.json is missing, and --help. All validated against the real, installed evalport-sdk 1.3.1's openeval.validate.validate_result_set() — not a mock. Also clean under this repo's own ruff/pyright config.

CHANGELOG.md updated under [Unreleased] per CONTRIBUTING.md.

Honest background on where this landed

I built and tested this adapter's logic once already, then hit a real tooling wall: at the time, I had no way to fork this repo or open a branch directly against it, so — rather than sit on tested work — I shipped the logic as a standalone package in EvalPort's own repo (adapters/clawbench-openeval-adapter/) as an interim home, while saying plainly in that package's README that the intent was still to land it here once that was possible. That's now resolved, so this PR ports the same tested logic into the shape you actually asked for on #322 (scripts/, not a new package) rather than leaving the interim version as the final answer.

— Sahi, independent contributor (not affiliated with TIGER-AI-Lab)

🤖 Generated with Claude Code

https://claude.ai/code/session_01RtocdH3tKifGdkiCZxxV3F

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.