Tracked datasets for public smokes, case-class demos, catalog Agents, and one abbreviated popular-bench conversion.
examples/
├── agents/ # ageval.agent/1 (dsh / nooa / miniswe / terminus-2 / openhands; builtin ACP is --agent pi)
└── datasets/
├── minimal-demo/ # dataset official/minimal-demo — case-class fidelity
├── tau3-airline-5/ # dataset official/tau3-airline-5 — 5-task airline cut
└── terminus-2-min/ # dataset examples/terminus-2-min — tmux host loop
There is no product executor: mock. Offline lock uses the real kinds; a missing
credential fails preflight and does not start the run. Bind a real Agent with --agent or --profiles.
CLI path is always the dataset root (ageval.yaml):
uv run ageval lock examples/datasets/<dataset> --task <task_id>
uv run ageval run examples/datasets/<dataset> --task <task_id>
uv run ageval tasks examples/datasets/<dataset># Full dataset (omit --task); concurrency from CLI or dataset defaults
uv run ageval run examples/datasets/minimal-demo --max-concurrent-tasks 2
uv run ageval run examples/datasets/minimal-demo --task terminal-jsonl-aggDefault minimal-demo profiles use environment: docker. --probe is lock +
preflight only.
uv run ageval lock examples/datasets/minimal-demo --task terminal-jsonl-agg
uv run ageval lock examples/datasets/tau3-airline-5 --task airline-00 # conversion; tau2 pin for run
uv run ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg
uv run ageval tasks examples/datasets/minimal-demo
# Expected failures
uv run ageval lock examples/datasets/minimal-demo --task does-not-exist # exit ≠ 0| Task | Case class |
|---|---|
multiagent-env-min |
Multi-session + SQL tools |
tau2-dialog-min |
Dual-role dialog + tools |
terminal-jsonl-agg |
workspace file + clean eval |
NVIDIA OO Agents path: real LiteLLM
calls via profile model / base_url / api_key (env locator). Install never
rewrites dataset profiles — bind with a separate profiles file:
uv sync --extra nooa
uv run ageval plugin install plugins/nooa
# repo/.env: litellm_api_key (+ litellm_base_url) or set profile base_url
unset AGEVAL_OFFLINE_AGENT
uv run ageval run examples/datasets/minimal-demo --profiles examples/datasets/minimal-demo/profiles.nooa.yamlPackage agents under each task’s lib/agents.py are nooa.Agent subclasses
(generation methods). Invoke runs the in-environment worker through host.exec and
projects locators into that exec env. Docker bake installs nooa so the
environment Python can import it.
DeepSeek Harness path: official
JSON-RPC SDK (deepseek-harness-sdk), not ACP. Same minimal-demo run.py; bind
executor: dsh + extensions: [{plugin: dsh}] + model + locator
deepseek_api_key. Invoke runs the in-environment worker through host.exec. Docker
bake installs the wheels in the Attempt image — --extra dsh is for the local
kind's interpreter. executor: alone does not bake.
uv run ageval plugin install plugins/dsh
# repo/.env: deepseek_api_key (projected as DEEPSEEK_API_KEY)
unset AGEVAL_OFFLINE_AGENT
uv run ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg \
--profiles examples/datasets/minimal-demo/profiles.dsh.yamlThis task writes aggregates.json, so omit options.permission or use
workspace-write. read-only fences DSH file-tool writes only; bash can still
write on the bundled jsonrpc runtime. That is not ageval isolation.
The OpenHands Software Agent SDK runs inside the Attempt environment. The instruction is a file in that environment. Install never rewrites dataset profiles. The SDK is imported only inside the environment.
ageval plugin install plugins/openhands
# repo/.env: litellm_api_key and litellm_base_url
ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg \
--profiles examples/datasets/minimal-demo/profiles.openhands.yamlDocker bake pins openhands-sdk and openhands-tools. An environment that
does not already have that pin installs it at setup. A failed install stops
the run with openhands_sdk_install_failed.
uv run ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg --probe
# missing extras or keys: --probe reports ready: false / started: false
# live ACP stdio over ssh A is unsupportedPopular-bench port of tau2-bench
airline (τ³-bench) as one domain = one dataset. Dual-role dialog
(user + service via profiles.yaml → openai-http GLM Coding Plan) with package-local tools/DB
bridge and independent evaluator (tau2 ENV+COMMUNICATE). Default environment is local.
In-repo this is a five-task cut (airline-00 … airline-04). The upstream domain
has 50 tasks; that full suite is not checked in.
| Item | Notes |
|---|---|
| Upstream pin | tau2-bench @ v1.0.1 (fc0055dc…); paper 2506.07982 |
| Members | 5 tasks: airline-00 … airline-04 (upstream ids 0…4) |
| Layout | Dataset-level shared/lib + shared/assets; no per-task lib/ copies |
| Gold | Under each tasks/airline-NN/evaluation/ only — not under shared/ |
| Host deps | tau2==1.0.1 (see datasets/tau3-airline-5/requirements.txt) for run/eval |
| Evidence | Not a public smoke upgrade path; package / Hub publish ≠ real-benchmark-verified |
uv run ageval lock examples/datasets/tau3-airline-5 --task airline-00
uv run ageval tasks examples/datasets/tau3-airline-5
uv run python scripts/check_shared_lib_collisions.py examples/datasets/tau3-airline-5
# Five-task in-repo suite (needs agent credentials + tau2):
# uv run ageval run examples/datasets/tau3-airline-5Package-local detail: datasets/tau3-airline-5/README.md.
Regenerate the in-repo cut:
python examples/datasets/tau3-airline-5/scripts/generate_package.py --ids 0,1,2,3,4.
One Attempt, six child openai-http sessions, at most four active. The coordinator writes report.json. The evaluator scores that file. Not a benchmark score.
ageval lock examples/datasets/concurrent-research --task fanout
ageval run examples/datasets/concurrent-research --task fanoutNeeds glm_coding_api_key. Detail: datasets/concurrent-research/README.md.
One docker task. The model stays on the parent; the agent types into tmux inside the attempt environment and writes hello.txt. Install plugins/terminus-2 first. PASS is evaluator.py.
ageval plugin install plugins/terminus-2
ageval lock examples/datasets/terminus-2-min --task hello
ageval run examples/datasets/terminus-2-min --task helloDetail: datasets/terminus-2-min/README.md.
Custom overlay packages for executors that are not builtin catalog cards:
dsh-default, nooa-default, miniswe-default, terminus-2-default, openhands-default. Built-in Agent packages
(pi, opencode, codex, claude-code, grok-build, openai-http,
anthropic-http) ship with ageval — --agent pi needs no install.
model.id is the package default, not identity. --agent is mutually
exclusive with --profiles. Optional --model overrides this run:
uv run ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg --agent pi --model glm-4.7
uv run ageval agent install examples/agents/dsh-default
uv run ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg --agent local/dsh-default@0.1.0
ageval agent install examples/agents/openhands-default
ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg --agent local/openhands-default@0.1.0This monorepo keeps minimal-demo plus a five-task tau3-airline-5 cut under
examples/datasets/. Larger popular-bench ports (including the full 50-task airline
domain) stay out of examples/ and ship as Hub packages (publish + suite
upload), so clone size and CI paths stay bounded:
| Upstream | Hub package id (org my-lab) |
Notes |
|---|---|---|
| Terminal-Bench 2.0 | terminal-bench-2 / light terminal-bench-2-10 |
Docker + Harbor pytest-style verifier |
| MARBLE coding | marble-coding / light marble-coding-10 |
shared-container multi-agent coding |
Package presence, Hub publish, or a suite job on the board does not make the evaluation results more credible.
uv run ageval lock examples/datasets/minimal-demo --task terminal-jsonl-agg
uv run ageval run examples/datasets/minimal-demo --task terminal-jsonl-agg
uv run ageval lock examples/datasets/tau3-airline-5 --task airline-00