AGENTS.md

Repository Guidelines

Project Structure & Module Organization

BenchKit is a Python 3.11 package using a src/ layout. src/benchkit/cli.py parses flags and either launches the TUI or runs headless. engine.py owns benchmark execution (jobs, slices, pause/skip/stop controls, events) and is shared by both front-ends; client.py talks to inference servers, executor.py sandboxes generated code, report.py writes artifacts, and runner.py renders headless progress. The terminal UI lives in src/benchkit/tui/ (app.py, app.tcss, theme.py, widgets.py, screens/), and demo.py provides the offline client behind --demo. Benchmark implementations belong in src/benchkit/benchmarks/, and bundled benchmark data lives in src/benchkit/datasets/ as JSONL files. Runtime outputs are written to results/<timestamp>/ and should not be committed.

Build, Test, and Development Commands

• uv sync: install and lock project dependencies from pyproject.toml and uv.lock.
• uv run benchkit: launch the terminal app against the configured host.
• uv run benchkit --demo: drive the whole UI offline with fake models - the fastest way to check TUI changes.
• OLLAMA_HOST=http://localhost:11434 uv run benchkit: run against an explicit local or remote Ollama endpoint.
• uv run benchkit --headless --models MODEL --benchmarks sanity:5 --verbose: scripted run that prints per-task prompts and responses.
• uv run benchkit --list: print the benchmark registry with task counts.
• uv sync --extra browser && uv run playwright install chromium: install the headless browser used by the treejs-arena suite.
• uv run pre-commit install: install the hooks (ruff on commit, pytest on push).
• uv run ruff check . / uv run ruff format .: lint and format.
• uv run pre-commit run -a: run every hook against the whole tree.

The package metadata and CLI entrypoint are defined in pyproject.toml; uv build produces the sdist and wheel. Two workflows run on every pull request: .github/workflows/lint.yml (ruff check and format) and .github/workflows/tests.yml (the test suite on Python 3.11-3.13, an offline --demo smoke run and the package build).

Coding Style & Naming Conventions

Ruff owns formatting and linting; its configuration lives in pyproject.toml (88 columns, py311 target). Run uv run ruff format . rather than hand-wrapping, and fence hand-aligned data tables with # fmt: off / # fmt: on when the layout matters. Use 4-space indentation, type hints, and short module docstrings consistent with the existing codebase. Prefer snake_case for modules and functions, PascalCase for classes, and lowercase benchmark registry keys such as sanity or humaneval. Keep new benchmark code narrow in scope: implement load_tasks(), build_prompt(), and evaluate(), store dataset payloads as JSONL, and register new classes in src/benchkit/benchmarks/__init__.py.

Testing Guidelines

Streaming, parser, and loop-detection behavior is covered by tests/; run it with uv run pytest. Everything else depends on manual validation. Before opening a PR, run uv run benchkit, confirm model discovery and benchmark execution, and inspect generated results.json, results.csv, and results.md for correctness. For scoring or parser changes, validate with a small benchmark slice before running larger suites. TUI changes can be driven without a terminal via Textual's pilot (async with BenchKitApp(demo=True).run_test() as pilot), which also captures SVG screenshots with app.save_screenshot().

Commit & Pull Request Guidelines

See CONTRIBUTING.md for the tooling walkthrough. Recent commits use short, imperative subjects such as Add per-task details and prompt/response to report and Format code and improve CLI selection. Follow that pattern and keep each commit focused on one change. PRs should describe user-visible behavior, list touched benchmark or dataset files, include the command used for validation, and attach a terminal or report snippet when CLI output changes.

Configuration & Data

Copy .env.example to .env and set OLLAMA_HOST before local runs. Do not commit .env, local caches, or generated results/ artifacts. Treat bundled dataset files as source data: update benchmark logic and dataset contents together when schema or evaluation behavior changes.

Product & Feature Map

Keep the public README short and human-oriented. Put implementation guidance, edge cases, and agent-facing operational detail here or next to the relevant code. Treat the CLI's --help, .env.example, and the benchmark registry as the canonical, version-matched references rather than duplicating large option tables in the README.

User Flows

• The TUI moves through Connect, Setup, Run, and Results screens. It supports model and benchmark filters, per-benchmark slices, live task inspection, pause, skip, stop, sortable results, and report drill-down.
• --demo exercises the TUI offline, including healthy and deliberately looping traces. Demo mode does not support the Pi harness.
• --headless uses the same engine and report pipeline as the TUI. --verbose prints prompts, available reasoning traces, and responses.
• benchkit history serves completed benchmark and performance reports from one or more results directories on localhost. It also serves the files inside those run directories, so a run's gallery and HTML report open from the dashboard. Artifact URLs name their results root by index and are resolved and checked to stay inside it; pages the model wrote (pages/) are served with a sandbox CSP so they cannot read anything else the server exposes.
• benchkit perf MODEL profiles prompt processing, generation speed, time to first token, wall time, and client overhead across configurable contexts.

Generation, Loops, and Retries

• Generation streams have no BenchKit token cap. BENCHKIT_TIMEOUT is the hard per-task deadline; partial traces are retained on timeout, loop kill, stop, and other recoverable terminal states.
• Reasoning comes from Ollama thinking, OpenAI-compatible reasoning_content, or inline <think> blocks. Providers that hide it must be reported as NO TRACE, not as producing no reasoning.
• Loop killing is controlled by BENCHKIT_LOOP_KILL, BENCHKIT_LOOP_KILL_PERCENT, and BENCHKIT_LOOP_KILL_SECONDS. Detection remains visible when killing is disabled. Only a confirmed, continuously growing suffix cycle is actionable; global repetition and code similarity are advisory.
• Transient gateway, rate-limit, DNS, connection, and dropped-stream failures use configurable exponential backoff. Never replay a generation after it has emitted tokens. Client errors are not retried. See .env.example and client.py for the current knobs and exact policy.

Harnesses and Perturbations

• --harness direct, pi, and both compare raw generation with the stock Pi coding agent. Paired scoring uses only items valid on both sides.
• Pi requires Docker. Each task gets a fresh persistent /workspace in an isolated container plus a restricted inference proxy. Do not add host mounts, the Docker socket, direct network egress, hidden answers, or hidden tests to the agent environment.
• --repair-attempts 1 gives an incorrect answer one sanitized verifier message and one full replacement attempt. Feedback must never expose the expected answer or hidden test bodies.
• aider-polyglot is Pi-only and uses pinned task content and toolchains. It measures the stock Pi protocol on Aider tasks, not Aider's edit formats.
• --perturbation choice-order runs supported MCQ tasks clean and with a deterministic permutation whose correct option moves. Perturbed jobs are paired with the baseline and excluded from the overall model score.

Creative Rendering (treejs-arena)

• treejs-arena has no ground truth. Frozen prompts ask for one self-contained HTML file each; the file is opened in headless Chromium and the automatic score is binary: no uncaught exception, and a sized canvas that acquired a drawing context. Console output and blocked hosts are recorded and fed back for repair but do not decide the verdict - a scene that draws has rendered even if a library logged a warning. The screenshots are the real output and are meant for human comparison.
• The prompt set is versioned (PROMPT_SET_VERSION). Never edit a shipped prompt in place; add a version so old screenshots stay comparable.
• Generated pages are untrusted. Requests are aborted unless the host is on the module-CDN allowlist (BENCHKIT_RENDER_ALLOWED_HOSTS, or nothing at all with BENCHKIT_RENDER_OFFLINE), downloads and service workers are blocked, and navigation, settle and capture each run under an explicit deadline. Do not widen the allowlist to make a scene pass, and do not add host mounts or file access to the browser context.
• --repair-attempts feeds the captured console and network diagnostics back as ordinary verifier feedback and re-renders the replacement file.
• Headless machines without a browser, without a working WebGL stack (checked once per process by browser.probe_environment), or without a route to an allowlisted module CDN are ordinary places to run BenchKit. The render check is skipped there and the task keeps its credit: render_status is skipped with a skip_reason, never a failure and never a harness error. Only faults inside the page the model wrote are scored failures. Skips are excluded from the gallery's render rate so they cannot inflate it, and benchkit render-check reports both preconditions directly. Render rates stay out of the overall average, like RULER.
• Every scored task keeps an artifact: the extracted page, or the raw answer when no HTML could be found. Tasks that never reached the browser (timeout, loop kill, output limit) still appear in the gallery with that status, so the denominator is the tasks attempted rather than the ones that got far enough to render.
• Screenshots and generated pages are staged under results/.artifacts/ during the run and collected into screenshots/ and pages/ inside the run directory at save time, with the paths in results.json rewritten relative to the report.
• The gallery is its own page, arena.html (template templates/arena.html), written next to the assets it links to whenever a run has render-scored rows. It carries the pass rate, per-model render rates, and large previews, so it is deliberately not standalone. results.html keeps the numbers and links to the gallery rather than embedding images.
• A preview runs the generated page itself, in an iframe sandboxed to allow-scripts allow-pointer-lock - an opaque origin, because the page is the model's own code - so a scene the benchmark machine could not render still plays in whatever browser opens the gallery. Frames load and unload with an IntersectionObserver and are capped, because browsers keep only a handful of WebGL contexts alive; the captured screenshot sits underneath as the poster and the record of what the run actually saw. Open ↗ opens the page in a full tab, and the header toggle turns live previews off.

Minecraft Builds (mc-arena)

• mc-arena has no ground truth either, but "it rendered" is deliberately not the score: a block list renders whether it is a watchtower or three blocks of dirt. Frozen prompts ask for one self-contained PEP 723 Python script that prints a JSON array of {x, y, z, block} into a fixed 32x32x32 volume with its origin at (0, 0, 0). The prompt shows a full block id, because without one smaller models answer oak plank and every block fails validation for the wrong reason.
• The script is untrusted and runs through sandbox.run_python_script: a throwaway container with no network, no capabilities, a read-only rootfs and memory, process and time limits. Do not give it network, host mounts, or the answer to anything. A SandboxError is a harness error, not a wrong answer.
• Scoring is deterministic and never uses a judge. Passing means the block list is usable: the script ran, the ids all exist, everything stayed in the volume, the entries were well formed and something was built. Quality is reported rather than scored - block count, palette size and evenness, the dominant block's share, floating (unsupported) fraction, duplicate positions, bounding box and fill. Keep it that way: these numbers are the signal, and a pass rate on its own would say almost nothing.
• Repairs come from the engine's --repair-attempts, fed by the verifier feedback (a traceback, a parse failure, or the offending ids). mc-arena is excluded from the overall average, like RULER and treejs-arena.
• Block ids are validated against datasets/mc_blocks_1_20_1.jsonl, the same Minecraft version the renderer draws, so a build cannot pass with an id the renderer would silently drop. Update both together.
• Rendering is prismarine-viewer, pinned and committed under mc_viewer/; see mc_viewer/build/README.md. Each build is photographed from three fixed cameras (isometric, side, top-down) at a fixed distance, lens and size, and the rig lives in mc_viewer/build/entry.js. Changing the rig or the prompt set makes old screenshots incomparable - version it, do not edit in place.
• The viewer is served over a loopback HTTP server because Chromium refuses to start a web worker from a file:// page. The page is BenchKit's own code; the model contributes block coordinates, never markup or script.
• A machine with no browser or no WebGL is an ordinary place to run mc-arena. The render is skipped, render_status is skipped with a skip_reason, and the build still scores from its block list - the picture is an illustration, never the verdict.
• The gallery can also explore a build in 3D: viewer.html?interactive=1&build= drops the three-camera rig for one full-window canvas with orbit, pan and zoom. The card loads it in place of a page preview, with the screenshot as the poster underneath. Unlike a treejs scene this frame is BenchKit's own code - the model contributed block coordinates, not markup - so it gets allow-scripts allow-same-origin; prismarine-viewer meshes in a web worker and an opaque origin cannot start one.
• Interactive viewing only works over HTTP, for the same reason the renderer runs its own loopback server: browsers refuse to start a worker from a file:// page. benchkit history serves the viewer straight from the installed package at /viewer/, so it is never copied into a run directory. A gallery opened straight off disk keeps the screenshots and loses only the 3D - never the scores, which never depended on it.

Concurrency and Metrics

• Request concurrency is detected from server slot endpoints or explicit metadata and is bounded by task count. Model jobs remain sequential; tasks within the active job may run concurrently.
• Pause waits for active requests to drain before withholding new work. Skip, stop, and interrupt cancel in-flight requests.
• Reports distinguish aggregate throughput, per-stream throughput, effective concurrency, and throughput coverage. Preserve raw timing/token denominators and the legacy tok_s compatibility alias when changing report schemas.
• Harness errors are excluded from scores and paired comparisons. Ordinary incorrect answers, timeouts, length limits, and loop kills remain scored failures when applicable.

Reports and Benchmarks

• Every run writes results.json, results.csv, results.md, and a standalone results.html. Performance runs write matching perf.* artifacts. Keep old reports readable when adding fields; missing historical fields mean “not captured,” not zero.
• Benchmark metadata, counts, perturbation support, and descriptions are canonical in src/benchkit/benchmarks/__init__.py; inspect it or run uv run benchkit --list instead of maintaining a second table.
• RULER is generated deterministically at runtime and reported by context bucket. It is excluded from overall-score averaging so its degradation curve stays visible.
• EvalPlus datasets may be downloaded and cached on first use. Generated code is untrusted; preserve evaluator guards and process isolation.