feat: make Legal Agent Bench agent harness and sandbox-backends configurable

[rystewart-nvidia]

Summary

This PR makes the Legal Agent Bench (LAB) agent harness configurable. Currently, LAB can only be run using the Harbor harness.

This adds three additional variants (in addition to Harbor) that enables running the same 1,749 tasks, skills, verifier, and reward logic with Gym's Hermes, Claude Code, and Codex harnesses:

• legal_agent_bench/config_hermes
• legal_agent_bench/config_claude_code
• legal_agent_bench/config_codex

Builds on PRs #1976 and #2075. Related to issue #1974.

Design

• Adds a LAB-owned configurable agent runner.
• Provisions pinned harness dependencies automatically.
• Routes every harness through the configured Gym policy model endpoint; no vendor subscription or CLI login is required.
• Keeps source documents and skills read-only while providing isolated writable output and scratch directories.
• Runs the agent and verifier in separate Docker containers so rubric files and judge credentials are unavailable during the agent phase.
• Uses an agent-neutral task index and stamps the selected agent during benchmark collation.
• Persists transcripts, output files, verifier reports, and failure diagnostics under harness- and model-specific result directories.
• Keeps Harbor as the default benchmark configuration.

Validation

• 82 scoped benchmark, resource-server, and configurable-runner tests
• Scoped pre-commit checks
• One-task Hermes Docker smoke test
• One-task Claude Code Docker smoke test
• One-task Codex Docker smoke test
• All three configurable smoke tests completed without model, sandbox, judge, or verifier failures
• All three smoke tests reported token usage and persisted LAB artifacts

Harness validation

Ran three LAB tasks for each harness using
nemotron-3-ultra-nvfp4 as policy and judge model.

Harness	Completed	Mean full_task reward	Mean criteria pass rate	Judge errors	Infrastructure/verifier failures
Harbor	3/3	0.0	89.07%	0	0
Hermes	3/3	0.0	89.74%	0	0
Claude Code	3/3	0.0	85.71%	0	0
Codex	3/3	0.0	75.67%	0	0

All 12 rollouts produced nonzero token usage, completed output files, agent
trajectories, and verifier artifacts. The 0.0full_task rewards are expected as a full task pass requires 100% criteria pass rate

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[roclark]

This looks good to me, thanks for opening @rystewart-nvidia! Do we have any quick validations on running the different harnesses against any model for verification? Otherwise, I think these changes are good.

[ritaneves]

Hi @rystewart-nvidia, what are your plans to move the PR from draft, so we can assign reviewers?

[rystewart-nvidia]

@roclark yes, I've run validations across all 4, will add some example results to the PR description shortly

@ritaneves moved to ready for review

[Glorf]

Very elegant implementation, I left a couple of comments, mostly related to possible race-conditions.
Some additional recommendations:

• testing on other sandbox env than docker to check stability, eg. opensandbox, ECS
• I'm not sure if harbor should be the default - usually we aim at one of the native configs for this; but I don't have enough knowledge about this benchmark to fully recommend it, if you feel harbor should be default then please keep it

[Glorf]

Gym mounts /logs/agent writable, so the agent can make stdout.log a symlink to another host file. Gym later follows that link when writing the log and may overwrite the target as the host user. Please keep host-written files outside agent mounts, write them only after sandbox teardown, and add a symlink regression test.

[rystewart-nvidia]

I updated it to remove writable host mounts and to write the host-side artifacts after sandbox teardown. Also added the requested symlink regression test. Addressed as part of this commit - 5942214#diff-8f3dbaaa67a8f1bae7d997d01053d82cb9a87a594832a35b17a3d38766af14cb

[Glorf]

same as above

[rystewart-nvidia]

Addressed in commit linked in comment above

[Glorf]

Runtime provisioning mounts the entire repository into a networked container that runs pip and npm installers. That mount can include gitignored env.yaml with judge credentials. Please stage and mount only the files required for installation, and test that credential-bearing paths are unavailable.

[rystewart-nvidia]

Addressed in this commit - 168bf83 - now mounts only the files needed for installation, and added tests to assert that credential files are not mounted

[Glorf]

This check uses trajectory content to infer success, but the selected harness can return partial output while dropping its failure state. The failed run can then reach the verifier and be scored. Please propagate exit, timeout, and adapter errors separately, set mask_sample=true, skip verification, and test partial output followed by failure for each harness.

[rystewart-nvidia]

Handled here - 486c6ea - exit, timeout, and adapter errors are propogated separately, mask_sample is set and verification is skipped on agent failures, and I added tests for each harness to confirm expected behavior after partial output + failure

[rystewart-nvidia]

Added similar failure handling for Harbor here - c687a5d

[Glorf]

The lock protects only one server process, but all processes share .deps/<harness>. Two evaluations can delete or rewrite the same runtime while another is building or using it.

[rystewart-nvidia]

Addressed in 168bf83 - runtime deps now built under os-level cross process lock and published in immutable temporary dir unique to each build

[Glorf]

The README recommends raising --concurrency, but every configurable agent keeps its server-side concurrency at 1, so the requests remain serial. Please document the matching server override or make both settings use one limit, and verify that two requests can overlap.

[rystewart-nvidia]

Documentation change here - 257688c. Also included test in this commit - 5942214 - to assert that concurrency requests actually result in overlapping runs

[Glorf]

The agent runs as container root, so on native Linux it can leave root-owned files in the host-mounted output directory. Please run the agent with the host UID/GID or repair ownership before teardown, and add a Linux ownership test.

[rystewart-nvidia]

Addressed in 5942214 - agent now runs as nobody, files are pulled from the sandbox through the sandbox api, and added tests to verify that files are owned by the UID/GID that invoked the process.

[Glorf]

Two blocking concerns before merge:

1. Operational failures need _ng_failure_class, not just mask_sample

mask_sample is a training hint, not a routing signal. Without _ng_failure_class, connection errors, sandbox crashes, and verifier failures land in the main rollout JSONL as completed zero-reward rows — they don't route to the failure sidecar and won't retry.

• Set _ng_failure_class for operational failures (connection, sandbox, verifier)
• Set _ng_failure_terminal=true for deterministic task/config errors that should never retry
• Max-turn and context-limit hits should not be masked — they are valid incomplete outcomes that should still be verified and scored

2. Default benchmark baseline changed without validation data

config.yaml now points the unqualified legal_agent_bench at the native loop, but the PR only includes 3-task smoke results per harness. Please either:

• Keep Harbor as the default, or
• Include full-scale native numbers that justify the switch

[rystewart-nvidia]

@Glorf Addressed _ng_failure_class tagging and sidecar routing, and unmasked the max-turn and context limit stoppages so that they will be scored.

The added native harness is directly aligned with Harvey official benchmark repo, so that's the sensible default. I've attached aggregate metrics from a full run across 1,749 tasks using Nemotron Ultra as policy and GPT 5.5 as judge. Full-task pass rate and criteria pass rate are in line with published metrics for Nemotron Ultra on Artificial Analysis and Vals.

Full-run validation results (Nemotron 3 Ultra):
Full-task pass rate: 35/1,749 = 2.0011%
Mean criteria pass rate: 81.5983%
Median criteria pass rate: 84.7059%
Judge/verifier errors: 0
Masked or operational-failure rows: 0
Note: 8 tasks still hit 64k max output cap, but are still scored and included in pass rates.

Also added some README updates to clarify recommendations on max output token and timeout settings, as there are no official/canonical values.

legal_agent_bench_native_nemotron_3_ultra_gpt_5_5_medium_judge_full_v4_aggregate_metrics.json

[cmunley1]

/claude review

[claude]

SHIP WITH CARE — no BLOCKER found. Large, thoroughly-tested addition (native LAB loop + configurable Hermes/Claude Code/Codex/Harbor harnesses, sandbox isolation, failure routing). I read the full diff and verified the failure-classification and scoring paths against the head files.

What I checked and found solid:

• Failure masking / reward=0 routing (harbor_agent, legal_agent_bench_agent): the priority order (model_connection_failed → sandbox_failed → verifier_failed → agent_timed_out → agent_failed) and the context-limit override (context exhaustion is a scoreable incomplete outcome, not masked) are correct and covered by test_harbor_context_limit_is_verified_and_scored and the per-harness incomplete-outcome test. mask_sample correctly drives training exclusion while _ng_failure_class/_ng_failure_terminal route retryability. No silent corruption of scored zeros.
• Verifier metric validation (_normalized_reward_data): rewards/pass-rate bounded to finite [0,1], counts to non-negative ints, bool rejected — bad verifier output can't poison the reward before it's flagged.
• Async correctness: model calls go through server_client.post (global aiohttp); ray.get is wrapped in asyncio.to_thread; native loop uses aiohttp for transport classification. No httpx, no bare ray.get in async, no missing awaits spotted.
• Sandbox isolation: agent/verifier phases use separate containers; judge creds staged only at verifier exec; untrusted output archive rejects links/devices/traversal (filter="data" + explicit member validation). Well tested.
• Legacy index migration (_migrate_legacy_agent_index) and _replace_directory flock serialization are atomic and have concurrency tests.
• ecs_fargate disk fix: disk_gib > 20 guard matches Fargate's implicit-20GiB / explicit-21–200 constraint; tested.

Two low-severity notes left inline (scoring .xml/.rels skip scope; verifier_result is None contract). Neither blocks merge.

Operability caveat for reviewers: CLI harness pins (claude_code_version 2.1.211, codex_version 0.144.4) and Node 22.15.0 are fetched from nodejs.org/npm at first-rollout provisioning — first run is network-dependent and slow, as documented.

[claude]

RISK — scoring change: adding .xml/.rels to SKIP_EXTENSIONS globally excludes those files from _load_all_output, which feeds the judge for every criterion that specifies no deliverables.

WHAT BREAKS: for a deliverable-less criterion whose intended output is legitimately an .xml file, that content is now silently dropped from the judge prompt, lowering the score. Deliverable-based criteria are unaffected (_match_deliverables/direct reads bypass SKIP_EXTENSIONS), so the blast radius is narrow: deliverable-less criteria + genuine .xml/.rels output across the 1,749-task set.

The intent (drop raw OOXML working files from an unzipped docx in the workdir — confirmed by test_full_output_ignores_raw_ooxml_working_files) is reasonable, and for docx/pptx/xlsx/pdf legal deliverables the collision is unlikely. FIX: confirm no LAB task's deliverable-less criterion expects .xml/.rels full-output content; if any do, scope the skip to the workdir/ OOXML tree rather than by extension globally.

[claude]

NOTE — verifier_failed fires when verifier_result is None and the agent didn't fail and no context-limit. But a genuine zero-reward run where the verifier legitimately produced no result object (vs. produced reward=0) is indistinguishable here. Verified against the test matrix this matches intended semantics (a missing verifier_result with a started/completed agent = infra failure → mask). Flagging only so it's a conscious contract: any future task type whose verifier can legitimately emit no verifier_result would be masked as verifier_failed and excluded from scoring. No change needed if that can't happen for LAB.

[rystewart-nvidia]

Validation run results with criteria pass rates for each combination of harness and sandbox-provider across five tasks

Provider	Native	Hermes	Claude Code	Codex
OpenSandbox	88.03%	91.74%	81.50%	83.27%
Apptainer	90.99%	91.15%	86.36%	67.99%
Enroot	88.41%	87.63%	86.31%	82.70%
OpenShell	85.78%	89.95%	83.33%	72.98%