diff --git a/docs/evaluator/agent-eval/harbor-runner.mdx b/docs/evaluator/agent-eval/harbor-runner.mdx index e5d87d756c..dcf1e973f7 100644 --- a/docs/evaluator/agent-eval/harbor-runner.mdx +++ b/docs/evaluator/agent-eval/harbor-runner.mdx @@ -21,8 +21,17 @@ Unlike the quickstart, this runner is **not** zero-dependency — it shells out - **Python ≥ 3.12** - **Docker and docker daemon** installed and running -- **Harbor**, installed separately: `uv pip install "harbor>=0.16.1"`. Harbor is intentionally **not** - a dependency of `nemo-platform[nemo-evaluator-sdk]`, so the rest of the SDK stays lightweight. +- **Harbor**, installed from a NeMo Platform source checkout as shown below. Harbor is + imported lazily, so the base SDK remains usable on Python 3.11 without it; Harbor execution and + existing-result adaptation require Python 3.12 or newer. + +The nemo-evaluator-sdk is not published as a standalone PyPI package. Use a NeMo Platform source checkout +(see the repository's [setup guide](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/SETUP.md) +for toolchain prerequisites). From the repository root, install the Harbor extra: + +```bash +uv sync --frozen --package nemo-evaluator-sdk --extra harbor +``` The runner raises a clear error pointing at this install step if `harbor` is missing. @@ -107,7 +116,7 @@ scores each trial's named rewards with `HarborRewardMetric`. **`dataset_path` is input; `jobs_dir` is output.** The dataset is your read-only task suite. `jobs_dir` is a directory the runner writes into — Harbor's per-trial results land under `jobs_dir//`, -and that directory doubles as a re-run cache (see [below](#attempts-concurrency-and-caching)). +and that directory doubles as a re-run cache (see [below](#caching)). @@ -139,9 +148,13 @@ Swap `agent_name` (or `agent_import_path`) for your own agent to get a real scor Mapping order and alphabetical order never select the primary. 3. The primary output is required. On a scoreable trial, a finite numeric value is emitted unchanged; a missing or unusable primary emits `0.0` and a diagnostic instead of skipping the trial. -4. Other reward keys discovered for that task become optional secondary outputs. Finite numeric - values are emitted. Missing, Boolean, nonnumeric, NaN, or infinite values are omitted with a - diagnostic. A secondary discovered for one task does not become applicable to another task. +4. Other keys from that task's Harbor-valid results become optional secondaries: + - Finite numbers are emitted. + - Missing or Boolean values are omitted with a diagnostic; usable siblings are kept. + - A `null`, nonnumeric string, or object in the reward mapping fails Harbor's `TrialResult` + check, so the whole attempt is skipped and sibling rewards are not scored. Harbor writes + `NaN` and infinity as `null`, which hits this gate. + - A secondary reward key discovered for one task does not apply to another task. 5. `result.summary` aggregates each named output, `result.trials` holds each trial's status and evidence, and `result.persist()` writes the standard run bundle. @@ -156,14 +169,18 @@ and validates that file before writing `result.json`: - A JSON Boolean is coerced to `1.0` or `0.0`. - `NaN` and infinity pass Harbor's numeric validation, but serialize as `null` in `result.json`. -The SDK adapter reads `result.json` per key. It keeps finite numbers and accepts numeric strings in -imported or manually authored result files. It omits a Boolean, `null`, unparseable value, NaN, or -infinity with a diagnostic while retaining usable sibling rewards. +The SDK first validates the whole `result.json` with Harbor's `TrialResult`. A `null`, nonnumeric +string, or object in the reward mapping invalidates the attempt; its sibling rewards are not scored +or cached. For a Harbor-valid result, the SDK parses each reward per key. It emits finite numbers, +including numeric strings normalized by Harbor, and omits Boolean or non-finite values with a +diagnostic while retaining usable siblings. Emit finite JSON numbers from the verifier. Use `1` and `0` when a reward represents pass/fail; do not rely on Harbor's coercion of strings or Booleans. -### Sparse secondary example +### Sparse secondary and errored reward examples + +#### Missing secondary reward For two attempts on task A: @@ -172,24 +189,115 @@ a1: reward=1.0, format_ok=1.0 a2: reward=0.0, format_ok omitted ``` -| Consumer | Expected result | +**Resulting aggregates and coverage** + +| Result field | Expected value | |---|---| | `harbor_reward.reward` | `mean=0.5`, `count=2`, `nan_count=0` | | `harbor_reward.format_ok` | `mean=1.0`, `count=1`, `nan_count=1` | | `format_ok` coverage | `total=2`, `scored=1`, `missing=1`, `failed=0` | -| `harbor_reward.format_ok.pass@1` | `mean=1.0`; measured `n=1` | +| `harbor_reward.format_ok.pass@1` | `mean=1.0`, `count=1`, `nan_count=0`; computed from one measured attempt | | `harbor_reward.format_ok.pass@2` | unestimable: `count=0`, `nan_count=1`, `mean=None` | -Omission means unmeasured, not failure. It does not create a persisted null, NaN, or zero. See -[Reading Results](/documentation/evaluate-models/agent-eval/reading-results) for denominator, failed -trial, semantic-view, and persistence behavior. +Omission means unmeasured, not failure: + +- It does not create a null, NaN, or zero per-trial metric output. +- Coverage records the missing measurement. +- Derived aggregates can therefore be unestimable (`mean=None`). + +See [Reading Results](/documentation/evaluate-models/agent-eval/reading-results) for denominator, +failed-trial, semantic-view, and persistence behavior. + +#### Errored attempt with a valid primary reward + +For two attempts on task A: + +```text +a1: reward=1.0, no error +a2: reward=0.8, error=RuntimeError +``` + +**Resulting statuses and aggregate** + +| Result field | Expected value | +|---|---| +| SDK trial `a1` | `COMPLETED` | +| SDK trial `a2` | `PARTIAL`; remains scoreable | +| `harbor_reward.reward` | `mean=0.9`, `count=2`, `nan_count=0` | +| `error_trial_ids` | `{"RuntimeError": ["a2"]}` | + +The error changes `a2`'s status and error rollup; it does not discard or replace its finite reward. + +## Attempts and concurrency + +- **`n_attempts`** — desired attempts per task. On resume, Harbor runs only the missing attempts. +- **`n_concurrent_trials`** — maximum trials Harbor runs concurrently. + +## Retries + +- **`max_retries`** — maximum extra attempts per trial during the current Harbor run; defaults to `0`. +- **Error policy** — Harbor retries only allowed errors. Its + [default non-retryable errors](https://github.com/harbor-framework/harbor/blob/v0.20.0/src/harbor/models/job/config.py#L288-L301) + include `AgentTimeoutError`. +- **Repeated SDK calls** — `max_retries` never reopens cached errored trials. Cache behavior is + described below. -## Attempts, concurrency, and caching +## Caching -- **`n_attempts`** — trials Harbor runs per task; **`n_concurrent_trials`** — how many run at once. -- The job directory doubles as a **cache**. Pin a stable `job_name` and a completed job whose results - already cover every requested task is re-scored instead of re-run. The default timestamped `job_name` - always runs fresh; set `force_rerun=True` to delete an existing job dir first. +Caching activates only when `HarborRuntimeConfig.job_name` is pinned. Without it, every call creates a +fresh timestamped job directory. + +A cache hit requires both: + +- A usable matching stamp for the requested inputs. +- At least `n_attempts` Harbor-valid results for every requested task. + +On a repeated SDK call: + +- **Cache hit** — skips Harbor and Docker, then re-scores existing results. Harbor-valid errored results + count as completed, become `PARTIAL`, and are not rerun. +- **Incomplete matching cache** — preserves valid results, including errored results, then runs only + attempts with missing or invalid results. +- **Stale or unusable cache** — deletes the job directory and reruns every requested attempt. This + includes changed inputs, a missing or malformed stamp, and an unresolved requested task directory. +- **`force_rerun=True`** — deletes the entire pinned job directory, then reruns every requested attempt. +- **Selective cached-error rerun** — unsupported. The SDK cannot rerun cached attempts by error type. +- **Concurrent processes** — must not share a pinned `job_name`; neither the SDK nor Harbor locks the + job directory. + +### Cache identity + +The SDK stores the cache stamp in `jobs_dir//.nemo-eval-harbor-cache.json`: + +- **`version`** — stamp schema; must match exactly. +- **`options`** — SHA-256 of result-affecting `HarborRuntimeConfig` fields; must match exactly. +- **`agent`** — digest of `agent_dir` contents, or `""` when unset; must match exactly. +- **`tasks`** — every requested task digest must match; extra cached tasks are ignored. Cached A, B, C + can serve A, but cached A cannot serve A and B. +- **Scoring and scheduling settings** — `reward_key`, selected metrics, `quiet`, and + `n_concurrent_trials` do not invalidate Harbor execution results. +- **Installed agents** — when `agent_dir` is unset, the stamp covers the agent selection or import path, + not the installed package contents. Change the import path, use `agent_dir`, or force a rerun after + changing installed agent code. + +### Force a complete rerun + +```python +import asyncio +from pathlib import Path + +from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval + +config = HarborRuntimeConfig( + jobs_dir=Path("./harbor-jobs"), + job_name="my-suite", + agent_name="oracle", + force_rerun=True, +) +result = asyncio.run(run_harbor_eval(config, dataset_path=Path("path/to/my-suite"))) +``` + +- **Keep the previous job** — use a new `job_name`, or omit it to create a fresh timestamped directory. ## Shortcut: `run_harbor_eval` @@ -197,13 +305,16 @@ When a run is exactly "one Harbor suite, scored by its reward," `run_harbor_eval steps above — discover, run, score — into a single call: ```python +import asyncio from pathlib import Path from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval -result = await run_harbor_eval( - HarborRuntimeConfig(jobs_dir=Path("./harbor-jobs"), agent_name="oracle"), - dataset_path=Path("path/to/my-suite"), +result = asyncio.run( + run_harbor_eval( + HarborRuntimeConfig(jobs_dir=Path("./harbor-jobs"), agent_name="oracle"), + dataset_path=Path("path/to/my-suite"), + ) ) ``` diff --git a/packages/nemo_evaluator_sdk/examples/harbor/README.md b/packages/nemo_evaluator_sdk/examples/harbor/README.md index 2418e7fa1c..1aef62b3f7 100644 --- a/packages/nemo_evaluator_sdk/examples/harbor/README.md +++ b/packages/nemo_evaluator_sdk/examples/harbor/README.md @@ -8,6 +8,17 @@ Run a Harbor **local dataset directory**, example scores with Harbor's deterministic **oracle** agent, so it needs no model or API key — only the `harbor` extra installed and a working Docker daemon. +## Install + +The base SDK supports Python ≥ 3.11, while Harbor-backed execution and result +adaptation require Python ≥ 3.12. The SDK is not published as a standalone PyPI +package. Use a NeMo Platform source checkout; see [SETUP.md](../../../../SETUP.md) +for toolchain prerequisites. From the repository root, install the optional extra: + +```bash +uv sync --frozen --package nemo-evaluator-sdk --extra harbor +``` + ## Minimal plumbing The SDK owns the Harbor plumbing. Apart from imports, running a whole dataset is @@ -24,18 +35,8 @@ result = await run_harbor_eval(config, "hello_world_dataset") # loads tasks, ru `run_harbor_eval` discovers the tasks, builds and runs Harbor's `JobConfig`, and scores each task with `HarborRewardMetric` — the caller never imports `harbor` or -assembles a job. `harbor` is imported lazily inside the runtime, so importing the -SDK never requires it. - -## Install - -Harbor is imported lazily and is **not** in the SDK's locked dependencies (it -requires Python ≥ 3.12 while the workspace supports ≥ 3.11, like `nemo_fabric`). -Install it separately into the environment that runs the example: - -```bash -uv pip install "harbor>=0.16.1" -``` +assembles a job. `harbor` is imported lazily: the base SDK does not require it, +but Harbor execution and existing-result adaptation do. ## The dataset directory (how Harbor tasks are found) @@ -74,8 +75,10 @@ The runtime is [`harbor_runtime.py`](../../src/nemo_evaluator_sdk/agent_eval/run ### Re-running and caching In native mode the `job_dir` doubles as a cache: if every requested task already -has `n_attempts` completed (non-errored) results there, the Harbor run is skipped -and the results are re-adapted instead. This only engages when you **pin a stable +has `n_attempts` Harbor-valid results there, the Harbor run is skipped +and the results are re-adapted instead. Valid errored results count because Harbor +also treats them as completed attempts; their SDK trials remain `PARTIAL` and +scoreable. This only engages when you **pin a stable `job_name`** on the config — the default `job_name` is a timestamp, so each run writes a fresh dir and never hits the cache. Set `force_rerun=True` to delete the job dir and re-run unconditionally. diff --git a/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md b/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md index 08760d8355..09319279d8 100644 --- a/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md +++ b/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/README.md @@ -9,6 +9,24 @@ runner. LAB ships **raw** tasks (`tasks/**/task.json` + `documents/`); this exam — it downloads the pinned source and *generates* the Harbor suite itself, then runs and scores it with one `AgentEvaluator` call. +## Prerequisites, seams & caveats + +- **Not zero-dependency**: Python ≥ 3.12, Docker, and a NeMo Platform source checkout + (see [SETUP.md](../../../../SETUP.md) for toolchain prerequisites). The SDK is not + published as a standalone PyPI package. From the repository root, install Harbor with + `uv sync --frozen --package nemo-evaluator-sdk --extra harbor`. Harbor native runtime is early-access. +- **Reproducing LAB's official reference-agent number** additionally requires wiring **LAB's reference + agent** (as an `--agent-import-path` adapter) and + LAB's **exact** `rubric_criterion` judge prompt into `lab_verify.py`. Out of the box this generates a + *runnable, faithful-in-shape* suite; treat scores as comparable-in-method until you drop those in. +- **Agent-output seam**: `prepare_lab_suite.py --run-dir` sets where the verifier reads the agent's + deliverables (default `/logs/agent/artifacts/lab-run`, LAB's reference-agent location). Point it at + wherever your chosen Harbor agent writes. +- **`scores.json` schema**: `LabCriteriaMetric` reads `n_criteria`, `n_passed`, `all_pass`, + `judge_error_count`, `criteria_results[].verdict` — exactly what `lab_verify.py` writes. +- **Scale**: the SDK runs tasks with async concurrency locally (or a single-container platform job). + For the full 1,749-task sweep, prefer the governed platform job over a local run. + ## Files - [`prepare_lab_suite.py`](prepare_lab_suite.py) — self-contained: downloads + SHA-verifies the pinned @@ -69,21 +87,5 @@ lab_criteria.judge_error_count: mean=0.0 # treat > 0 as an infra failure, no view.legal_quality: mean=0.60 # MEAN(reward, criteria_pass_rate) ``` -## Prerequisites, seams & caveats - -- **Not zero-dependency**: Python ≥ 3.12, Docker, and `harbor` installed separately - (`uv pip install "harbor>=0.16.1"`). Harbor native runtime is early-access. -- **Reproducing LAB's official reference-agent number** additionally requires wiring **LAB's reference - agent** (as an `--agent-import-path` adapter) and - LAB's **exact** `rubric_criterion` judge prompt into `lab_verify.py`. Out of the box this generates a - *runnable, faithful-in-shape* suite; treat scores as comparable-in-method until you drop those in. -- **Agent-output seam**: `prepare_lab_suite.py --run-dir` sets where the verifier reads the agent's - deliverables (default `/logs/agent/artifacts/lab-run`, LAB's reference-agent location). Point it at - wherever your chosen Harbor agent writes. -- **`scores.json` schema**: `LabCriteriaMetric` reads `n_criteria`, `n_passed`, `all_pass`, - `judge_error_count`, `criteria_results[].verdict` — exactly what `lab_verify.py` writes. -- **Scale**: the SDK runs tasks with async concurrency locally (or a single-container platform job). - For the full 1,749-task sweep, prefer the governed platform job over a local run. - For the **task-driven, bring-your-own-agent** counterpart (native `AgentEvalTask`s + Fabric + a rubric *metric* instead of an in-container verifier), see [`../legal_agent_bench_fabric`](../legal_agent_bench_fabric). diff --git a/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py b/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py index 616fc4c1e8..ec2c293c29 100644 --- a/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py +++ b/packages/nemo_evaluator_sdk/examples/legal_agent_bench_harbor/run_legal_agent_bench.py @@ -25,8 +25,10 @@ Prerequisites: * Python >= 3.12 and a running Docker daemon. -* Harbor, installed separately: ``uv pip install "harbor>=0.16.1"`` (kept out of the - SDK's lock so importing the SDK stays lightweight). +* A NeMo Platform source checkout (see SETUP.md for toolchain prerequisites). + From the repository root, install Harbor with + ``uv sync --frozen --package nemo-evaluator-sdk --extra harbor``. The SDK is not + published as a standalone PyPI package. Harbor is imported lazily. * A prepared LAB Harbor suite on disk — a directory of task folders. Generate it with the bundled, self-contained ``prepare_lab_suite.py`` (pinned download + Harbor-task generation; see the example README). diff --git a/packages/nemo_evaluator_sdk/pyproject.toml b/packages/nemo_evaluator_sdk/pyproject.toml index 056f6e6d34..06c6026658 100644 --- a/packages/nemo_evaluator_sdk/pyproject.toml +++ b/packages/nemo_evaluator_sdk/pyproject.toml @@ -57,14 +57,12 @@ agent-runtimes = [ "openai-agents[docker]>=0.17.3,<0.18", ] engine = [] -# The native Harbor runtime (HarborAgentTaskRunner / run_harbor_eval) needs `harbor`, which requires -# Python >=3.12 while this workspace floor is >=3.11. The `python_version >= '3.12'` marker (matching -# harbor's own requires-python) scopes the dependency to the 3.12+ fork of the universal lock, so it -# installs on 3.12+ and is simply omitted on 3.11 — no floor bump for the rest of the workspace. -# `harbor` is imported lazily (only inside HarborAgentTaskRunner.run_tasks), so importing the SDK and -# resolving a HarborRunnerTarget stay valid on 3.11; only *executing* a Harbor job needs this extra. +# Native Harbor execution and result adaptation need `harbor`, which requires Python >=3.12 while +# the SDK floor remains >=3.11. The marker scopes the dependency to compatible interpreters. +# Harbor is imported lazily, so importing the SDK and constructing Harbor configuration remain +# valid on Python 3.11; invoking either execution or result adaptation requires this extra. harbor = [ - "harbor>=0.16.1; python_version >= '3.12'", + "harbor>=0.20,<0.21; python_version >= '3.12'", ] nemo-platform = [ "nemo-platform-sdk", diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py index 3df2a07ab5..8459f47d9e 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py @@ -433,7 +433,13 @@ class AgentEvalSummary(BaseModel): ], ) task_count: int = Field(default=0, description="Number of tasks represented in the run.") - trial_count: int = Field(default=0, description="Number of distinct trials scored.") + trial_count: int = Field( + default=0, + description=( + "Number of trial records in the run. Uses len(trials) when supplied, including duplicate " + "ids and trials without scores; otherwise counts distinct (task_id, trial_id) score pairs." + ), + ) score_count: int = Field(default=0, description="Total number of metric scores.") error_count: int = Field( default=0, @@ -523,7 +529,8 @@ def from_scores( :attr:`error_trial_ids` empty rather than raising -- the same silent-skip contract ``tasks`` already has for pass@k. It may legitimately be *wider* than ``scores`` (a caller re-aggregating a subset), so the rollup can name trial ids absent from - :attr:`task_metric_values`. + :attr:`task_metric_values`. :attr:`trial_count` uses ``len(trials)`` when supplied; otherwise + it counts distinct ``(task_id, trial_id)`` pairs among ``scores``. """ score_list = list(scores) task_list = list(tasks) if tasks is not None else None @@ -533,6 +540,10 @@ def from_scores( observations_by_output = _group_by(observations, _by_output) task_metric_values = _task_metric_values(observations, task_list) error_trial_ids = _error_trial_ids(trials) + if trials is None: + trial_count = len({(score.task_id, score.trial_id) for score in score_list}) + else: + trial_count = len(trials) return AgentEvalSummary( scores=_aggregate_scores( observations, @@ -544,7 +555,7 @@ def from_scores( task_metric_values=task_metric_values, error_trial_ids=error_trial_ids, task_count=len(task_list) if task_list is not None else len({score.task_id for score in score_list}), - trial_count=len({score.trial_id for score in score_list}), + trial_count=trial_count, score_count=len(score_list), error_count=sum(len(ids) for ids in error_trial_ids.values()), ) diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_runtime.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_runtime.py index a5b69a51a0..6554b30830 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_runtime.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_runtime.py @@ -25,8 +25,9 @@ * **Injected / offline** — pass a ``job_dir`` (and optionally a ``run_job`` callback) to adapt an already-completed job dir or to run a caller-built job. -Trial *adaptation* only ever reads Harbor's on-disk ``result.json`` files, so -that half stays dependency-free regardless of how the job was produced. +Trial adaptation validates Harbor's on-disk ``result.json`` files with Harbor's +own model. The import remains lazy, but invoking native execution or offline +adaptation requires the optional Harbor extra and Python >=3.12. """ from __future__ import annotations @@ -55,7 +56,11 @@ from nemo_evaluator_sdk.agent_eval.results import AgentEvalResult from nemo_evaluator_sdk.agent_eval.reward_keys import ParsedHarborRewards, validate_reward_key -from nemo_evaluator_sdk.agent_eval.runtimes.harbor_trial_adapter import _trial_from_harbor_result +from nemo_evaluator_sdk.agent_eval.runtimes.harbor_trial_adapter import ( + _HARBOR_EXTRA_REQUIRED_MESSAGE, + _iter_harbor_trial_results, + _trial_from_harbor_result, +) from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask, AgentEvalTaskset from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrial, RunnerInfo from nemo_evaluator_sdk.enums import MetricType @@ -194,8 +199,8 @@ class HarborAgentTaskRunner: :func:`discover_harbor_tasks`), or from an explicit ``dataset_path`` override, so it isn't repeated. ``task_names`` optionally restricts the run to a subset of tasks, and the ``config``'s ``job_dir`` doubles as a cache: - an existing run whose results already cover every requested task (with - ``n_attempts`` completed, non-errored trials each) is re-adapted instead of + an existing run whose Harbor-valid results cover every requested task (with + ``n_attempts`` attempts each) is re-adapted instead of re-run (unless ``force_rerun`` is set). Caching only takes effect when a stable ``job_name`` is set on the config — the default timestamped ``job_name`` writes a fresh dir per run and never hits the cache. @@ -265,16 +270,16 @@ async def run_tasks( :func:`discover_harbor_tasks`) unless a ``dataset_path`` override was given, so callers don't repeat it. - ``job_dir`` doubles as a cache. Results are served straight off it, without - importing Harbor at all, only when **both** hold: every requested task already - has ``n_attempts`` completed, non-errored results there, *and* the directory + ``job_dir`` doubles as a cache. Results are served straight off it only when + **both** hold: every requested task already has ``n_attempts`` Harbor-valid + results there, *and* the directory carries a cache stamp matching this run's inputs (agent contents, task contents, result-affecting options). Otherwise Harbor runs, and what happens to the directory depends on *which* check failed. A **stamp mismatch** discards it first: those results came from different inputs, so there is nothing safe to resume onto. A directory that - merely lacks **coverage** — stamp matches, but not enough completed results — + merely lacks **coverage** — stamp matches, but not enough valid results — is handed to Harbor intact so its per-trial resume keeps the finished trials and runs only what is missing. Harbor may still refuse a directory on its own (stricter) terms; :func:`_build_native_job` then discards it and re-runs. @@ -423,28 +428,21 @@ def _harbor_folder_names(tasks: Sequence[AgentEvalTask]) -> list[str] | None: def _all_tasks_cached(job_dir: Path, tasks: Sequence[AgentEvalTask], *, n_attempts: int) -> bool: - """Return True when every requested task already has ``n_attempts`` completed results. - - Lets ``job_dir`` act as a cache so a native run whose results are all present - is re-adapted instead of re-run. The cache is **success-aware**: only trials - that finished without an ``exception_info`` count, and a task must have at - least ``n_attempts`` of them, so an interrupted, errored, or under-sampled run - is re-run rather than silently served from a partial cache. Caching only takes - effect when a stable ``job_name`` is set on the config; with the default - timestamped ``job_name`` every run writes a fresh dir and never hits the cache. + """Return True when each requested task has ``n_attempts`` Harbor-valid results. + + Every valid result counts, including one with ``exception_info``; + Harbor itself treats such a result as an existing completed attempt. Missing, + unreadable, and schema-invalid results do not count. Caching only takes effect + when a stable ``job_name`` is set on the config; with the default timestamped + name every run writes a fresh directory and never hits the cache. """ if not job_dir.is_dir(): return False + requested_task_ids = {task.id for task in tasks} counts: dict[str, int] = {} - for result_path in job_dir.glob("*/result.json"): - try: - data = json.loads(result_path.read_text()) - except (json.JSONDecodeError, OSError): - continue - if data.get("exception_info") is not None: - continue + for _trial_dir, data in _iter_harbor_trial_results(job_dir): name = data.get("task_name") - if isinstance(name, str): + if name in requested_task_ids: counts[name] = counts.get(name, 0) + 1 return all(counts.get(task.id, 0) >= n_attempts for task in tasks) @@ -846,10 +844,7 @@ async def run_job() -> None: VerifierConfig, ) except ModuleNotFoundError as exc: - raise ModuleNotFoundError( - "the native Harbor runtime needs `harbor`, which is not an SDK dependency " - '(it requires Python >=3.12). Install it separately: uv pip install "harbor>=0.16.1"' - ) from exc + raise ModuleNotFoundError(_HARBOR_EXTRA_REQUIRED_MESSAGE) from exc if effective_force_rerun and job_dir.exists(): shutil.rmtree(job_dir) @@ -1163,19 +1158,14 @@ def build_trials_from_job_dir( job_path = Path(job_dir) known_task_ids = {task.id for task in tasks} trials: list[AgentEvalTrial] = [] - for result_path in sorted(job_path.glob("*/result.json")): - try: - data = json.loads(result_path.read_text()) - except (json.JSONDecodeError, OSError) as exc: - logger.warning("Skipping unreadable Harbor trial result %s: %s", result_path, exc) - continue + for trial_dir, data in _iter_harbor_trial_results(job_path): task_id = data.get("task_name") if task_id not in known_task_ids: # Trial for a task we weren't asked to score (e.g. a wider dataset run). continue trials.append( _trial_from_harbor_result( - result_path.parent, + trial_dir, data, reward_key=reward_key, ) diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_trial_adapter.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_trial_adapter.py index 1a8bc06b16..c8431b01cf 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_trial_adapter.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_trial_adapter.py @@ -4,19 +4,21 @@ """Adapt one Harbor result directory into an SDK :class:`AgentEvalTrial`. This module owns the complete Harbor trial-data seam: identity, reward and error -normalization, measurements, and collision-safe evidence discovery. Harbor job -execution and result-file discovery remain in :mod:`harbor_runtime`. +normalization, measurements, Harbor-valid result-file discovery, and +collision-safe evidence discovery. Harbor job execution and cache orchestration +remain in :mod:`harbor_runtime`. """ from __future__ import annotations import contextlib +import json import logging import math -from collections.abc import Mapping +from collections.abc import Iterator, Mapping from datetime import datetime from pathlib import Path -from typing import Any, Literal +from typing import Any, Literal, cast from nemo_evaluator_sdk.agent_eval.reward_keys import ( HarborRewardValueRejection, @@ -49,6 +51,7 @@ resource_spans_from_text, ) from opentelemetry.proto.trace.v1.trace_pb2 import ResourceSpans +from pydantic import ValidationError logger = logging.getLogger(__name__) @@ -84,6 +87,32 @@ "verifier/test-stdout.txt": "Verifier stdout captured while Harbor runs the task tests from the /tests directory.", } _MAX_TRACEBACK_CHARS = 8192 +_HARBOR_EXTRA_REQUIRED_MESSAGE = ( + "Harbor execution and result adaptation require the optional `harbor` extra on Python >=3.12. " + "From a NeMo Platform source checkout's repository root, run: " + "uv sync --frozen --package nemo-evaluator-sdk --extra harbor" +) + + +def _iter_harbor_trial_results(job_dir: Path) -> Iterator[tuple[Path, Mapping[str, Any]]]: + """Yield Harbor-valid result mappings in deterministic path order. + + Harbor remains a lazy import. The yielded mapping stays uncoerced because the + SDK intentionally applies stricter reward rules than Harbor's Pydantic model. + """ + try: + from harbor.models.trial.result import TrialResult # ty: ignore[unresolved-import,unused-ignore-comment] + except ModuleNotFoundError as exc: + raise ModuleNotFoundError(_HARBOR_EXTRA_REQUIRED_MESSAGE) from exc + + for result_path in sorted(job_dir.glob("*/result.json")): + try: + result_data = json.loads(result_path.read_bytes()) + TrialResult.model_validate(result_data) + except (OSError, UnicodeError, json.JSONDecodeError, ValidationError) as exc: + logger.warning("Skipping invalid Harbor trial result %s: %s", result_path, exc) + continue + yield result_path.parent, cast(dict[str, Any], result_data) def _trial_from_harbor_result( @@ -116,9 +145,8 @@ def _trial_from_harbor_result( } metadata.update(_trial_measurements(data)) - # An errored trial (or one with no reward) stays PARTIAL so it is still scored - # as 0 and counted in the summary; FAILED would exclude it from scoring. - status = AgentEvalTrialStatus.COMPLETED if error is None and reward is not None else AgentEvalTrialStatus.PARTIAL + is_complete = error is None and reward is not None + status = AgentEvalTrialStatus.COMPLETED if is_complete else AgentEvalTrialStatus.PARTIAL extension_descriptors, atif_trace, otlp_trace = _harbor_extension_evidence(trial_dir) descriptors = standard_evidence_descriptors( diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_error_propagation.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_error_propagation.py index 0740087b11..ee149f07f3 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_error_propagation.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_error_propagation.py @@ -9,8 +9,9 @@ Marked ``integration`` rather than ``e2e``/``slow`` on purpose: that combination (used by ``test_harbor_runtime_e2e.py``) is selected by no make target and no CI job. ``integration`` at least -runs wherever the plugin's ``test_harbor_plugin_run.py`` does. It still skips in CI today, because -``harbor`` is an optional SDK extra and nothing depends on ``nemo-evaluator-sdk[harbor]``. +runs wherever the plugin's ``test_harbor_plugin_run.py`` does. This older error-rollup check remains +optional; the Experimentalist integration suite owns the required Harbor 0.20 error-plus-reward +parity contract. """ from __future__ import annotations diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_resume_reconciliation.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_resume_reconciliation.py new file mode 100644 index 0000000000..1b67e834f7 --- /dev/null +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_resume_reconciliation.py @@ -0,0 +1,415 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pin Harbor 0.20.0 resume behavior and the aligned SDK cache contract.""" + +from __future__ import annotations + +import inspect +import json +from collections.abc import Iterator +from contextlib import contextmanager +from pathlib import Path +from typing import Any + +import pytest + +pytest.importorskip("harbor", reason="Harbor tests require the optional harbor extra") + +import harbor +from harbor.cli.jobs import jobs_app, resume +from harbor.job import Job +from harbor.models.job.config import DatasetConfig, JobConfig, RetryConfig +from harbor_fixtures import write_harbor_trial_result +from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import ( + _CACHE_IRRELEVANT_OPTIONS, + HarborAgentTaskRunner, + HarborRewardMetric, + HarborRuntimeConfig, + _all_tasks_cached, + _build_native_job, + _cache_stamp, + _write_cache_stamp, + build_trials_from_job_dir, +) +from nemo_evaluator_sdk.agent_eval.runtimes.harbor_trial_adapter import _iter_harbor_trial_results +from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalTask +from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrialStatus + +_DATASET_DIR = Path(__file__).resolve().parents[2] / "examples" / "harbor" / "hello_world_dataset" + + +def _write_harbor_result(trial_dir: Path, *, exception_type: str | None = None, text: str | None = None) -> None: + if text is not None: + trial_dir.mkdir(parents=True, exist_ok=True) + (trial_dir / "result.json").write_text(text, encoding="utf-8") + return + write_harbor_trial_result( + trial_dir, + task_name="harbor/hello-world", + rewards={"reward": 0.0}, + exception=exception_type, + ) + + +@contextmanager +def _halt_harbor_cli_after_filter() -> Iterator[None]: + """Let ``jobs resume`` delete matching trials, then stop before Docker/Job.run.""" + from unittest.mock import patch + + def abandon(coro: Any) -> object: + coro.close() + return object() + + with ( + patch("harbor.environments.factory.EnvironmentFactory.run_preflight"), + patch("harbor.cli.jobs.run_async", side_effect=abandon), + patch("harbor.cli.jobs.print_job_results_tables"), + ): + yield + + +def _cli_job_dir(tmp_path: Path) -> Path: + jobs_dir = tmp_path / "jobs" + job_dir = jobs_dir / "resume-job" + job_dir.mkdir(parents=True) + (job_dir / "config.json").write_text( + JobConfig(job_name="resume-job", jobs_dir=jobs_dir, quiet=True).model_dump_json(), + encoding="utf-8", + ) + return job_dir + + +def _sdk_errored_job( + tmp_path: Path, +) -> tuple[HarborRuntimeConfig, Path, AgentEvalTask]: + dataset_path = tmp_path / "dataset" + task_dir = dataset_path / "t" + task_dir.mkdir(parents=True) + (task_dir / "task.toml").write_text('[task]\nname = "t"\n') + jobs_dir = dataset_path / "jobs" + job_dir = jobs_dir / "cached-job" + write_harbor_trial_result( + job_dir / "t__aaa", + task_name="t", + rewards={"reward": 0.8}, + exception={"exception_type": "AgentTimeoutError", "exception_message": "timed out"}, + ) + config = HarborRuntimeConfig(jobs_dir=jobs_dir, job_name="cached-job") + task = AgentEvalTask( + id="t", + intent="x", + inputs={"instruction": "x"}, + metrics=[HarborRewardMetric()], + metadata={"harbor_dataset_path": str(dataset_path), "harbor_task_dir": str(task_dir)}, + ) + _write_cache_stamp(job_dir, _cache_stamp(config, dataset_path, [task])) + return config, job_dir, task + + +# --- Harbor CLI ``jobs resume`` ------------------------------------------------- + + +def test_harbor_cli_resume_defaults_to_cancelled_error_and_hides_it() -> None: + from typer.testing import CliRunner + + assert harbor.__version__ == "0.20.0" + parameter = inspect.signature(resume).parameters["filter_error_types"] + assert parameter.default == ["CancelledError"] + source = inspect.getsource(resume) + assert "show_default=False" in source + help_text = CliRunner().invoke(jobs_app, ["resume", "--help"]).output + assert "--filter-error-type" in help_text + assert "CancelledError" not in help_text + + +def test_harbor_cli_resume_exact_matches_exception_type_and_skips_bad_results( + tmp_path: Path, +) -> None: + job_dir = _cli_job_dir(tmp_path) + _write_harbor_result(job_dir / "cancelled", exception_type="CancelledError") + _write_harbor_result(job_dir / "timeout", exception_type="AgentTimeoutError") + _write_harbor_result(job_dir / "clean", exception_type=None) + _write_harbor_result(job_dir / "empty", text=" ") + _write_harbor_result(job_dir / "invalid", text="{not json") + (job_dir / "no-result").mkdir() + source = inspect.getsource(resume) + assert "trial_result.exception_info.exception_type in filter_error_types_set" in source + + with _halt_harbor_cli_after_filter(): + resume(job_path=job_dir) + + remaining = sorted(path.name for path in job_dir.iterdir() if path.is_dir()) + assert remaining == ["clean", "empty", "invalid", "no-result", "timeout"] + + +def test_harbor_cli_filter_flag_replaces_the_cancelled_error_default(tmp_path: Path) -> None: + from typer.testing import CliRunner + + job_dir = _cli_job_dir(tmp_path) + _write_harbor_result(job_dir / "cancelled", exception_type="CancelledError") + _write_harbor_result(job_dir / "timeout", exception_type="AgentTimeoutError") + source = inspect.getsource(resume) + assert inspect.signature(resume).parameters["filter_error_types"].default == ["CancelledError"] + assert "filter_error_types_set = set(filter_error_types)" in source + + with _halt_harbor_cli_after_filter(): + result = CliRunner().invoke( + jobs_app, + ["resume", "--job-path", str(job_dir), "--filter-error-type", "AgentTimeoutError"], + ) + + assert result.exit_code == 0, result.output + remaining = sorted(path.name for path in job_dir.iterdir() if path.is_dir()) + assert remaining == ["cancelled"], "passing -f replaces CancelledError rather than appending to it" + + +def test_harbor_job_create_has_no_filter_error_types() -> None: + assert "filter_error_type" not in inspect.signature(Job.create).parameters + assert "filter_error_type" not in inspect.getsource(Job) + + +def test_empty_resume_filter_deletes_nothing(tmp_path: Path) -> None: + job_dir = _cli_job_dir(tmp_path) + _write_harbor_result(job_dir / "cancelled", exception_type="CancelledError") + + with _halt_harbor_cli_after_filter(): + resume(job_path=job_dir, filter_error_types=[]) + + assert (job_dir / "cancelled").is_dir() + + +# --- Harbor Job reconciliation -------------------------------------------------- + + +async def _harbor_job(tmp_path: Path, *, n_attempts: int = 1) -> tuple[Job, Path]: + jobs_dir = tmp_path / "jobs" + job_name = "spike" + job_dir = jobs_dir / job_name + job_dir.mkdir(parents=True, exist_ok=True) + config = JobConfig( + job_name=job_name, + jobs_dir=jobs_dir, + n_attempts=n_attempts, + quiet=True, + datasets=[DatasetConfig(path=_DATASET_DIR, task_names=["hello-world"])], + ) + (job_dir / "config.json").write_text(config.model_dump_json(), encoding="utf-8") + return await Job.create(config), job_dir + + +def _plant_errored_trial(job: Any, job_dir: Path, exception_type: str) -> Path: + planned = job._trial_configs[0] + trial_dir = job_dir / planned.trial_name + write_harbor_trial_result( + trial_dir, + task_name="harbor/hello-world", + rewards={"reward": 0.0}, + exception=exception_type, + config=planned, + ) + return trial_dir + + +@pytest.mark.asyncio +async def test_harbor_job_treats_an_errored_result_json_as_complete(tmp_path: Path) -> None: + job, job_dir = await _harbor_job(tmp_path) + _plant_errored_trial(job, job_dir, "AgentTimeoutError") + source = inspect.getsource(Job._maybe_init_existing_job) + assert "exception_info" not in source + + resumed = await Job.create(job.config) + + [existing_result] = resumed._existing_trial_results + assert existing_result.exception_info is not None + assert existing_result.exception_info.exception_type == "AgentTimeoutError" + assert resumed._remaining_trial_configs == [] + + +@pytest.mark.asyncio +async def test_harbor_job_rmtrees_a_trial_dir_without_result_json_and_reruns_it( + tmp_path: Path, +) -> None: + job, job_dir = await _harbor_job(tmp_path) + orphan = job_dir / "empty-attempt" + orphan.mkdir() + + resumed = await Job.create(job.config) + + assert not orphan.exists() + assert resumed._existing_trial_results == [] + assert len(resumed._remaining_trial_configs) == 1 + + +@pytest.mark.asyncio +async def test_harbor_cancelled_error_is_stats_only_and_is_not_rerun(tmp_path: Path) -> None: + job, job_dir = await _harbor_job(tmp_path) + trial_dir = _plant_errored_trial(job, job_dir, "CancelledError") + source = inspect.getsource(Job._init_progress_tracking) + assert "_is_cancelled_result" in source + assert "rmtree" not in inspect.getsource(Job._is_cancelled_result) + + resumed = await Job.create(job.config) + + assert resumed._remaining_trial_configs == [] + assert trial_dir.name in resumed._cancelled_trial_names + assert resumed._existing_stats.n_cancelled_trials == 1 + assert (trial_dir / "result.json").exists() + + +# --- SDK cache alignment -------------------------------------------------------- + + +def test_all_tasks_cached_accepts_an_errored_only_n_attempts_of_one(tmp_path: Path) -> None: + config, job_dir, task = _sdk_errored_job(tmp_path) + + assert _all_tasks_cached(job_dir, [task], n_attempts=1) is True + assert config.job_name == "cached-job" + + +@pytest.mark.asyncio +async def test_errored_stamped_job_is_served_without_invoking_harbor( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + from nemo_evaluator_sdk.agent_eval.runtimes import harbor_runtime + + config, job_dir, task = _sdk_errored_job(tmp_path) + calls: list[bool] = [] + + def fake_build(runtime_config, _dataset_path, _task_names, *, job_name=None, force_rerun=None): + async def run_job() -> None: + calls.append(bool(force_rerun)) + + return runtime_config.jobs_dir / (job_name or "job"), run_job + + monkeypatch.setattr(harbor_runtime, "_build_native_job", fake_build) + + trials = await HarborAgentTaskRunner(config=config).run_tasks([task]) + + assert calls == [], "a stamp hit with Harbor-valid errored coverage must be served directly" + assert [trial.id for trial in trials] == ["t__aaa"] + assert trials[0].status == AgentEvalTrialStatus.PARTIAL + assert trials[0].error is not None + assert trials[0].error.type == "AgentTimeoutError" + assert trials[0].metadata["reward"] == 0.8 + + +_REQUIRED_RESULT_FIELDS = ("task_name", "trial_name", "trial_uri", "task_id", "task_checksum", "config", "agent_info") +_VALID_RESULT_INPUTS: dict[str, tuple[dict[str, float | int], str | None]] = { + "valid-clean": ({"reward": 1.0}, None), + "valid-runtime-error": ({"reward": 0.8}, "RuntimeError"), + "valid-timeout-without-primary": ({"format_ok": 1.0}, "AgentTimeoutError"), + "valid-cancelled": ({"reward": 0.0}, "CancelledError"), +} +_INVALID_RESULT_TEXTS = { + "empty": "", + "null": "null", + "array": "[]", + "scalar": '"result"', + "fragment": '{"task_name": "t"}', +} +_INVALID_FIELD_VALUES: dict[str, tuple[str, object]] = { + "invalid-task-name": ("task_name", 17), + "invalid-task-id": ("task_id", {}), + "invalid-config": ("config", {}), + "invalid-agent-info": ("agent_info", {"name": "oracle"}), + "invalid-exception-info": ("exception_info", {"exception_type": "RuntimeError"}), + "invalid-verifier-result": ("verifier_result", {"rewards": "not-a-mapping"}), + "invalid-agent-result": ("agent_result", {"n_input_tokens": {"not": "an integer"}}), + "invalid-timing": ("environment_setup", {"started_at": []}), +} +_RESULT_VALIDITY_CASES = ( + *_VALID_RESULT_INPUTS, + "unreadable", + *_INVALID_RESULT_TEXTS, + *(f"missing-{field}" for field in _REQUIRED_RESULT_FIELDS), + *_INVALID_FIELD_VALUES, +) + + +def _plant_result_validity_case(job_dir: Path, case: str) -> None: + trial_dir = job_dir / f"t__{case}" + if case == "unreadable": + (trial_dir / "result.json").mkdir(parents=True) + return + if case in _INVALID_RESULT_TEXTS: + trial_dir.mkdir(parents=True) + (trial_dir / "result.json").write_text(_INVALID_RESULT_TEXTS[case], encoding="utf-8") + return + + if case in _VALID_RESULT_INPUTS: + rewards, exception = _VALID_RESULT_INPUTS[case] + else: + rewards, exception = {"reward": 1.0}, None + + result = write_harbor_trial_result( + trial_dir, + task_name="t", + rewards=rewards, + exception=exception, + ) + if case in _VALID_RESULT_INPUTS: + return + + payload = result.model_dump(mode="json") + if case.startswith("missing-"): + del payload[case.removeprefix("missing-")] + elif case in _INVALID_FIELD_VALUES: + field, value = _INVALID_FIELD_VALUES[case] + payload[field] = value + else: # pragma: no cover - the parametrization is exhaustive + raise AssertionError(f"unhandled result-validity case {case!r}") + (trial_dir / "result.json").write_text(json.dumps(payload), encoding="utf-8") + + +@pytest.mark.parametrize("case", _RESULT_VALIDITY_CASES) +def test_harbor_result_validity_is_shared_by_loader_cache_and_adaptation(tmp_path: Path, case: str) -> None: + assert harbor.__version__ == "0.20.0" + job_dir = tmp_path / "job" + job_dir.mkdir() + _plant_result_validity_case(job_dir, case) + task = AgentEvalTask(id="t", intent="x", inputs={"instruction": "x"}, metrics=[HarborRewardMetric()]) + expected_valid = case in _VALID_RESULT_INPUTS + + loaded = list(_iter_harbor_trial_results(job_dir)) + cached = _all_tasks_cached(job_dir, [task], n_attempts=1) + trials = build_trials_from_job_dir(job_dir, [task]) + + assert bool(loaded) is expected_valid + assert cached is expected_valid + assert bool(trials) is expected_valid + + if case == "valid-clean": + assert trials[0].status is AgentEvalTrialStatus.COMPLETED + assert trials[0].metadata["reward"] == 1.0 + elif case == "valid-runtime-error": + assert trials[0].status is AgentEvalTrialStatus.PARTIAL + assert trials[0].metadata["reward"] == 0.8 + elif case == "valid-timeout-without-primary": + assert trials[0].status is AgentEvalTrialStatus.PARTIAL + assert trials[0].metadata["reward"] is None + + +def test_sdk_does_not_expose_resume_filter_error_types_today() -> None: + assert "resume_filter_error_types" not in HarborRuntimeConfig.model_fields + assert "resume_filter_error_types" not in _CACHE_IRRELEVANT_OPTIONS + + +# --- Retry vs resume ------------------------------------------------------------ + + +def test_in_run_retry_and_cli_resume_filter_are_separate_surfaces() -> None: + assert harbor.__version__ == "0.20.0" + fields = RetryConfig.model_fields + assert set(fields) >= {"max_retries", "include_exceptions", "exclude_exceptions"} + default_exclude = RetryConfig().exclude_exceptions + assert default_exclude is not None + assert "AgentTimeoutError" in default_exclude + assert "CancelledError" not in default_exclude + + source = inspect.getsource(_build_native_job) + assert "RetryConfig(max_retries=config.max_retries)" in source + assert "include_exceptions" not in source + assert "exclude_exceptions" not in source + assert "filter_error_type" not in source + assert "resume_filter" not in source diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_runtime.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_runtime.py index 59c1d29df7..3baead7574 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_runtime.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_runtime.py @@ -2,20 +2,23 @@ # SPDX-License-Identifier: Apache-2.0 import asyncio -import builtins import hashlib import importlib import json import logging import os import sys -from collections.abc import Awaitable, Callable +from collections.abc import Awaitable, Callable, Mapping from datetime import datetime, timezone from pathlib import Path from types import ModuleType from typing import cast import pytest + +pytest.importorskip("harbor", reason="Harbor tests require the optional harbor extra") + +from harbor_fixtures import ErrorAwareQualityMetric, write_harbor_trial_result from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator from nemo_evaluator_sdk.agent_eval.metrics import AgentPhaseSuccessMetric from nemo_evaluator_sdk.agent_eval.results import AgentEvalSummary @@ -66,38 +69,47 @@ def __len__(self) -> int: def _write_trial( - job_dir: Path, trial_name: str, task_name: str, *, reward: float | None, exception: object | None = None + job_dir: Path, + trial_name: str, + task_name: str, + *, + reward: float | None, + exception: str | Mapping[str, object] | None = None, ) -> None: - """Write one Harbor trial dir. ``exception`` is stored verbatim as ``exception_info``. + """Write one complete Harbor-valid trial result.""" + write_harbor_trial_result( + job_dir / trial_name, + task_name=task_name, + rewards=None if reward is None else {"reward": reward}, + exception=exception, + ) - Typed loosely on purpose: Harbor writes a mapping there, older runs wrote a bare - string, so the adapter has to cope with both. - """ - trial_dir = job_dir / trial_name - (trial_dir / "agent").mkdir(parents=True) - (trial_dir / "verifier").mkdir(parents=True) - (trial_dir / "agent" / "trajectory.json").write_text("{}") - payload = { - "task_name": task_name, - "trial_name": trial_name, - "verifier_result": None if reward is None else {"rewards": {"reward": reward}}, - "exception_info": exception, - "agent_result": {"n_input_tokens": 100, "n_output_tokens": 10, "n_cache_tokens": 5, "cost_usd": 0.25}, - } - (trial_dir / "result.json").write_text(json.dumps(payload)) +def _write_rewards_trial(job_dir: Path, trial_name: str, task_name: str, rewards: Mapping[str, float | int]) -> None: + write_harbor_trial_result(job_dir / trial_name, task_name=task_name, rewards=rewards) -def _write_rewards_trial(job_dir: Path, trial_name: str, task_name: str, rewards: dict[object, object]) -> None: - trial_dir = job_dir / trial_name - (trial_dir / "agent").mkdir(parents=True, exist_ok=True) - (trial_dir / "verifier").mkdir(parents=True, exist_ok=True) - payload = { - "task_name": task_name, - "trial_name": trial_name, - "verifier_result": {"rewards": rewards}, - "exception_info": None, - } - (trial_dir / "result.json").write_text(json.dumps(payload)) + +def _adapt_raw_trial( + tmp_path: Path, + *, + rewards: Mapping[object, object] | None, + exception_info: object | None = None, + reward_key: str = "reward", +) -> AgentEvalTrial: + """Exercise defensive normalization below the Harbor-valid file boundary.""" + trial_dir = tmp_path / "raw__trial" + (trial_dir / "agent").mkdir(parents=True) + (trial_dir / "verifier").mkdir() + return _trial_from_harbor_result( + trial_dir, + { + "task_name": "t", + "trial_name": trial_dir.name, + "verifier_result": None if rewards is None else {"rewards": rewards}, + "exception_info": exception_info, + }, + reward_key=reward_key, + ) @pytest.mark.parametrize( @@ -180,12 +192,7 @@ async def test_primary_reward_matrix_survives_adaptation_and_metric_diagnostics( rewards: dict[object, object] = {"sibling": 0.5} if raw is not _MISSING: rewards["score"] = raw - job_dir = tmp_path / "job" - job_dir.mkdir() - _write_rewards_trial(job_dir, "t__a", "t", rewards) - task = AgentEvalTask(id="t", intent="t", inputs={"instruction": "t"}, metrics=[HarborRewardMetric()]) - - trial = build_trials_from_job_dir(job_dir, [task], reward_key="score")[0] + trial = _adapt_raw_trial(tmp_path, rewards=rewards, reward_key="score") result = await HarborRewardMetric(output_name="score", reward_keys=("score",)).compute_scores( MetricInput(row=DatasetRow(data={}), candidate=CandidateOutput(metadata=trial.metadata)) ) @@ -203,6 +210,29 @@ async def test_primary_reward_matrix_survives_adaptation_and_metric_diagnostics( assert trial.status is AgentEvalTrialStatus.PARTIAL +@pytest.mark.asyncio +async def test_harbor_valid_boolean_primary_remains_unusable_after_file_validation(tmp_path: Path) -> None: + job_dir = tmp_path / "job" + job_dir.mkdir() + _write_rewards_trial(job_dir, "t__a", "t", {"score": 1.0, "sibling": 0.5}) + result_path = job_dir / "t__a" / "result.json" + payload = json.loads(result_path.read_text()) + payload["verifier_result"]["rewards"]["score"] = True + result_path.write_text(json.dumps(payload)) + task = AgentEvalTask(id="t", intent="t", inputs={"instruction": "t"}, metrics=[HarborRewardMetric()]) + + trial = build_trials_from_job_dir(job_dir, [task], reward_key="score")[0] + result = await HarborRewardMetric(output_name="score", reward_keys=("score",)).compute_scores( + MetricInput(row=DatasetRow(data={}), candidate=CandidateOutput(metadata=trial.metadata)) + ) + + assert trial.status is AgentEvalTrialStatus.PARTIAL + assert trial.metadata["reward"] is None + assert trial.metadata["reward_rejections"] == {"score": "boolean"} + assert [(output.name, output.value) for output in result.outputs] == [("score", 0.0)] + assert [diagnostic.details for diagnostic in result.diagnostics] == [{"output": "score", "reason": "boolean"}] + + @pytest.mark.asyncio @pytest.mark.parametrize( ("raw", "expected_secondary", "reason"), @@ -225,12 +255,7 @@ async def test_secondary_reward_matrix_survives_adaptation_and_metric_diagnostic rewards: dict[object, object] = {"score": 1.0} if raw is not _MISSING: rewards["format_ok"] = raw - job_dir = tmp_path / "job" - job_dir.mkdir() - _write_rewards_trial(job_dir, "t__a", "t", rewards) - task = AgentEvalTask(id="t", intent="t", inputs={"instruction": "t"}, metrics=[HarborRewardMetric()]) - - trial = build_trials_from_job_dir(job_dir, [task], reward_key="score")[0] + trial = _adapt_raw_trial(tmp_path, rewards=rewards, reward_key="score") result = await HarborRewardMetric(output_name="score", reward_keys=("score", "format_ok")).compute_scores( MetricInput(row=DatasetRow(data={}), candidate=CandidateOutput(metadata=trial.metadata)) ) @@ -428,6 +453,80 @@ async def test_harbor_exception_stats_is_read_straight_off_the_summary(tmp_path: assert by_trial["gamma__a"].status is AgentEvalTrialStatus.PARTIAL +@pytest.mark.asyncio +async def test_errored_harbor_rewards_and_metric_owned_exclusions_are_independent(tmp_path: Path) -> None: + job_dir = tmp_path / "job" + job_dir.mkdir() + _write_rewards_trial(job_dir, "t__a_success", "t", {"reward": 1.0, "format_ok": 1.0}) + _write_trial(job_dir, "t__b_runtime_reward", "t", reward=0.8, exception="RuntimeError") + _write_trial(job_dir, "t__c_runtime_missing", "t", reward=None, exception="RuntimeError") + _write_trial(job_dir, "t__d_timeout", "t", reward=0.6, exception="AgentTimeoutError") + task = AgentEvalTask( + id="t", + intent="test", + inputs={"instruction": "test"}, + metrics=[HarborRewardMetric(), ErrorAwareQualityMetric()], + views={ + "quality": SemanticView( + reducer=SemanticReducer.MEAN, + signals=[ViewSignal(metric="error_aware_quality", output="quality")], + ) + }, + ) + + result = await AgentEvaluator().run(tasks=[task], target=HarborAgentTaskRunner(job_dir=job_dir)) + + assert [(trial.id, trial.status) for trial in result.trials] == [ + ("t__a_success", AgentEvalTrialStatus.COMPLETED), + ("t__b_runtime_reward", AgentEvalTrialStatus.PARTIAL), + ("t__c_runtime_missing", AgentEvalTrialStatus.PARTIAL), + ("t__d_timeout", AgentEvalTrialStatus.PARTIAL), + ] + harbor_scores = [score for score in result.scores if score.metric_type == "harbor_reward"] + assert [(score.trial_id, score.outputs[0].value) for score in harbor_scores] == [ + ("t__a_success", 1.0), + ("t__b_runtime_reward", 0.8), + ("t__c_runtime_missing", 0.0), + ("t__d_timeout", 0.6), + ] + missing_reward_score = next(score for score in harbor_scores if score.trial_id == "t__c_runtime_missing") + assert {"output": "reward", "reason": "absent"} in [ + diagnostic.details for diagnostic in missing_reward_score.diagnostics + ] + assert result.summary.score("harbor_reward.reward").mean == pytest.approx(0.6) + assert result.summary.metric_coverage["harbor_reward"]["format_ok"].model_dump() == { + "total": 4, + "scored": 1, + "failed": 0, + "missing": 3, + } + + quality_scores = [score for score in result.scores if score.metric_type == "error_aware_quality"] + assert [[output.value for output in score.outputs] for score in quality_scores] == [[1.0], [], [], [1.0]] + assert [ + diagnostic.details + for score in quality_scores + for diagnostic in score.diagnostics + if diagnostic.details is not None + ] == [ + {"output": "quality", "reason": "excluded_error_type"}, + {"output": "quality", "reason": "excluded_error_type"}, + ] + assert result.summary.metric_coverage["error_aware_quality"]["quality"].model_dump() == { + "total": 4, + "scored": 2, + "failed": 0, + "missing": 2, + } + quality_view = result.summary.score("view.quality") + assert (quality_view.count, quality_view.nan_count, quality_view.mean) == (2, 2, 1.0) + assert result.summary.error_trial_ids == { + "RuntimeError": ["t__b_runtime_reward", "t__c_runtime_missing"], + "AgentTimeoutError": ["t__d_timeout"], + } + assert result.summary.trial_count == 4 + + def test_reward_with_no_matching_reward_key_is_partial_and_warns(tmp_path: Path, caplog) -> None: # Verifier emitted a reward, but under a key we didn't ask for: no guessing — # the trial is treated as having no reward (None -> PARTIAL, scores 0.0) and warns. @@ -445,24 +544,18 @@ def test_reward_with_no_matching_reward_key_is_partial_and_warns(tmp_path: Path, def test_rejected_primary_is_distinguished_and_trial_metadata_is_sanitized(tmp_path: Path, caplog) -> None: - job_dir = tmp_path / "job" - job_dir.mkdir() - _write_rewards_trial( - job_dir, - "t__a", - "t", - { - "score": True, - "format_ok": "1", - "shape_ok": "bad", - "": 1, - "derived.pass@2": 1, - }, - ) - tasks = [AgentEvalTask(id="t", intent="x", inputs={"instruction": "p"}, metrics=[HarborRewardMetric()])] - with caplog.at_level(logging.WARNING): - trial = build_trials_from_job_dir(job_dir, tasks, reward_key="score")[0] + trial = _adapt_raw_trial( + tmp_path, + rewards={ + "score": True, + "format_ok": "1", + "shape_ok": "bad", + "": 1, + "derived.pass@2": 1, + }, + reward_key="score", + ) assert trial.metadata["reward"] is None assert trial.metadata["reward_details"] == {"format_ok": 1.0} @@ -473,10 +566,10 @@ def test_rejected_primary_is_distinguished_and_trial_metadata_is_sanitized(tmp_p @pytest.mark.asyncio -async def test_harbor_runner_finalizes_sparse_outputs_per_task_and_keeps_all_rejected_keys(tmp_path: Path) -> None: +async def test_harbor_runner_finalizes_sparse_outputs_per_task(tmp_path: Path) -> None: job_dir = tmp_path / "job" job_dir.mkdir() - _write_rewards_trial(job_dir, "a__1", "A", {"score": 1, "format_ok": 1, "shape.ok": "bad"}) + _write_rewards_trial(job_dir, "a__1", "A", {"score": 1, "format_ok": 1, "shape.ok": 0.5}) _write_rewards_trial(job_dir, "a__2", "A", {"score": 0}) _write_rewards_trial(job_dir, "b__1", "B", {"score": 1}) @@ -512,18 +605,15 @@ async def test_harbor_runner_finalizes_sparse_outputs_per_task_and_keeps_all_rej format_score = result.summary.score("harbor_reward.format_ok") shape_score = result.summary.score("harbor_reward.shape.ok") assert (format_score.count, format_score.nan_count, format_score.mean) == (1, 1, 1.0) - assert (shape_score.count, shape_score.nan_count, shape_score.mean) == (0, 2, None) + assert (shape_score.count, shape_score.nan_count, shape_score.mean) == (1, 1, 0.5) assert result.summary.metric_coverage["harbor_reward"]["format_ok"].missing == 1 - assert result.summary.metric_coverage["harbor_reward"]["shape.ok"].missing == 2 + assert result.summary.metric_coverage["harbor_reward"]["shape.ok"].missing == 1 a_scores = [score for score in result.scores if score.task_id == "A" and score.metric_type == "harbor_reward"] assert [[output.name for output in score.outputs] for score in a_scores] == [ - ["score", "format_ok"], + ["score", "format_ok", "shape.ok"], ["score"], ] - assert any( - diagnostic.details == {"output": "shape.ok", "reason": "non_numeric"} for diagnostic in a_scores[0].diagnostics - ) @pytest.mark.asyncio @@ -693,30 +783,51 @@ def _seed_cached_job(tmp_path: Path, *, task_id: str = "t") -> tuple[HarborRunti return config, job_dir, task +@pytest.mark.asyncio +async def test_metric_selection_changes_cached_scoring_but_not_execution_cache_identity( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import _cache_stamp + + config, _job_dir, task = _seed_cached_job(tmp_path) + dataset_path = Path(str(task.metadata["harbor_dataset_path"])) + with_quality = task.model_copy(update={"metrics": [HarborRewardMetric(), ErrorAwareQualityMetric()]}) + run_calls: list[bool] = [] + _spy_on_run_job(monkeypatch, run_calls) + + without_result = await AgentEvaluator().run(tasks=[task], target=HarborAgentTaskRunner(config=config)) + with_result = await AgentEvaluator().run(tasks=[with_quality], target=HarborAgentTaskRunner(config=config)) + + assert _cache_stamp(config, dataset_path, [task]) == _cache_stamp(config, dataset_path, [with_quality]) + assert run_calls == [] + assert [trial.id for trial in without_result.trials] == [trial.id for trial in with_result.trials] + without_descriptors = without_result.tasks[0].model_dump(mode="json")["metrics"] + with_descriptors = with_result.tasks[0].model_dump(mode="json")["metrics"] + assert [descriptor["type"] for descriptor in without_descriptors] == ["harbor_reward"] + assert [descriptor["type"] for descriptor in with_descriptors] == ["harbor_reward", "error_aware_quality"] + assert with_descriptors[0] == without_descriptors[0] + assert with_descriptors[1]["outputs"] == [ + {"name": "quality", "description": None, "value_schema": "ContinuousScore", "required": False} + ] + assert [score.metric_type for score in without_result.scores] == ["harbor_reward"] + assert [score.metric_type for score in with_result.scores] == ["harbor_reward", "error_aware_quality"] + assert [(output.name, output.value) for output in with_result.scores[1].outputs] == [("quality", 1.0)] + + @pytest.mark.asyncio async def test_native_runner_uses_job_dir_as_cache(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: # A native run whose job_dir covers every requested task AND carries a matching - # cache stamp is re-adapted, not re-run: run_job is never awaited, so Harbor is - # never imported here (which is why this test needs no harbor install). + # cache stamp is re-adapted, not re-run. Adaptation still imports Harbor lazily + # to validate the persisted result against Harbor's own schema. config, _job_dir, task = _seed_cached_job(tmp_path) - # Watch the lazy import directly instead of mutating sys.modules: popping only - # "harbor" would leave already-imported harbor.* submodules parentless and - # corrupt the module identity other suites monkeypatch. - imported: list[str] = [] - real_import = builtins.__import__ - - def recording_import(name, *args, **kwargs): - if name == "harbor" or name.startswith("harbor."): - imported.append(name) - return real_import(name, *args, **kwargs) - - monkeypatch.setattr(builtins, "__import__", recording_import) + run_calls: list[bool] = [] + _spy_on_run_job(monkeypatch, run_calls) trials = await HarborAgentTaskRunner(config=config).run_tasks([task]) - monkeypatch.undo() assert [trial.task_id for trial in trials] == ["t"] assert trials[0].metadata["reward"] == 1.0 - assert imported == [], f"a cache hit must not import harbor, but imported {imported}" + assert run_calls == [] @pytest.mark.asyncio @@ -1006,7 +1117,7 @@ def test_multiple_attempts_map_to_one_trial_each(tmp_path: Path) -> None: assert sorted(trial.metadata["reward"] for trial in trials) == [0.0, 1.0] -def test_cache_is_attempt_and_success_aware(tmp_path: Path) -> None: +def test_cache_counts_harbor_valid_physical_attempts(tmp_path: Path) -> None: from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import _all_tasks_cached job_dir = tmp_path / "job" @@ -1018,11 +1129,11 @@ def test_cache_is_attempt_and_success_aware(tmp_path: Path) -> None: assert _all_tasks_cached(job_dir, tasks, n_attempts=1) is True assert _all_tasks_cached(job_dir, tasks, n_attempts=2) is False - # An errored attempt does not count, so the run is not served from a partial cache. + # Harbor considers a valid errored result complete, so it is the second attempt. _write_trial(job_dir, "t__bbb", "t", reward=0.0, exception="NonZeroAgentExitCodeError") - assert _all_tasks_cached(job_dir, tasks, n_attempts=2) is False + assert _all_tasks_cached(job_dir, tasks, n_attempts=2) is True - # A second clean attempt satisfies n_attempts=2. + # Extra valid attempts do not invalidate already-satisfied coverage. _write_trial(job_dir, "t__ccc", "t", reward=1.0) assert _all_tasks_cached(job_dir, tasks, n_attempts=2) is True @@ -1913,35 +2024,20 @@ def test_exception_info_shapes_all_resolve_to_a_type( actually writes — went untested. Resolving to None here would silently promote a crashed trial to COMPLETED and let it score. """ - job_dir = tmp_path / "job" - job_dir.mkdir() - _write_trial(job_dir, "t__aaa", "t", reward=1.0, exception=exception_info) - - trials = build_trials_from_job_dir( - job_dir, [AgentEvalTask(id="t", intent="x", inputs={"instruction": "p"}, metrics=[HarborRewardMetric()])] - ) + trial = _adapt_raw_trial(tmp_path, rewards={"reward": 1.0}, exception_info=exception_info) - assert len(trials) == 1 - assert trials[0].error is not None - assert trials[0].error.type == expected_type - assert trials[0].status is AgentEvalTrialStatus.PARTIAL + assert trial.error is not None + assert trial.error.type == expected_type + assert trial.status is AgentEvalTrialStatus.PARTIAL def test_non_string_message_and_traceback_are_dropped_rather_than_carried(tmp_path: Path) -> None: # Same totality requirement as the parametrization above: a producer that put a number (or a # nested object) where a string belongs must not take down the whole job dir. - job_dir = tmp_path / "job" - job_dir.mkdir() - _write_trial( - job_dir, - "t__aaa", - "t", - reward=1.0, - exception={"exception_type": "RuntimeError", "exception_message": 42, "exception_traceback": {"a": 1}}, - ) - - [trial] = build_trials_from_job_dir( - job_dir, [AgentEvalTask(id="t", intent="x", inputs={"instruction": "p"}, metrics=[HarborRewardMetric()])] + trial = _adapt_raw_trial( + tmp_path, + rewards={"reward": 1.0}, + exception_info={"exception_type": "RuntimeError", "exception_message": 42, "exception_traceback": {"a": 1}}, ) assert trial.error is not None diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_trial_error_rollup.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_trial_error_rollup.py index 22c5983bb5..c0fb5cb05f 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_trial_error_rollup.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_trial_error_rollup.py @@ -95,25 +95,48 @@ def test_membership_ignores_trial_status() -> None: def test_duplicate_trial_ids_stay_two_entries() -> None: # Nothing enforces trial-id uniqueness (Gym derives ids from a rollout index in two separate # loops), so the rollup must append rather than collect into a set — collapsing them would - # understate the error count. + # understate the error count. trial_count is len(trials) for the same reason. summary = AgentEvalSummary.from_scores([], trials=[_trial("dup", error="E"), _trial("dup", error="E")]) assert summary.error_trial_ids == {"E": ["dup", "dup"]} assert summary.error_count == 2 + assert summary.trial_count == 2 def test_trials_may_be_wider_than_the_scores() -> None: # A caller re-aggregating a subset can hand over more trials than scores. The rollup names them - # regardless: it reads trials, not scores, so the two need not line up. + # regardless: it reads trials, not scores, so the two need not line up. trial_count is + # len(trials), so the unmeasured t1 still contributes even though it has no score. + unmeasured = _trial("t1", task_id="task-b", error="RuntimeError") summary = AgentEvalSummary.from_scores( [_score("task-a", "t0", 1.0)], - trials=[_trial("t0"), _trial("t1", task_id="task-b", error="RuntimeError")], + trials=[_trial("t0"), unmeasured], ) assert summary.error_trial_ids == {"RuntimeError": ["t1"]} + assert summary.trial_count == 2 + assert summary.score_count == 1 assert "task-b" not in summary.task_metric_values +def test_trial_count_without_trials_is_distinct_score_pairs() -> None: + # Same trial_id on two tasks is two attempts. Two score rows for one pair are one attempt. + two_tasks = AgentEvalSummary.from_scores([_score("task-a", "shared", 1.0), _score("task-b", "shared", 0.0)]) + two_metrics = AgentEvalSummary.from_scores([_score("task-a", "t0", 1.0), _score("task-a", "t0", 0.0)]) + + assert two_tasks.trial_count == 2 + assert two_metrics.trial_count == 1 + + +def test_trial_count_with_trials_is_the_supplied_list_length() -> None: + summary = AgentEvalSummary.from_scores( + [_score("task-a", "shared", 1.0), _score("task-b", "shared", 0.0)], + trials=[_trial("shared"), _trial("shared", task_id="task-b")], + ) + + assert summary.trial_count == 2 + + def test_error_count_must_agree_with_the_rollup() -> None: # The model is public and directly constructible; a count contradicting the rollup beside it is # worse than no count at all. diff --git a/packages/nemo_evaluator_sdk/tests/harbor_fixtures.py b/packages/nemo_evaluator_sdk/tests/harbor_fixtures.py new file mode 100644 index 0000000000..d69dd70c36 --- /dev/null +++ b/packages/nemo_evaluator_sdk/tests/harbor_fixtures.py @@ -0,0 +1,146 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Shared Harbor 0.20 result builders for evaluator SDK tests.""" + +from __future__ import annotations + +from collections.abc import Mapping +from pathlib import Path +from types import MappingProxyType +from typing import Any, Literal + +from harbor.models.trial.config import TrialConfig +from harbor.models.trial.result import TrialResult +from nemo_evaluator_sdk.metrics.protocol import ( + MetricDiagnostic, + MetricInput, + MetricOutput, + MetricOutputSpec, + MetricResult, +) +from pydantic import BaseModel + +_ERROR_RESULT = Path(__file__).parent / "agent_eval" / "fixtures" / "harbor_error_result.json" +_FIXTURE_AGENT_RESULT: Mapping[str, float | int | None] = MappingProxyType( + { + "n_input_tokens": 100, + "n_output_tokens": 10, + "n_cache_tokens": 5, + "cost_usd": 0.25, + } +) + + +class ErrorAwareQualityMetric(BaseModel): + """Test-only metric proving that error exclusions belong to each metric.""" + + type: Literal["error_aware_quality"] = "error_aware_quality" + + def output_spec(self) -> list[MetricOutputSpec]: + return [MetricOutputSpec.continuous_score("quality", required=False)] + + async def compute_scores(self, input: MetricInput) -> MetricResult: + trial = input.row.data["trial"] + error = trial["error"] + if isinstance(error, dict) and error.get("type") == "RuntimeError": + return MetricResult( + outputs=[], + diagnostics=[ + MetricDiagnostic( + message="quality excluded for RuntimeError", + details={"output": "quality", "reason": "excluded_error_type"}, + ) + ], + ) + return MetricResult(outputs=[MetricOutput(name="quality", value=1.0)]) + + +def harbor_trial_result( + trial_dir: Path, + *, + task_name: str, + rewards: Mapping[str, float | int] | None, + exception: str | Mapping[str, Any] | None = None, + config: TrialConfig | None = None, + task_path: Path | None = None, + source: str | None = None, + agent_result: Mapping[str, float | int | None] | None = _FIXTURE_AGENT_RESULT, +) -> TrialResult: + """Build one coherent Harbor-valid result from the captured 0.20 fixture.""" + payload = TrialResult.model_validate_json(_ERROR_RESULT.read_bytes()).model_dump(mode="json") + trial_name = trial_dir.name + if task_path is not None: + resolved_task_path = task_path + elif config is not None and config.task.path is not None: + resolved_task_path = config.task.path + else: + resolved_task_path = trial_dir.parent / "tasks" / task_name + + if source is not None: + resolved_source = source + elif config is not None and config.task.source is not None: + resolved_source = config.task.source + else: + resolved_source = "nemo-evaluator-sdk-tests" + + payload.update( + { + "task_name": task_name, + "trial_name": trial_name, + "trial_uri": trial_dir.resolve().as_uri(), + "task_id": {"path": str(resolved_task_path)}, + "source": resolved_source, + "verifier_result": None if rewards is None else {"rewards": dict(rewards)}, + "agent_result": None if agent_result is None else dict(agent_result), + } + ) + result_config = config.model_dump(mode="json") if config is not None else payload["config"] + assert isinstance(result_config, dict) + result_config.update({"trial_name": trial_name, "trials_dir": str(trial_dir.parent)}) + task_config = result_config["task"] + assert isinstance(task_config, dict) + task_config.update({"path": str(resolved_task_path), "source": resolved_source}) + payload["config"] = result_config + + if exception is None: + payload["exception_info"] = None + else: + template = payload["exception_info"] + assert isinstance(template, dict) + updates = {"exception_type": exception} if isinstance(exception, str) else dict(exception) + payload["exception_info"] = {**template, **updates} + + return TrialResult.model_validate(payload) + + +def write_harbor_trial_result( + trial_dir: Path, + *, + task_name: str, + rewards: Mapping[str, float | int] | None, + exception: str | Mapping[str, Any] | None = None, + config: TrialConfig | None = None, + task_path: Path | None = None, + source: str | None = None, + agent_result: Mapping[str, float | int | None] | None = _FIXTURE_AGENT_RESULT, +) -> TrialResult: + """Write a coherent trial ``config.json`` and Harbor-valid ``result.json``.""" + trial_dir.mkdir(parents=True, exist_ok=True) + (trial_dir / "agent").mkdir(exist_ok=True) + (trial_dir / "verifier").mkdir(exist_ok=True) + (trial_dir / "agent" / "trajectory.json").write_text("{}", encoding="utf-8") + + result = harbor_trial_result( + trial_dir, + task_name=task_name, + rewards=rewards, + exception=exception, + config=config, + task_path=task_path, + source=source, + agent_result=agent_result, + ) + (trial_dir / "config.json").write_text(result.config.model_dump_json(), encoding="utf-8") + (trial_dir / "result.json").write_text(result.model_dump_json(), encoding="utf-8") + return result diff --git a/packages/nemo_evaluator_sdk/tests/test_lazy_public_api.py b/packages/nemo_evaluator_sdk/tests/test_lazy_public_api.py index 771251e1c9..55065a2a54 100644 --- a/packages/nemo_evaluator_sdk/tests/test_lazy_public_api.py +++ b/packages/nemo_evaluator_sdk/tests/test_lazy_public_api.py @@ -22,10 +22,13 @@ a top-level name so that it imports nothing. """ +import asyncio +import builtins import importlib.util import json import subprocess import sys +from pathlib import Path import pytest @@ -73,6 +76,21 @@ """ +def _is_harbor_module(name: str) -> bool: + return name == "harbor" or name.startswith("harbor.") + + +def _block_harbor_import(monkeypatch: pytest.MonkeyPatch) -> None: + real_import = builtins.__import__ + + def blocked_harbor_import(name, *args, **kwargs): + if _is_harbor_module(name): + raise ModuleNotFoundError("blocked Harbor import") + return real_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", blocked_harbor_import) + + def test_agent_eval_import_does_not_pull_the_execution_stack() -> None: """The optimizer imports only ``agent_eval``; it must not pay for backends and benchmarks. @@ -97,6 +115,8 @@ def test_agent_eval_import_does_not_pull_the_execution_stack() -> None: # cached_property, so it was never on this path and asserting it would prove nothing. heavy = {"openai", "sacrebleu", "zstandard", "pyarrow", "numpy", "jinja2", "jsonschema"} & modules assert heavy == set(), f"heavy dependencies pulled into the agent_eval path: {sorted(heavy)}" + harbor_modules = sorted(name for name in modules if _is_harbor_module(name)) + assert harbor_modules == [], f"the lazy Harbor result validator was imported eagerly: {harbor_modules}" # A canary, not a spec: measured at 300 modules once both barrels went lazy, down from 1416. # The bound is deliberately close — the assertions above enumerate known offenders, so only @@ -105,6 +125,37 @@ def test_agent_eval_import_does_not_pull_the_execution_stack() -> None: assert len(modules) < 380, f"agent_eval import surface grew to {len(modules)} modules" +def test_harbor_adapter_invocation_without_extra_has_actionable_error(monkeypatch: pytest.MonkeyPatch) -> None: + from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import build_trials_from_job_dir + + _block_harbor_import(monkeypatch) + with pytest.raises(ModuleNotFoundError, match=r"optional `harbor` extra on Python >=3\.12") as exc_info: + build_trials_from_job_dir(".", []) + + assert "repository root" in str(exc_info.value) + assert "uv sync --frozen --package nemo-evaluator-sdk --extra harbor" in str(exc_info.value) + + +def test_harbor_execution_without_extra_has_actionable_error( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, _build_native_job + + _block_harbor_import(monkeypatch) + config = HarborRuntimeConfig(jobs_dir=tmp_path / "jobs") + _job_dir, run_job = _build_native_job(config, tmp_path / "dataset", None) + + async def invoke_run_job() -> None: + await run_job() + + with pytest.raises(ModuleNotFoundError, match=r"optional `harbor` extra on Python >=3\.12") as exc_info: + asyncio.run(invoke_run_job()) + + assert "repository root" in str(exc_info.value) + assert "uv sync --frozen --package nemo-evaluator-sdk --extra harbor" in str(exc_info.value) + + @pytest.mark.parametrize( ("module_name", "submodule_name"), [ diff --git a/packages/nemo_platform/pyproject.toml b/packages/nemo_platform/pyproject.toml index 83ca710f85..dfef35cc03 100644 --- a/packages/nemo_platform/pyproject.toml +++ b/packages/nemo_platform/pyproject.toml @@ -368,7 +368,7 @@ nemo-evaluator-sdk = [ nemo-experimentalist-plugin = [ "pydantic>=2", "httpx", - "harbor>=0.16", + "harbor>=0.20,<0.21", "nemo-evaluator-sdk", "opentelemetry-proto>=1.42.1", "protobuf>=6.0.0", diff --git a/plugins/nemo-experimentalist/pyproject.toml b/plugins/nemo-experimentalist/pyproject.toml index ffa9ff1eb4..008f4855c2 100644 --- a/plugins/nemo-experimentalist/pyproject.toml +++ b/plugins/nemo-experimentalist/pyproject.toml @@ -9,7 +9,7 @@ requires-python = ">=3.12,<3.14" dependencies = [ "pydantic>=2", "httpx", - "harbor>=0.16", + "harbor>=0.20,<0.21", "nemo-evaluator-sdk", "opentelemetry-proto>=1.42.1", "protobuf>=6.0.0", diff --git a/plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor_evaluator.py b/plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor_evaluator.py index bf260000d4..3d6b0b08cc 100644 --- a/plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor_evaluator.py +++ b/plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/evaluator/harbor_evaluator.py @@ -5,7 +5,7 @@ ``HarborEvaluator`` builds Harbor's ``JobConfig`` and drives ``Job`` itself. This evaluator hands that job to the SDK's ``HarborAgentTaskRunner`` instead: the SDK -owns the ``JobConfig``, the success-aware job-directory cache, and the scoped +owns the ``JobConfig``, the Harbor-valid result cache, and the scoped agent import. Harbor still does the work underneath — the difference is who owns the orchestration. @@ -84,7 +84,7 @@ class HarborRunnerConfig(EvaluatorConfig): default=None, description=( "Harbor job name. Defaults to the loop's deterministic '-', " - "which is what makes the SDK's success-aware job-dir cache usable." + "which is what makes the SDK's Harbor-valid result cache usable." ), ) jobs_dir: Path = Field( diff --git a/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/environment/Dockerfile b/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/environment/Dockerfile new file mode 100644 index 0000000000..0251107d10 --- /dev/null +++ b/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/environment/Dockerfile @@ -0,0 +1,8 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +FROM alpine:3.23 + +RUN apk add --no-cache bash + +WORKDIR /app diff --git a/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/instruction.md b/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/instruction.md new file mode 100644 index 0000000000..b30e0dc101 --- /dev/null +++ b/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/instruction.md @@ -0,0 +1,4 @@ + + + +Complete the task using the reference solution. diff --git a/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/solution/solve.sh b/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/solution/solve.sh new file mode 100755 index 0000000000..fb1d47aa0d --- /dev/null +++ b/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/solution/solve.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +sleep 10 diff --git a/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/task.toml b/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/task.toml new file mode 100644 index 0000000000..63214b745e --- /dev/null +++ b/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/task.toml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +schema_version = "1.1" + +[task] +name = "harbor/timeout-with-reward" +authors = [{ name = "NVIDIA" }] +keywords = ["error-scoring", "timeout"] + +[agent] +timeout_sec = 1.0 + +[verifier] +timeout_sec = 60.0 + +[environment] +build_timeout_sec = 600.0 +cpus = 1 +memory_mb = 512 diff --git a/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/tests/test.sh b/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/tests/test.sh new file mode 100755 index 0000000000..01811b9b65 --- /dev/null +++ b/plugins/nemo-experimentalist/tests/experimentalist/fixtures/harbor_sdk_error_scoring/dataset/timeout-with-reward/tests/test.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +mkdir -p /logs/verifier +printf '{"reward": 0.8}\n' > /logs/verifier/reward.json +cat /logs/verifier/reward.json diff --git a/plugins/nemo-experimentalist/tests/experimentalist/integration/test_sdk_harbor_error_parity.py b/plugins/nemo-experimentalist/tests/experimentalist/integration/test_sdk_harbor_error_parity.py new file mode 100644 index 0000000000..dedee07215 --- /dev/null +++ b/plugins/nemo-experimentalist/tests/experimentalist/integration/test_sdk_harbor_error_parity.py @@ -0,0 +1,111 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Real-Harbor parity for an errored attempt that still has a verifier reward.""" + +from __future__ import annotations + +import os +import subprocess +from pathlib import Path + +import pytest +from harbor.models.trial.result import TrialResult +from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import ( + HarborRewardMetric, + HarborRuntimeConfig, + run_harbor_eval, +) +from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrialStatus + +from packages.nemo_evaluator_sdk.tests.harbor_fixtures import ErrorAwareQualityMetric + +pytestmark = [ + pytest.mark.asyncio, + pytest.mark.integration, + pytest.mark.timeout(300), + pytest.mark.xdist_group("harbor-evaluator-parity"), +] + +_FIXTURE_ROOT = Path(__file__).parents[1] / "fixtures" / "harbor_sdk_error_scoring" +_TASK_NAME = "harbor/timeout-with-reward" + + +def _require_docker() -> None: + """Skip locally, but fail CI, when genuine Harbor execution is unavailable.""" + try: + completed = subprocess.run( + ["docker", "info"], + capture_output=True, + check=False, + text=True, + timeout=10, + ) + except (FileNotFoundError, subprocess.TimeoutExpired) as exc: + reason = f"Docker is unavailable for the Harbor error-scoring parity test: {exc}" + else: + if completed.returncode == 0: + return + reason = "Docker is unavailable for the Harbor error-scoring parity test" + if completed.stderr.strip(): + reason = f"{reason}: {completed.stderr.strip()}" + + if os.environ.get("CI"): + pytest.fail(reason) + pytest.skip(reason) + + +async def test_sdk_preserves_a_real_harbor_error_and_finite_reward(tmp_path: Path) -> None: + _require_docker() + jobs_dir = tmp_path / "jobs" + config = HarborRuntimeConfig( + jobs_dir=jobs_dir, + job_name="sdk-error-parity", + agent_name="oracle", + n_concurrent_trials=1, + ) + + result = await run_harbor_eval( + config, + _FIXTURE_ROOT / "dataset", + metrics=[HarborRewardMetric(), ErrorAwareQualityMetric()], + ) + + result_paths = list((jobs_dir / "sdk-error-parity").glob("*/result.json")) + assert len(result_paths) == 1 + harbor_result = TrialResult.model_validate_json(result_paths[0].read_bytes()) + assert harbor_result.task_name == _TASK_NAME + assert harbor_result.exception_info is not None + assert harbor_result.exception_info.exception_type == "AgentTimeoutError" + assert harbor_result.verifier_result is not None + assert harbor_result.verifier_result.rewards is not None + assert harbor_result.verifier_result.rewards["reward"] == 0.8 + + [trial] = result.trials + assert trial.status is AgentEvalTrialStatus.PARTIAL + assert trial.error is not None + assert trial.error.type == "AgentTimeoutError" + + harbor_rewards = { + (harbor_result.task_name, harbor_result.trial_name): harbor_result.verifier_result.rewards["reward"] + } + sdk_rewards = { + (score.task_id, score.trial_id): output.value + for score in result.scores + if score.metric_type == "harbor_reward" + for output in score.outputs + if output.name == "reward" + } + assert sdk_rewards == harbor_rewards + assert result.summary.score("harbor_reward.reward").mean == 0.8 + [quality_score] = [score for score in result.scores if score.metric_type == "error_aware_quality"] + assert quality_score.trial_id == trial.id + assert [(output.name, output.value) for output in quality_score.outputs] == [("quality", 1.0)] + assert quality_score.diagnostics == [] + assert result.summary.metric_coverage["error_aware_quality"]["quality"].model_dump() == { + "total": 1, + "scored": 1, + "failed": 0, + "missing": 0, + } + assert result.summary.error_trial_ids == {"AgentTimeoutError": [trial.id]} diff --git a/plugins/nemo-experimentalist/tests/experimentalist/test_evaluator_harbor_evaluator.py b/plugins/nemo-experimentalist/tests/experimentalist/test_evaluator_harbor_evaluator.py index 3e255f99da..c8812fecac 100644 --- a/plugins/nemo-experimentalist/tests/experimentalist/test_evaluator_harbor_evaluator.py +++ b/plugins/nemo-experimentalist/tests/experimentalist/test_evaluator_harbor_evaluator.py @@ -11,7 +11,6 @@ from __future__ import annotations import inspect -import json import sys from pathlib import Path from typing import Any @@ -36,6 +35,8 @@ ) from pydantic import ValidationError +from packages.nemo_evaluator_sdk.tests.harbor_fixtures import write_harbor_trial_result + def _write(path: Path, text: str) -> None: path.parent.mkdir(parents=True, exist_ok=True) @@ -69,18 +70,21 @@ def _write_trial( agent_result: dict[str, int] | None = None, ) -> None: trial_dir = job_dir / trial_name - _write( - trial_dir / "result.json", - json.dumps( - { - "trial_name": trial_name, - "task_name": task_name, - "task_id": {"path": str(task_dir.resolve())}, - "verifier_result": {"rewards": rewards if rewards is not None else {}}, - "exception_info": exception_info, - "agent_result": agent_result, - } - ), + exception = None + if exception_info is not None: + exception = { + "exception_type": exception_info["exception_type"], + "exception_message": exception_info.get("exception_message", ""), + "exception_traceback": exception_info.get("exception_traceback", ""), + } + write_harbor_trial_result( + trial_dir, + task_name=task_name, + rewards=rewards if rewards is not None else {}, + exception=exception, + task_path=task_dir.resolve(), + source="nemo-experimentalist-tests", + agent_result=agent_result, ) @@ -428,30 +432,32 @@ async def test_complete_cached_job_is_not_rerun( assert {trial.task_id for trial in trials} == {"sum-two", "sum-three"} -async def test_errored_cached_job_is_rerun( +async def test_harbor_valid_errored_cached_job_is_reused( tmp_path: Path, dataset: HarborDataset, agent_dir: Path, cached_job_dir: Path, fake_job: type[_FakeJob], ) -> None: - """An errored trial must force a rerun even when the cache is otherwise valid. - - Built on the *stamped* `cached_job_dir` on purpose. A hand-rolled job dir has - no fingerprint, so it is rejected as untrusted and the run happens for that - reason instead — the assertion would then hold even if error-awareness were - completely broken. Mutating one trial in place keeps the stamp valid, so the - error is the only thing left that can trigger the rerun. - """ - errored = json.loads((cached_job_dir / "sum-three__0" / "result.json").read_text(encoding="utf-8")) - errored["exception_info"] = {"exception_type": "TimeoutError"} - _write(cached_job_dir / "sum-three__0" / "result.json", json.dumps(errored)) + """A Harbor-valid errored result is a complete attempt and remains cached.""" + _write_trial( + cached_job_dir, + trial_name="sum-three__0", + task_name="hello/sum-three", + task_dir=_dataset_root(dataset) / "sum-three", + rewards={"reward": 0.8}, + exception_info={"exception_type": "TimeoutError", "exception_message": "boom"}, + ) - await HarborRunnerOutcomeEvaluator(experiment_dir=tmp_path)._run( + trials = await HarborRunnerOutcomeEvaluator(experiment_dir=tmp_path)._run( agent_dir, dataset, HarborRunnerConfig(jobs_dir=Path("jobs")) ) - assert len(fake_job.calls) == 1, "an errored cached trial must not be served from cache" + assert fake_job.calls == [] + by_task = {trial.task_id: trial for trial in trials} + assert by_task["sum-three"].status == "failed" + assert by_task["sum-three"].metrics["reward"].value == 0.8 + assert by_task["sum-three"].error == {"type": "TimeoutError", "message": "boom", "traceback": ""} async def test_under_sampled_cached_job_is_rerun( @@ -702,7 +708,7 @@ def write_results(config: Any) -> None: by_task = {trial.task_id: trial for trial in trials} assert by_task["sum-three"].status == "failed" - assert by_task["sum-three"].error == {"type": "TimeoutError", "message": "boom"} + assert by_task["sum-three"].error == {"type": "TimeoutError", "message": "boom", "traceback": ""} assert by_task["sum-two"].status == "completed" assert by_task["sum-two"].attempt == 0 diff --git a/skills/nemo-evaluator-plugin/references/agent-evaluation.md b/skills/nemo-evaluator-plugin/references/agent-evaluation.md index 6fbef91e2e..f2620712cf 100644 --- a/skills/nemo-evaluator-plugin/references/agent-evaluation.md +++ b/skills/nemo-evaluator-plugin/references/agent-evaluation.md @@ -318,11 +318,17 @@ target = HarborRunnerTarget( ) ``` -`reward_key` selects the required primary reward by name; mapping order and alphabetical order do not -select it. On a scoreable trial, a missing or unusable primary emits `0.0` with a diagnostic. Other -task-local reward keys become optional secondary outputs: finite numeric values are emitted, while -missing, Boolean, nonnumeric, NaN, or infinite values are omitted with diagnostics. A secondary -discovered for one task does not become applicable to another task. +- `reward_key` selects the required primary reward by name (default `reward`). Mapping order and + alphabetical order do not select it. +- The SDK scores only Harbor-valid `result.json` files (Harbor's `TrialResult`). A `null`, + nonnumeric string, or object in the reward mapping fails that check, so the whole attempt is + skipped and sibling rewards are not scored. Harbor writes `NaN` and infinity as `null`, which + hits this gate. +- On a Harbor-valid trial, a finite primary is emitted unchanged. A missing or unusable primary + (including Boolean) emits `0.0` with a diagnostic; the trial is still scored. +- Other keys from that task's Harbor-valid results become optional secondaries. Finite numbers are + emitted. Missing or Boolean values are omitted with a diagnostic; usable siblings are kept. +- A secondary reward discovered for one task does not apply to another task. Use `agent_import_path` for a custom Harbor agent and `agent_model_name` when the agent requires a model. The module must be importable in the execution diff --git a/skills/nemo-evaluator-plugin/references/execution.md b/skills/nemo-evaluator-plugin/references/execution.md index 3cea260f2e..5033e2bf80 100644 --- a/skills/nemo-evaluator-plugin/references/execution.md +++ b/skills/nemo-evaluator-plugin/references/execution.md @@ -267,10 +267,10 @@ job = client.evaluator.submit( ) ``` -`bundle_metric` preserves the v1 metric-bundle wire shape while encoding optional outputs: +`bundle_metric` keeps `bundle_format_version: v1` and encodes optional outputs as an additive `required` field on output entries: -- `required=True` is the default and is omitted from serialized output entries, preserving existing bundle identity. -- `required=False` is serialized explicitly on optional output entries. +- `required=True` is the default and is omitted from serialized output entries, so bundles without optional outputs are unchanged and keep their existing identity. +- `required=False` is serialized explicitly on optional output entries. Evaluator releases that predate optional outputs reject such a bundle with `outputs.N.required: Extra inputs are not permitted`, so the submitting client and the service must both run a release that supports optional outputs. - `required` must be a Boolean when supplied. Let the packager serialize the output contract. Do not add `required=True` to generated bundles by hand. diff --git a/skills/nemo-evaluator-plugin/references/troubleshooting.md b/skills/nemo-evaluator-plugin/references/troubleshooting.md index fcb0c19c43..f88f1f273f 100644 --- a/skills/nemo-evaluator-plugin/references/troubleshooting.md +++ b/skills/nemo-evaluator-plugin/references/troubleshooting.md @@ -21,7 +21,7 @@ nemo evaluator agent-evaluate explain | Agent-eval metric fails every trial with a missing template key | The metric uses the dataset-driven `item.*` context in a task-driven run | Use `inputs.*`, `reference.*`, `task.*`, `trial.*`, or `sample.output_text` | | Metric validation reports a missing required output | `compute_scores` omitted an output whose spec defaults to `required=True` | Emit the output on every scoreable trial, or set `required=False` only when absence means unmeasured or not expected on every trial | | An optional output has `missing > 0` or `nan_count > 0` | The output was omitted, the metric or trial failed, or an emitted value was non-finite | Compare `missing` and `failed` coverage, then inspect score diagnostics and the effective `count`; do not fill omissions with zero | -| Metric bundle rejects `required` or `bundle_format_version` | `required` is not Boolean, or the bundle format is not v1 | Regenerate with `bundle_metric`; it omits `required=True` and writes `required=False` explicitly | +| Metric bundle rejects `required` or `bundle_format_version` | `required` is not Boolean, the bundle format is not v1, or the reader is an evaluator release that predates optional outputs (`outputs.N.required: Extra inputs are not permitted`) | Regenerate with `bundle_metric`; it omits `required=True` and writes `required=False` explicitly. If the error names an extra `required` input, upgrade the service or job image instead | | Spec validation error | Fields do not match the current job schema | Run the matching `explain` command and validate against the spec class before submission | | Dataset row has missing fields | Jinja templates or `field_mapping` do not match row keys | Inspect one row and every referenced template before rerunning | | Standalone model/agent authentication fails | `api_key_secret` names a platform secret instead of an environment variable, or the variable is unset | Use the name of a populated local environment variable | diff --git a/uv.lock b/uv.lock index bb664f36b5..e65705a229 100644 --- a/uv.lock +++ b/uv.lock @@ -4592,7 +4592,7 @@ dev = [ [package.metadata] requires-dist = [ - { name = "harbor", marker = "python_full_version >= '3.12' and extra == 'harbor'", specifier = ">=0.16.1" }, + { name = "harbor", marker = "python_full_version >= '3.12' and extra == 'harbor'", specifier = ">=0.20,<0.21" }, { name = "httpx", specifier = ">=0.27.0,<1" }, { name = "jinja2", specifier = ">=3.1.6" }, { name = "jsonpath-ng", specifier = ">=1.7.0" }, @@ -4645,7 +4645,7 @@ dependencies = [ [package.metadata] requires-dist = [ - { name = "harbor", specifier = ">=0.16" }, + { name = "harbor", specifier = ">=0.20,<0.21" }, { name = "httpx" }, { name = "nemo-evaluator-sdk", editable = "packages/nemo_evaluator_sdk" }, { name = "nemo-insights-plugin", editable = "plugins/nemo-insights" }, @@ -5924,10 +5924,10 @@ requires-dist = [ { name = "gunicorn", marker = "extra == 'nemo-safe-synthesizer-plugin'", specifier = ">=23.0.0" }, { name = "gunicorn", marker = "extra == 'plugins'", specifier = ">=23.0.0" }, { name = "gunicorn", marker = "extra == 'services'", specifier = ">=23.0.0" }, - { name = "harbor", marker = "extra == 'all'", specifier = ">=0.16" }, - { name = "harbor", marker = "extra == 'nemo-experimentalist-plugin'", specifier = ">=0.16" }, - { name = "harbor", marker = "extra == 'plugins'", specifier = ">=0.16" }, - { name = "harbor", marker = "extra == 'services'", specifier = ">=0.16" }, + { name = "harbor", marker = "extra == 'all'", specifier = ">=0.20,<0.21" }, + { name = "harbor", marker = "extra == 'nemo-experimentalist-plugin'", specifier = ">=0.20,<0.21" }, + { name = "harbor", marker = "extra == 'plugins'", specifier = ">=0.20,<0.21" }, + { name = "harbor", marker = "extra == 'services'", specifier = ">=0.20,<0.21" }, { name = "httpx", marker = "extra == 'all'" }, { name = "httpx", marker = "extra == 'all'", specifier = ">=0.27.0" }, { name = "httpx", marker = "extra == 'all'", specifier = ">=0.27.2" },