NVIDIA-NeMo · ngoncharenko · Sep 9, 2026 · Sep 9, 2026
@@ -21,8 +21,17 @@ Unlike the quickstart, this runner is **not** zero-dependency — it shells out
 
 - **Python ≥ 3.12**
 - **Docker and docker daemon** installed and running
-- **Harbor**, installed separately: `uv pip install "harbor>=0.16.1"`. Harbor is intentionally **not**
-  a dependency of `nemo-platform[nemo-evaluator-sdk]`, so the rest of the SDK stays lightweight.
+- **Harbor**, installed from a NeMo Platform source checkout as shown below. Harbor is
+  imported lazily, so the base SDK remains usable on Python 3.11 without it; Harbor execution and
+  existing-result adaptation require Python 3.12 or newer.
+
+The nemo-evaluator-sdk is not published as a standalone PyPI package. Use a NeMo Platform source checkout
+(see the repository's [setup guide](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/SETUP.md)
+for toolchain prerequisites). From the repository root, install the Harbor extra:
+
+```bash
+uv sync --frozen --package nemo-evaluator-sdk --extra harbor
+```
 
 The runner raises a clear error pointing at this install step if `harbor` is missing.
 
@@ -107,7 +116,7 @@ scores each trial's named rewards with `HarborRewardMetric`.
 
 **`dataset_path` is input; `jobs_dir` is output.** The dataset is your read-only task suite. `jobs_dir`
 is a directory the runner writes into — Harbor's per-trial results land under `jobs_dir/<job_name>/`,
-and that directory doubles as a re-run cache (see [below](#attempts-concurrency-and-caching)).
+and that directory doubles as a re-run cache (see [below](#caching)).
 
 </Note>
 
@@ -139,9 +148,13 @@ Swap `agent_name` (or `agent_import_path`) for your own agent to get a real scor
    Mapping order and alphabetical order never select the primary.
 3. The primary output is required. On a scoreable trial, a finite numeric value is emitted unchanged;
    a missing or unusable primary emits `0.0` and a diagnostic instead of skipping the trial.
-4. Other reward keys discovered for that task become optional secondary outputs. Finite numeric
-   values are emitted. Missing, Boolean, nonnumeric, NaN, or infinite values are omitted with a
-   diagnostic. A secondary discovered for one task does not become applicable to another task.
+4. Other keys from that task's Harbor-valid results become optional secondaries:
+   - Finite numbers are emitted.
+   - Missing or Boolean values are omitted with a diagnostic; usable siblings are kept.
+   - A `null`, nonnumeric string, or object in the reward mapping fails Harbor's `TrialResult`
+     check, so the whole attempt is skipped and sibling rewards are not scored. Harbor writes
+     `NaN` and infinity as `null`, which hits this gate.
+   - A secondary reward key discovered for one task does not apply to another task.
 5. `result.summary` aggregates each named output, `result.trials` holds each trial's status and
    evidence, and `result.persist()` writes the standard run bundle.
 
@@ -156,14 +169,18 @@ and validates that file before writing `result.json`:
 - A JSON Boolean is coerced to `1.0` or `0.0`.
 - `NaN` and infinity pass Harbor's numeric validation, but serialize as `null` in `result.json`.
 
-The SDK adapter reads `result.json` per key. It keeps finite numbers and accepts numeric strings in
-imported or manually authored result files. It omits a Boolean, `null`, unparseable value, NaN, or
-infinity with a diagnostic while retaining usable sibling rewards.
+The SDK first validates the whole `result.json` with Harbor's `TrialResult`. A `null`, nonnumeric
+string, or object in the reward mapping invalidates the attempt; its sibling rewards are not scored
+or cached. For a Harbor-valid result, the SDK parses each reward per key. It emits finite numbers,
+including numeric strings normalized by Harbor, and omits Boolean or non-finite values with a
+diagnostic while retaining usable siblings.
 
 Emit finite JSON numbers from the verifier. Use `1` and `0` when a reward represents pass/fail; do
 not rely on Harbor's coercion of strings or Booleans.
 
-### Sparse secondary example
+### Sparse secondary and errored reward examples
+
+#### Missing secondary reward
 
 For two attempts on task A:
 
@@ -172,38 +189,132 @@ a1: reward=1.0, format_ok=1.0
 a2: reward=0.0, format_ok omitted
 ```
 
-| Consumer | Expected result |
+**Resulting aggregates and coverage**
+
+| Result field | Expected value |
 |---|---|
 | `harbor_reward.reward` | `mean=0.5`, `count=2`, `nan_count=0` |
 | `harbor_reward.format_ok` | `mean=1.0`, `count=1`, `nan_count=1` |
 | `format_ok` coverage | `total=2`, `scored=1`, `missing=1`, `failed=0` |
-| `harbor_reward.format_ok.pass@1` | `mean=1.0`; measured `n=1` |
+| `harbor_reward.format_ok.pass@1` | `mean=1.0`, `count=1`, `nan_count=0`; computed from one measured attempt |
 | `harbor_reward.format_ok.pass@2` | unestimable: `count=0`, `nan_count=1`, `mean=None` |
 
-Omission means unmeasured, not failure. It does not create a persisted null, NaN, or zero. See
-[Reading Results](/documentation/evaluate-models/agent-eval/reading-results) for denominator, failed
-trial, semantic-view, and persistence behavior.
+Omission means unmeasured, not failure:
+
+- It does not create a null, NaN, or zero per-trial metric output.
+- Coverage records the missing measurement.
+- Derived aggregates can therefore be unestimable (`mean=None`).
+
+See [Reading Results](/documentation/evaluate-models/agent-eval/reading-results) for denominator,
+failed-trial, semantic-view, and persistence behavior.
+
+#### Errored attempt with a valid primary reward
+
+For two attempts on task A:
+
+```text
+a1: reward=1.0, no error
+a2: reward=0.8, error=RuntimeError
+```
+
+**Resulting statuses and aggregate**
+
+| Result field | Expected value |
+|---|---|
+| SDK trial `a1` | `COMPLETED` |
+| SDK trial `a2` | `PARTIAL`; remains scoreable |
+| `harbor_reward.reward` | `mean=0.9`, `count=2`, `nan_count=0` |
+| `error_trial_ids` | `{"RuntimeError": ["a2"]}` |
+
+The error changes `a2`'s status and error rollup; it does not discard or replace its finite reward.
+
+## Attempts and concurrency
+
+- **`n_attempts`** — desired attempts per task. On resume, Harbor runs only the missing attempts.
+- **`n_concurrent_trials`** — maximum trials Harbor runs concurrently.
+
+## Retries
+
+- **`max_retries`** — maximum extra attempts per trial during the current Harbor run; defaults to `0`.
+- **Error policy** — Harbor retries only allowed errors. Its
+  [default non-retryable errors](https://github.com/harbor-framework/harbor/blob/v0.20.0/src/harbor/models/job/config.py#L288-L301)
+  include `AgentTimeoutError`.
+- **Repeated SDK calls** — `max_retries` never reopens cached errored trials. Cache behavior is
+  described below.
 
-## Attempts, concurrency, and caching
+## Caching
 
-- **`n_attempts`** — trials Harbor runs per task; **`n_concurrent_trials`** — how many run at once.
-- The job directory doubles as a **cache**. Pin a stable `job_name` and a completed job whose results
-  already cover every requested task is re-scored instead of re-run. The default timestamped `job_name`
-  always runs fresh; set `force_rerun=True` to delete an existing job dir first.
+Caching activates only when `HarborRuntimeConfig.job_name` is pinned. Without it, every call creates a
+fresh timestamped job directory.
+
+A cache hit requires both:
+
+- A usable matching stamp for the requested inputs.
+- At least `n_attempts` Harbor-valid results for every requested task.
+
+On a repeated SDK call:
+
+- **Cache hit** — skips Harbor and Docker, then re-scores existing results. Harbor-valid errored results
+  count as completed, become `PARTIAL`, and are not rerun.
+- **Incomplete matching cache** — preserves valid results, including errored results, then runs only
+  attempts with missing or invalid results.
+- **Stale or unusable cache** — deletes the job directory and reruns every requested attempt. This
+  includes changed inputs, a missing or malformed stamp, and an unresolved requested task directory.
+- **`force_rerun=True`** — deletes the entire pinned job directory, then reruns every requested attempt.
+- **Selective cached-error rerun** — unsupported. The SDK cannot rerun cached attempts by error type.
+- **Concurrent processes** — must not share a pinned `job_name`; neither the SDK nor Harbor locks the
+  job directory.
+
+### Cache identity
+
+The SDK stores the cache stamp in `jobs_dir/<job_name>/.nemo-eval-harbor-cache.json`:
+
+- **`version`** — stamp schema; must match exactly.
+- **`options`** — SHA-256 of result-affecting `HarborRuntimeConfig` fields; must match exactly.
+- **`agent`** — digest of `agent_dir` contents, or `"<none>"` when unset; must match exactly.
+- **`tasks`** — every requested task digest must match; extra cached tasks are ignored. Cached A, B, C
+  can serve A, but cached A cannot serve A and B.
+- **Scoring and scheduling settings** — `reward_key`, selected metrics, `quiet`, and
+  `n_concurrent_trials` do not invalidate Harbor execution results.
+- **Installed agents** — when `agent_dir` is unset, the stamp covers the agent selection or import path,
+  not the installed package contents. Change the import path, use `agent_dir`, or force a rerun after
+  changing installed agent code.
+
+### Force a complete rerun
+
+```python
+import asyncio
+from pathlib import Path
+
+from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval
+
+config = HarborRuntimeConfig(
+    jobs_dir=Path("./harbor-jobs"),
+    job_name="my-suite",
+    agent_name="oracle",
+    force_rerun=True,
+)
+result = asyncio.run(run_harbor_eval(config, dataset_path=Path("path/to/my-suite")))
+```
+
+- **Keep the previous job** — use a new `job_name`, or omit it to create a fresh timestamped directory.
 
 ## Shortcut: `run_harbor_eval`
 
 When a run is exactly "one Harbor suite, scored by its reward," `run_harbor_eval` collapses the three
 steps above — discover, run, score — into a single call:
 
 ```python
+import asyncio
 from pathlib import Path
 
 from nemo_evaluator_sdk.agent_eval.runtimes.harbor_runtime import HarborRuntimeConfig, run_harbor_eval
 
-result = await run_harbor_eval(
-    HarborRuntimeConfig(jobs_dir=Path("./harbor-jobs"), agent_name="oracle"),
-    dataset_path=Path("path/to/my-suite"),
+result = asyncio.run(
+    run_harbor_eval(
+        HarborRuntimeConfig(jobs_dir=Path("./harbor-jobs"), agent_name="oracle"),
+        dataset_path=Path("path/to/my-suite"),
+    )
 )
 ```
 

@@ -8,6 +8,17 @@ Run a Harbor **local dataset directory**,
 example scores with Harbor's deterministic **oracle** agent, so it needs no model
 or API key — only the `harbor` extra installed and a working Docker daemon.
 
+## Install
+
+The base SDK supports Python ≥ 3.11, while Harbor-backed execution and result
+adaptation require Python ≥ 3.12. The SDK is not published as a standalone PyPI
+package. Use a NeMo Platform source checkout; see [SETUP.md](../../../../SETUP.md)
+for toolchain prerequisites. From the repository root, install the optional extra:
+
+```bash
+uv sync --frozen --package nemo-evaluator-sdk --extra harbor
+```
+
 ## Minimal plumbing
 
 The SDK owns the Harbor plumbing. Apart from imports, running a whole dataset is
@@ -24,18 +35,8 @@ result = await run_harbor_eval(config, "hello_world_dataset")  # loads tasks, ru
 
 `run_harbor_eval` discovers the tasks, builds and runs Harbor's `JobConfig`, and
 scores each task with `HarborRewardMetric` — the caller never imports `harbor` or
-assembles a job. `harbor` is imported lazily inside the runtime, so importing the
-SDK never requires it.
-
-## Install
-
-Harbor is imported lazily and is **not** in the SDK's locked dependencies (it
-requires Python ≥ 3.12 while the workspace supports ≥ 3.11, like `nemo_fabric`).
-Install it separately into the environment that runs the example:
-
-```bash
-uv pip install "harbor>=0.16.1"
-```
+assembles a job. `harbor` is imported lazily: the base SDK does not require it,
+but Harbor execution and existing-result adaptation do.
 
 ## The dataset directory (how Harbor tasks are found)
 
@@ -74,8 +75,10 @@ The runtime is [`harbor_runtime.py`](../../src/nemo_evaluator_sdk/agent_eval/run
 ### Re-running and caching
 
 In native mode the `job_dir` doubles as a cache: if every requested task already
-has `n_attempts` completed (non-errored) results there, the Harbor run is skipped
-and the results are re-adapted instead. This only engages when you **pin a stable
+has `n_attempts` Harbor-valid results there, the Harbor run is skipped
+and the results are re-adapted instead. Valid errored results count because Harbor
+also treats them as completed attempts; their SDK trials remain `PARTIAL` and
+scoreable. This only engages when you **pin a stable
 `job_name`** on the config — the default `job_name` is a timestamp, so each run
 writes a fresh dir and never hits the cache. Set `force_rerun=True` to delete the
 job dir and re-run unconditionally.

@@ -9,6 +9,24 @@ runner. LAB ships **raw** tasks (`tasks/**/task.json` + `documents/`); this exam
 — it downloads the pinned source and *generates* the Harbor suite itself, then runs and scores it with
 one `AgentEvaluator` call.
 
+## Prerequisites, seams & caveats
+
+- **Not zero-dependency**: Python ≥ 3.12, Docker, and a NeMo Platform source checkout
+  (see [SETUP.md](../../../../SETUP.md) for toolchain prerequisites). The SDK is not
+  published as a standalone PyPI package. From the repository root, install Harbor with
+  `uv sync --frozen --package nemo-evaluator-sdk --extra harbor`. Harbor native runtime is early-access.
+- **Reproducing LAB's official reference-agent number** additionally requires wiring **LAB's reference
+  agent** (as an `--agent-import-path` adapter) and
+  LAB's **exact** `rubric_criterion` judge prompt into `lab_verify.py`. Out of the box this generates a
+  *runnable, faithful-in-shape* suite; treat scores as comparable-in-method until you drop those in.
+- **Agent-output seam**: `prepare_lab_suite.py --run-dir` sets where the verifier reads the agent's
+  deliverables (default `/logs/agent/artifacts/lab-run`, LAB's reference-agent location). Point it at
+  wherever your chosen Harbor agent writes.
+- **`scores.json` schema**: `LabCriteriaMetric` reads `n_criteria`, `n_passed`, `all_pass`,
+  `judge_error_count`, `criteria_results[].verdict` — exactly what `lab_verify.py` writes.
+- **Scale**: the SDK runs tasks with async concurrency locally (or a single-container platform job).
+  For the full 1,749-task sweep, prefer the governed platform job over a local run.
+
 ## Files
 
 - [`prepare_lab_suite.py`](prepare_lab_suite.py) — self-contained: downloads + SHA-verifies the pinned
@@ -69,21 +87,5 @@ lab_criteria.judge_error_count:  mean=0.0    # treat > 0 as an infra failure, no
 view.legal_quality:              mean=0.60   # MEAN(reward, criteria_pass_rate)
 ```
 
-## Prerequisites, seams & caveats
-
-- **Not zero-dependency**: Python ≥ 3.12, Docker, and `harbor` installed separately
-  (`uv pip install "harbor>=0.16.1"`). Harbor native runtime is early-access.
-- **Reproducing LAB's official reference-agent number** additionally requires wiring **LAB's reference
-  agent** (as an `--agent-import-path` adapter) and
-  LAB's **exact** `rubric_criterion` judge prompt into `lab_verify.py`. Out of the box this generates a
-  *runnable, faithful-in-shape* suite; treat scores as comparable-in-method until you drop those in.
-- **Agent-output seam**: `prepare_lab_suite.py --run-dir` sets where the verifier reads the agent's
-  deliverables (default `/logs/agent/artifacts/lab-run`, LAB's reference-agent location). Point it at
-  wherever your chosen Harbor agent writes.
-- **`scores.json` schema**: `LabCriteriaMetric` reads `n_criteria`, `n_passed`, `all_pass`,
-  `judge_error_count`, `criteria_results[].verdict` — exactly what `lab_verify.py` writes.
-- **Scale**: the SDK runs tasks with async concurrency locally (or a single-container platform job).
-  For the full 1,749-task sweep, prefer the governed platform job over a local run.
-
 For the **task-driven, bring-your-own-agent** counterpart (native `AgentEvalTask`s + Fabric + a rubric
 *metric* instead of an in-container verifier), see [`../legal_agent_bench_fabric`](../legal_agent_bench_fabric).
@@ -25,8 +25,10 @@
 Prerequisites:
 
 * Python >= 3.12 and a running Docker daemon.
-* Harbor, installed separately: ``uv pip install "harbor>=0.16.1"`` (kept out of the
-  SDK's lock so importing the SDK stays lightweight).
+* A NeMo Platform source checkout (see SETUP.md for toolchain prerequisites).
+  From the repository root, install Harbor with
+  ``uv sync --frozen --package nemo-evaluator-sdk --extra harbor``. The SDK is not
+  published as a standalone PyPI package. Harbor is imported lazily.
 * A prepared LAB Harbor suite on disk — a directory of task folders. Generate it
   with the bundled, self-contained ``prepare_lab_suite.py`` (pinned download +
   Harbor-task generation; see the example README).

@@ -57,14 +57,12 @@ agent-runtimes = [
   "openai-agents[docker]>=0.17.3,<0.18",
 ]
 engine = []
-# The native Harbor runtime (HarborAgentTaskRunner / run_harbor_eval) needs `harbor`, which requires
-# Python >=3.12 while this workspace floor is >=3.11. The `python_version >= '3.12'` marker (matching
-# harbor's own requires-python) scopes the dependency to the 3.12+ fork of the universal lock, so it
-# installs on 3.12+ and is simply omitted on 3.11 — no floor bump for the rest of the workspace.
-# `harbor` is imported lazily (only inside HarborAgentTaskRunner.run_tasks), so importing the SDK and
-# resolving a HarborRunnerTarget stay valid on 3.11; only *executing* a Harbor job needs this extra.
+# Native Harbor execution and result adaptation need `harbor`, which requires Python >=3.12 while
+# the SDK floor remains >=3.11. The marker scopes the dependency to compatible interpreters.
+# Harbor is imported lazily, so importing the SDK and constructing Harbor configuration remain
+# valid on Python 3.11; invoking either execution or result adaptation requires this extra.
 harbor = [
-  "harbor>=0.16.1; python_version >= '3.12'",
+  "harbor>=0.20,<0.21; python_version >= '3.12'",
 ]
 nemo-platform = [
   "nemo-platform-sdk",