Add repeat-level variability statistics to reward profiling

.agents/skills/nemo-gym-reward-profiling/references/output-format.md

@@ -8,6 +8,7 @@ Reward profiling is built around joining materialized inputs with rollout result
 - `rollouts.jsonl`: completed rollout results. Each row should have matching `_ng_task_index` and `_ng_rollout_index`.
 - `*_reward_profiling.jsonl`: task-level summaries produced by `gym eval profile`.
 - `*_agent_metrics.json`: agent/global aggregate metrics.
+- `*_repeat_level_metrics.json`: per-repeat summaries.
 
 ## Task Profile Rows
 
@@ -23,6 +24,32 @@ Task-level profile rows include:
 - aggregate metric keys such as `mean/reward`, `max/reward`, `min/reward`, `median/reward`, and `std/reward`.
 - token usage aggregate keys such as `mean/input_tokens`, `std/output_tokens`, and `mean/total_tokens`, when those fields are present in `response.usage`.
 
+## Repeat-Level Metrics
+
+`*_repeat_level_metrics.json` holds one entry per `(agent, _ng_rollout_index)`. Agents with a single repeat are skipped entirely.
+
+Each entry includes:
+
+- `agent_ref`, `_ng_rollout_index`
+- `sample_count`: tasks with a completed rollout in this repeat.
+- `missing_count`: tasks present in some *other* repeat but not this one. Not a count against the expected task list — see below.
+- per numeric field: `mean/`, `median/`, `std/` (ddof=1), `sem/`, `min/`, `max/`, `p25/`, `p75/`, and `ci_low_95/`+`ci_high_95/` (Student's t, omitted when `n <= 1`).
+
+`missing_count` derives its denominator from the tasks that completed in at least one repeat, not from the materialized inputs. A task that fails in *every* repeat therefore appears in no repeat, is counted nowhere, and leaves `missing_count` at `0`. Those rollouts live in `<output>_failures.jsonl`, or nowhere at all for `kill_shaped` failures (Slurm SIGTERM, Ray actor died, OOM), where the absent row is itself the signal.
+
+This makes the worst case silent: when the same tasks fail in every repeat, all repeats share an identical sample count, `missing_count` is `0` throughout, and the unequal-sample-size warning never fires — even though every statistic was computed over only the tasks that finished. For true coverage read `expected_num_rollouts`/`missing_num_rollouts` in the task profile rows, or the completion summary printed at the end of `gym eval profile`; both are measured against the materialized inputs.
+
+The per-repeat `mean/{field}` values are also summarized across repeats and merged into `*_agent_metrics.json` as `mean_across_repeats/mean/{field}`, `median_across_repeats/mean/{field}`, `se_across_repeats/mean/{field}`, and `ci_low_95_across_repeats/mean/{field}`+`ci_high_95_across_repeats/mean/{field}`.
+
+Read these keys outward-in: `se_across_repeats/mean/reward` is the standard error, across repeats, of the per-repeat mean reward. The `_across_repeats` half is the cross-repeat stat; the half after the slash is the per-repeat estimate being aggregated. 
+
+Two cases emit a `UserWarning`:
+
+- Unequal `sample_count` across repeats — statistics come from different task sets and are not directly comparable; agent metrics skew toward whichever tasks completed. Fires off `missing_count`, so it inherits the blind spot above: tasks that failed in every repeat do not trigger it.
+- Zero `sem` (all values identical) — the CI collapses to `(mean, mean)`. Reported explicitly because SciPy's `t.interval` returns `NaN` at `scale=0`.
+
+Confidence intervals are unbounded, so a 95% CI on a 0–1 reward over few tasks can fall outside `[0, 1]`. That is expected for a t-interval.
+
 `rollout_infos` are intentionally compact. They can include:
 
 - `rollout_id`, usually `task_idx:rollout_idx`

fern/versions/latest/pages/evaluation/aggregate-metrics.mdx

@@ -13,7 +13,8 @@ After rollout collection, NeMo Gym computes **aggregate metrics** for each agent
 2. **Group by agent** — responses are partitioned by agent name.
 3. **Call `/aggregate_metrics`** — for each agent, the stripped verify responses are POSTed to the agent's `/aggregate_metrics` endpoint.
 4. **Compute stats** — per-task and overall statistics (`mean`, `max`, `min`, `median`, `std`) are computed for every numeric field. If the resources server overrides `compute_metrics()` or `get_key_metrics()`, those are called to add additional metrics.
-5. **Write results** — all per-agent metrics are written to `<output>_aggregate_metrics.json`.
+5. **Compute variability stats** — when an agent has two or more repeats, per-repeat statistics are computed for every numeric field and summarized across repeats. See [Repeat-Level Metrics](#repeat-level-metrics).
+6. **Write results** — all per-agent metrics are written to `<output>_aggregate_metrics.json`.
 
 ## Output Format
 
@@ -28,14 +29,24 @@ The output file is a JSON array with one entry per agent:
       "max/reward": 1.0,
       "min/reward": 0.0,
       "median/reward": 1.0,
-      "std/reward": 0.433
+      "std/reward": 0.433,
+
+      "mean_across_repeats/mean/reward": 0.75,
+      "median_across_repeats/mean/reward": 0.75,
+      "se_across_repeats/mean/reward": 0.042,
+      "ci_low_95_across_repeats/mean/reward": 0.66,
+      "ci_high_95_across_repeats/mean/reward": 0.84
     },
     "key_metrics": {
       "mean/reward": 0.75
     },
     "group_level_metrics": [
       {"mean/reward": 1.0, "sample": {"...": "..."}},
       {"mean/reward": 0.5, "sample": {"...": "..."}}
+    ],
+    "repeat_level_metrics": [
+      {"_ng_rollout_index": 0, "mean/reward": 0.72, "sem/reward": 0.09, "...": "..."},
+      {"_ng_rollout_index": 1, "mean/reward": 0.78, "sem/reward": 0.09, "...": "..."}
     ]
   }
 ]
@@ -44,9 +55,90 @@ The output file is a JSON array with one entry per agent:
 | Field | Description |
 |---|---|
 | `agent_ref` | Agent identity (`{"name": "..."}`) |
-| `agent_metrics` | Overall stats across all rollouts, plus any custom metrics from `compute_metrics()` |
+| `agent_metrics` | Overall stats across all rollouts, plus any custom metrics from `compute_metrics()`, plus cross-repeat aggregates |
 | `key_metrics` | Headline numbers (default: all `mean/*` entries from `agent_metrics`) |
 | `group_level_metrics` | Per-task breakdown — one entry per task with stats across that task's rollouts |
+| `repeat_level_metrics` | Per-repeat breakdown — one entry per `_ng_rollout_index` with stats across that repeat's tasks. Empty unless the agent has two or more repeats |
+
+---
+
+## Repeat-Level Metrics
+
+A single run is a point estimate. Collecting repeats (`--num-repeats`) lets you separate a real score difference from sampling noise, and NeMo Gym reports that variability at three levels.
+
+### Within a repeat, across tasks
+
+`repeat_level_metrics` holds one entry per `(agent, repeat)`. It is produced **only for agents with two or more repeats** — an agent with a single repeat has nothing to compare against and is skipped. When no agent qualifies, the list is empty.
+
+Each entry carries:
+
+| Field | Description |
+|---|---|
+| `agent_ref` | Agent identity |
+| `_ng_rollout_index` | Which repeat this entry summarizes |
+| `sample_count` | Tasks with a completed rollout in this repeat |
+| `missing_count` | Tasks present in some **other** repeat but not this one — see the caveat below |
+
+plus, for **every numeric field** (`reward`, token usage, and any numeric field your verifier returns):
+
+| Key | Description |
+|---|---|
+| `mean/{field}`, `median/{field}` | Central tendency across this repeat's tasks |
+| `std/{field}` | Sample standard deviation (`ddof=1`) |
+| `sem/{field}` | Standard error of the mean — `std / sqrt(n)` |
+| `min/{field}`, `max/{field}` | Range |
+| `p25/{field}`, `p75/{field}` | Quartiles |
+| `ci_low_95/{field}`, `ci_high_95/{field}` | 95% confidence interval (Student's t). Omitted when `n <= 1` |
+
+<Warning>
+`missing_count` is **not** measured against the expected task list. Its denominator is the set of tasks that completed in at least one repeat, so it only catches tasks that succeeded somewhere else and are absent here.
+
+A task that fails in *every* repeat never appears in any repeat, so it is counted nowhere and `missing_count` stays `0` for it. Those rollouts are in `<output>_failures.jsonl` — or, for `kill_shaped` failures (Slurm SIGTERM, Ray actor died, OOM), nowhere at all, since the absence of a row is itself the signal.
+
+To check real coverage, use the numbers that *are* measured against the materialized inputs: `expected_num_rollouts` and `missing_num_rollouts` in `group_level_metrics`, and the completion summary `gym eval profile` prints at the end of a run.
+</Warning>
+
+### Across repeats
+
+The per-repeat `mean/{field}` values are themselves summarized and merged into `agent_metrics`, treating each repeat as one observation. This answers a different question than the keys above: not "how much do tasks vary within a repeat" but "how much does the headline score move if I run the whole benchmark again."
+
+| Key | Description |
+|---|---|
+| `mean_across_repeats/mean/{field}` | Mean of the per-repeat means |
+| `median_across_repeats/mean/{field}` | Median of the per-repeat means |
+| `se_across_repeats/mean/{field}` | Standard error across repeats  |
+| `ci_low_95_across_repeats/mean/{field}`, `ci_high_95_across_repeats/mean/{field}` | 95% confidence interval of that mean |
+
+
+<Note>
+`mean_across_repeats/mean/{field}` and the per-rollout `mean/{field}` answer different questions but coincide numerically when every repeat covers the same tasks. 
+</Note>
+
+### Per task, across repeats
+
+`group_level_metrics` reports `num_rollouts`, `mean`, `median`, and `std` per task. It deliberately carries **no confidence interval**: a CI here would require assuming a distribution for a single task's repeated outcomes, which are frequently binomial rather than normal, and the Central Limit Theorem does not rescue it because these are raw outcomes rather than averages.
+
+### Two cases worth knowing about
+
+Both emit a `UserWarning`, so you will see them in your terminal:
+
+- **Unequal sample counts across repeats.** If a task is missing from some repeats (a crashed rollout, an interrupted run), each repeat's statistics are computed over a different task set, so they are not directly comparable — and `agent_metrics` skews toward whichever tasks happened to complete. Collect the missing rollouts before drawing conclusions. Note this fires off `missing_count`, so it inherits the blind spot above: tasks that failed in *every* repeat do not trigger it.
+- **Zero standard error.** If every value in a sample is identical, the confidence interval collapses to the single point `(mean, mean)`. This is reported rather than left null, because SciPy's `t.interval` computes an indeterminate `±inf * 0` at `scale=0` and returns `NaN`.
+
+<Tip>
+Confidence intervals are unbounded, so on a small number of tasks a 95% CI for a 0–1 reward can extend below 0 or above 1. That is expected for a t-interval and is a signal that you need more tasks or repeats, not a bug.
+</Tip>
+
+### Where it lands on disk
+
+The same statistics are laid out differently depending on which command produced them:
+
+| Command | File | Layout |
+|---|---|---|
+| `gym eval run`, `gym eval aggregate` | `<output>_aggregate_metrics.json` | `repeat_level_metrics` nested inside each agent's object |
+| `gym eval profile` | `<rollouts>_repeat_level_metrics.json` | A separate flat list, each entry carrying its own `agent_ref` |
+
+`gym eval profile` always writes its file, containing `[]` when there is only one repeat. Either way, the cross-repeat aggregates (`mean_across_repeats/mean/*`, `se_across_repeats/mean/*`, and the CI bounds) live in `agent_metrics`.
 
 ---
 

fern/versions/latest/pages/reference/cli-commands.mdx

@@ -719,6 +719,16 @@ gym eval profile \
     --rollouts rollouts.jsonl
 ```
 
+Writes three files next to the rollouts, named from its stem:
+
+| File | Contents |
+| --- | --- |
+| `<rollouts>_reward_profiling.jsonl` | One row per task. |
+| `<rollouts>_agent_metrics.json` | One entry per agent. |
+| `<rollouts>_repeat_level_metrics.json` | One entry per repeat. |
+
+See [Repeat-Level Metrics](/evaluation/aggregate-metrics#repeat-level-metrics) for the full field list and how to read the variability statistics.
+
 ### `gym eval reverify`
 
 Recompute rewards from existing rollouts by replaying them through a resources server's `/verify` endpoint — without re-running model inference. Starts the resources server automatically from the provided config. Requires the `*_materialized_inputs.jsonl` and `rollouts.jsonl` artifacts produced by `gym eval run`.

nemo_gym/cli/eval.py

@@ -463,16 +463,17 @@ def reward_profile():  # pragma: no cover
     results.sort(key=lambda r: (r[TASK_INDEX_KEY_NAME], r[ROLLOUT_INDEX_KEY_NAME]))
 
     rp = RewardProfiler()
-    group_level_metrics, agent_level_metrics = rp.profile_from_data(
+    group_level_metrics, agent_level_metrics, repeat_level_metrics = rp.profile_from_data(
         rows, results, allow_partial_rollouts=config.allow_partial_rollouts
     )
     completion_summary = rp.profile_completion_summary(rows, results)
-    reward_profiling_fpath, agent_level_metrics_fpath = rp.write_to_disk(
-        group_level_metrics, agent_level_metrics, Path(config.rollouts_jsonl_fpath)
+    reward_profiling_fpath, agent_level_metrics_fpath, repeat_level_metrics_fpath = rp.write_to_disk(
+        group_level_metrics, agent_level_metrics, repeat_level_metrics, Path(config.rollouts_jsonl_fpath)
     )
 
     print(f"""Profiling outputs:
 Reward profile completion: {completion_summary["completed_rollout_rows"]}/{completion_summary["expected_rollout_rows"]} rollout rows ({completion_summary["reward_profile_completion_pct"]:.2f}%)
 Input rows: {completion_summary["total_input_rows"]} total; {completion_summary["complete_input_rows"]} complete; {completion_summary["partial_input_rows"]} partial; {completion_summary["missing_input_rows"]} without rollouts dropped from output.
 Reward profiling outputs: {reward_profiling_fpath}
-Agent-level metrics: {agent_level_metrics_fpath}""")
+Agent-level metrics: {agent_level_metrics_fpath}
+Repeat-level metrics: {repeat_level_metrics_fpath}""")

nemo_gym/config_types.py

@@ -788,6 +788,10 @@ class AggregateMetrics(BaseModel):
         default_factory=dict,
         description="Headline metrics for this benchmark. Subset of agent_metrics.",
     )
+    repeat_level_metrics: List[Dict[str, Any]] = Field(
+        default_factory=list,
+        description="Per-repeat summary stats (one dict per rollout_index).",
+    )
 
 
 ########################################