Add repeat-level variability statistics to reward profiling - #2474

Open

mpatel31415 wants to merge 33 commits into

mainfrom

mpatelka/add-variability-statistics

mpatel31415 commented Aug 11, 2026 •

edited

Loading

Contributor

Issue this PR addresses
Related new bug

Adds a per-repeat statistics to reward profiling / aggregation, so a score can be
reported with a measure of how much it moves between whole runs rather than as
a bare point estimate.

What's new

repeat_level_metrics — one entry per (agent, repeat), computed when an
agent has two or more completed repeats. Each entry carries sample_count,
missing_count, and per numeric field: mean, median, std (ddof=1), sem,
min/max, p25/p75, and a 95% Student's-t confidence interval.

Cross-repeat aggregates — the per-repeat means are summarized across repeats
and merged into agent_metrics as mean_across_repeats/mean/{field},
median_across_repeats/…, se_across_repeats/…, and the 95% CI bounds.

Note

Per task across samples aggregates deliberately gets no CI — a per-task distribution over
repeated outcomes can't be assumed normal, and the CLT doesn't apply to raw
outcomes, so their computation / formula is benchmark and metric specific.


          feat(eval): initial changes

f7928f8

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>

copy-pr-bot Bot commented Aug 11, 2026

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

mpatel31415 added 10 commits

August 12, 2026 11:40


          feat(eval): add confidence intervals

751c4bf

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          Merge branch 'mpatelka/add-variability-statistics-2' into mpatelka/ad…

fcde208

…d-variability-statistics

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          Merge branch 'main' into mpatelka/add-variability-statistics

4b520aa

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          feat(eval): final draft of changes

b6ca9dd

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          feat(eval): added grouping by agent

24c5a2f

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          added new metrics to tests

3737bf9

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          feat(eval): added warning if data are missing

856024b

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          final version of the code

82d7b61

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          feat(eval): saving the data in rsults

36c2423

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          feat(eval): adding docs

54c61c0

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>

github-actions Bot commented Aug 18, 2026

Contributor

🌿 Preview your docs: https://nvidia-preview-mpatelka-add-variability-statistics.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

mpatel31415 and others added 2 commits

August 18, 2026 10:54


          fix(eval): manual cleaning of the code

8d592ae

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          Merge branch 'main' into mpatelka/add-variability-statistics

15cbf1b

mpatel31415 changed the title ~~feat(eval): initial changes~~ Add repeat-level variability statistics to reward profiling

mpatel31415 marked this pull request as ready for review

August 18, 2026 09:51

copy-pr-bot Bot temporarily deployed to public

August 18, 2026 09:51

Inactive

copy-pr-bot Bot temporarily deployed to public

August 18, 2026 09:53

Inactive

copy-pr-bot Bot temporarily deployed to public

August 18, 2026 09:53

Inactive

copy-pr-bot Bot temporarily deployed to public

August 18, 2026 09:54

Inactive

mpatel31415 added 2 commits

August 18, 2026 13:04


          fix(eval): removing constant agent ref

25a1b0d

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          Merge remote-tracking branch 'refs/remotes/upstream/mpatelka/add-vari…

f8fa29e

…ability-statistics' into mpatelka/add-variability-statistics

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>

mpatel31415 requested a review from a team as a code owner

August 18, 2026 11:05

copy-pr-bot Bot temporarily deployed to public

August 18, 2026 11:06

Inactive

copy-pr-bot Bot temporarily deployed to public

August 18, 2026 11:07

Inactive

copy-pr-bot Bot temporarily deployed to public

August 18, 2026 11:07

Inactive

copy-pr-bot Bot temporarily deployed to public

August 18, 2026 11:08

Inactive


          fix(eval): change scipy version

d84dd91

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>

copy-pr-bot Bot temporarily deployed to public

August 18, 2026 11:50

Inactive

mpatel31415 added 6 commits

August 25, 2026 09:04


          final version of the code

f7bb410

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          feat(eval): saving the data in rsults

2f25e23

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          feat(eval): adding docs

1558e1d

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          fix(eval): manual cleaning of the code

73c1d7d

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          fix(eval): removing constant agent ref

0b64f09

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          fix(eval): change scipy version

38bb207

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>

mpatel31415 force-pushed the mpatelka/add-variability-statistics branch from 78b9cae to 38bb207 Compare

August 25, 2026 07:04

copy-pr-bot Bot deployed to public

August 25, 2026 07:04

Active

copy-pr-bot Bot deployed to public

August 25, 2026 07:06

Active

copy-pr-bot Bot deployed to public

August 25, 2026 07:06

Active

copy-pr-bot Bot deployed to public

August 25, 2026 07:06

Active

mpatel31415 commented Aug 25, 2026

Contributor Author

Non-blocking scope note: #2528 requests three aggregation levels. This PR covers the main repeat-level mechanics, but a few requested outputs remain absent:

within-repeat output has sample_count and observed-task missing_count, but no separate valid/failed counts;

across-repeat output does not emit the calculated between-repeat standard deviation, min, or max;

existing per-task output covers repeat count, mean, median, and standard deviation, but the requested per-task CI is explicitly left out.

I do not think this PR needs to grow to include all of these, but could we state explicitly that it implements a subset of #2528 and leave the remaining pieces for follow-ups? That would prevent the issue from appearing fully satisfied when this merges.

I added across-repeat output between-repeat standard deviation, min, or max;
Linear issue comment describes things that are not present

mpatel31415 added 2 commits

August 25, 2026 10:01


          fix(eval): adding missing stats after review

7ded3d4

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>


          Merge remote-tracking branch 'refs/remotes/upstream/mpatelka/add-vari…

4cbece5

…ability-statistics' into mpatelka/add-variability-statistics

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>

copy-pr-bot Bot deployed to public

August 25, 2026 08:06

Active

copy-pr-bot Bot deployed to public

August 25, 2026 08:07

Active

copy-pr-bot Bot deployed to public

August 25, 2026 08:08

Active

copy-pr-bot Bot deployed to public

August 25, 2026 08:08

Active

giuliolovisotto approved these changes

View reviewed changes


          Merge branch 'main' into mpatelka/add-variability-statistics

f4939d9

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>

copy-pr-bot Bot deployed to public

August 25, 2026 11:00

Active

copy-pr-bot Bot deployed to public

August 25, 2026 11:02

Active

copy-pr-bot Bot deployed to public

August 25, 2026 11:02

Active

copy-pr-bot Bot deployed to public

August 25, 2026 11:03

Active


          Merge branch 'main' into mpatelka/add-variability-statistics

9116eda

Signed-off-by: Martyna Patelka <mpatelka@nvidia.com>

copy-pr-bot Bot deployed to public

August 26, 2026 07:41

Active

copy-pr-bot Bot deployed to public

August 26, 2026 07:43

Active

copy-pr-bot Bot deployed to public

August 26, 2026 07:43

Active

copy-pr-bot Bot deployed to public

August 26, 2026 07:44

Active

ko3n1g approved these changes

View reviewed changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sla:review-overdue