Skip to content

fix(arc_agi_3): exit non-zero when the harness produces no result.json - #180

Merged
esarafian merged 2 commits into
NVIDIA-NeMo:mainfrom
ShichengRao:fix/run-solver-exit-code
Sep 10, 2026
Merged

fix(arc_agi_3): exit non-zero when the harness produces no result.json#180
esarafian merged 2 commits into
NVIDIA-NeMo:mainfrom
ShichengRao:fix/run-solver-exit-code

Conversation

@ShichengRao

Copy link
Copy Markdown
Contributor

What does this PR do?

run_solver.main() prints an error when the harness exits without writing result.json — and then falls through to return 0. Any wrapper or CI job reading the exit code sees success for a run that produced nothing. (We hit this integrating run_solver into an offline evaluation rig: a fully failed run looked successful to our automation until we checked artifacts.)

This returns 1 on that path so the printed failure and the exit status agree. The success path is unchanged, so test_offline_e2e's returncode == 0 assertion still holds — and now actually discriminates.

Related issues

None filed; happy to open one if you prefer tracking it that way.

Checklist

  • One-line change; python -m ast parses the file (ruff/pytest not run locally — no formatting-sensitive lines touched)
  • Tests added/updated (none — existing e2e covers the success path; the failure path has no harness-free test hook)
  • No public API or docs impact beyond the corrected exit status
  • No new source files

run_solver.main() prints an error when the harness exits without writing
result.json, but then falls through to 'return 0'. Any wrapper or CI job
reading the exit code sees success for a run that produced nothing.

Return 1 on that path so the printed failure and the exit status agree.
The success path is unchanged, so the e2e test's returncode assertion
still holds.

Signed-off-by: Shicheng Rao <122049441+ShichengRao@users.noreply.github.com>
Signed-off-by: Elad Sarafian <esarafian@nvidia.com>

@esarafian esarafian left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The production fix remains a single return 1 after the existing cleanup and durable-result check. Added a credential-free, normally collected regression covering missing/valid result.json with harness exit 0/1: all four cases pass; the original source fails both missing-result cases. Valid-result behavior, artifact copy-back, and cleanup are preserved. Ruff lint and format checks pass. Fork CI is approved and running; merge will wait for successful checks. 🤖🤖🤖

@esarafian
esarafian merged commit bf351b3 into NVIDIA-NeMo:main Sep 10, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants