Skip to content

fix(escalation): require fresh consistent failure evidence - #632

Closed
pst2154 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
pst2154:codex/escalation-judge-confirmations
Closed

fix(escalation): require fresh consistent failure evidence#632
pst2154 wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
pst2154:codex/escalation-judge-confirmations

Conversation

@pst2154

@pst2154 pst2154 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

What

  • Normalize duplicated Terminus command serialization in the escalation judge's transcript without changing model-facing history.
  • Require consecutive escalation verdicts to cite fresh evidence for the same bounded failure category before latching to the strong tier.
  • Log the parsed category, freshness flag, and a bounded reason; update the packaged schema, prompt, tests, and routing documentation.

Why

Agent harnesses can represent one command both as assistant JSON/text and as a structured tool call. The trajectory judge could count those same-turn representations as repeated failed attempts. More generally, boolean-only confirmations could combine stale evidence or unrelated failure modes into a false escalation.

The new confirmation rule keeps the existing one-way latch while requiring a coherent multi-turn failure pattern. This is a focused correctness improvement related to the signal-composition direction discussed in #601.

Live SWE-Atlas diagnostics with GLM 5.2 as the efficient tier and Opus 4.8 as the capable tier:

  • Positive path, RF task-697e7458be1623d850a88838: two consecutive fresh repetition verdicts caused exactly one GLM-to-Opus transition; the run used 28 GLM and 36 Opus worker calls and received verifier reward 1.
  • Recovery control on the same task: two repetition votes separated by a decline did not latch; the GLM-only run received verifier reward 1.
  • Prior false-positive cases RF task-69391d8d1ce51c407be1e531, RF task-694b4b99829f00e24fd11889, and TW task-6902ef3ab97fe23e2ad271f3 all remained GLM-only under the new rule. In the first case, positive votes changed from repetition to false_progress, so they did not combine into a latch.

These are routing diagnostics, not an aggregate accuracy claim: the positive-path task also passed in a separate direct-GLM rollout, and the false-positive controls still received reward 0. The evidence shows correct confirmation/latch behavior and avoided strong-tier calls on previously wasteful trajectories.

Notes for reviewers

Start with crates/libsy/src/algorithms/escalation.rs for the confirmation state machine and crates/libsy/src/algorithms/util/escalation.rs for judge transcript normalization and typed verdicts.

Validation on Rust 1.96.1:

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo test --workspace --all-features
  • uv run ruff check .
  • uv run mypy switchyard
  • wheel build/install through maturin
  • uv run pytest tests/ -v (117 passed, 2 subtests passed)

Signed-off-by: Alex Steiner <asteiner@nvidia.com>
@pst2154

pst2154 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #637, which carries the current fresh-evidence implementation and paired SWE-Atlas report. The separate directional/model-order follow-up is #638.

@pst2154 pst2154 closed this Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant