Skip to content

fix(escalation): require fresh consistent failure evidence - #637

Draft
pst2154 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
pst2154:codex/escalation-fresh-evidence
Draft

fix(escalation): require fresh consistent failure evidence#637
pst2154 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
pst2154:codex/escalation-fresh-evidence

Conversation

@pst2154

@pst2154 pst2154 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • deduplicate equivalent Terminus raw-command and structured-tool serialization only in the escalation judge transcript
  • require fresh, same-category failure evidence across distinct turns before switching
  • preserve the maximum-one-switch and post-switch latch invariants
  • add focused Rust coverage and a 40-task paired SWE-Atlas report

Benchmark evidence

On 20 RF and 20 TW tasks, the patched router scored 15/40 versus 10/40 for the original router and reduced switches from 9 to 5. Under the documented cached-token-aware synthetic rates, estimated cost fell from $1.171/task to $0.889/task. Direct GLM also scored 15/40 at $0.501/task, so this PR claims a false-positive fix and improvement over the original escalation policy, not superiority over the best fixed model.

Full methodology, task-level observations, formulas, limitations, and validation evidence are in benchmark/SWE_ATLAS_ESCALATION_REPORT.md.

Validation

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo test --workspace --all-features
  • uv run ruff check .
  • uv run mypy switchyard
  • uv run maturin develop --uv
  • uv run pytest tests/ -v

All passed on Rust 1.96.1. Pytest reported 115 passed, 2 deselected, and 2 subtests passed.

Signed-off-by: Alex Steiner <asteiner@nvidia.com>
Signed-off-by: Alex Steiner <asteiner@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant