Skip to content

feat(escalation): gate switching on calibrated utility - #638

Draft
pst2154 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
pst2154:codex/directional-escalation-calibration
Draft

feat(escalation): gate switching on calibrated utility#638
pst2154 wants to merge 2 commits into
NVIDIA-NeMo:mainfrom
pst2154:codex/directional-escalation-calibration

Conversation

@pst2154

@pst2154 pst2154 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add optional expected_capable_gain calibration to the escalation route
  • bypass trajectory judging and serve the efficient target when the configured gain is nonpositive
  • preserve existing behavior when the value is unset or positive, including normal transport/context fallback
  • expose the setting through TOML and Python and document the deployment utility formula
  • include a sanitized 100-cell SWE-Atlas TW reproducer and full findings report

This is an independent PR based directly on main. It complements the fresh-evidence fix in #637 but does not include that PR's commits.

Held-out evidence

Twenty deterministic held-out SWE-Atlas TW tasks were run once under each of five arms (100/100 final valid cells):

Arm Correct Synthetic total Judge/classifier calls Opus worker trajectories
Direct GLM 4/20 $6.422 0 0
Always Opus 7/20 $10.838 0 20
Patched escalation (#637) 7/20 $12.479 674 2
Capability selector 9/20 $5.036 20 0
Calibrated escalation (-0.10) 8/20 $7.520 0 0

The calibrated arm's routing invariant held across all 800 calls: every call was a GLM worker call, with no judge or Opus call. The uncalibrated patched arm had one long-tail task that made 146 judge calls and 144 GLM calls before a valid late switch, then 30 latched Opus calls; it scored 0 and cost $7.392 synthetically by itself.

The accuracy point estimates are not treated as causal. Three independently sampled GLM-worker arms scored 4/20, 8/20, and 9/20, and held-out Opus beat direct GLM despite the calibration sample showing the reverse. The report therefore concludes that this gate is a mechanical safety constraint when external calibration is reliable, not an automatic model-order solution. A full #601-style solution needs pool-owned, task-conditioned, uncertainty-aware utility estimates.

Synthetic costs include uncached input, cached input, output, and classifier/judge tokens. They are comparative assumptions, not provider prices. See the full report for formulas, rates, methodology, limitations, and reproduction assets.

Validation

  • uv sync --locked
  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo test --workspace --all-features
  • uv run ruff check .
  • uv run mypy switchyard
  • uv run maturin develop --uv
  • uv run pytest tests/ -v -m "not integration" — 116 passed, 2 deselected, 2 subtests passed
  • make -C docs publish

All commands passed on Rust 1.96.1 in a Slurm compute job.

Signed-off-by: Alex Steiner <asteiner@nvidia.com>
Signed-off-by: Alex Steiner <asteiner@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant