Skip to content

fix(eval): enforce pre-edit fallback route assertions - #258

Merged
Helweg merged 1 commit into
mainfrom
fix/pre-edit-route-outcome-gates
Aug 7, 2026
Merged

fix(eval): enforce pre-edit fallback route assertions#258
Helweg merged 1 commit into
mainfrom
fix/pre-edit-route-outcome-gates

Conversation

@Helweg

@Helweg Helweg commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Review finding

Independent review of the three latest merged PRs found one functional omission in #257: the unresolved-symbol golden query recorded expectedRoute: "search" and expectedOutcome: "results", but those values were only reported as metrics. The pre-edit budget did not enforce them, so a routing regression could still pass Hit@5/MRR and CI.

Fix

  • Add optional minRouteAccuracy and minOutcomeAccuracy budget thresholds.
  • Parse and enforce them in evaluateBudgetGate.
  • Require both at 1.0 in benchmarks/budgets/pre-edit.json.
  • Add unit/schema coverage and document the gates.

Validation

  • npm audit --audit-level=moderate reports 0 vulnerabilities.
  • Focused eval suites pass: 46 tests across budget, schema, and metrics.
  • npm run typecheck and npm run lint pass.
  • Independent end-to-end pre-edit mock-embedding gate passes with no violations: Hit@5 1.0, MRR@10 1.0, graph-neighbor recall 1.0, route accuracy 1.0, outcome accuracy 1.0.
  • npm run build passes. The full local test run retained four known environment-sensitive timing failures in unrelated identity/watcher tests. The changed eval suites pass in isolation and the prior merged PR checks were green.

@Helweg Helweg added the bug Bug fix label Aug 6, 2026
@Helweg
Helweg force-pushed the fix/pre-edit-route-outcome-gates branch 3 times, most recently from ccf374b to 4322040 Compare August 6, 2026 18:25
@Helweg
Helweg force-pushed the fix/pre-edit-route-outcome-gates branch from 4322040 to 0a1b037 Compare August 7, 2026 00:57
@Helweg
Helweg merged commit 57ca706 into main Aug 7, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant