diff --git a/CHANGELOG.md b/CHANGELOG.md index 14b8989..df6482e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - **Context diagnostics**: `codebase_context` now supports an optional `diagnostic` flag across MCP, OpenCode, and Pi. It exposes bounded routing, retrieval, and evidence-pack traces for troubleshooting without changing normal text output. -- **Pre-edit evaluation gate**: `codebase_edit_context` now has a documented CI evaluation gate. The budget gate supports a `minGraphNeighborRecall` threshold, and `npm run eval:pre-edit:ci` runs the deterministic mock-embedding baseline (Hit@5 1.0, MRR@10 1.0, graph-neighbor recall 1.0) with thresholds in `benchmarks/budgets/pre-edit.json`. +- **Pre-edit evaluation gate**: `codebase_edit_context` now has a documented CI evaluation gate. The budget gate supports a `minGraphNeighborRecall` threshold, and `npm run eval:pre-edit:ci` runs the deterministic mock-embedding baseline (Hit@5 1.0, MRR@10 1.0, graph-neighbor recall 1.0) with thresholds in `benchmarks/budgets/pre-edit.json`. The golden dataset covers a resolvable caller lookup, a target-only lookup, and an unresolved-symbol fallback that must route to the conceptual search pack. ### Fixed diff --git a/benchmarks/golden/pre-edit-context.json b/benchmarks/golden/pre-edit-context.json index 9e531c6..73ba4ee 100644 --- a/benchmarks/golden/pre-edit-context.json +++ b/benchmarks/golden/pre-edit-context.json @@ -39,6 +39,22 @@ "filePath": "src/eval/runner.ts", "symbol": "runEvaluation" } + }, + { + "id": "edit-context-unresolved-fallback", + "query": "how is context evidence about a code target assembled for an agent before editing", + "queryType": "definition", + "retrievalMode": "edit-context", + "args": { + "symbol": "resolveDefinitelyNonExistentSymbol", + "tokenBudget": 1200 + }, + "expected": { + "filePath": "src/tools/edit-context.ts", + "symbol": "formatResolutionRisk", + "expectedRoute": "search", + "expectedOutcome": "results" + } } ] } diff --git a/docs/pre-edit-context-design.md b/docs/pre-edit-context-design.md index 7af4b74..0f8f9db 100644 --- a/docs/pre-edit-context-design.md +++ b/docs/pre-edit-context-design.md @@ -44,21 +44,21 @@ The acceptance gate is no regression in the existing `agent-context` and represe ## Baseline (mock embeddings, 2026-08-06) -The gate runs in CI through `npm run eval:pre-edit:ci` with the deterministic mock-embedding provider and `benchmarks/budgets/pre-edit.json`. +The gate runs in CI through `npm run eval:pre-edit:ci` with the deterministic mock-embedding provider and `benchmarks/budgets/pre-edit.json`. The dataset (`benchmarks/golden/pre-edit-context.json`) covers three cases: a resolvable caller lookup, a resolvable target-only lookup, and an unresolved-symbol fallback that must route to the conceptual `search` pack and surface the expected implementation chunk. | Metric | Baseline | Budget threshold | |---|---|---| | Hit@5 | 1.0000 | `minHitAt5` 1.0 | | MRR@10 | 1.0000 | `minMrrAt10` 1.0 | | Graph-neighbor recall | 1.0000 | `minGraphNeighborRecall` 1.0 | -| p95 latency | 687 ms | `p95LatencyMaxAbsoluteMs` 5000 | -| Response tokens average | 358.5 | `maxContextResponseTokensAverage` 600 | -| Response tokens p95 | 399.5 | `maxContextResponseTokensP95` 800 | +| p95 latency | 672 ms | `p95LatencyMaxAbsoluteMs` 5000 | +| Response tokens average | 314.3 | `maxContextResponseTokensAverage` 600 | +| Response tokens p95 | 394.9 | `maxContextResponseTokensP95` 800 | | Response tokens max | 404 | `maxContextResponseTokensMax` 1000 | | Duplicate candidate ratio | 0.0 | `maxContextDuplicateCandidateRatio` 0.5 | -| Selected-file ratio | 0.7857 | `minContextSelectedFileRatio` 0.5 | -| Hit@5 per 1k response tokens | 2.789 | `minContextHitAt5Per1kResponseTokens` 1.0 | -| MRR@10 per 1k response tokens | 2.789 | `minContextMrrAt10Per1kResponseTokens` 1.0 | +| Selected-file ratio | 0.8571 | `minContextSelectedFileRatio` 0.5 | +| Hit@5 per 1k response tokens | 3.181 | `minContextHitAt5Per1kResponseTokens` 1.0 | +| MRR@10 per 1k response tokens | 3.181 | `minContextMrrAt10Per1kResponseTokens` 1.0 | The budget gate supports the `minGraphNeighborRecall` threshold for datasets whose queries assert graph neighbors; datasets without graph-neighbor expectations are unaffected because the metric is only compared when present.