feat(search): commit-policy answer prompt to cut over-abstention - #71
Merged
Conversation
Benchmark on a 50-q sample (SimpleQA+FRAMES) showed crw is extremely well-calibrated (near-zero hallucination: 0 wrong on SimpleQA, 2 on FRAMES) but ABSTAINS too much (32% not-attempted) — the entire accuracy gap. ~6 of the misses are "present-but-hedged": the answer IS in the sources but the model hedged a range or withheld the exact field instead of committing. Rewrite the answer SYSTEM_PROMPT to a commit policy: when the sources contain the answer, reconcile minor disagreements / pick the most authoritative value and state the exact value the query asks for (no ranges/approximations, give the precise field); lead with the direct answer; abstain ONLY when the fact is genuinely absent from every source. Preserves grounding + abstain-when-absent (keeps the near-zero hallucination), targets the hedge-when-present misses. Zero added latency/cost (same single chat call).
There was a problem hiding this comment.
Pull request overview
Rewrites the answer model's SYSTEM_PROMPT to a "commit policy" that instructs the model to state exact values when present in sources (reconciling minor differences) rather than hedging or abstaining, while preserving the no-outside-knowledge constraint. Aimed at reducing over-abstention observed in benchmark cycle 1 without adding latency.
Changes:
- Replaces the "no outside knowledge / abstain / 3–6 sentences" rules with explicit commit-to-answer guidance.
- Requires leading with the direct answer and tightens length to 1–4 sentences.
- Narrows the abstention condition to "genuinely absent from every source."
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
us
added a commit
that referenced
this pull request
Jun 1, 2026
…ed) (#72) The commit-policy prompt (#71) was benchmarked and NET-REGRESSED: on the 50-q SimpleQA+FRAMES sample, overall accuracy fell 64%->48% and INCORRECT exploded 2->17 — forcing commits made the model confidently wrong on multi-hop chains and weak retrieval, destroying the near-zero-hallucination property that was crw's strongest asset. Reverting to the baseline prompt. The real accuracy lever is retrieval recall (get the right source into context), not prompt-forced commits — pursued next.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Benchmark (50-q SimpleQA+FRAMES sample) baseline: 64% accuracy, but crw is near-zero hallucination (0 wrong SimpleQA / 2 FRAMES) and over-abstains (32% not-attempted). ~6 misses are 'present-but-hedged' (answer in sources, model hedged a range / withheld exact field). This rewrites the answer SYSTEM_PROMPT to a commit policy (reconcile + state the exact value when present; abstain only when truly absent), preserving the near-zero hallucination. Zero added latency. Cycle 1 of the benchmark improvement loop.