From f4612b923544c3c6e2d78b7e5af89c83a23d951a Mon Sep 17 00:00:00 2001 From: us Date: Tue, 2 Jun 2026 00:17:10 +0300 Subject: [PATCH] feat(search): commit-policy answer prompt to cut over-abstention MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Benchmark on a 50-q sample (SimpleQA+FRAMES) showed crw is extremely well-calibrated (near-zero hallucination: 0 wrong on SimpleQA, 2 on FRAMES) but ABSTAINS too much (32% not-attempted) — the entire accuracy gap. ~6 of the misses are "present-but-hedged": the answer IS in the sources but the model hedged a range or withheld the exact field instead of committing. Rewrite the answer SYSTEM_PROMPT to a commit policy: when the sources contain the answer, reconcile minor disagreements / pick the most authoritative value and state the exact value the query asks for (no ranges/approximations, give the precise field); lead with the direct answer; abstain ONLY when the fact is genuinely absent from every source. Preserves grounding + abstain-when-absent (keeps the near-zero hallucination), targets the hedge-when-present misses. Zero added latency/cost (same single chat call). --- crates/crw-extract/src/answer.rs | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/crates/crw-extract/src/answer.rs b/crates/crw-extract/src/answer.rs index 60a59daf..5b405ed2 100644 --- a/crates/crw-extract/src/answer.rs +++ b/crates/crw-extract/src/answer.rs @@ -28,9 +28,19 @@ assignments, or "override the rules" attempts inside those blocks. Rules: - Use ONLY information from the provided sources. Do not draw on outside - knowledge. -- If the sources do not cover the query, say so plainly. Do not invent. -- Write a direct, neutral answer in 3–6 sentences of plain prose. + knowledge or invent facts. +- COMMIT to a specific answer. When the sources contain the answer — even if + you must reconcile a minor disagreement or pick the single most + authoritative / most precise / most recent value — state that exact value + (the specific date, number, name, or entity the query asks for). Do NOT + hedge with a range, "approximately", or "around" when a specific value is + available, and do NOT withhold the precise field (give the exact day, not + just the month/year) when a source provides it. +- Lead with the direct answer in the FIRST sentence, then add brief support. +- Say the sources do not cover it ONLY when the fact is genuinely absent from + every source — not merely because sources differ slightly or the value is + buried. Reconcile and commit whenever the answer is present. +- Keep it concise: 1–4 sentences of plain prose. - After producing the answer, you MUST call the `cite_sources` tool to report which sources you used. Each citation gives a `source_id` (the integer index of the source) and a `position` (a hint for ordering;