fix(prompt): score confidence on whether data answers the question

[Naseem77]

Problem

The self-reported confidence score could be misleading: when the graph had no data for a question (e.g. "What is the population of city A?"), the model answered "not available" with CONFIDENCE: 100, because it was 100% certain the data was absent.

Fix

Reworded the confidence instruction in templates/last_request_prompt.txt: confidence now measures whether the retrieved data actually answers the user's question. Explicitly states that "the information is not available" is a non-answer and must get low confidence.

Verification (gpt-4o-mini, local graphs)

Question	Before	After
Population of city A (no such data)	100 ⚠️	0 ✅
Travel time A → D (no such data)	10	0
Are the products fresh? (no such data)	70	0
Shortest path A → D	100	100
How many products?	100	100
Who won 2022 WC final? (world knowledge, not in graph)	100	5

Data-backed answers stay at 100; non-answers drop to 0-5, with intermediate values (20, 30) for partially supported answers.

Summary by CodeRabbit

• Chores
  • Updated the response instructions to require a final CONFIDENCE: <0-100> line that reflects whether the available data fully answers the user’s question.
  • Clarified that confidence should be lower when information is incomplete or partial.
  • Added a stricter output rule so nothing appears after the confidence line.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 1365d246-7ddd-4c69-b129-bf8468126b76

📥 Commits

Reviewing files that changed from the base of the PR and between aeb4e0d and c386065.

📒 Files selected for processing (1)

• templates/last_request_prompt.txt

---

Walkthrough

A single instruction line in the prompt template was revised so the required trailing confidence line reflects whether retrieved data actually answers the user's question, explicitly guiding lower confidence for empty or partial data, while keeping the fixed CONFIDENCE: <0-100> output format.

Changes

Prompt confidence instruction

Layer / File(s)	Summary
Update confidence instruction text templates/last_request_prompt.txt	The final confidence-line instruction now ties the score to data answerability, requiring lower confidence for missing/partial data, while retaining the CONFIDENCE: <0-100> format and disallowing trailing text.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

• FalkorDB/text-to-cypher#160: Both PRs modify templates/last_request_prompt.txt to control the required trailing CONFIDENCE: <0-100> line.

Poem

A single line, refined with care,
Confidence now knows when data's bare.
No borrow checker, no lifetime fight,
Just words tuned to answer right. 🐇
docs: sharpen the confidence line

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the prompt change to score confidence based on whether the data answers the question.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch naseem77/confidence-grounded-in-data

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}