Skip to content

fix(adapters): tighten hallucination_detection contract and DecodeSentences bounds (#1719) - #1754

Merged
planetf1 merged 2 commits into
generative-computing:mainfrom
CG-5228:oss-agent/issue-1719
Oct 28, 2026
Merged

planetf1 merged 2 commits into
generative-computing:mainfrom
CG-5228:oss-agent/issue-1719

Conversation

@CG-5228

@CG-5228 CG-5228 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1719

Summary

  • Tighten the hallucination_detection output contract (mellea/backends/adapters/io_contracts.py):
    • Replace the plain _ListContract("hallucination_detection", _HALLUCINATION_REQUIRED_KEYS) entry in _INTRINSIC_IO_CONTRACTS with _HallucinationDetectionContract (registered in io_contracts.py only, keeping _core.py capability-agnostic).
    • Enforce the closed faithfulness label set {"faithful", "partial", "unfaithful", "NA"} (fail-closed): non-str values (such as None when the model omits f and the project step supplies None) raise AdapterSchemaMismatchError naming the field and the received type (got NoneType), while unknown str values raise AdapterSchemaMismatchError naming the invalid value, the valid labels, and an upgrade hint for mellea.
    • Validate that response_begin and response_end are int (specifically rejecting bool, since isinstance(True, int) is True) with field- and type-specific error messages (got <type>) and satisfy response_begin <= response_end.
    • Validate that explanation is a str.
    • Deduplicate repeated (response_begin, response_end) spans that agree on faithfulness (keeping the first record), raise AdapterSchemaMismatchError when duplicate spans disagree on faithfulness, and raise AdapterSchemaMismatchError when distinct spans have overlapping half-open intervals [response_begin, response_end).
    • Update the flag_hallucinated_content docstring in mellea/stdlib/components/intrinsic/rag.py to document all four faithfulness values ("faithful", "partial", "unfaithful", "NA").
  • Enforce sentence-index bounds and reject bool in DecodeSentences._transform (mellea/formatters/granite/intrinsics/output.py):
    • Reject bool sentence indices before the isinstance(json_input, int) check (TypeError).
    • Raise ValueError naming the offending index and sentence count when json_input < 0 or json_input >= num_sentences (closing silent negative-index wraparound on citations and context-attribution, which do not run DropInvalid before DecodeSentences).
    • Note on downstream callers: find_citations and find_context_attributions now raise ValueError / TypeError instead of returning citations or attributions silently resolved to the wrong sentence (GroundednessRequirement calls find_citations inside try ... except Exception in mellea/stdlib/requirements/rag.py:254-257, so a bad citation index now returns a failed ValidationResult instead of a possibly wrong verdict).
  • Unit tests (test/backends/test_adapters/test_io_contracts.py):
    • Add CPU-only unit tests covering every validation branch and deduplication rule of _HallucinationDetectionContract and DecodeSentences._transform, plus regression tests verifying valid indices against recorded citations and context-attribution canned fixtures.

Checklist

  • uv run pytest test/backends/test_adapters/test_io_contracts.py test/stdlib/components/intrinsic/test_rag_contracts.py passes (83 passed)
  • uv run pre-commit run --all-files passes
  • DCO sign-off included on commits

@CG-5228
CG-5228 requested a review from a team as a code owner October 21, 2026 00:01
@github-actions github-actions Bot added the bug Something isn't working label Oct 21, 2026
…tences bounds (generative-computing#1719)

- Replace the generic list[dict] contract for hallucination_detection with _HallucinationDetectionContract in mellea/backends/adapters/io_contracts.py.
- Validate per-record faithfulness against {"faithful", "partial", "unfaithful", "NA"}, integer response_begin/response_end (rejecting bool) with response_begin <= response_end, and str explanation.
- Deduplicate identical (response_begin, response_end) spans when faithfulness matches (keeping first record) and raise AdapterSchemaMismatchError on conflicting duplicates or overlapping [begin, end) spans.
- Update DecodeSentences._transform to reject bool indices with TypeError and out-of-range sentence indices with ValueError naming the index and sentence count.
- Update flag_hallucinated_content, find_citations, and find_context_attributions docstrings and add unit tests in test_io_contracts.py.

Assisted-by: Google Antigravity
Signed-off-by: CG-5228 <g15222152017@gmail.com>
@CG-5228
CG-5228 force-pushed the oss-agent/issue-1719 branch from a57bc52 to fd3c363 Compare October 21, 2026 21:32

@planetf1 planetf1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good, apart from the inline comments. On head fd3c363, mypy is clean and the contract tests pass. I also ran flag_hallucinated_content against granite-4.1-3b and granite-4.1-8b on this branch. Every record passed the new contract, all four labels turned up across the two models, and the spans were adjacent with no duplicates or overlaps. The five Granite Switch io_configs/hallucination_detection/io.yaml files I had cached list the same four labels, so the fail-closed set looks safe there too.

Two things that don't attach to a diff line:

  • For the record, one line in the description isn't quite right: GroundednessRequirement doesn't raise. It calls find_citations inside except Exception (mellea/stdlib/requirements/rag.py:254-257), so a bad citation index now comes back as a failed ValidationResult instead of a possibly wrong verdict. That's fine as it is, so no change is needed in the code or the docs. #1718 covers how those failures get reported.
  • docs/docs/tutorials/04-making-agents-reliable.md:560 keeps records where faithfulness == "No", which isn't one of the four labels, so flagged is always empty and the example always prints "Grounded response". It predates this PR (#957), and the same line is in the 0.7.0 and 0.8.0 versioned docs. You could fix it here or leave it for a follow-up issue; which labels count as a failure (partial in particular) is still open in #1721.

…ct errors

Signed-off-by: CG-5228 <g15222152017@gmail.com>
@CG-5228

CG-5228 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review @planetf1! Applied the updates in bc4afc5:

  1. _HallucinationDetectionContract error messages & conditional upgrade hint:
    • Separated non-str faithfulness checks from unknown str label checks: non-str values (such as None when f is omitted and the project step fills None) now report "`faithfulness` must be a `str` (one of ...), got NoneType" without the upgrade hint, and only unknown str values include (upgrade \mellea` if the upstream adapter added a new label)`.
    • Split the response_begin and response_end checks so each error names the specific field and received type ("response_beginmust beint(notbool), got NoneType", etc.).
  2. Test coverage in test_io_contracts.py:
    • Added parametrized tests covering non-array JSON payloads, non-object array elements, missing required keys, and build_prompt() on _HallucinationDetectionContract (100% line/branch coverage on the class), along with assertions for the field/type-specific error reasons and conditional upgrade hint.
  3. PR description & tutorial note:

@planetf1 planetf1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both inline points are addressed in bc4afc5: the error reasons now name the field and received type, the upgrade hint only appears for unknown string labels, and the parse() branches have direct tests. Tests, mypy and ruff are clean on my side.

The tutorial filter (faithfulness == "No") is tracked in #1771, so nothing more needed here.

@planetf1

Copy link
Copy Markdown
Contributor

thanks. all looks good. will merge

@planetf1
planetf1 enabled auto-merge October 28, 2026 08:46
@planetf1
planetf1 added this pull request to the merge queue Oct 28, 2026
Merged via the queue into generative-computing:main with commit 3b47b51 Oct 28, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(adapters): hallucination_detection output isn't checked against its own contract

2 participants