Repository navigation
fix(adapters): tighten hallucination_detection contract and DecodeSentences bounds (#1719) - #1754
Conversation
…tences bounds (generative-computing#1719) - Replace the generic list[dict] contract for hallucination_detection with _HallucinationDetectionContract in mellea/backends/adapters/io_contracts.py. - Validate per-record faithfulness against {"faithful", "partial", "unfaithful", "NA"}, integer response_begin/response_end (rejecting bool) with response_begin <= response_end, and str explanation. - Deduplicate identical (response_begin, response_end) spans when faithfulness matches (keeping first record) and raise AdapterSchemaMismatchError on conflicting duplicates or overlapping [begin, end) spans. - Update DecodeSentences._transform to reject bool indices with TypeError and out-of-range sentence indices with ValueError naming the index and sentence count. - Update flag_hallucinated_content, find_citations, and find_context_attributions docstrings and add unit tests in test_io_contracts.py. Assisted-by: Google Antigravity Signed-off-by: CG-5228 <g15222152017@gmail.com>
a57bc52 to
fd3c363
Compare
planetf1
left a comment
There was a problem hiding this comment.
Overall this looks good, apart from the inline comments. On head fd3c363, mypy is clean and the contract tests pass. I also ran flag_hallucinated_content against granite-4.1-3b and granite-4.1-8b on this branch. Every record passed the new contract, all four labels turned up across the two models, and the spans were adjacent with no duplicates or overlaps. The five Granite Switch io_configs/hallucination_detection/io.yaml files I had cached list the same four labels, so the fail-closed set looks safe there too.
Two things that don't attach to a diff line:
- For the record, one line in the description isn't quite right:
GroundednessRequirementdoesn't raise. It callsfind_citationsinsideexcept Exception(mellea/stdlib/requirements/rag.py:254-257), so a bad citation index now comes back as a failedValidationResultinstead of a possibly wrong verdict. That's fine as it is, so no change is needed in the code or the docs. #1718 covers how those failures get reported. docs/docs/tutorials/04-making-agents-reliable.md:560keeps records wherefaithfulness == "No", which isn't one of the four labels, soflaggedis always empty and the example always prints "Grounded response". It predates this PR (#957), and the same line is in the 0.7.0 and 0.8.0 versioned docs. You could fix it here or leave it for a follow-up issue; which labels count as a failure (partialin particular) is still open in #1721.
…ct errors Signed-off-by: CG-5228 <g15222152017@gmail.com>
|
Thanks for the thorough review @planetf1! Applied the updates in bc4afc5:
|
planetf1
left a comment
There was a problem hiding this comment.
Both inline points are addressed in bc4afc5: the error reasons now name the field and received type, the upgrade hint only appears for unknown string labels, and the parse() branches have direct tests. Tests, mypy and ruff are clean on my side.
The tutorial filter (faithfulness == "No") is tracked in #1771, so nothing more needed here.
|
thanks. all looks good. will merge |
3b47b51
Fixes #1719
Summary
hallucination_detectionoutput contract (mellea/backends/adapters/io_contracts.py):_ListContract("hallucination_detection", _HALLUCINATION_REQUIRED_KEYS)entry in_INTRINSIC_IO_CONTRACTSwith_HallucinationDetectionContract(registered inio_contracts.pyonly, keeping_core.pycapability-agnostic).faithfulnesslabel set{"faithful", "partial", "unfaithful", "NA"}(fail-closed): non-strvalues (such asNonewhen the model omitsfand theprojectstep suppliesNone) raiseAdapterSchemaMismatchErrornaming the field and the received type (got NoneType), while unknownstrvalues raiseAdapterSchemaMismatchErrornaming the invalid value, the valid labels, and an upgrade hint formellea.response_beginandresponse_endareint(specifically rejectingbool, sinceisinstance(True, int)isTrue) with field- and type-specific error messages (got <type>) and satisfyresponse_begin <= response_end.explanationis astr.(response_begin, response_end)spans that agree onfaithfulness(keeping the first record), raiseAdapterSchemaMismatchErrorwhen duplicate spans disagree onfaithfulness, and raiseAdapterSchemaMismatchErrorwhen distinct spans have overlapping half-open intervals[response_begin, response_end).flag_hallucinated_contentdocstring inmellea/stdlib/components/intrinsic/rag.pyto document all fourfaithfulnessvalues ("faithful","partial","unfaithful","NA").boolinDecodeSentences._transform(mellea/formatters/granite/intrinsics/output.py):boolsentence indices before theisinstance(json_input, int)check (TypeError).ValueErrornaming the offending index and sentence count whenjson_input < 0orjson_input >= num_sentences(closing silent negative-index wraparound oncitationsandcontext-attribution, which do not runDropInvalidbeforeDecodeSentences).find_citationsandfind_context_attributionsnow raiseValueError/TypeErrorinstead of returning citations or attributions silently resolved to the wrong sentence (GroundednessRequirementcallsfind_citationsinsidetry ... except Exceptioninmellea/stdlib/requirements/rag.py:254-257, so a bad citation index now returns a failedValidationResultinstead of a possibly wrong verdict).test/backends/test_adapters/test_io_contracts.py):_HallucinationDetectionContractandDecodeSentences._transform, plus regression tests verifying valid indices against recordedcitationsandcontext-attributioncanned fixtures.Checklist
uv run pytest test/backends/test_adapters/test_io_contracts.py test/stdlib/components/intrinsic/test_rag_contracts.pypasses (83 passed)uv run pre-commit run --all-filespasses