Skip to content

fix(nli): return 1.0 for empty claims in score_faithfulness#208

Open
syf2211 wants to merge 1 commit into
OpenAgentHQ:mainfrom
syf2211:fix/evidence-finder-empty-claims-score
Open

fix(nli): return 1.0 for empty claims in score_faithfulness#208
syf2211 wants to merge 1 commit into
OpenAgentHQ:mainfrom
syf2211:fix/evidence-finder-empty-claims-score

Conversation

@syf2211

@syf2211 syf2211 commented Jul 22, 2026

Copy link
Copy Markdown

Description

Align EvidenceFinder.score_faithfulness() with the Faithfulness metric wrapper: when the claims list is empty, return a score of 1.0 (vacuously faithful) instead of 0.0.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Test update
  • CI/CD update

Related Issues

Fixes #76

How Has This Been Tested?

  • Unit tests pass (pytest tests/unit/test_metrics/test_nli.py)
  • Linter passes (ruff check on changed files — pre-existing warnings only)
  • Type checker passes (uv run mypy openagent_eval/)
  • Manual testing performed

Test output

pytest tests/unit/test_metrics/test_nli.py -v
# 36 passed in 0.06s

pytest tests/unit/test_metrics/test_nli.py::TestEvidenceFinder::test_score_faithfulness_empty_claims -v
# PASSED

Checklist

  • My code follows the project's coding standards
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Additional Notes

The Faithfulness metric already short-circuits on empty extracted claims and returns score=1.0 before calling score_faithfulness(). This fix ensures direct callers of EvidenceFinder.score_faithfulness([], contexts) get the same semantics.

Summary: One-line behavioral fix + corresponding unit test assertion update.

Motivation: Inconsistent API — direct NLI module usage returned 0.0 for empty claims while the Faithfulness wrapper returned 1.0.

Changes:

  • evidence_finder.py: return 0.0, []return 1.0, [] for empty claims
  • test_nli.py: update test_score_faithfulness_empty_claims expectation

Risk: Low — only affects direct score_faithfulness([]) callers; no existing code depends on the old 0.0 behavior.

Align EvidenceFinder.score_faithfulness() with the Faithfulness metric
wrapper: an empty claims list is vacuously faithful (score 1.0), not
unfaithful (score 0.0).

Fixes OpenAgentHQ#76
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

metrics(evidence_finder): score_faithfulness returns 0.0 for empty claims inconsistent with Faithfulness metric

1 participant