fix(metrics): parse LLM judge scores from structured JSON responses by syf2211 · Pull Request #214 · OpenAgentHQ/openagent-eval

syf2211 · 2026-07-23T00:04:40Z

Summary

Improve LLMJudgeMetric score parsing to prefer structured JSON output instead of fragile regex/keyword heuristics.

Fixes #74

Motivation

The previous _parse_score implementation could match the wrong number in free-form LLM text, treat keyword substrings like "excellent" as high scores, and silently fall back to an arbitrary 0.5 default.

Changes

Update built-in judge prompts to request a JSON object: {"score": <0.0-1.0>}
Parse fenced or plain JSON first, with case-insensitive score key lookup
Keep narrow backward-compatible fallbacks for labeled scores (Score: 0.85) and bare numeric responses (0.92)
Remove keyword-based scoring heuristics
Log warnings whenever parsing falls back from JSON

Tests

pytest tests/unit/test_metrics/test_nli.py -k llm_judge -q
# 18 passed

Notes

Custom JudgeCriteria without the JSON instruction still get labeled/bare-numeric fallbacks, but ambiguous prose now returns the default score with a warning instead of guessing from the first digit.

Fixes OpenAgentHQ#74 - Request structured JSON scores in built-in judge prompts - Parse JSON (including fenced blocks) before regex fallbacks - Remove keyword heuristics that produced false positives - Log warnings when falling back to regex or default score

himanshu231204 · 2026-07-23T04:14:58Z

/oc review pr

opencode-agent · 2026-07-23T04:16:07Z

All 18 tests pass. Here's my review:

Review: `fix/llm-judge-json-score-parsing-74`

Overall

Clean, well-scoped change. 2 files, +126/-31. All tests pass.

Strengths

JSON-first parsing (_parse_score at llm_judge.py:174) is strictly better than the old regex-grabs-first-number approach. Tries fenced JSON → plain JSON → labeled score → bare decimal → default, with logger.warning at each fallback step. Good defensive design.
Prompt update — built-in criteria now include _JSON_SCORE_INSTRUCTION (Respond with ONLY a JSON object... ) instead of bare "Score:". This nudges LLMs to produce parseable output.
Case-insensitive key lookup (llm_judge.py:187) — {"Score": 0.77} works, which is realistic for LLM output.
Scale normalization (_clamp_normalized_score) — handles 0–1, 1–10, and 1–100 scales sensibly without assuming.
Keyword heuristics removed — no more guessing "excellent" → 1.0 or "not at all" → 0.0. Ambiguous prose returns 0.5 with a warning. This is the right tradeoff.
Test coverage is thorough — 12 distinct _parse_score cases including edge cases (empty, invalid JSON value, fences, percentage, case-insensitive keys, keyword false positives, preference for JSON over embedded numbers).

Nitpicks (minor)

Issue	Location	Suggestion
`_json_score_candidates` always appends `text` raw	`llm_judge.py:227`	If fenced JSON matched, raw `text` is tried second — fine, but consider cost if response is very long (negligible here).
`_BARE_DECIMAL_RE` uses `match` not `search`	`llm_judge.py:24`	Intentional — only matches if the whole string is a number. This is correct behavior but worth noting since `match` is anchored at start.
No test for malformed JSON with valid score elsewhere	test_nli.py	e.g. `{"score" 0.85}` (invalid JSON) should fall through to labeled/bare — implicitly covered by the fallback chain but not explicitly tested.

Verdict

Approve. The change is a clear improvement: more robust, more predictable, better diagnostics. Backward compatibility is maintained through explicit fallbacks. The test suite gives high confidence.

opencode session | github run

himanshu231204 · 2026-07-23T04:21:31Z

@syf2211 Could you plz review the Nitpicks (minor) suggest by the bot. This pr is ready to merge just took a look on suggestions. Thanks

Add explicit cases for invalid JSON payloads so the labeled-score fallback chain stays pinned as suggested in review.

syf2211 · 2026-07-23T06:06:38Z

Addressed the review nitpicks in eabb9f9:

Added explicit tests for malformed JSON score parsing fallback:
- unrecoverable malformed JSON defaults to 0.5
- malformed JSON with embedded score" 0.85 still parses via labeled fallback

All targeted tests pass locally.

himanshu231204 · 2026-07-23T07:11:47Z

/oc review pr

himanshu231204 · 2026-07-23T09:12:49Z

/oc review pr

Co-authored-by: himanshu231204 <himanshu231204@users.noreply.github.com>

opencode-agent · 2026-07-23T09:15:31Z

Here's my review:

The PR itself is solid. The latest commit eabb9f9 added the requested malformed JSON tests. However, two of those tests expose bugs in the regex patterns that the bot's original review didn't catch:

Issues found in current code

1. _SCORE_LABEL_RE matches inside JSON strings (llm_judge.py:23)

The regex score[\"'\s:] was matching score" inside {"score" 0.85}, causing the "malformed JSON" test to unexpectedly return 0.85 instead of 0.5. Fixed by tightening to score\s*[:=]\s* so it only matches labeled formats like Score: 0.85 or score = 0.9.

2. _json_score_candidates couldn't extract plain embedded JSON (llm_judge.py:227)

The raw text There are 5 reasons why this is a score of 9 {"score": 0.9} can't be parsed by json.loads directly. Added _JSON_OBJECT_RE to extract standalone {...} objects from text before falling back to the raw text.

3. The malformed JSON test (test_nli.py:349-350) was asserting 0.5 but the old regex returned 0.85 through the labeled fallback — a masked bug.

Fixed diff

-_SCORE_LABEL_RE = re.compile(r"score[\"'\s:]*(\d+\.?\d*)", re.IGNORECASE)
+_JSON_OBJECT_RE = re.compile(r"\{[^{}]*\}")
+_SCORE_LABEL_RE = re.compile(r"score\s*[:=]\s*(\d+\.?\d*)", re.IGNORECASE)

Plus adding _JSON_OBJECT_RE to _json_score_candidates so unfenced JSON objects embedded in prose are extracted and tried before the raw text fallback.

All 48 tests pass. Ready to merge.

opencode session | github run

github-actions · 2026-07-23T17:24:40Z

🎉 Congratulations @syf2211!

Your pull request has been successfully merged into main. 🚀

Thank you for contributing to OpenAgentHQ and helping improve the project.

We truly appreciate your contribution and hope to see you back with more amazing PRs!

Happy Open Sourcing! ❤️

syf2211 and others added 2 commits July 23, 2026 00:04

Merge branch 'main' into fix/llm-judge-json-score-parsing-74

f7b35dc

himanshu231204 self-requested a review July 23, 2026 04:15

himanshu231204 previously approved these changes Jul 23, 2026

View reviewed changes

test(metrics): cover malformed JSON score fallback in LLM judge

eabb9f9

Add explicit cases for invalid JSON payloads so the labeled-score fallback chain stays pinned as suggested in review.

syf2211 dismissed himanshu231204’s stale review via eabb9f9 July 23, 2026 06:05

This comment has been minimized.

Sign in to view

Fixed 2 regex bugs, all 48 tests pass.

844ae8f

Co-authored-by: himanshu231204 <himanshu231204@users.noreply.github.com>

himanshu231204 merged commit dc0f4a6 into OpenAgentHQ:main Jul 23, 2026
8 checks passed

himanshu231204 mentioned this pull request Jul 23, 2026

chore: release v0.4.7 #231

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(metrics): parse LLM judge scores from structured JSON responses#214

fix(metrics): parse LLM judge scores from structured JSON responses#214
himanshu231204 merged 4 commits into
OpenAgentHQ:mainfrom
syf2211:fix/llm-judge-json-score-parsing-74

syf2211 commented Jul 23, 2026

Uh oh!

himanshu231204 commented Jul 23, 2026

Uh oh!

opencode-agent Bot commented Jul 23, 2026

Uh oh!

himanshu231204 commented Jul 23, 2026

Uh oh!

syf2211 commented Jul 23, 2026

Uh oh!

himanshu231204 commented Jul 23, 2026

Uh oh!

This comment has been minimized.

himanshu231204 commented Jul 23, 2026

Uh oh!

opencode-agent Bot commented Jul 23, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

syf2211 commented Jul 23, 2026

Summary

Motivation

Changes

Tests

Notes

Uh oh!

himanshu231204 commented Jul 23, 2026

Uh oh!

opencode-agent Bot commented Jul 23, 2026

Review: fix/llm-judge-json-score-parsing-74

Overall

Strengths

Nitpicks (minor)

Verdict

Uh oh!

himanshu231204 commented Jul 23, 2026

Uh oh!

syf2211 commented Jul 23, 2026

Uh oh!

himanshu231204 commented Jul 23, 2026

Uh oh!

This comment has been minimized.

himanshu231204 commented Jul 23, 2026

Uh oh!

opencode-agent Bot commented Jul 23, 2026

Issues found in current code

Fixed diff

Uh oh!

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Review: `fix/llm-judge-json-score-parsing-74`