fix(metrics): parse LLM judge scores from structured JSON responses

[syf2211]

Summary

Improve LLMJudgeMetric score parsing to prefer structured JSON output instead of fragile regex/keyword heuristics.

Fixes #74

Motivation

The previous _parse_score implementation could match the wrong number in free-form LLM text, treat keyword substrings like "excellent" as high scores, and silently fall back to an arbitrary 0.5 default.

Changes

• Update built-in judge prompts to request a JSON object: {"score": <0.0-1.0>}
• Parse fenced or plain JSON first, with case-insensitive score key lookup
• Keep narrow backward-compatible fallbacks for labeled scores (Score: 0.85) and bare numeric responses (0.92)
• Remove keyword-based scoring heuristics
• Log warnings whenever parsing falls back from JSON

Tests

pytest tests/unit/test_metrics/test_nli.py -k llm_judge -q
# 18 passed

Notes

Custom JudgeCriteria without the JSON instruction still get labeled/bare-numeric fallbacks, but ambiguous prose now returns the default score with a warning instead of guessing from the first digit.

[himanshu231204]

/oc review pr

[opencode-agent]

All 18 tests pass. Here's my review:

---

Review: fix/llm-judge-json-score-parsing-74

Overall

Clean, well-scoped change. 2 files, +126/-31. All tests pass.

Strengths

1. JSON-first parsing (_parse_score at llm_judge.py:174) is strictly better than the old regex-grabs-first-number approach. Tries fenced JSON → plain JSON → labeled score → bare decimal → default, with logger.warning at each fallback step. Good defensive design.

2. Prompt update — built-in criteria now include _JSON_SCORE_INSTRUCTION (Respond with ONLY a JSON object... ) instead of bare "Score:". This nudges LLMs to produce parseable output.

3. Case-insensitive key lookup (llm_judge.py:187) — {"Score": 0.77} works, which is realistic for LLM output.

4. Scale normalization (_clamp_normalized_score) — handles 0–1, 1–10, and 1–100 scales sensibly without assuming.

5. Keyword heuristics removed — no more guessing "excellent" → 1.0 or "not at all" → 0.0. Ambiguous prose returns 0.5 with a warning. This is the right tradeoff.

6. Test coverage is thorough — 12 distinct _parse_score cases including edge cases (empty, invalid JSON value, fences, percentage, case-insensitive keys, keyword false positives, preference for JSON over embedded numbers).

Nitpicks (minor)

Issue	Location	Suggestion
_json_score_candidates always appends text raw	llm_judge.py:227	If fenced JSON matched, raw text is tried second — fine, but consider cost if response is very long (negligible here).
_BARE_DECIMAL_RE uses match not search	llm_judge.py:24	Intentional — only matches if the whole string is a number. This is correct behavior but worth noting since match is anchored at start.
No test for malformed JSON with valid score elsewhere	test_nli.py	e.g. {"score" 0.85} (invalid JSON) should fall through to labeled/bare — implicitly covered by the fallback chain but not explicitly tested.

Verdict

Approve. The change is a clear improvement: more robust, more predictable, better diagnostics. Backward compatibility is maintained through explicit fallbacks. The test suite gives high confidence.

opencode session  |  github run

[himanshu231204]

@syf2211 Could you plz review the Nitpicks (minor) suggest by the bot. This pr is ready to merge just took a look on suggestions. Thanks