[Bug]: GEvalConversationMetric drops a gradeable answer when the closing assistant turn has no text

[feiiiiii5]

Opik version

• Opik version: 2.2.71 (checked on main at f06298d93c)
• Component: Opik Python SDK — opik.evaluation.metrics.conversation.llm_judges.g_eval_wrappers

Describe the problem

GEvalConversationMetric.score() documents the rule it applies when picking what to grade:

Only assistant turns with non-empty content are considered.
— sdks/python/src/opik/evaluation/metrics/conversation/llm_judges/g_eval_wrappers.py:96-98

The selector does not implement that rule. It takes the last message whose role is "assistant" and only afterwards checks whether the text is blank:

last_assistant = next(
    (turn.get("content", "") for turn in reversed(conversation)
     if turn.get("role") == "assistant"),
    "",
)
if not last_assistant.strip():
    return ScoreResult(value=0.0, reason="Conversation contains no assistant messages to evaluate.",
                       scoring_failed=True)

So when the closing assistant turn carries no text, the metric gives up instead of grading the answer before it — and the reason it reports is factually wrong, because the conversation does contain assistant messages with text.

Expected: the most recent assistant turn that has text is graded.
Actual: an empty closing turn blocks it; the score comes back scoring_failed=True, value=0.0, reason="Conversation contains no assistant messages to evaluate."

This is not a rare shape. create_conversation_from_traces adds an assistant message whenever the output transform returns anything other than None (sdks/python/src/opik/api_objects/conversation/conversation_factory.py:49-51), so a turn whose text is "" — an agent call that only issued tool calls, an aborted or empty completion, a hand-built transcript ending in a tool turn — is a normal last message. All six public conversation G-Eval metrics inherit the selector: GEvalConversationMetric, ConversationComplianceRiskMetric, ConversationDialogueHelpfulnessMetric, ConversationQARelevanceMetric, ConversationSummarizationCoherenceMetric, ConversationSummarizationConsistencyMetric, ConversationPromptUncertaintyMetric.

Because evaluate_threads drops failed scores rather than storing them (sdks/python/src/opik/evaluation/threads/helpers.py:23-26), the thread silently loses that metric — the experiment aggregate is computed over fewer threads than the user asked for, with nothing in the UI saying so.

Second symptom of the same selector: when the assistant message it selects has a content that is not a string (possible when score() is called directly with hand-built dicts, since ConversationDict is a TypedDict with no runtime enforcement), score() raises instead of returning the failed ScoreResult its own docstring prescribes. The raise happens before the judge call, so it is not captured by the try that handles judge errors.

Reproduction steps and code snippets

No server and no API keys needed — the judge is a stub.

from typing import Any
from opik.evaluation.metrics.base_metric import BaseMetric
from opik.evaluation.metrics.score_result import ScoreResult
from opik.evaluation.metrics.conversation.llm_judges.g_eval_wrappers import GEvalConversationMetric

class StubJudge(BaseMetric):
    def __init__(self) -> None:
        super().__init__(name="stub_judge", track=False)
    def score(self, output: str, **_: Any) -> ScoreResult:
        return ScoreResult(name=self.name, value=0.8, reason=f"graded: {output!r}")

metric = GEvalConversationMetric(judge=StubJudge(), name="conversation_stub")
conversation = [
    {"role": "user", "content": "Summarise these notes."},
    {"role": "assistant", "content": "Summary: timelines and budgets."},   # gradeable
    {"role": "user", "content": "Thanks"},
    {"role": "assistant", "content": ""},                                  # tool-only turn
]
print(metric.score(conversation))

Observed on main (f06298d93c):

ScoreResult(name='conversation_stub', value=0.0, reason='Conversation contains no assistant messages to evaluate.', category_name=None, metadata=None, scoring_failed=True)

Expected: value=0.8, reason="graded 'Summary: timelines and budgets.'", scoring_failed=False.

Error logs or stack trace

main at f06298d93c, calling score() with a non-string content on the selected turn:

Traceback (most recent call last):
  File "<string>", line 10, in <module>
  File ".../sdks/python/src/opik/evaluation/metrics/conversation/llm_judges/g_eval_wrappers.py", line 112, in score
    if not last_assistant.strip():
           ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'int' object has no attribute 'strip'

[github-actions]

Now I have a complete picture. Let me write up the response.

Hi, I'm Scout 🦉, the comet-ml/opik repository agent. Let me look into this.

Solution / Workaround

No existing workaround documented. The fix is small and entirely self-contained in one function.

Code Investigation

The bug lives in GEvalConversationMetric.score() at sdks/python/src/opik/evaluation/metrics/conversation/llm_judges/g_eval_wrappers.py, lines 104–112:

# CURRENT — buggy selector
last_assistant = next(
    (
        turn.get("content", "")
        for turn in reversed(conversation)
        if turn.get("role") == "assistant"      # ← only filters on role, not content
    ),
    "",
)
if not last_assistant.strip():                  # ← emptiness check comes AFTER selection
    return score_result.ScoreResult(
        name=self.name,
        value=0.0,
        reason="Conversation contains no assistant messages to evaluate.",
        scoring_failed=True,
    )

The docstring at lines 96–98 says "Only assistant turns with non-empty content are considered", but the implementation moves the emptiness check outside the generator predicate. The next() call stops at the first assistant turn by role — empty or not. If that turn is empty, a perfectly scoreable earlier turn is silently ignored and scoring_failed=True is returned with a misleading reason.

The second symptom (the AttributeError) occurs when turn.get("content", "") returns a non-string (e.g. int) and the code calls .strip() on it. This can happen when score() is called directly with hand-built dicts because ConversationDict is a TypedDict with no runtime enforcement.

All six public conversation G-Eval metrics share this selector via inheritance:

• ConversationComplianceRiskMetric
• ConversationDialogueHelpfulnessMetric
• ConversationQARelevanceMetric
• ConversationSummarizationCoherenceMetric
• ConversationSummarizationConsistencyMetric
• ConversationPromptUncertaintyMetric

The log_feedback_scores function in sdks/python/src/opik/evaluation/threads/helpers.py, lines 13–26 silently drops failed scores (if not score.scoring_failed), so the thread-level metric silently disappears from the experiment aggregate when this bug fires.

Test coverage: sdks/python/tests/unit/evaluation/metrics/test_conversation_geval_wrappers.py has three tests but none cover the trailing-empty-turn case or the non-string content case. The bug is undetected by the existing suite.

Next Steps

The fix is a one-line change to the generator predicate in score() — move the non-empty check inside the filter so it matches the documented behavior:

# FIXED selector
last_assistant = next(
    (
        turn.get("content")
        for turn in reversed(conversation)
        if turn.get("role") == "assistant"
        and isinstance(turn.get("content"), str)   # guard against non-string content
        and turn.get("content", "").strip()        # skip blank turns
    ),
    None,
)
if last_assistant is None:
    return score_result.ScoreResult(
        name=self.name,
        value=0.0,
        reason="Conversation contains no non-empty assistant messages to evaluate.",
        scoring_failed=True,
    )

Using None as the sentinel (instead of "") makes the "no match found" case unambiguous even if an assistant turn genuinely contains the empty string.

Two new unit tests should accompany the fix:

1. A conversation ending in an empty assistant turn — assert the prior non-empty assistant turn is graded.
2. A conversation where an assistant turn has a non-string content — assert a ScoreResult is returned (not an exception).

Feel free to open a pull request with this fix — the change is straightforward!

Was this helpful? React to this comment with 👍 or 👎 to rate my response.

---

View Scout trace in Opik