ConversationDegenerationMetric silently drops punctuation/whitespace-only assistant turns instead of scoring them as degenerate

[shaurya416]

sdks/python/src/opik/evaluation/metrics/conversation/heuristics/degeneration/metric.py:100-104

prev_tokens: Optional[List[str]] = None
for content in assistant_turns:
    tokens = _tokenize(content)
    if not tokens:
        continue

_tokenize is re.findall(r"\b\w+\b", text.lower()), which returns [] for any assistant turn whose content is punctuation-only ("...", "???", "!!!") or whitespace-only. That turn is then skipped completely: it contributes nothing to degeneracy_scores, per_turn_metadata, peak_score, or average_score, and it doesn't even update prev_tokens for the next real turn's overlap calculation. It is as if the turn never happened.

The class docstring and the published docs both describe the opposite behaviour. The docstring (metric.py:29-37) says the metric "inspects each assistant turn" so callers "can quickly flag sections where the assistant got stuck or stopped being helpful," and conversation_threads_metrics.mdx:39 calls it "a lightweight guard against models that fall into loops or short-circuit the dialogue." A punctuation-only reply is the textbook case of an assistant that stopped being helpful / short-circuited the dialogue — the one case this guard exists for — yet it is invisible to the scoring loop.

What happens (reproduced on the real module)

ConversationDegenerationMetric.score() (metric.py:100-104) tokenizes each assistant turn with _tokenize (word-boundary regex) and does if not tokens: continue for any turn whose content is punctuation- or whitespace-only. Such a turn does not increment degeneracy_scores, per_turn_metadata, or prev_tokens.

A conversation that degenerates partway through, e.g. one real reply followed by several '...'/'???'/'!!!' turns, therefore reports a near-zero peak/average score computed only over the surviving real turns. This contradicts the class docstring ('inspects each assistant turn... can quickly flag sections where the assistant got stuck or stopped being helpful') and the public docs ('a lightweight guard against models that fall into loops or short-circuit the dialogue', conversation_threads_metrics.mdx:39).

Nothing else in the code compensates:

• No caller in evaluate_threads/score_statistics/evaluation_result or the ConversationThreadMetric base class filters, pads, or otherwise guards against this.
• No existing test exercises the mixed real+garbage-turn case.
• The all-garbage path (MetricComputationError when every turn tokenizes empty) shows the codebase already treats token-less content as a distinct, meaningful case elsewhere, just not consistently.

Fix: either score a token-less turn as maximal degeneration risk, or at minimum record the count of excluded turns in ScoreResult.metadata so threshold consumers aren't silently blind to them.

Happy to open the PR.

[github-actions]

Now I have a complete picture. Let me write my response.

Hi, I'm Scout 🦉, the comet-ml/opik repository agent. Let me look into this.

Solution / Workaround

No existing workaround is in place. Callers who want correct behavior today would need to pre-filter their conversations and manually flag punctuation/whitespace-only turns before calling score(). There is no option on the metric itself to change this behavior.

Code Investigation

The bug is exactly as described. The relevant code lives in:

sdks/python/src/opik/evaluation/metrics/conversation/heuristics/degeneration/metric.py

The tokenizer (line 17):

def _tokenize(text: str) -> List[str]:
    return re.findall(r"\b\w+\b", text.lower())

This returns [] for any content that contains no word characters — e.g. "...", "???", "!!!", " ".

The scoring loop (lines 100–104):

prev_tokens: Optional[List[str]] = None
for content in assistant_turns:
    tokens = _tokenize(content)
    if not tokens:
        continue   # <-- silently skips the turn entirely

The continue does four things that are all wrong:

1. Does not append to degeneracy_scores — the turn is invisible to average_score and peak_score.
2. Does not append to per_turn_metadata — the metadata count mismatches the actual turn count.
3. Does not update prev_tokens — the next real turn's overlap_with_previous is computed against the last non-empty turn, not the immediately preceding one.
4. Does not surface anything to the caller — no flag, no count in metadata, nothing.

There is also a second guard at score() line 88 that already filters at collection time:

assistant_turns = [
    turn["content"]
    for turn in conversation
    if turn.get("role") == "assistant" and turn.get("content")
]

turn.get("content") is falsy for "" but truthy for "..." or " ", so punctuation/whitespace-only turns do reach the loop — they just get silently dropped by if not tokens: continue.

Test coverage gap: test_conversation_metrics.py has no test case for mixed real + punctuation-only turns. The test_conversation_degeneration_detects_repetition test asserts len(result.metadata["per_turn"]) == 3 which would silently pass even if a punctuation-only turn were added to that conversation.

Next Steps

Two approaches, with the simpler one being most defensible:

Option A (recommended) — score token-less turns as maximum degeneration:

for content in assistant_turns:
    tokens = _tokenize(content)
    if not tokens:
        # Punctuation/whitespace-only reply is textbook degeneration — score it as 1.0
        per_turn_metadata.append({
            "repetition_ratio": 1.0,
            "overlap_previous": 1.0,
            "fallback_hit": 0.0,
            "normalized_entropy": 1.0,
            "degeneration_score": 1.0,
        })
        degeneracy_scores.append(1.0)
        # Do NOT update prev_tokens; the previous context is still the last real turn
        continue

Option B (lighter) — keep skipping but record excluded turn count in metadata:

excluded_turns += 1

…and surface "excluded_turns": excluded_turns in the final ScoreResult.metadata. This at minimum stops callers being silently blind to the problem.

Option A is the correct fix per the docstring contract: "inspects each assistant turn… flag sections where the assistant got stuck or stopped being helpful." A "..." reply is unambiguously degenerate.

The fix should also add a test case like:

conversation = [
    {"role": "user", "content": "Tell me more."},
    {"role": "assistant", "content": "Of course, here is some information."},
    {"role": "user", "content": "And more?"},
    {"role": "assistant", "content": "..."},   # punctuation-only turn
    {"role": "assistant", "content": "???"},   # punctuation-only turn
]
result = metric.score(conversation)
assert result.value == 1.0  # peak should be maximal
assert len(result.metadata["per_turn"]) == 3  # all 3 assistant turns counted

Feel free to open a pull request with this fix — the change is straightforward!

Was this helpful? React to this comment with 👍 or 👎 to rate my response.

---

View Scout trace in Opik