diagnosis(analyzer): ChunkingQualityAnalyzer receives raw unvalidated contexts from item dict

[himanshu231204]

Description

In analyzer.py, the analyze() method passes raw item.get("contexts", []) directly to the ChunkingQualityAnalyzer, bypassing the type validation that _extract_scores performs:

# analyzer.py line 123-128
if scores.context_count > 0:
    chunking_issues = self._chunking_analyzer.analyze(
        question,
        item.get("contexts", []) or [],   # <-- raw, unvalidated
        item.get("metadata"),
    )

Meanwhile, _extract_scores (line 192-194) does validate contexts:

contexts = item.get("contexts", []) or []
if not isinstance(contexts, list):
    contexts = []

But this validated list is only used for ComponentScores.context_count and context_lengths — it's never passed to the chunking analyzer. The chunking analyzer receives the raw contexts from the item dict instead.

If the evaluation pipeline output contains contexts as a list of dicts (e.g., [{"content": "...", "metadata": {...}}, ...]) rather than list[str], the chunking analyzer will crash:

• _check_empty_chunks calls ctx.strip() on line 127 → AttributeError on a dict
• _check_content_gaps calls " ".join(contexts).lower() on line 168 → TypeError from dict in join

Severity

Medium — Silent failure in edge case. If contexts are list[str] (standard pipeline output), no crash occurs. But the mismatch between _extract_scores (which validates) and the chunking analyzer call (which doesn't reuse the validated data) is a fragility that will bite anyone with a non-standard input format.

Affected Code

• openagent_eval/diagnosis/analyzer.py lines 123-128
• openagent_eval/diagnosis/analyzer.py lines 192-194 (validated but unused by chunking analyzer)

Suggested Fix

Reuse the already-validated contexts from _extract_scores instead of re-fetching from the raw item:

scores = self._extract_scores(item)
question = scores.question

# Use scores.context_count (already validated), and pass validated contexts
if scores.context_count > 0:
    # Reconstruct or store validated contexts — currently _extract_scores discards them
    ...

Better: have _extract_scores also return the validated contexts list so the caller can reuse them.

[Nithin00614]

/claim

[github-actions]

🎉 Thanks @Nithin00614!

This issue has been assigned to you.

Happy coding! 🚀