[Bug]: Re-scoring with evaluate_experiment() silently drops every metric that needs task_span

[feiiiiii5]

What component(s) are affected?

• Opik Python SDK
• Opik Typescript SDK
• Opik Agent Optimizer SDK
• Opik UI
• Opik Server
• Documentation

Opik version

opik 2.2.70, and current main at 5cfff7b013107d44a449bd2b085c1b16f8d92eea. Both carry the same line quoted below.

Describe the problem

evaluate_experiment() re-scores an existing experiment. Its docstring lists task_span as one of the inputs a scoring function may take ("task_span - the data collected during the LLM task execution [optional]"), and evaluate() does support such metrics (covered by tests/e2e/evaluation/test_evaluate_task_span.py). In the re-scoring path, however, every metric that requires task_span is discarded before it can be scored, and nothing reports that it was dropped:

sdks/python/src/opik/evaluation/engine/engine.py:653 (EvaluationEngine.score_test_cases)

regular_metrics, _ = metrics_evaluator.split_into_regular_and_task_span_metrics(
    scoring_metrics
)

The second half of the split is thrown away, and only regular_metrics reaches _compute_test_result_for_test_case. So a user who adds a span-based scorer to an existing experiment gets a run that completes normally, a report that lists only the other metrics, no log line, no failed score, and no entry in aggregate_evaluation_scores(). The experiment looks fully evaluated.

score_test_cases has exactly one caller — evaluator.py:930, inside evaluate_experiment — so this is confined to that entry point. Every other evaluation entry point goes through run_and_score, which does handle span metrics.

This is the failure class the SDK already ruled on elsewhere:

• tests/unit/evaluation/test_silently_skipped_scores.py — "OPIK-6925: an evaluation the user asked for must never vanish without a trace."
• sdks/python/src/opik/evaluation/types.py (ErrorTolerance.METRIC_ERRORS) — "Of the failures that happen before score is entered, a missing required score argument and an item-level evaluator that cannot be built abort the run".

score_test_cases is the remaining place where a configured metric is dropped before argument validation is entered, so neither documented outcome happens. evaluate_experiment builds its engine with ErrorTolerance.METRIC_ERRORS ("This entrypoint does not expose the setting; it runs strict"), i.e. the strictest level, which makes the silence worse rather than better.

Reproduction steps and code snippets

At the public entry point (needs a running Opik instance):

import opik
from opik.evaluation.metrics.score_result import ScoreResult


def answer_is_long_enough(dataset_item, task_outputs):
    return ScoreResult(
        name="answer_is_long_enough", value=float(len(task_outputs["output"]) > 20)
    )


def task_ran_under_the_token_budget(dataset_item, task_outputs, task_span):
    # task_span carries the LLM call metadata collected while the task ran
    return ScoreResult(
        name="task_ran_under_the_token_budget",
        value=float((task_span.usage or {}).get("total_tokens", 0) <= 4096),
    )


opik.evaluate_experiment(
    experiment_name="my-experiment",
    scoring_metrics=[],
    scoring_functions=[answer_is_long_enough, task_ran_under_the_token_budget],
)
# Completes without error. The experiment receives "answer_is_long_enough" only;
# "task_ran_under_the_token_budget" appears nowhere, and nothing says it could not run.

Without a server, the same thing at engine level (fixtures and import style copied from tests/unit/evaluation/):

from typing import Any

import pytest

from opik import exceptions
from opik.evaluation.metrics import base_metric, score_result
from opik.evaluation.types import ErrorTolerance


class RequiresTaskSpan(base_metric.BaseMetric):
    def __init__(self) -> None:
        super().__init__(name="requires_task_span", track=False)

    def score(self, task_span: Any, **ignored: Any) -> score_result.ScoreResult:
        return score_result.ScoreResult(name=self.name, value=1.0)


# `score_test_cases` with a metric that needs the span, both tolerances:
#   METRIC_ERRORS        -> expected ScoreMethodMissingArguments, nothing raised
#   ALL_SCORING_ERRORS   -> expected a failed ScoreResult, key absent from results

Error logs or stack trace

Running that expectation against main at 5cfff7b01:

E  Failed: DID NOT RAISE <class 'opik.exceptions.ScoreMethodMissingArguments'>   # ErrorTolerance.METRIC_ERRORS
E  KeyError: 'requires_task_span'                                                 # ErrorTolerance.ALL_SCORING_ERRORS

The KeyError is the bug: the score result the user configured is not in the returned results at all, so there is nothing to inspect.

Proposed fix — needs a direction call

Two shapes, and I would rather you pick than have me guess:

1. Minimal, no new semantics. Stop discarding the split result: let task-span metrics reach argument validation with nothing bound to task_span. _compute_metric_scores already handles ScoreMethodMissingArguments exactly as documented — abort below ALL_SCORING_ERRORS, otherwise append _build_failed_score_result(...) with error_info, and keep it out of the persisted feedback scores. That is the same code path evaluate() uses for any other unsatisfiable argument, so the change stays inside score_test_cases (plus how it builds its evaluator) and the existing test file patterns cover it. One detail worth a call: a scoring_functions scorer is wrapped as ScorerWrapperMetricTaskSpan, whose score defaults task_span=None, so it would not trip argument validation and would instead fail inside the wrapped user function. I would make that shape report the same way rather than inherit a TypeError.
2. Full support. Reconstruct the task span for each test case from the stored trace and score it properly. This is a better outcome for users but a bigger change: it needs a TracePublic/SpanPublic → models.SpanModel route, a rule for which span of a stored trace is "the" task span (run_and_score currently uses the first span of the locally recorded tree), and one extra read per item.

A related question for option 1: evaluate_experiment hard-codes ErrorTolerance.METRIC_ERRORS and does not expose the parameter, so a user who wants the run to continue with a reported failure has no way to ask for it. Should this entry point take error_tolerance like evaluate() does? That is a public-API decision, so I am not folding it into a fix unless you want it.

Healthcheck results

Not applicable — reproduced at unit level against source, with no server involved.

Disclosure: this was found by an AI-assisted code read of score_test_cases and the OPIK-6925 tests, and confirmed by running the reproduction above locally against 2.2.70 and main; I have reviewed the cited code and the claimed behavior myself.

[github-actions]

I now have enough information to write a thorough and accurate response. Let me compile everything I've found:

1. The bug is confirmed: In engine.py:score_test_cases (around line 653), split_into_regular_and_task_span_metrics is called and the task_span metrics are silently discarded.
2. The fix path is clear: The minimal fix (Option 1 from the reporter) keeps task_span metrics in the pipeline and lets the existing _build_failed_score_result / argument validation machinery handle them.
3. The ScorerWrapperMetricTaskSpan subtlety is real: its score() defaults task_span=None and silently skips passing it to the user function when None, so it won't trip argument validation — it'll just silently score without span data.
4. No existing duplicate issue found.
5. Test coverage gap: test_evaluate_experiment.py mocks score_test_cases entirely, so the split-and-discard bug is invisible to the existing test suite.

Hi, I'm Scout 🦉, the comet-ml/opik repository agent. Let me look into this.

Solution / Workaround

No existing workaround. The silent drop happens unconditionally in score_test_cases, so there's no way for a user to opt out of it from the public API — the metric simply never reaches scoring.

Code Investigation

The drop site — sdks/python/src/opik/evaluation/engine/engine.py, around line 653, inside EvaluationEngine.score_test_cases:

regular_metrics, _ = metrics_evaluator.split_into_regular_and_task_span_metrics(
    scoring_metrics
)

The _ (task_span_metrics) is never used. Only regular_metrics is forwarded to _compute_test_result_for_test_case → build_metrics_evaluator.

The split function — sdks/python/src/opik/evaluation/engine/metrics_evaluator.py:

def split_into_regular_and_task_span_metrics(
    scoring_metrics: List[base_metric.BaseMetric],
) -> Tuple[List[base_metric.BaseMetric], List[base_metric.BaseMetric]]:
    for metric in scoring_metrics:
        if _has_evaluation_span_parameter(metric.score):
            task_span_metrics.append(metric)
        else:
            regular_metrics.append(metric)
    return regular_metrics, task_span_metrics

The classifier correctly identifies task_span metrics — they're just never used after the split in the score_test_cases path.

The wrapper subtlety — sdks/python/src/opik/evaluation/scorers/scorer_wrapper_metric.py:

ScorerWrapperMetricTaskSpan.score() defaults task_span=None and has this guard:

if task_span is not None and scorer_function.has_task_span_in_parameters(self.scorer):
    return self.scorer(..., task_span=task_span)
return self.scorer(dataset_item=dataset_item, task_outputs=task_outputs)

This means if a ScorerWrapperMetricTaskSpan were allowed to reach scoring with task_span=None, it would silently score without the span rather than raising ScoreMethodMissingArguments. A bare BaseMetric subclass with def score(self, task_span, **kwargs) (no default) would correctly fail argument validation. The two cases need unified handling.

The evaluate() path correctly handles this — in run_and_score, task_span metrics get the span object bound from the live execution context. The score_test_cases path has no live task, so there is no span to bind.

Test coverage gap: test_evaluate_experiment.py mocks engine.EvaluationEngine.score_test_cases entirely, so the split-and-discard never executes in tests. test_silently_skipped_scores.py covers the evaluate() path only. There are no tests exercising score_test_cases with a task_span metric.

Next Steps

The reporter's Option 1 (minimal fix) is the right call. Here's the concrete sketch:

In engine.py / score_test_cases:
Replace the silent discard with a call that produces a _build_failed_score_result for each task_span metric, using a new EvaluationError explaining that task_span is unavailable in re-scoring. Respect the existing error_tolerance:

• < ALL_SCORING_ERRORS (i.e. METRIC_ERRORS, the current hard-coded level): raise, matching the documented behaviour for "missing required score argument".
• >= ALL_SCORING_ERRORS: append a failed ScoreResult with scoring_failed=True and a clear reason.

In scorer_wrapper_metric.py / ScorerWrapperMetricTaskSpan.score():
When task_span is None and the wrapped function requires it, raise ScoreMethodMissingArguments (or a suitable EvaluationError) rather than silently calling the scorer without the argument. That makes ScorerWrapperMetricTaskSpan behave identically to a bare BaseMetric in this failure mode, so the engine's existing handler takes over in both cases.

New test in test_silently_skipped_scores.py (or a new test_score_test_cases_task_span.py): construct a RequiresTaskSpan metric, call score_test_cases directly (without mocking it), and assert:

• ErrorTolerance.METRIC_ERRORS → raises (not silently passes).
• ErrorTolerance.ALL_SCORING_ERRORS → result contains a ScoreResult with scoring_failed=True and the metric name present.

Option 2 (full span reconstruction from stored trace) is a larger change that requires fetching spans from the server per test case and determining which span is "the" task span. This is a product-level decision about whether evaluate_experiment() should support span-based metrics at all, rather than just reporting the limitation cleanly.

The error_tolerance exposure question is secondary — the minimal fix works entirely within the existing hard-coded METRIC_ERRORS level and does not require a public API change.

Was this helpful? React to this comment with 👍 or 👎 to rate my response.

---

View Scout trace in Opik

[feiiiiii5]

Opened #8404 (draft) implementing option 1, with the before/after measurement on 5cfff7b01 and a mutation table for each edit.

Two calls I made while writing it, both easy to revisit:

1. A scorer whose task_span parameter has a default keeps running without the span, rather than being reported. The docstring calls the argument optional, and my first version keyed off the parameter name instead, which turned that case from "works" into "aborts the run" — caught by measuring the whole unit suite, not by reasoning. The span_is_optional test on the branch pins the narrower behaviour.
2. The abort still happens per item, so other items may already have been scored and persisted before the run stops. That is what evaluate() does today for any other unsatisfiable score argument, so I kept it instead of adding a pre-flight check unique to this entry point. If you would rather this fail before any work starts, that is a small change on top.

Neither touches the open question in the issue body: evaluate_experiment still hard-codes ErrorTolerance.METRIC_ERRORS, so a user who wants to continue with a reported failure has no way to ask for it. I left the parameter alone rather than deciding the public API here.