Skip to content

[kbn-evals] Small correctness (UX) fixes #42

Description

@milistu

Summary

Seven small fixes, each a few lines.

  1. run_experiment() has no summary output. Add a verbose: bool = True parameter that prints a result table at the end using rich. (executor/client.py)
  2. 207 partial-ingest failures are logged but the run proceeds as if everything worked. The caller should know which scores failed. (scores_client.py:45-54)
  3. task_output.pop("_interaction_trace_id", ...) mutates the dict returned by the task. Take a shallow copy first. (client.py:152)
  4. evaluation_runs[-1] in client.py:208 assumes the last item in the list belongs to the current example. No await exists between the append and this read today, but a single await added later would silently corrupt results. Capture the object directly at append time instead.
  5. validate_log_level runs after Pydantic's Literal coercion so the check always passes. It's dead code. Remove it. (config.py)
  6. CI build provenance is hardcoded to None. Read the standard Buildkite env vars (BUILDKITE_BUILD_ID, BUILDKITE_JOB_ID, etc.) when present. (client.py:111)
  7. except httpx.HTTPError: raise in scores_client.py:42-43 catches an exception and immediately re-raises it unchanged. Remove it.

Done when

  • All seven items above are fixed
  • Existing tests pass

Metadata

Metadata

Assignees

No one assigned

    Labels

    Team:nightshift-context-and-researchNightshift Context & Research Teamkbn-evalsIssue related to the work on Kibana's LLM evaluation framework.

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions