Skip to content

[kbn-evals] Replace evaluator factories with named classmethods #36

Description

@milistu

Summary

The factory functions have two problems. First, combining two factories of the same family (e.g. analysis + quantitative) silently doubles the Kibana judge API calls per trace because each factory creates its own _EvaluationBatch. Second, the four CODE evaluator factories are byte-for-byte identical except for three strings (docstring, function name, and the name= argument), split across eight files.

Proposed API

kibana_evaluators([
    KibanaEvaluatorConfig.correctness(connector_id="..."),
    KibanaEvaluatorConfig.groundedness(connector_id="..."),
    KibanaEvaluatorConfig.latency(),
    KibanaEvaluatorConfig.input_tokens(),
], client=client)

One kibana_evaluators() call = one HTTP request per trace, regardless of how many evaluators are in the list. IDE autocomplete on KibanaEvaluatorConfig. shows everything available. The double-cost problem is impossible by design.

Done when

  • KibanaEvaluatorConfig has named classmethods for all built-in evaluator types
  • The standalone create_* factory functions are removed
  • README and evaluator reference show the composable API as the primary approach
  • One kibana_evaluators() call with multiple configs produces one HTTP request per trace

Metadata

Metadata

Assignees

No one assigned

    Labels

    Team:nightshift-context-and-researchNightshift Context & Research Teamkbn-evalsIssue related to the work on Kibana's LLM evaluation framework.

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions