Summary
The factory functions have two problems. First, combining two factories of the same family (e.g. analysis + quantitative) silently doubles the Kibana judge API calls per trace because each factory creates its own _EvaluationBatch. Second, the four CODE evaluator factories are byte-for-byte identical except for three strings (docstring, function name, and the name= argument), split across eight files.
Proposed API
kibana_evaluators([
KibanaEvaluatorConfig.correctness(connector_id="..."),
KibanaEvaluatorConfig.groundedness(connector_id="..."),
KibanaEvaluatorConfig.latency(),
KibanaEvaluatorConfig.input_tokens(),
], client=client)
One kibana_evaluators() call = one HTTP request per trace, regardless of how many evaluators are in the list. IDE autocomplete on KibanaEvaluatorConfig. shows everything available. The double-cost problem is impossible by design.
Done when
Summary
The factory functions have two problems. First, combining two factories of the same family (e.g. analysis + quantitative) silently doubles the Kibana judge API calls per trace because each factory creates its own
_EvaluationBatch. Second, the four CODE evaluator factories are byte-for-byte identical except for three strings (docstring, function name, and thename=argument), split across eight files.Proposed API
One
kibana_evaluators()call = one HTTP request per trace, regardless of how many evaluators are in the list. IDE autocomplete onKibanaEvaluatorConfig.shows everything available. The double-cost problem is impossible by design.Done when
KibanaEvaluatorConfighas named classmethods for all built-in evaluator typescreate_*factory functions are removedkibana_evaluators()call with multiple configs produces one HTTP request per trace