Skip to content

[kbn-evals] Documentation: prerequisites, quick start, env vars, evaluator reference #34

Description

@milistu

Summary

Several gaps in the getting-started experience that block new users from running a first experiment.

Principle

The README should link to existing Elastic docs rather than duplicate them. If the connector creation flow changes, that change should happen in Kibana docs, not here. One sentence per concept, one link for depth.

Problems

  1. No prerequisites section: users don't know they need a Kibana connector before running anything (README.md)
  2. Missing connector raises KeyError with doubled quotes and no fix hint (config.py:21-25)
  3. Env var table has three problems: ELASTIC_EVALS_TRACING_EXPORTER is documented but never read by from_env() so setting it as an env var has no effect (only the CLI flag works); ELASTIC_EVALS_SUITE_ID is read but undocumented; EDOT_ENDPOINT is missing from the table. The default Kibana port is 5601 in config.py but Quick start uses 5620 and examples vary between the two (config.py:72-107)
  4. The evaluator reference omits kibana_evaluators, KibanaEvaluatorConfig, and all four CODE evaluators. About 29% of the README is a low-level API smoke test with no explanatory value. The Correctness evaluator docs describe 3 sub-scores but 4 rows appear in Kibana (the qualitative analysis row is always emitted alongside them)
  5. Quick start doesn't mention that each run replaces the dataset on Kibana
  6. examples/README.md is a one-line placeholder. claude_code_eval is not listed in the main README examples section
  7. examples/claude_code_eval/README.md mixes host-side uv run with Docker-only hostnames. A real connector ID is hardcoded at line 32. The URL includes an unexplained /dev path segment that causes 404s on a standard Kibana setup

Done when

  • Prerequisites section explains what a connector is, links to Kibana docs for creation steps, and states the minimum stack version
  • Missing connector raises a ConfigurationError with a clear message and a link to the prerequisites section
  • Env var table matches what from_env() actually reads (no undocumented vars, no dead vars); ELASTIC_EVALS_TRACING_EXPORTER is either wired into from_env() or removed from the table
  • Default Kibana port is consistent across all docs and examples
  • Evaluator reference covers all built-in evaluators and the composable API; Correctness evaluator section explains the 4-row structure in Kibana
  • Quick start notes that running an experiment replaces the existing dataset
  • examples/README.md is filled in and claude_code_eval appears in the main README
  • claude_code_eval README uses localhost, a CONNECTOR_ID placeholder, and explains the /dev basePath or removes it

Metadata

Metadata

Assignees

No one assigned

    Labels

    Team:nightshift-context-and-researchNightshift Context & Research Teamkbn-evalsIssue related to the work on Kibana's LLM evaluation framework.

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions