Summary
Several gaps in the getting-started experience that block new users from running a first experiment.
Principle
The README should link to existing Elastic docs rather than duplicate them. If the connector creation flow changes, that change should happen in Kibana docs, not here. One sentence per concept, one link for depth.
Problems
- No prerequisites section: users don't know they need a Kibana connector before running anything (
README.md)
- Missing connector raises
KeyError with doubled quotes and no fix hint (config.py:21-25)
- Env var table has three problems:
ELASTIC_EVALS_TRACING_EXPORTER is documented but never read by from_env() so setting it as an env var has no effect (only the CLI flag works); ELASTIC_EVALS_SUITE_ID is read but undocumented; EDOT_ENDPOINT is missing from the table. The default Kibana port is 5601 in config.py but Quick start uses 5620 and examples vary between the two (config.py:72-107)
- The evaluator reference omits
kibana_evaluators, KibanaEvaluatorConfig, and all four CODE evaluators. About 29% of the README is a low-level API smoke test with no explanatory value. The Correctness evaluator docs describe 3 sub-scores but 4 rows appear in Kibana (the qualitative analysis row is always emitted alongside them)
- Quick start doesn't mention that each run replaces the dataset on Kibana
examples/README.md is a one-line placeholder. claude_code_eval is not listed in the main README examples section
examples/claude_code_eval/README.md mixes host-side uv run with Docker-only hostnames. A real connector ID is hardcoded at line 32. The URL includes an unexplained /dev path segment that causes 404s on a standard Kibana setup
Done when
Summary
Several gaps in the getting-started experience that block new users from running a first experiment.
Principle
The README should link to existing Elastic docs rather than duplicate them. If the connector creation flow changes, that change should happen in Kibana docs, not here. One sentence per concept, one link for depth.
Problems
README.md)KeyErrorwith doubled quotes and no fix hint (config.py:21-25)ELASTIC_EVALS_TRACING_EXPORTERis documented but never read byfrom_env()so setting it as an env var has no effect (only the CLI flag works);ELASTIC_EVALS_SUITE_IDis read but undocumented;EDOT_ENDPOINTis missing from the table. The default Kibana port is5601inconfig.pybut Quick start uses5620and examples vary between the two (config.py:72-107)kibana_evaluators,KibanaEvaluatorConfig, and all four CODE evaluators. About 29% of the README is a low-level API smoke test with no explanatory value. The Correctness evaluator docs describe 3 sub-scores but 4 rows appear in Kibana (the qualitative analysis row is always emitted alongside them)examples/README.mdis a one-line placeholder.claude_code_evalis not listed in the main README examples sectionexamples/claude_code_eval/README.mdmixes host-sideuv runwith Docker-only hostnames. A real connector ID is hardcoded at line 32. The URL includes an unexplained/devpath segment that causes 404s on a standard Kibana setupDone when
ConfigurationErrorwith a clear message and a link to the prerequisites sectionfrom_env()actually reads (no undocumented vars, no dead vars);ELASTIC_EVALS_TRACING_EXPORTERis either wired intofrom_env()or removed from the tableexamples/README.mdis filled in andclaude_code_evalappears in the main READMEclaude_code_evalREADME useslocalhost, aCONNECTOR_IDplaceholder, and explains the/devbasePath or removes it