feat(verifier): dashboard fidelity verification framework - #212
Conversation
The 5-tier verifier proves a panel's ES|QL text does not mutate across the pipeline, but not that the Lens visualization wiring is internally consistent - the defect class behind multi-label legend collapse (#189). This adds a layered, oracle-backed fidelity framework that catches those defects deterministically. - Layer 9 (invariants.py): offline linter over migration_report.json that flags broken accessors, silently merged XY series, legend/breakdown mismatches, and silent markdown placeholders. Column truth from an embedded ES|QL parser, upgradeable to the live ES POST /_query columns oracle (Oracle 1). - Layer 7 (test_promql_property.py): Hypothesis no-crash, determinism, and label-conservation properties over generated PromQL. - Layer 8 (test_promql_metamorphic.py): semantics-preserving mutations must not change translation structure. - Layer 11 (test_panel_matrix.py): combinatorial panel matrix through the real translate_panel pipeline + invariant lint (300 cells, 0 silent errors). - Layer 12 (test_verifier_invariants.py): self-test that corrupts artifacts and asserts the exact category fires. - Layer 13 (scorecard.py): fidelity scorecard + one-way regression ratchet. Wired into obs-migrate verify-panels (--no-invariants / --live-oracle / --fail-on-invariant) and a standalone verifier.scorecard CLI. The verifier package keeps its zero-dependency-on-observability_migration contract. Refs #211
…ositives Linting node-exporter-full / prometheus-all and a real user corpus end-to-end surfaced three false-positive classes in the Layer-9 linter (the product pipeline itself produced no silent fidelity errors): - ES|QL escapes dotted fields with backticks (`service.instance.id`); the breakdown references the unescaped name. Normalize identifiers on both sides so escaped columns are not reported as ACCESSOR_BROKEN. - The emitter reorders/expands panels, so an externally-zipped yaml panel can belong to a different panel. report_panel_from_translation now uses the pipeline-associated panel_result.visual_ir, falling back to the yaml panel only when visual_ir is absent. - Grafana text/row/dashlist/etc. panels legitimately migrate to markdown; exclude these non-data source types from the PLACEHOLDER_DROPPED check. Added regressions covering backtick columns, text/row->markdown, and visual_ir-preferred association. Re-running the linter across all bundled + corpus dashboards now reports 0 ERROR findings (only disclosed merged-series warnings, the known XY single-breakdown limitation). Refs #211
Deep-testing pass: ran the framework against real dashboards end-to-endPointed the Layer-9 linter at every bundled fixture ( Outcome: 0 silent fidelity errors from the product pipeline. All 82 findings are disclosed merged-series warnings - the known XY single-breakdown limitation, which the translator always discloses (composite The pass did surface three false-positive bugs in the linter itself, now fixed in
Regressions added for all three. Net: the linter is now accurate on real-world ES|QL output, and confirms the pipeline degrades honestly. |
Elasticsearch is the real ES|QL parser; the compiler/lint are heuristic. This module executes each emitted panel query against a live cluster and classifies the response so genuine translator bugs are separated from unseeded-data noise: - real_bug: parsing / type / argument / function errors (the emitted ES|QL is invalid) - e.g. one-arg PERCENTILE (#213), the ^ power operator (#214). - data_gap: unknown column / index (well-formed query, telemetry absent). - ok / other. The executor is injectable, so the classifier and driver are fully unit-tested with no cluster (error strings are real captured 9.5 responses). A CLI runs over a migration_report.json and can gate CI with --fail-on-bug. This is the oracle that found the quantile and power-operator bugs fixed in the engine PR; structural compile/lint checks cannot catch runtime-only defects. Refs #211
The live ES|QL oracle treated any error body containing "Found N problem" as a data gap. Elasticsearch also wraps real verification/type failures in that phrase (for example RATE() on a non-counter field), so the oracle could hide invalid emitted ES|QL from --fail-on-bug. Classify data gaps only on explicit unknown-column/index/no-index signals, and add a regression for a captured "Found 1 problem ... first argument ... must be [counter]" response so it stays a real_bug. Refs #211
…racles Expands the dashboard verification framework with four trust-building layers: - Typed Dashboards API conformance oracle: maps emitted migration visual_ir into Kibana's typed /api/dashboards payload for common ES|QL xy/metric/markdown panels, validates via a live API client (fake-client tested), and reports server-side schema rejection as UI-contract gaps. - Frozen corpus gate: evaluates one or more obs-migrate compare reports against explicit FAIL/ERROR/SHAPE_PASS budgets for repeatable semantic CI. - Mutation harness: deliberately corrupts otherwise-good reports (broken accessors, broken composite legends, silent placeholders) and asserts the invariant linter catches the expected category. - Lens fixture oracle scaffold: defines and validates the consumer-side contract for Kibana LensConfigBuilder-generated fixtures, plus coverage checks for chart families we claim to support. These are deliberately independent oracles: schema/UI contract, semantic corpus budgeting, verifier self-falsification, and authoritative Lens fixture coverage. Refs #211
…rtifacts Exercising the new oracles against a real migrated artifact and live Kibana surfaced two verifier issues: - The typed Dashboards API ES|QL xy schema uses `breakdown_by`, not `breakdown`. Update the conformance converter and regression tests; the oracle now passes live against a real migrated dashboard. - The mutation harness mutated the first panel in a report. Native PROMQL passthrough panels have statically-unknown output columns, so accessor/legend mutations were skipped on reports that only use native PROMQL panels. Select a statically-inferable ES|QL panel, or append a tiny sentinel panel when none exists, so mutation self-tests remain meaningful on any artifact set. Refs #211
Adds a gate for the exact success metrics tracked by the benchmark UI: - dashboards migrated % - dashboards clean % - panels migrated % - panels clean % - panels verified % - optional duration increase The gate loads benchmark_history.json-style data, picks the most recent compatible baseline (same Grafana/Datadog config and same schema-discovery class), and fails when the latest run drops beyond the configured percentage point budget. It also aggregates grafana/datadog leg metrics when `overall` is missing, matching the tools UI formulas. This turns the trend chart into a pre-merge/CI guard instead of an after-merge surprise. Refs #211
Reviewing the PM benchmark gate against the dashboard UI/server formulas exposed two ways a gate could miss or misclassify regressions: - Comparing a run against an earlier run of the same CLI hash can turn rerun noise into a baseline. The gate now skips same-hash baselines by default and compares the latest run to the most recent compatible different hash, with an explicit --allow-same-hash-baseline override. - Percentage metrics can stay flat while the corpus denominator drops (for example fewer panels/dashboards verified). The gate now also tracks count regressions for dashboards, panels_total, and verification_total via --max-count-drop. Adds regressions for same-hash reruns, denominator drops, verification coverage drops, count tolerances, and current-index bounds. Refs #211
Add screenshot-like benchmark-history cases for the observed 9864c90 -> c658c9a regression: the gate fails on the material dashboard migrated/clean and panel verified drops even with a 0.5pp percentage tolerance and small count-drop allowance. Also cover the adjacent stable 14c0a94 -> 9864c90 transition, where a 2-panel verification denominator drift should pass with a small count tolerance. Refs #211
Add a deterministic corpus-manifest builder so benchmark size can grow without turning into a noisy "top N today" sample. The manifest combines: - top Grafana dashboards by downloads - deterministic long-tail rank slices - datasource-stratified quotas using the tools repo datasource map - pinned bug-seed dashboard IDs The output is plain JSON with de-duplicated all_ids, suitable for PR/nightly benchmark runners. Tests cover deterministic long-tail sampling, datasource quotas, bug seed preservation, catalog loading, and CLI shape. Refs #211
The PM benchmark UI can filter history by source (Grafana/Datadog) and by Grafana datasource. Overall metrics alone do not protect those filtered slices. Add source and Grafana datasource filtering to benchmark_gate, mirroring the UI: - --source grafana/datadog selects benchmark legs. - --grafana-datasource plus --grafana-datasource-map recomputes Grafana metrics from per-dashboard results for the selected datasource slice. - legacy grafana_esql_* rows and Datadog rows are excluded from Grafana datasource slices, matching the UI. Adds tests for source filtering, datasource-slice recomputation, no-match skips, legacy row exclusion, and a datasource-specific regression. Refs #211
Add command-contract documentation for the layered verifier tools: live ES|QL validation, typed Dashboards API conformance, semantic corpus gates, PM benchmark-history gates, mutation checks, Lens fixtures, and stratified corpus manifest generation. Also update agent guidance so dashboard regression work uses these gates instead of relying on one migrated/clean percentage. Call out denominator drops, datasource-filtered slices, and pinned stratified corpora as required practice. Refs #211
The typed Dashboards API oracle could pass while mapping only a small subset of panels and reporting the rest as informational unsupported findings. Add explicit coverage budgets so callers can gate on the amount of the dashboard that was actually checked: - --max-unsupported limits unsupported_by_api_oracle findings. - --min-mapped-panels requires a minimum mapped panel count. - summary now reports mapped_panels and unsupported counts. Validated live against a migrated dashboard with --min-mapped-panels 3 and --max-unsupported 0. Refs #211
stefans-elastic
left a comment
stefans-elastic
left a comment
There was a problem hiding this comment.
Findings:
- High: invalid ES|QL can be classified as
data_gap.
In parity-rig/verifier/live_validate.py, _DATA_GAP matches verification_exception.*Unknown and classify_error() checks it before _REAL_BUG. A normal Elasticsearch error like verification_exception ... Unknown function [FOO] matches both, but returns data_gap, so --fail-on-bug would pass despite invalid emitted ES|QL. Please add a wrapped unknown-function regression and make the data-gap regex specific to unknown columns/indices.
- High: Datadog reports are effectively skipped by the new oracles.
queries_from_report() in parity-rig/verifier/live_validate.py only reads visual_ir.presentation.config.query or top-level esql, while Datadog reports write translated queries as esql_query. Similarly, lint_report_panel() in parity-rig/verifier/invariants.py returns early without visual_ir.presentation. A Datadog migration_report.json can therefore produce 0 queries / 0 findings and pass. If these tools are Grafana-only, the CLI/docs should say so; otherwise add esql_query support and Datadog-shaped tests.
- Medium: datasource-slice benchmark gates can pass without evaluating anything.
In parity-rig/verifier/benchmark_gate.py, using --grafana-datasource without a datasource map makes every dashboard miss, _filtered_metrics() returns None, and evaluate_history() returns ok=True with a skipped reason. That is risky for CI because a misconfigured slice gate looks green. Please treat “filter requested but no current/baseline metrics matched” as a failure unless an explicit skip flag is set.
- Medium: corpus gate treats zero evaluated panels as success.
parity-rig/verifier/corpus_gate.py leaves ok=True when reports contain no panels or no panel verdicts. A broken/empty compare artifact can pass the frozen corpus gate. Please add a minimum evaluated panel check or fail zero-total reports by default.
I did not run the full PR test suite; this was a remote diff review. I did locally confirm the live classifier regex overlap for the wrapped Unknown function case.
Whole-dashboard typed API validation proves that Kibana accepts the mapped payload, but a 400 response only identifies paths like panels.0 and can be hard to triage on large dashboards. Add --per-panel mode to validate each mapped panel in its own scratch dashboard, preserving delete-on-success cleanup and attributing rejection findings to the exact panel title/index. This trades speed for diagnosability and is intended for debugging suspicious Dashboards API failures or high-value corpus runs. Refs #211
The public LensConfigBuilder fixture generator writes raw Lens attributes, not only the wrapper shape originally assumed by the fixture oracle. Teach lens_fixtures to accept both forms: - wrapper fixtures with explicit chart_type/data_source/attributes metadata - raw Lens attributes generated by Kibana, inferred from visualizationType, datasourceStates, visualization layers, and filenames The oracle now reports source-format and data-source counts and has regressions using real raw fixture shapes (metric and xy ES|QL). Also smoke-tested against actual v9.2.2 generated fixtures from strawgate/kb-yaml-to-lens-fixtures. Refs #211
Strengthen the LensConfigBuilder fixture oracle beyond presence/coverage checks: for raw textBased Lens attributes, collect generated column IDs and ensure visualization accessors (metricAccessor, xAccessor, accessors, yConfig, split/breakdown accessors) reference existing columns. Testing against real generated fixtures also found two oracle issues: - xy fixtures can have preferredSeriesType=line while the actual layer is bar; coverage should use layer.seriesType. - rerunning the CLI in the fixture directory should not consume its own report.json output as a fixture. Adds regressions for both and smoke-tests actual metric/xy fixtures from the LensConfigBuilder fixture generator. Refs #211
Reviewer feedback exposed verifier paths that could pass without evaluating Datadog reports or filtered corpus slices. Tighten those gates so CI failures represent real missing coverage instead of silent success.
|
Addressed the review findings in
Verification on the PR branch: |
stefans-elastic
left a comment
stefans-elastic
left a comment
There was a problem hiding this comment.
Findings:
- High:
obs-migrate verify-panelscannot run the new invariant gate flags.
parity-rig/verifier/cli.py defines --no-invariants, --live-oracle, and --fail-on-invariant, but the package-level wrapper in observability_migration/app/cli.py never registers those options on verify-panels and _run_verify_panels() never forwards them. That means the user-facing command advertised by the PR cannot actually enable the new CI gate; obs-migrate verify-panels --fail-on-invariant is rejected/unsupported while only python -m verifier.cli can use it. Please add the three flags to the umbrella CLI and cover the forwarding behavior in the app CLI tests.
- High: explicit markdown panels can be reinterpreted as ES|QL when stale query fields are present.
invariants.lint_report_panel(), dashboards_api._visual_presentation(), and live_validate.queries_from_report() all fall back to panel["esql"] / panel["esql_query"] whenever the presentation is not ES|QL or has no config. That includes panels whose visual_ir.presentation.kind is explicitly markdown. If a placeholder/manual-review markdown panel retains a top-level translated query for audit purposes, the new oracles lint/API-validate/live-execute the stale query instead of the emitted markdown, hiding placeholder regressions and validating the wrong artifact. The fallback should only run when presentation is absent/unknown, not when it is an explicit non-query kind, and this needs a markdown+retained-query regression.
- High: Datadog report-only invariant linting still does not verify Lens accessor wiring.
Datadog migration_report.json panels persist esql_query and query_ir, but not visual_ir (observability_migration/adapters/source/datadog/report.py). The new _fallback_esql_config() reconstructs only {query, type} from that shape, with no dimension, metrics, breakdown, or breakdowns; then referenced_fields() returns no field references, so _check_accessor_fields() cannot catch broken Datadog Lens/YAML accessors. This makes Layer 9 look enabled for Datadog while the core accessor invariant is skipped. Please either persist Datadog visual_ir in the report or merge the emitted YAML panel config into the linter, plus add a Datadog-shaped bad-accessor test.
- Medium:
verifier.benchmark_gatestill exits success when no comparison happened.
evaluate_history() returns ok=True for empty history and for no compatible baseline, and main() converts that to exit code 0. The docs correctly say “no compatible baseline” is not a quality pass, but a CI gate that only checks exit status will still go green if the history file is empty, reset, or keyed differently. Please fail closed for skipped comparisons unless an explicit opt-out such as --allow-skip is provided, and document the exit-code contract.
I did not rerun the full PR suite locally. The GitHub checks were green at review time.
Follow-up review found that the user-facing CLI and report-only fallbacks could still skip the new gates. Forward invariant flags, preserve emitted Datadog panel config, and fail skipped benchmark comparisons unless explicitly allowed.
|
Addressed the follow-up review findings in
Verification on the PR branch: |
Bugbot review found that --limit capped the five-tier loop but invariant linting still ran over the whole report, so --fail-on-invariant could trip on out-of-sample panels and the report mixed limited panels with whole-dashboard findings. Scope linting to the sampled panels when --limit is set.
|
Fixed in
Added regressions for the scoping helper (
|
2 participants
Summary
Adds a layered, oracle-backed dashboard fidelity verification framework on top of the existing 5-tier panel verifier. The 5-tier verifier proves a panel's ES|QL text does not mutate across the pipeline; this proves the Lens visualization wiring is internally consistent - the defect class behind multi-label legend collapse (#189), which text-drift,
smoke.py, andcomparators.pyall miss.Guiding principle: borrow real engines as oracles instead of re-implementing correctness.
What's added
parity-rig/verifier/invariants.py): offline checks overmigration_report.json(intendedquery_irvs emittedvisual_ir):ACCESSOR_BROKEN- a dimension/metric/breakdown references a column the query never produces.VISUAL_SEMANTIC_DRIFT- >1 grouping dimension collapsed onto a single XY breakdown; ERROR if silent, WARNING if disclosed.BREAKDOWN_LEGEND_MISMATCH- breakdown bound tolegendwith noEVAL legend = CONCAT(...).PLACEHOLDER_DROPPED- a chart silently became a markdown placeholder.POST /_querycolumns(Oracle 1) via--live-oracle.test_promql_property.py) - Hypothesis no-crash / determinism / label-conservation over generated PromQL.test_promql_metamorphic.py) - semantics-preserving mutations (whitespace,by()/ matcher reordering) must not change translation structure.test_panel_matrix.py) - combinatorial matrix (300 cells) through the realtranslate_panelpipeline + invariant lint.test_verifier_invariants.py) - self-test: corrupt an artifact, assert the exact category fires.scorecard.py) - fidelity scorecard + one-way regression ratchet for CI.Wiring
obs-migrate verify-panelsnow emitsinvariant_findings/invariant_summaryand gains--no-invariants,--live-oracle,--fail-on-invariant.python -m verifier.scorecardfor the ratchet gate.observability_migrationcontract (embedded parser; package parser used only when importable; live result adapter is duck-typed).What the tooling found
Run across the 300-cell matrix and the property/metamorphic suites: 0 silent fidelity errors. The matrix surfaced 24 disclosed merged-series warnings - all the documented XY single-breakdown degradation, consistently disclosed (never silent), so no new bug to file. The metamorphic relations found no ordering bugs.
Test plan
pytest tests/test_verifier_invariants.py tests/test_promql_property.py tests/test_promql_metamorphic.py tests/test_panel_matrix.py tests/test_verifier_scorecard.py(validated against a cleanorigin/mainworktree)tests/test_verifier*.pystill pass (no regressions)ruff+ source-header checks cleanverify-panels --fail-on-invariantgates correctly on a broken-accessor fixtureOut of scope (follow-ups, tracked in #211)
Closes #211