fix(grafana): disclose source metrics silently dropped from migrated queries - #362
Merged
giorgi-imerlishvili-elastic merged 3 commits intoAug 19, 2026
Conversation
…queries Two related gaps let a panel report itself as cleanly "migrated" while a real Grafana target's metric never made it into the emitted ES|QL: - Curated-pack overrides (issue elastic#349): `status_override: migrated` unconditionally set status/confidence, bypassing any check that the hand-written query actually covers every metric the panel's real targets reference. A pack author could forget a metric and the panel would still claim confidence 1.0. - The general multi-target fusion path (issue elastic#352): no check compared a panel's original PromQL target metrics against what actually survived into the final fused query, so a target judged "mergeable" could still have its metric silently dropped during query construction. Both paths now share `_source_metrics_absent_from_query`, gated on successful live field-caps discovery to avoid false positives from an unverified schema. `status_override` is now a ceiling, not an unconditional assignment. The curated-path check excludes hidden (`hide: true`) targets (a disabled/legacy fallback query Grafana itself never renders) and the general-path check covers every Lens layer of a cross-index panel, not just the first -- both false-positive traps found via live Kibana verification and an independent model review before merge. Also fixes the two Node Exporter Full (1860) curated-pack gaps issue elastic#349 named directly: `node_pressure_irq_stalled_seconds_total` was missing from the Pressure panel's PSI tiles and `node_cpu_guest_seconds_total` was missing from the CPU panel's per-mode breakdown. The CPU panel's new Guest aggregation is CASE-wrapped to match its sibling assignments, avoiding the STATS_CASE_BARE_TS_MIX structural-oracle class of query that Elasticsearch can reject.
6 tasks
…ueries Adding irq/guest to the 1860 pack referenced unknown columns on clusters without those fields, which made Elasticsearch reject the whole panel. Treat them as live-optional, strip leftover unpivot EVAL/WHERE aliases, and stop reporting stripped optional metrics as pack omissions.
shmsr
approved these changes
Aug 19, 2026
Merged
4 tasks
… incidental sample data A same-day commit (4cb7726) reserved the literal axis-label text "percentage" as an opaque Grafana-unit-id alias whose title is intentionally suppressed (so unit-inferred titles like "%" can take over instead) and added a correct, dedicated test for that behavior. It didn't touch this unrelated, pre-existing test, which happened to reuse the same literal string purely as incidental sample text for testing something else entirely (that bar charts keep axis config while omitting line/area-only appearance keys) -- breaking it on main and therefore on every open PR whose CI merges against main. Swap the incidental fixture text for an ordinary, non-reserved label so the test again exercises its own actual intent without colliding with the new opaque-alias behavior.
giorgi-imerlishvili-elastic
added a commit
to giorgi-imerlishvili-elastic/observability-migration-platform
that referenced
this pull request
Aug 21, 2026
… verification script (elastic#350) All 6 dashboard_sha256 pins in curated_packs/registry.yaml were wrong (didn't match a canonical-JSON hash of the pinned grafana.com revision), and gnet_id 11835 pinned gnet_revision 4, which has never existed on grafana.com (that dashboard only has revision 1) -- silently defeating the provenance check these fields exist for. - Correct dashboard_sha256 for the 5 packs whose gnet_revision resolves to a real download; re-pin 11835 to revision 1 with an explicit callout that this can't be independently confirmed to match the pack authors' original source, since revision 4 never existed. - Add scripts/verify_curated_pack_pins.py, a maintainer/CI-only command (network required, not part of `make test`) that re-downloads each pinned revision and re-checks its canonical hash, mirroring the existing fetch_community_corpus.py pattern. Hardened against a malformed download response aborting the whole run, and against an empty/unparseable registry silently reporting a false "0/0 verified" pass. - Document these fields as maintainer-verified provenance pins, not a migration-time gate: a pristine grafana.com download structurally differs from any real Grafana-instance export (mutated id/uid/version), so comparing against operator dashboards would mismatch on every real migration. Rewrote docs/design/curated-dashboard-packs.md's "drift detection" section, which described a revision-comparison mechanism and two CLI commands that were never actually implemented, to match this. - Fix a stale duplicate gnet_revision in the 11835 pack's own fidelity_manifest.yaml left behind by the re-pin, and add a test guarding every pack's manifest against the registry to prevent this drift from recurring silently. Depends on elastic#362 (dropped-source-metric disclosure) for the new docs/sources/grafana.md wording to describe behavior that actually exists on main; should land at or after that PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issues fixed
status_override: migratedreports full confidence while the hand-written ES|QL drops source series #349 — curated-packstatus_overrideunconditionally sets status/confidence, hiding a dropped source metricValidity
Both issues are valid. Confirmed empirically against the real "Node Exporter Full" (grafana.com id 1860, revision 37) dashboard: the curated pack's "Pressure" and "CPU" panel overrides genuinely omit two metrics present in the panel's real Grafana targets (
node_pressure_irq_stalled_seconds_total,node_cpu_guest_seconds_total) while still reportingstatus_override: migrated/ confidence1.0.Root cause
status_override: migratedreports full confidence while the hand-written ES|QL drops source series #349: the curated-override branch oftranslate_paneltreatsstatus_overrideas an unconditional assignment rather than a ceiling — a hand-written pack query can omit a target's metric and the panel still reports full-confidence "migrated".Code paths checked
observability_migration/adapters/source/grafana/panels.py: curated-override branch oftranslate_panel(status/confidence assignment), the multi-target fusion pipeline (fused_series, cross-index layer construction), and the existing "live-missing-metric" / "incompatible-grouping-drop" warning paths (to avoid double-reporting the same gap under two messages).observability_migration/adapters/source/grafana/curated_packs/grafana_1860_node_exporter_full/pack.yaml: thePressureandCPUpanel overrides.observability_migration/core/verification/translation_oracle/structure.py: theSTATS_CASE_BARE_TS_MIXstructural-oracle rule, relevant to a bug found during review (see below).Fix
_source_metrics_absent_from_query(source_exprs, query_text, resolver)inpanels.py: given a panel's original PromQL target expressions and the final emitted query text, flags any source metric (or itsresolve_metric_field-resolved candidate name) that never appears in the output. Gated on successful live field-caps discovery (resolver.discovery_status()["status"] == "ok") — without it, a bare substring match against an unverified schema would be unreliable, so the helper no-ops.status_override: migratedreports full confidence while the hand-written ES|QL drops source series #349): called right beforePanelResultconstruction.status_override: migratedis now a ceiling — a detected gap downgrades tomigrated_with_warningsand caps confidence at0.6, with a"Target telemetry missing from curated override: <metric>"reason. Excludeshide: truetargets (a disabled/legacy fallback query Grafana itself never renders — found via live verification, see below).fused_series(to avoid double-reporting a gap already explained by an existing live-missing-metric or incompatible-grouping warning) against all emitted query layers, including every layer of a cross-index panel (a metric legitimately landing in the second Lens layer must not look "dropped" just because the first layer's query was checked alone — also found via review, see below). Appends"Dropped from migrated query: <metric>".status_override: migratedreports full confidence while the hand-written ES|QL drops source series #349 named directly: addednode_pressure_irq_stalled_seconds_totalas a 4th "Irq" tile to thePressurebargauge, andnode_cpu_guest_seconds_totalas a "Guest" series to theCPUper-mode breakdown.docs/sources/grafana.md.Side effects considered
hide: truetarget's metric (node_memory_MemFree_bytes, a legacy MemFree-based formula the real "RAM Used" panel keeps disabled behind a newer MemAvailable-based one). Fixed by excluding hidden targets from the source-metric set; addedtest_curated_override_ignores_hidden_target_when_checking_dropped_metrics.test_cross_index_xy_layers.pytest with a regression assertion (confirmed it fails without the fix).guest_lhsaggregation used a bareSUM(IRATE(...))alongside seven CASE-wrapped sibling assignments in the sameSTATS, tripping the codebase's ownSTATS_CASE_BARE_TS_MIXstructural-oracle rule (an ERROR-level check created after a real historical incident where Elasticsearch rejected this exact mix — seedocs/design/open-problems.md). It executed fine against my local ES version, which is exactly the kind of environment-dependent landmine that rule exists to catch. Fixed by CASE-wrapping it to match its siblings; confirmed the structural oracle now reports zero errors on the fully resolved query.status_override: migratedreports full confidence while the hand-written ES|QL drops source series #349/General translator drops source panel targets without any warning #352 asked for.status_override: migratedand no gap staysmigrated/confidence1.0with no new warning.Tests
test_source_metrics_absent_from_query-style unit tests intests/test_grafana_extended.py(TestSourceMetricsAbsentFromQuery),test_curated_override_downgrades_when_source_metric_dropped,test_curated_override_status_ceiling_not_downgraded_when_no_gap,test_curated_override_ignores_hidden_target_when_checking_dropped_metricsintests/test_curated_packs.py, and a strengthened assertion intests/core/metric_mapping/test_cross_index_xy_layers.py.make lintandmake typecheckclean.Kibana visual verification
Performed directly (with prior authorization) against a local Kibana/Elasticsearch stack: downloaded the real "Node Exporter Full" (1860, revision 37) dashboard from grafana.com, migrated it on this branch, uploaded the resulting native artifact, and seeded synthetic telemetry matching the new contract via
obs-migrate seed-sample-data.Review
Independent review performed via a subagent on a different model (GPT 5.6 Sol xhigh) per the team's cross-model review workflow. First pass found the two false-positive/structural bugs described above (both fixed and re-verified live); second pass confirmed Ready to merge.