grafana-migrate: multi-label legend `{{verb}} {{resource_kind}}` collapses to a single breakdown — series visually merged

[miguel-sanchez-elastic]

Summary

When migrating a Grafana time-series panel whose legend references two or more labels (e.g. {{verb}} {{resource_kind}}), the migrated Kibana panel ends up with only the first label in the Breakdown. The chart then collapses all series that share the first label, so it looks visibly wrong versus the source. Adding the second field to the Breakdown manually in Kibana fixes it.

Source

• Dashboard: ArgoCD (official argo-cd-mixin), Grafana Labs #14584
• Panel: "K8s API Activity"
• Query: sum(increase(argocd_app_k8s_request_total{namespace=~"$namespace",server=~"$cluster"}[$interval])) by (verb, resource_kind)
• Legend: {{verb}} {{resource_kind}}

Expected

The migrated panel should break the series down by both verb and resource_kind (Grafana shows one line per verb×resource_kind combination). The engine already has logic for this — for 2+ legend labels it is supposed to synthesize a composite EVAL legend = CONCAT(...) column and set breakdown.field = "legend".

Actual

Only verb lands in the Kibana Breakdown. The chart shows one series per verb (verb_1, verb_2, verb_3) instead of per verb×resource_kind, so series that differ only by resource_kind are merged and the shape no longer matches the source. Manually adding resource_kind to the Breakdown in the Kibana panel editor restores the correct chart — confirming the field is present and queryable; it just wasn't wired into the breakdown.

Likely area

The composite-legend path (_apply_composite_legend_to_xy_panel, observability_migration/adapters/source/grafana/panels.py) appears to have bailed out for this panel and fallen back to the single-field breakdown selected in select_xy_dimension_fields (observability_migration/targets/kibana/emit/esql_utils.py), which only ever takes the second by column. The "extra grouping dimension(s) … visually merged" warning likely fired. Needs investigation into why the composite legend didn't apply when both labels are valid output columns.

Acceptance criteria

• Migrating the ArgoCD "K8s API Activity" panel produces a Kibana chart broken down by both verb and resource_kind (composite legend), matching the source.
• Multi-label legends where all labels resolve to real output columns no longer silently collapse to a single breakdown field.
• Regression test covering a {{a}} {{b}} legend over a by (a, b) query.

[shmsr]

Additional corpus pass confirms this is broader than the ArgoCD panel in the issue description.

I scanned the six downloaded dashboards plus the repo's representative benchmark corpus (Node Exporter Full, Apache Full, HAProxy, NFS, Bind9 Full, Unbound Full). The broader corpus adds 60 multi-label legend / multi-label grouping targets, mostly native PROMQL timeseries panels.

Common affected shapes include:

• ArgoCD: {{verb}} {{resource_kind}} over by (verb, resource_kind) and multiple {{grpc_code}} {{grpc_method}} panels.
• HAProxy: {{proxy}} {{server}}, {{code}} {{proxy}}, {{code}} {{proxy}} {{server}}.
• Node Exporter: {{chip_name}} {{sensor}}, {{name}} {{type}}.
• Apache: {{balancer}} {{worker}}, {{balancer}} {{worker}} {{status}}.
• Bind9: {{type}} {{view}}, {{error}} {{view}}, {{le}} {{view}}.

The ES|QL path often already emits EVAL legend = CONCAT(...) and breakdown.field = legend. The weak path is native PROMQL XY panel generation, where labels are valid output columns (from explicit by(...) or _timeseries extraction) but the Kibana panel config still chooses only one breakdown field.

Planned fix: teach native PROMQL XY panel generation to synthesize a composite legend column and use breakdown.field = legend for 2+ legend placeholders, preserving the underlying label columns for parity/debugging.

[shmsr]

Follow-up after schema + real Kibana verification:

I checked docs/dashboards/schema.json and the compiler output rather than relying on the UI alone. The YAML schemas for Lens XY charts (line, area, bar) expose a singular breakdown, while multi-breakdowns arrays are present for datatable/pie/treemap-style schemas. That means the schema-supported way to represent a Grafana multi-label series identity in an XY chart is to synthesize one composite field, e.g.:

| EVAL legend = CONCAT(COALESCE(TO_STRING(verb), ""), " ", COALESCE(TO_STRING(resource_kind), ""))

and then emit:

breakdown:
  field: legend

I also verified this against the actual uploaded ArgoCD dashboard, not just YAML text:

• The saved Lens panel for K8s API Activity contains the legend textBased column.
• The ES|QL/PROMQL query returns rows with legend values such as verb_1 resource_kind_3.
• In a clean Kibana view-mode session (after clearing stale dashboard edit state), the visible legend shows combined labels like verb_1 resource_kind_3, not just verb_1.

One caveat from debugging: Kibana can retain stale unsaved dashboard edit state in local/session storage, which can make the browser appear to show the old split. Clean view-mode validation is required before judging the uploaded saved object.

I am also doing an empirical compiler experiment with breakdowns/multi-breakdown YAML in case the docs are behind the implementation; will update if the compiler proves a first-class multi-breakdown XY shape exists.

[shmsr]

Empirical compiler check confirms the schema reading:

I created minimal XY line-chart YAMLs and ran them through the actual kb-dashboard-cli compile path:

1. esql.type: line with:

breakdowns:
- field: verb
- field: resource_kind

Compiler result:

dashboards[0].panels[0].esql.line.breakdowns: Extra inputs are not permitted

2. esql.type: line with:

breakdown:
- field: verb
- field: resource_kind

Compiler result:

dashboards[0].panels[0].esql.line.breakdown: Input should be a valid dictionary or instance of ESQLBreakdown

So even if the Kibana UI visually offers “Add a field,” the YAML/compiler contract for ES|QL XY charts currently supports only one breakdown dimension. The schema-correct migration fix remains: synthesize a single composite series field (legend = CONCAT(...)) and use that one field as breakdown.field.

[shmsr]

More explicit problem statement and verification notes:

What users see

For Grafana time-series panels whose legend is built from multiple labels, for example:

{{verb}} {{resource_kind}}
{{grpc_code}} {{grpc_method}}
{{proxy}} {{server}}

Grafana renders one series per full label tuple. So verb=GET, resource_kind=Pod and verb=GET, resource_kind=Deployment are distinct lines.

The migrated Kibana Lens XY panel previously selected only one field as the visual breakdown, for example:

breakdown:
  field: verb

That means all series sharing verb=GET were visually merged even if their resource_kind values differed. This is a correctness bug: the query can return correct rows, and smoke can pass, while the chart is still visually wrong.

Why not use multiple YAML breakdown fields?

I checked both docs/dashboards/schema.json and the actual kb-dashboard-cli compile behavior. For ES|QL/Lens XY chart schemas (line, area, bar), the contract supports exactly one breakdown object. It does not support a breakdowns array.

Empirical compiler checks:

esql:
  type: line
  breakdowns:
    - field: verb
    - field: resource_kind

fails with:

esql.line.breakdowns: Extra inputs are not permitted

and:

esql:
  type: line
  breakdown:
    - field: verb
    - field: resource_kind

fails with:

esql.line.breakdown: Input should be a valid dictionary or instance of ESQLBreakdown

Multi-breakdown arrays exist elsewhere in the schema, such as datatable/pie/treemap-style panels, but not XY charts.

Correct migration strategy for XY charts

For XY charts, the schema-compatible way to preserve multi-label series identity is to synthesize one composite field:

| EVAL legend = CONCAT(
    COALESCE(TO_STRING(verb), ""),
    " ",
    COALESCE(TO_STRING(resource_kind), "")
  )

and then use:

breakdown:
  field: legend

The underlying label columns should remain in the query output where possible, so parity/debugging can still inspect verb and resource_kind separately. Lens uses legend as the single split field, satisfying the XY schema while preserving Grafana's full series identity.

Scope is broader than ArgoCD

A broader scan found this pattern in multiple dashboard families:

• ArgoCD: {{verb}} {{resource_kind}}, {{grpc_code}} {{grpc_method}}
• HAProxy: {{proxy}} {{server}}, {{code}} {{proxy}} {{server}}
• Node Exporter: {{chip_name}} {{sensor}}, {{name}} {{type}}
• Apache: {{balancer}} {{worker}}, {{balancer}} {{worker}} {{status}}
• Bind9: {{type}} {{view}}, {{error}} {{view}}, {{le}} {{view}}

Verification requirement

This bug cannot be proven fixed by query success alone. Required verification is:

1. Generated YAML has breakdown.field: legend.
2. Compiled saved object's Lens splitAccessor points at the legend column.
3. Direct _query returns a legend column with combined values, e.g. verb_1 resource_kind_3.
4. A clean Kibana view-mode browser session shows combined legend labels.

Important: Kibana can retain stale dashboard edit state in localStorage/sessionStorage, so browser checks must use a clean view-mode session before judging the uploaded saved object.

[shmsr]

Tooling / schema limitations discovered while debugging this:

What kb-dashboard-cli + dashboard YAML supports for XY charts

For Lens XY charts (line, area, bar), the YAML/compiler path supports:

esql:
  type: line
  dimension:
    field: time_bucket
  metrics:
    - field: value
  breakdown:
    field: some_single_field

That compiles to a Lens saved object with a single splitAccessor. This is the only supported XY split dimension in the current YAML contract.

What it does NOT support for XY charts

It does not support multiple breakdown fields for XY charts, even though the Kibana UI may show an “Add a field” affordance.

These shapes fail at compile time:

esql:
  type: line
  breakdowns:
    - field: verb
    - field: resource_kind

esql.line.breakdowns: Extra inputs are not permitted

and:

esql:
  type: line
  breakdown:
    - field: verb
    - field: resource_kind

esql.line.breakdown: Input should be a valid dictionary or instance of ESQLBreakdown

So the tool’s schema has a hard limitation: one XY breakdown only.

Where multi-breakdown is available

The schema does contain breakdowns arrays, but they are for other chart families, e.g. datatable/pie/treemap-style panels. They are not accepted by ES|QL/Lens XY chart configs.

Why this is dangerous for migration validation

This limitation can make a migration look superficially successful:

• The ES|QL/PROMQL query runs.
• The YAML validates.
• The dashboard uploads.
• Smoke tests can pass.
• Kibana renders a chart.

But the chart can still be semantically wrong because Lens uses only one split field and visually merges series that Grafana kept separate.

For example, if Grafana legend identity is (verb, resource_kind), splitting only by verb collapses all resource kinds for the same verb into one series. This is a visual correctness bug, not a query execution bug.

Practical workaround / correct migration shape

Because the tool only accepts one XY breakdown, the migration engine must create one field that encodes the full Grafana series identity:

| EVAL legend = CONCAT(
    COALESCE(TO_STRING(verb), ""),
    " ",
    COALESCE(TO_STRING(resource_kind), "")
  )

Then the YAML uses the one supported split:

breakdown:
  field: legend

This is not just a workaround; it is the schema-compatible way to preserve Grafana’s multi-label legend semantics through the current toolchain.

Tooling gap to keep in mind

The UI can imply capability that the YAML/compiler contract does not expose. For migration work we should treat the compiler/schema as the source of truth, and verify by inspecting the compiled saved object (splitAccessor, textBased columns), not only the visual editor controls.