Summary
PromQL aggregations written without a by clause collapse across all series by definition. When such a target shares a panel with a non-aggregated target, the translator groups every target by the union of all BY fields, so the aggregation is computed inside each individual series instead of across them. The resulting series are not the ones Grafana draws.
In Grafana community dashboard 9852 "node-exporter disk graphs", IO Wait per core has four targets:
# 0, legend "{{ instance }} CPU {{ cpu }}/{{ mode }}"
rate(node_cpu_seconds_total{cpu=~"$CPU", instance=~"$Node", mode="iowait"}[$RateInterval])
# 1, legend "Min"
min( rate(node_cpu_seconds_total{cpu=~"$CPU", instance=~"$Node", mode="iowait"}[$RateInterval]) )
# 2, legend "Avg"
avg( rate(...) )
# 3, legend "Max"
max( rate(...) )
Targets 1–3 have no by, so in Grafana each yields exactly one series — the min/avg/max across all CPUs. The panel shows 5 lines: one per CPU, plus Min, Avg, Max.
The migrated ES|QL groups all four by the same key, which includes labels.cpu:
| STATS CPU = AVG(RATE(metrics.node_cpu_seconds_total)),
Min = MIN(RATE(metrics.node_cpu_seconds_total)),
Avg = AVG(RATE(metrics.node_cpu_seconds_total)),
Max = MAX(RATE(metrics.node_cpu_seconds_total))
BY time_bucket = TBUCKET(100, ?_tstart, ?_tend), labels.instance, labels.cpu, labels.mode
Each (time_bucket, instance, cpu, mode) group holds a single rate value, so MIN, AVG and MAX over it all reduce to that same value. The three summary series stop being cross-CPU aggregates and become per-CPU duplicates of the raw series.
Observed on the uploaded dashboard — 8 legend entries instead of 5:
node-1:9100 CPU 0/iowait - CPU / node-1:9100 CPU 1/iowait - CPU /
node-1:9100 CPU 0/iowait - Min node-1:9100 CPU 1/iowait - Min
node-1:9100 CPU 0/iowait - Avg node-1:9100 CPU 1/iowait - Avg
node-1:9100 CPU 0/iowait - Max node-1:9100 CPU 1/iowait - Max
Grafana, same data and time range:
node-1:9100 CPU 0/iowait node-1:9100 CPU 1/iowait Min Avg Max
On a 2-core box this is merely wrong. On a 64-core host the panel would render 256 series instead of 67, and the Min/Avg/Max lines an operator relies on to spot a hot core would no longer exist.
Disclosure today
The panel is reported migrated_with_warnings, confidence 0.6, reason:
Unioned BY fields across multi-target series with mismatched grouping
So the degradation is flagged, which is the right instinct. But the wording describes a mechanism, not a consequence: it does not tell the operator that Min/Avg/Max no longer mean min/avg/max across cores. Given AGENTS.md's "degrade gracefully rather than hide semantic gaps" rule, this one deserves a reason that names the changed semantics.
Environment
Reproduction
obs-migrate migrate \
--source grafana \
--input-mode api \
--grafana-url "$GRAFANA_URL" \
--field-profile prometheus_native \
--es-url "$ES_URL" \
--es-api-key "$ES_API_KEY" \
--kibana-url "$KIBANA_URL" \
--kibana-api-key "$KIBANA_API_KEY" \
--validate --upload --ensure-data-views
Compare the IO Wait per core legend against the Grafana original.
Note the CPU control must include the .* option for this panel to render at all — otherwise it hits the numeric-parameter compile error filed separately.
Suggested fix
Track each target's aggregation scope through translation instead of unioning the BY fields:
- A target with a bare
min/avg/max/sum (no by) must be grouped by time only.
- Where one ES|QL statement cannot host both groupings, emit the aggregated targets as a separate Lens layer, or compute them in a second
STATS over the first result.
- If neither is feasible, keep the current union but make the reason explicit, e.g. "Min/Avg/Max are computed per cpu, not across cpus, because the panel mixes grouped and ungrouped targets".
Related
Summary
PromQL aggregations written without a
byclause collapse across all series by definition. When such a target shares a panel with a non-aggregated target, the translator groups every target by the union of allBYfields, so the aggregation is computed inside each individual series instead of across them. The resulting series are not the ones Grafana draws.In Grafana community dashboard 9852 "node-exporter disk graphs",
IO Wait per corehas four targets:Targets 1–3 have no
by, so in Grafana each yields exactly one series — the min/avg/max across all CPUs. The panel shows 5 lines: one per CPU, plus Min, Avg, Max.The migrated ES|QL groups all four by the same key, which includes
labels.cpu:Each
(time_bucket, instance, cpu, mode)group holds a single rate value, soMIN,AVGandMAXover it all reduce to that same value. The three summary series stop being cross-CPU aggregates and become per-CPU duplicates of the raw series.Observed on the uploaded dashboard — 8 legend entries instead of 5:
Grafana, same data and time range:
On a 2-core box this is merely wrong. On a 64-core host the panel would render 256 series instead of 67, and the Min/Avg/Max lines an operator relies on to spot a hot core would no longer exist.
Disclosure today
The panel is reported
migrated_with_warnings, confidence 0.6, reason:So the degradation is flagged, which is the right instinct. But the wording describes a mechanism, not a consequence: it does not tell the operator that
Min/Avg/Maxno longer mean min/avg/max across cores. GivenAGENTS.md's "degrade gracefully rather than hide semantic gaps" rule, this one deserves a reason that names the changed semantics.Environment
feat/curated-dashboard-packs(PR feat!: curated dashboard packs, PromQL fidelity, and native-only dashboard artifacts #346) at5db742f9.5.0-SNAPSHOTnode_exporter, 2 cores--field-profile prometheus_native, no curated pack appliesReproduction
Compare the
IO Wait per corelegend against the Grafana original.Note the
CPUcontrol must include the.*option for this panel to render at all — otherwise it hits the numeric-parameter compile error filed separately.Suggested fix
Track each target's aggregation scope through translation instead of unioning the
BYfields:min/avg/max/sum(noby) must be grouped by time only.STATSover the first result.Related