Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/sources/grafana.md
Original file line number Diff line number Diff line change
Expand Up @@ -869,6 +869,7 @@ is available at `examples/cue/grafana-rule-pack.cue`.
- `label_join(v, dst, separator, src1, src2, ...)` translates to a post-`STATS` `| EVAL dst = CONCAT(src1, "separator", src2, ...)` when all source labels appear in the inner expression's `by()` clause. If any source label is absent from the `by()` clause, the panel stays `not_feasible` (the column would not exist in the `STATS` output and `CONCAT` cannot reference it).
- `label_replace(v, dst, replacement, src, regex)` follows the same rule when the replacement references the source label (a `$1`-style capture): the post-`STATS` `| EVAL`/`| GROK` can only read `src` if the inner expression's result still carries it, so forms that aggregate it away stay `not_feasible` rather than emitting a query that fails on an unknown column. This includes nested aggregations, whose grouping comes from the source's own `by()` clauses (issue #382) and whose outer `STATS` drops the inner grouping regardless — `label_replace(max(sum by (namespace) (m)), "ns", "$1", "namespace", "(.*)")` is not feasible even though `namespace` appears in an inner `by()`. Forms that read no source column are unaffected: a literal replacement emits `EVAL dst = "literal"`, and an identity copy (`dst` resolving to the same target field as `src`) emits nothing at all.
- Histogram mean idioms `sum(increase|rate(m_sum) / increase|rate(m_count))` approximate as a ratio of aggregates (`sum(m_sum)/sum(m_count)`) with an explicit warning; unrelated per-element ratios stay `not_feasible`.
- An aggregation **wrapping** a binary operator (`agg(A op B)`) is only feasible when a rewrite can prove the operator survives, because PromQL evaluates `A op B` per matching series pair — matching on the operands' full label set — before the aggregation reduces the survivors, and ES|QL has no equivalent stage. The rewrites are: `sum(A ± B)` push-down (`sum` is linear), scalar hoisting (`max(A * 8)`, `avg(A / 100)`), the histogram mean idiom above, and the co-located per-document renderer (`count(A + B)` → `COUNT((A + B))`, for `+ - * /` over operands that share a label set). Everything else stays `not_feasible` with a warning naming the operator and the aggregation: set operators `and` / `unless` (`count(A and B >= threshold)` — issue #377), comparisons between two series (`count(A < B)`; compare against a scalar threshold instead), and arithmetic the renderer does not cover (`%`, `^`, `atan2`, or operands that carry their own `by()` reduction). The refusal is an allowlist of operators that are handed on rather than a list of operators that are refused, so an operator nobody enumerated fails closed instead of silently reaching the fallback. Previously these fell through to the generic single-metric fallback, which rebuilt `agg(<first metric>)` and shipped a plausible but wrong number as `migrated_with_warnings`. `agg(A or B)` is deliberately excluded from this refusal: it keeps its established reductions (the same-metric range-window fallback `max_over_time(M[$interval]) or max_over_time(M[5m])`, and the live-absent operand drop).
- Multi-target XY panels fuse when series share a compatible ES|QL shape. Summary panels (`stat` / `singlestat` / `gauge` / `bargauge` / table) use the same compatibility group and approximate multi-series stats as a summary table when needed. Grouping mismatches where a bare aggregation (no `by()`, e.g. `min`/`avg`/`max`/`sum`) shares its metric field with a grouped sibling target — the disk-graphs "IO Wait per core" Min/Avg/Max-over-CPUs shape (issue #355) — render as two separate ES|QL layers instead of unioning the `BY` fields: one layer keeps the grouped target's own dimensions, the other re-aggregates the bare target across every group (time-bucket only), matching what Grafana always draws as a single cross-series line. When the bare target instead broadcasts an *unrelated* metric alongside a grouped breakdown (e.g. QoS `by (qos_class)` + an ungrouped fleet-wide total) there is no shared per-series computation to split out, so that target's `BY` fields still union with the grouped target's, but the warning now names the changed semantics (which series is affected, the fields it is now grouped by) instead of only describing the mechanism; an unrelated bare target sharing the panel does not disable the split for the pair that *does* share a metric. A lone bare target with no Min/Avg/Max siblings still gets its own summary layer, aliased to its own `legendFormat` text (not the raw metric field name); the same split applies within a data stream even when another fused target lands on a different index partition. Divergent label filters on otherwise identical measures CASE-inline into the shared `STATS` (including window-less `LAST_OVER_TIME`, used by Express-style status-class counters). `legendFormat` `{{label}}` placeholders on `rate`/`irate`/`increase` (and other TS paths covered by issue #99) are display hints — they become series aliases, not `BY` dimensions — so overlays like Redis in/out rates can share one panel. When fused targets carry *different* multi-placeholder `legendFormat` templates (issue #354), the composite `EVAL legend` expression is suppressed so the `STATS` column name carries series identity instead of mislabelling every series with the first target's literal text; targets whose templates agree, or that use only a single placeholder, are unaffected. Targets that remain incompatible (Windows vs Linux metrics, complex `or`/`label_replace` trees) still keep the largest compatible group and warn; Windows-specific drop wording only applies when every dropped target is a `windows_*` metric.
- Grafana **stat/gauge tiles are one-per-series** (a status grid). Translation preserves the source PromQL expression's grouping and cardinality rather than inferring `BY` dimensions from `legendFormat` text alone. An ungrouped `up` (or similar) that Grafana would still fan out per scrape target is migrated **with a warning**; outer aggregations such as `sum(rate(...))` stay scalars even when the legend contains `{{job}}`. The same rule governs **nested aggregations** on every panel type: the grouping of `max(sum(m))` comes from the source's own `by()` clauses and nothing else — legend placeholders, legacy table column patterns, and dashboard-wide label inference are all display hints. In a nested aggregation the inner grouping additionally decides what the *outer* aggregation reduces over, so a hint-derived inner `BY` made `max(sum(m))` report the largest single label value instead of the collapsed total, and a hint-derived outer `BY` turned the scalar `count(count by (cpu) (m))` into a constant `1` per CPU (issue #382).
- Composite/multi-target series (a fused native-PROMQL `value` column, the general ES|QL translator's scalar-expression `computed_value` column, or a curated-pack override that folds several source metrics into one `value` column with a `series_group` breakdown) never surface that internal column name to the operator. An unambiguous static `legendFormat` shared by every visible target takes priority; mixed or templated legends fall back to the panel title as the metric's label. A single such metric with a breakdown uses that label as the Y-axis title when Grafana left the axis unnamed and no uniform unit title (`%`, `Bytes`, …) can be inferred (issue #351). An explicit Grafana axis label still wins; opaque aliases such as `percentage` are ignored so a unit title can apply. A panel with no usable title/legend text falls back to the prior hidden-title behavior.
Expand Down
79 changes: 72 additions & 7 deletions observability_migration/adapters/source/grafana/promql.py
Original file line number Diff line number Diff line change
Expand Up @@ -2925,6 +2925,65 @@ def _join_rhs_not_plain_selector_reason(right_frag):
)


_COMPARISON_OPERATORS = frozenset({"==", "!=", ">", "<", ">=", "<="})

# Operators the closing guard below hands on instead of refusing. Only ``or``
# qualifies: unlike ``and``/``unless`` it has established reductions the later
# stages own -- the Grafana same-metric range-window fallback
# (``max_over_time(M[$interval]) or max_over_time(M[5m])``) and the live-absent
# operand drop in ``colocated_binary_agg_plan`` -- both of which need a resolver
# that parse time does not have.
#
# This is an allowlist rather than a list of refused operators so an operator
# nobody enumerated fails closed. A deny-list silently omitted ``atan2``, which
# reintroduced exactly the mistranslation the guard exists to stop.
_AGG_OVER_BINARY_DEFERRED_OPS = frozenset({"or"})


def _agg_over_binary_not_feasible_reason(outer_agg, op):
"""Explain why ``agg(A <op> B)`` has no honest ES|QL rendering.

PromQL evaluates the inner operator per matching series pair *before* the
aggregation reduces them, and matching is on the operands' full label set.
ES|QL has no equivalent stage, so unless a dedicated rewrite applies the
only faithful answer is to refuse.
"""
agg = outer_agg or "aggregation"
op = (op or "").strip()
if not op:
return (
f"Aggregating over an unrecognised PromQL binary expression ({agg}(A op B)) "
"cannot be expressed accurately in ES|QL; the operands must be matched on "
"their full label set before the aggregation reduces them"
)
if op.lower() in _SET_OPERATORS:
return (
f"PromQL set operator '{op.lower()}' inside an aggregation "
f"({agg}(A {op.lower()} B)) has no honest ES|QL translation; it selects which "
"series survive by matching the operands on their full label set before "
f"{agg}() reduces them, so the operand cannot be dropped without changing "
"the result; marked not_feasible"
)
if op in _COMPARISON_OPERATORS:
return (
f"PromQL comparison '{op}' between two time-series inside an aggregation "
f"({agg}(A {op} B)) filters one series by another and has no honest ES|QL "
"translation; compare against a scalar threshold instead of a second "
"series; marked not_feasible"
)
if op in {"*", "/"}:
return (
f"Aggregating over a per-element {op} between two time-series "
f"({agg}(A {op} B)) cannot be expressed accurately in ES|QL; "
"rewrite as a ratio of aggregates if the series are label-aligned"
)
return (
f"Aggregating over a per-element {op} between two time-series "
f"({agg}(A {op} B)) cannot be expressed accurately in ES|QL; the operands "
"must be matched on their full label set before the aggregation reduces them"
)


def _ast_aggregate_fragment(node, expr):
child = _ast_from_node(node.expr, _ast_node_expr(node.expr))
frag = _copy_fragment_summary(_new_fragment(expr), child)
Expand Down Expand Up @@ -3131,8 +3190,8 @@ def _ast_aggregate_fragment(node, expr):
# Handle aggregation over a binary expression between two time-series.
# SUM is linear so sum(A ± B) = sum(A) ± sum(B); push the aggregation
# down to each operand and return a binary_expr the pipeline can handle.
# Division and multiplication are not linear: sum(A/B) ≠ sum(A)/sum(B),
# so those patterns are marked not_feasible rather than silently dropped.
# Every other shape either has a dedicated rewrite below or is refused by
# the closing guard, because no rewrite means the operator is dropped.
if child.family == "binary_expr":
inner_left = child.extra.get("left_frag")
inner_right = child.extra.get("right_frag")
Expand Down Expand Up @@ -3204,13 +3263,19 @@ def _ast_aggregate_fragment(node, expr):
new_binary.group_mode = frag.group_mode
new_binary.extra["approximated_agg_over_summary_ratio"] = True
return new_binary
# Two true time-series operands — multiplication/division is not
# linearisable: agg(A op B) ≠ agg(A) op agg(B).
# Nothing above could rewrite ``agg(A op B)`` honestly. Refuse instead
# of returning a bare ``unknown`` fragment: the generic
# ``fragment_extract``/``stats_expression`` fallback rebuilds
# ``agg(<first metric leaf>)`` from the fragment's summary fields, which
# discards the operator and every other operand and ships a plausible
# but wrong number (issue #377). ``colocated_binary_agg_unblock`` clears
# this reason again for the arithmetic the co-located renderer can
# express exactly, so the refusal only sticks where nothing else can.
op_lower = (child.binary_op or "").strip().lower()
if op_lower not in _AGG_OVER_BINARY_DEFERRED_OPS and not frag.extra.get("not_feasible_reasons"):
_append_not_feasible_reason(
frag,
f"Aggregating over a per-element {child.binary_op} between two time-series "
f"({frag.outer_agg}(A {child.binary_op} B)) cannot be expressed accurately in ES|QL; "
"rewrite as a ratio of aggregates if the series are label-aligned",
_agg_over_binary_not_feasible_reason(frag.outer_agg, child.binary_op),
)

return frag
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
source: count((max by (persistentvolumeclaim,namespace) (kubelet_volume_stats_used_bytes)) and ((max by (persistentvolumeclaim,namespace) (kubelet_volume_stats_used_bytes)) / (max by (persistentvolumeclaim,namespace) (kubelet_volume_stats_capacity_bytes))) >= (80 / 100)) or vector(0)
feasibility: not_feasible
warning: PromQL set operator 'and' inside an aggregation (count(A and B)) has no honest ES|QL translation; it selects which series survive by matching the operands on their full label set before count() reduces them, so the operand cannot be dropped without changing the result; marked not_feasible
---

Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
source: count(node_filesystem_avail_bytes < node_filesystem_size_bytes)
feasibility: not_feasible
warning: PromQL comparison '<' between two time-series inside an aggregation (count(A < B)) filters one series by another and has no honest ES|QL translation; compare against a scalar threshold instead of a second series; marked not_feasible
---

Loading
Loading