Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/sources/grafana.md
Original file line number Diff line number Diff line change
Expand Up @@ -631,6 +631,7 @@ is available at `examples/cue/grafana-rule-pack.cue`.
- `label_join(v, dst, separator, src1, src2, ...)` translates to a post-`STATS` `| EVAL dst = CONCAT(src1, "separator", src2, ...)` when all source labels appear in the inner expression's `by()` clause. If any source label is absent from the `by()` clause, the panel stays `not_feasible` (the column would not exist in the `STATS` output and `CONCAT` cannot reference it).
- Histogram mean idioms `sum(increase|rate(m_sum) / increase|rate(m_count))` approximate as a ratio of aggregates (`sum(m_sum)/sum(m_count)`) with an explicit warning; unrelated per-element ratios stay `not_feasible`.
- Multi-target XY panels fuse when series share a compatible ES|QL shape. Summary panels (`stat` / `singlestat` / `gauge` / `bargauge` / table) use the same compatibility group and approximate multi-series stats as a summary table when needed. Grouping mismatches where one target's groups are a subset of another's (e.g. QoS `by (qos_class)` + ungrouped total) union the BY fields with a warning. Divergent label filters on otherwise identical measures CASE-inline into the shared `STATS` (including window-less `LAST_OVER_TIME`, used by Express-style status-class counters). `legendFormat` `{{label}}` placeholders on `rate`/`irate`/`increase` (and other TS paths covered by issue #99) are display hints — they become series aliases, not `BY` dimensions — so overlays like Redis in/out rates can share one panel. Targets that remain incompatible (Windows vs Linux metrics, complex `or`/`label_replace` trees) still keep the largest compatible group and warn; Windows-specific drop wording only applies when every dropped target is a `windows_*` metric.
- Composite/multi-target series (a fused native-PROMQL `value` column, the general ES|QL translator's scalar-expression `computed_value` column, or a curated-pack override that folds several source metrics into one `value` column with a `series_group` breakdown) never surface that internal column name to the operator. An unambiguous static `legendFormat` shared by every visible target takes priority; mixed or templated legends fall back to the panel title as the metric's label. A single such metric with a breakdown uses that label as the Y-axis title when Grafana left the axis unnamed and no uniform unit title (`%`, `Bytes`, …) can be inferred (issue #351). An explicit Grafana axis label still wins; opaque aliases such as `percentage` are ignored so a unit title can apply. A panel with no usable title/legend text falls back to the prior hidden-title behavior.
- Kibana ES|QL visualizations are still effectively single-query / single data layer. Independent Grafana queries that cannot fuse into one wide ES|QL statement cannot be overlaid the way Grafana does; that is a platform limit, not a silent drop.
- Mixed-datasource and mixed-query-language panels are still weaker than single-source Prometheus or Loki paths.
- Verification is strongest when live Prometheus/Loki and Elasticsearch are available, but full measured source-vs-target comparison is still partial.
Expand Down
49 changes: 46 additions & 3 deletions observability_migration/adapters/source/grafana/panels.py
Original file line number Diff line number Diff line change
Expand Up @@ -2082,7 +2082,37 @@ def _static_legend_label(legend_format):
return label


def _label_native_promql_value_metric(yaml_panel, *, title, legend_format=""):
def _panel_static_legend_label(panel):
"""Return the one unambiguous static legend shared by visible targets."""
labels = []
for target in panel.get("targets") or []:
if not isinstance(target, dict) or target.get("hide"):
continue
label = _static_legend_label(target.get("legendFormat", ""))
if not label:
return ""
if label not in labels:
labels.append(label)
return labels[0] if len(labels) == 1 else ""


_PLACEHOLDER_VALUE_METRIC_FIELDS = frozenset({"value", "computed_value"})


def _label_placeholder_value_metric(yaml_panel, *, title, legend_format=""):
"""Give a synthetic ``value``/``computed_value`` metric column a real label.

Both the native-PROMQL path (single ``value=(...)`` column) and the
general ES|QL translator (single ``computed_value`` scalar-expression
column) collapse a panel's target(s) into one numeric column with a
placeholder name. Kibana's Lens then falls back to that raw column name
-- ``value``/``computed_value`` -- as the axis/legend label (issue #351).
Curated-pack overrides that fuse multiple metrics into one ``value``
column via ``EVAL value = ...`` hit the same gap.

Uses the same fallback as a single-target panel: the target's static
legend text if the operator set one, otherwise the panel title.
"""
esql = yaml_panel.get("esql")
if not isinstance(esql, dict):
return
Expand All @@ -2098,7 +2128,7 @@ def _label_native_promql_value_metric(yaml_panel, *, title, legend_format=""):
for metric in metrics:
if not isinstance(metric, dict):
continue
if metric.get("field") != "value":
if metric.get("field") not in _PLACEHOLDER_VALUE_METRIC_FIELDS:
continue
metric.setdefault("label", fallback_label)

Expand Down Expand Up @@ -2730,7 +2760,7 @@ def _translate_panel_native_promql(

yaml_panel["esql"] = native_panel
enrich_yaml_panel_display(yaml_panel, panel)
_label_native_promql_value_metric(yaml_panel, title=title, legend_format=legend_format)
_label_placeholder_value_metric(yaml_panel, title=title, legend_format=legend_format)
_apply_series_override_axes(yaml_panel, panel, [])

notes = list(panel_notes) + ["Native PROMQL: original PromQL reused via ES|QL PROMQL command"]
Expand Down Expand Up @@ -3783,6 +3813,11 @@ def translate_panel(panel, datasource_index="metrics-*", esql_index=None, rule_p
yaml_panel, panel, _override_warnings
)
enrich_yaml_panel_display(yaml_panel, panel)
_label_placeholder_value_metric(
yaml_panel,
title=title,
legend_format=_panel_static_legend_label(panel),
)
_score = 1.0 if _status == "migrated" else 0.7
_override_notes = list(panel_notes)
if _status == "migrated":
Expand Down Expand Up @@ -4383,6 +4418,14 @@ def translate_panel(panel, datasource_index="metrics-*", esql_index=None, rule_p
panel,
metric_labels=metric_labels or None,
)
_label_placeholder_value_metric(
yaml_panel,
title=title,
# Same rule as curated overrides: only an unambiguous static legend
# shared by every visible target is safe on a fused series. Mixed
# legends fall back to the panel title rather than the primary target.
legend_format=_panel_static_legend_label(panel) or static_legend_label or "",
)
_apply_series_override_axes(yaml_panel, panel, primary.warnings)
if yaml_panel.get("esql", {}).get("query"):
primary.esql_query = yaml_panel["esql"]["query"]
Expand Down
19 changes: 17 additions & 2 deletions observability_migration/targets/kibana/compile.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,15 +116,30 @@ def _sync_dimension(field_name):
breakdown["field"] = "series_group"
changed = True
format_cfg = None
label_cfg = None
metrics = esql_config.get("metrics")
if isinstance(metrics, list):
for item in metrics:
if isinstance(item, dict) and isinstance(item.get("format"), dict):
if not isinstance(item, dict):
continue
if format_cfg is None and isinstance(item.get("format"), dict):
format_cfg = copy.deepcopy(item["format"])
break
if (
label_cfg is None
and str(item.get("field") or item.get("column") or "").strip()
in {"value", "computed_value"}
and str(item.get("label") or "").strip()
):
label_cfg = item["label"]
new_metric_item = {"field": "value"}
if format_cfg:
new_metric_item["format"] = format_cfg
# Preserve a caller-derived label (panel title / static legend
# fallback for the placeholder ``value`` column, issue #351) across
# this rebuild -- otherwise a post-validation query swap silently
# regresses a labeled axis back to the raw column name.
if label_cfg:
new_metric_item["label"] = label_cfg
if esql_config.get("metrics") != [new_metric_item]:
esql_config["metrics"] = [new_metric_item]
changed = True
Expand Down
106 changes: 61 additions & 45 deletions observability_migration/targets/kibana/dashboards_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -740,11 +740,19 @@ def _api_legend(legend: Any, kind: str) -> dict[str, Any] | None:
}


# YAML/visual-IR spellings that put a series on the right-hand axis. Kept in
# sync with the ``axis`` mapping in :func:`_api_column` (role ``xy_y``).
_RIGHT_AXIS_VALUES = {"right", "y2"}


def _infer_y_axis_title(metrics: list[Any]) -> str:
"""Return a Y-axis title inferred from a uniform metric format, or ''."""
left_formats = set()
for m in metrics:
if not isinstance(m, dict) or m.get("axis") == "right":
if (
not isinstance(m, dict)
or str(m.get("axis") or "") in _RIGHT_AXIS_VALUES
):
continue
fmt = m.get("format")
if isinstance(fmt, dict):
Expand Down Expand Up @@ -774,11 +782,6 @@ def _xy_metrics(cfg: dict[str, Any]) -> list[Any]:
return metrics


# YAML/visual-IR spellings that put a series on the right-hand axis. Kept in
# sync with the ``axis`` mapping in :func:`_api_column` (role ``xy_y``).
_RIGHT_AXIS_VALUES = {"right", "y2"}


def _has_right_axis_series(cfg: dict[str, Any]) -> bool:
"""True when at least one series is plotted against the right-hand axis."""
return any(
Expand Down Expand Up @@ -870,48 +873,63 @@ def _ensure_temporal_x_scale(
return out


def _xy_config_has_breakdown(cfg: dict[str, Any]) -> bool:
"""True when the XY config (or any layer) declares a breakdown dimension."""
if cfg.get("breakdown") or cfg.get("breakdowns"):
return True
layers = cfg.get("layers")
if isinstance(layers, list):
for layer in layers:
if isinstance(layer, dict) and (layer.get("breakdown") or layer.get("breakdowns")):
return True
return False


def _xy_axis_from_cfg(cfg: dict[str, Any]) -> dict[str, Any] | None:
"""Return the XY axis config, falling back to format-inferred Y title."""
axis = _api_xy_axis(_cfg_axis_source(cfg))
if axis and (axis.get("y") or {}).get("title"):
return axis
metrics = cfg.get("metrics") if isinstance(cfg.get("metrics"), list) else []
# Include ``layers[*].metrics`` so cross-data-stream XY panels get the same
# left-axis title decision as a top-level ``metrics`` list.
metrics = _xy_metrics(cfg)
left = [
m for m in metrics
if (
isinstance(m, dict)
and str(m.get("axis") or "") not in _RIGHT_AXIS_VALUES
)
]

def _with_y_title(title: dict[str, Any]) -> dict[str, Any]:
if axis:
merged = dict(axis)
merged["y"] = {**merged.get("y", {}), **title}
return merged
return {"y": title}

inferred = _infer_y_axis_title(metrics)
if not inferred:
# No unit-derived title available. With two or more left-axis series
# Kibana falls back to the FIRST series' name, which mislabels the whole
# axis ("not expiring" on a chart plotting not-expiring AND expiring;
# "hits" on hits AND misses). Grafana showed no axis title at all here
# (axisLabel is empty), so hide it rather than let Kibana invent a
# wrong one. A single series keeps the default, where the column name
# does describe the axis.
left = [
m for m in metrics
if isinstance(m, dict) and m.get("axis") != "right"
]
if len(left) >= 2:
# ``visible: false`` alone hides the title; a companion ``text: ""``
# names nothing and Kibana does not store it (see _api_axis_title).
hidden: dict[str, Any] = {"title": {"visible": False}}
if axis:
merged_hidden = dict(axis)
merged_hidden["y"] = {**merged_hidden.get("y", {}), **hidden}
return merged_hidden
return {"y": hidden}
if len(left) == 1 and _xy_single_metric_uses_placeholder_name(cfg, left[0]):
hidden = {"title": {"visible": False}}
if axis:
merged_hidden = dict(axis)
merged_hidden["y"] = {**merged_hidden.get("y", {}), **hidden}
return merged_hidden
return {"y": hidden}
return axis
y_title: dict[str, Any] = {"title": {"text": inferred, "visible": True}}
if axis:
merged = dict(axis)
merged["y"] = {**merged.get("y", {}), **y_title}
return merged
return {"y": y_title}
if inferred:
# A uniform unit title ("%"/ "Bytes") is more useful as an axis name
# than repeating the panel title on a composite ``value`` series.
return _with_y_title({"title": {"text": inferred, "visible": True}})
if len(left) == 1 and _xy_single_metric_uses_placeholder_name(cfg, left[0]):
placeholder_label = str(left[0].get("label") or "").strip()
if placeholder_label:
return _with_y_title({"title": {"text": placeholder_label, "visible": True}})
return _with_y_title({"title": {"visible": False}})
# No unit-derived title available. With two or more left-axis series
# Kibana falls back to the FIRST series' name, which mislabels the whole
# axis ("not expiring" on a chart plotting not-expiring AND expiring;
# "hits" on hits AND misses). Grafana showed no axis title at all here
# (axisLabel is empty), so hide it rather than let Kibana invent a
# wrong one. A single series keeps the default, where the column name
# does describe the axis.
if len(left) >= 2:
# ``visible: false`` alone hides the title; a companion ``text: ""``
# names nothing and Kibana does not store it (see _api_axis_title).
return _with_y_title({"title": {"visible": False}})
return axis


def _xy_single_metric_uses_placeholder_name(cfg: dict[str, Any], metric: Any) -> bool:
Expand All @@ -924,9 +942,7 @@ def _xy_single_metric_uses_placeholder_name(cfg: dict[str, Any], metric: Any) ->
"""
if not isinstance(metric, dict):
return False
breakdown = cfg.get("breakdown")
breakdowns = cfg.get("breakdowns")
if not breakdown and not breakdowns:
if not _xy_config_has_breakdown(cfg):
return False
field_name = str(metric.get("field") or metric.get("column") or "").strip()
return field_name in {"value", "computed_value"}
Expand Down
16 changes: 14 additions & 2 deletions observability_migration/targets/kibana/emit/display.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@

_OPAQUE_AXIS_TITLE_ALIASES = {
"aqu-sz",
# Community dashboards often copy Grafana's percent unit id into axisLabel.
# "%"/inferred unit titles are better; "counter" is left alone because
# hiding it lets Lens name the whole axis after the first series.
"percentage",
"percent",
}

GRAFANA_UNIT_TO_YAML: dict[str, dict[str, Any]] = {
Expand Down Expand Up @@ -224,7 +229,7 @@ def sanitize_axis_title_text(label: str, *, unit: str = "") -> str:
text = str(label or "").strip()
if not text:
return ""
if text in _OPAQUE_AXIS_TITLE_ALIASES:
if text.casefold() in _OPAQUE_AXIS_TITLE_ALIASES:
return ""
return text

Expand Down Expand Up @@ -334,7 +339,14 @@ def humanize_metric_label(field_name: str, legend_format: str | None = None) ->
return None
text = re.sub(r"_+", " ", field_name).strip()
text = re.sub(r"\s{2,}", " ", text)
if not text or text.lower() in ("series", "value", "metric", "time bucket"):
# "computed value" is the ES|QL translator's own synthetic scalar-expression
# column name (``computed_value``, the ``by()``-less binary-expression
# counterpart to native PROMQL's ``value``) -- humanizing it into "Computed
# Value" is exactly the misleading placeholder-as-title leak issue #351
# covers, just re-capitalized rather than literal. Both placeholder names
# rely on the panel-title/legend fallback in
# ``_label_placeholder_value_metric`` instead.
if not text or text.lower() in ("series", "value", "computed value", "metric", "time bucket"):
return None
parts = text.split()
label = " ".join(p if p.isupper() else p.capitalize() for p in parts)
Expand Down
Loading
Loading