diff --git a/docs/design/curated-pack-315-kubernetes-cadvisor.md b/docs/design/curated-pack-315-kubernetes-cadvisor.md new file mode 100644 index 00000000..017a1175 --- /dev/null +++ b/docs/design/curated-pack-315-kubernetes-cadvisor.md @@ -0,0 +1,117 @@ +# Curated Pack — Grafana 315 "Kubernetes cluster monitoring (via Prometheus)" + +> Design + living discoveries for the 315 curated pack. Follows the general +> Curation Playbook in `curated-dashboard-packs-plan.md`. First of two stacked +> Kubernetes packs (315 cAdvisor, then 6417 kube-state-metrics). + +- Source: community **"Kubernetes cluster monitoring (via Prometheus)"**, + +- gnetId **315**, latest revision **3**. +- canonical sha256 (rev 3) = `6fb5e045bc6d860f0f22ce7e145f4da04d2e25fd4fbf9fda29355cef6d63aeae`. +- Datasource: Prometheus — curated-pack eligible. +- Schema: **v12 (old `rows[]` layout)**. 21 panels across 13 rows. +- Metric family: **cAdvisor** (`container_*`) + **machine_** (`machine_cpu_cores`, + `machine_memory_bytes`). NOT kube-state-metrics. +- Variables: `$Node` = `label_values(kubernetes_io_hostname)`. + +## Goal + +Ship a curated pack so 315 renders in Kibana against a modern cAdvisor scrape +ingested in the Elastic `prometheus_native` layout (`metrics.* + labels.*`), +bridging the pre-1.16 cAdvisor label conventions the dashboard was authored +against, and degrading honestly where the source series no longer exist. + +## The core problem: a pre-1.16 cAdvisor dashboard + +Unlike 12485 (an old dashboard mapped onto a *modern exporter the rig runs*), +315's PromQL is written against label/runtime conventions that modern Kubernetes +and the Elastic cAdvisor integration no longer emit: + +| Dashboard convention | Modern reality | Pack handling | +|---|---|---| +| `pod_name` / `container_name` grouping | cAdvisor is `pod` / `container` | `label_rewrites` | +| `kubernetes_io_hostname=~"^$Node$"` | relabel gone; container series carry no node label | `ignored_labels` (drop matcher); `$Node` control is inert → dropped | +| `image!=""`, `name=~"^k8s_.*"` | not emitted | `ignored_labels` (drop matcher) so the good panels are not filtered to empty | +| `name!~"^k8s_.*"` (docker), `rkt_container_name` | obsolete runtimes | **dropped as honest gaps** via curated `query_overrides` with `approximation_note` | +| `systemd_service_name` | pre-labelmap convention | NOT ignored → panel degrades to an honest empty (never a fake single aggregate) | +| `machine_cpu_cores`, `machine_memory_bytes`, `container_fs_*`, `id="/"` | present only on a full cAdvisor + machine scrape | cluster-KPI panels are `APPROXIMATE`; render when that telemetry exists | + +## Engine vs pack split + +The general pipeline already handles rate() counters, gauge sum, the +unary-minus **butterfly** net-I/O panels, `rows[]` → Kibana sections, singlestat +reducers, and control synthesis. The pack carries only: + +- **`metric_kinds`** — cAdvisor counters (`container_cpu_usage_seconds_total`, + `container_network_{receive,transmit}_bytes_total`) vs gauges + (`container_memory_working_set_bytes`, `container_fs_*`, `machine_*`). +- **`label_rewrites`** — `pod_name`→`labels.pod`, `container_name`→`labels.container`, + `id`→`labels.id`. +- **`ignored_labels`** — `kubernetes_io_hostname`, `image`, `name`. These appear + only as filters on the good container panels; dropping them lets those panels + resolve instead of filtering to empty. `systemd_service_name` / + `rkt_container_name` are deliberately NOT ignored so their panels degrade to + an honest empty rather than collapse into one misleading aggregate line. +- **`query_overrides`** — the three multi-runtime panels (Containers CPU / memory + / network) keep only the k8s pod/container series and disclose the dropped + docker/rkt runtimes via `approximation_note` (status capped at + `migrated_with_warnings`, never a clean `migrated`); Pods CPU/memory keep the + Lens breakdown on `labels.pod`; Pods/All-processes network name Received/Sent + (native butterfly otherwise labels transmit `Value B`); All-processes panels + group by `labels.id` excluding the root cgroup. + +No engine changes required — the pack reuses the APIs 12485 added. + +## Fidelity + +- **PERFECT** (render on any modern cAdvisor scrape after the label bridge): + Network I/O pressure, Pods CPU / memory / network (Received/Sent series names). +- **APPROXIMATE**: Containers CPU / memory / network (docker/rkt series dropped); + the cluster-KPI strip and Used/Total stats (require `machine_*` + + `container_fs_*` + the root-cgroup `id="/"` series); All-processes panels + (grouped by the cAdvisor cgroup `id`, excluding `id="/"`; the producer must + emit a per-container cgroup `id` or the legend is a single `(null)`). +- **GAP** (honest empty on modern data): System services CPU / memory + (`systemd_service_name` is a dead convention). + +See `fidelity_manifest.yaml` for the per-panel table. + +## Validation gates (UI testing) + +Live validation uses the shared curated rig (`parity-rig/curated/…`) extended +with a synthetic cAdvisor exporter (`container_*` + `machine_*` + `container_fs_*` +with modern `pod`/`container` labels, a root-cgroup `id="/"` series, and a +per-container cgroup `id` so All-processes legends are not `(null)`) so the +per-pod panels and the cluster KPIs both have real data. + +1. Migrate + upload to Kibana (`prometheus_native`, `--esql-index` = data view). +2. Render audit — 0 `render_error`; document `field_gap`/`data_gap` for the + `systemd`/all-processes gap panels in `fidelity_manifest.yaml`. +3. Side-by-side vs the provisioned Grafana 315. + +## Task checklist + +- [x] registry.yaml entry (315, rev 3, sha above) +- [x] pack.yaml + fidelity_manifest.yaml +- [x] offline fixture tests (`tests/test_curated_packs.py`) +- [x] rig: synthetic cAdvisor exporter (`machine_*`, `container_fs_*`, `id="/"`, per-container cgroup `id`) +- [x] live: migrate + upload + render audit + view-mode UI pass +- [x] docs: discoveries here + `docs/sources/grafana.md` + +## Discoveries + +- Offline translation confirms the label bridge: `Pods CPU usage` resolves to + `SUM(RATE(container_cpu_usage_seconds_total)) BY time_bucket, labels.pod` + with the `image`/`name`/`kubernetes_io_hostname` matchers stripped; the + Containers panels take the curated k8s-only ES|QL; `System services` uses an + impossible-filter override so Lens does not error on a missing + `systemd_service_name` column (honest empty). +- The cluster-KPI panels translate to a same-bucket ratio referencing + `machine_*` and the `id="/"` root cgroup — so live rendering needs those + series in the scrape (they are absent from a plain container-only ingest). +- Engine butterfly + a pod breakdown names transmit `Value B`; Pods network + I/O needs an explicit Received/Sent override (same class as Containers + network). +- All-processes panels group by cgroup `id`. A producer that only sets + `id="/"` on the root cgroup yields a single `(null)` legend; emit + `/kubepods//` on workload series. diff --git a/docs/sources/grafana.md b/docs/sources/grafana.md index 9db0abc0..a33920ad 100644 --- a/docs/sources/grafana.md +++ b/docs/sources/grafana.md @@ -207,6 +207,29 @@ target exporter runs the `stat_statements` + `postmaster` collectors and the `pg_stat_statements` extension is installed; otherwise they degrade to an honest field/data gap. +The Kubernetes cluster-monitoring (cAdvisor 315) pack bridges a pre-1.16 +cAdvisor lineage: `label_rewrites` map `pod_name`/`container_name` to the modern +`labels.pod`/`labels.container`, and `ignored_labels` strip dead selector +matchers (`kubernetes_io_hostname`, `image`, `name`) so the per-pod/per-container +panels are not filtered to empty. Its `$Node` variable is dropped as inert — +modern cAdvisor container series carry no node label. Because the native PROMQL +DSL rewrites a grouped metric but leaves the Lens breakdown accessor bound to the +pre-rewrite label (a `by (pod_name)` panel then fails with "invalid column" once +`pod_name` becomes `labels.pod`), the pack emits explicit `query_overrides` for +the Pods CPU/memory panels so the ES|QL output column and the Lens breakdown stay +aligned on `labels.pod`, and for the Pods/All-processes network panels so the +transmit series is named `Sent` rather than the fused `Value B`. All-processes +panels group by the cAdvisor cgroup `id` (`labels.id`, excluding the root +cgroup). The obsolete container-runtime series are honest gaps +rather than fabricated aggregates: docker (`name!~"^k8s_"`) and rkt +(`rkt_container_name`) targets are dropped from the multi-runtime container +panels (disclosed as an approximation), and the systemd system-service panels +(`systemd_service_name`, a pre-labelmap relabel that no longer exists) degrade to +an honest empty via an impossible-filter override instead of erroring on a +non-existent breakdown column. The cluster-total KPI strip additionally needs the +node `machine_*` metrics and the root-cgroup (`id="/"`) + `container_fs_*` series +to populate. + Each pack is registered in `curated_packs/registry.yaml` with a `gnet_revision` and `dashboard_sha256` — maintainer-verified provenance pins recording the exact grafana.com revision the pack authors read, re-checkable diff --git a/observability_migration/adapters/source/grafana/curated_packs/grafana_315_kubernetes_cadvisor/fidelity_manifest.yaml b/observability_migration/adapters/source/grafana/curated_packs/grafana_315_kubernetes_cadvisor/fidelity_manifest.yaml new file mode 100644 index 00000000..3c4f517b --- /dev/null +++ b/observability_migration/adapters/source/grafana/curated_packs/grafana_315_kubernetes_cadvisor/fidelity_manifest.yaml @@ -0,0 +1,108 @@ +# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one or more contributor license agreements. +# SPDX-License-Identifier: Elastic-2.0 +# +# Fidelity manifest — Grafana dashboard 315 (Kubernetes cluster monitoring via Prometheus) +# https://grafana.com/grafana/dashboards/315-kubernetes-cluster-monitoring-via-prometheus/ +# +# PERFECT = same information as Grafana on a modern cAdvisor scrape. +# APPROXIMATE = documented delta (dropped obsolete runtime series, id/device gaps). +# GAP = renders empty on modern data because the underlying series/label +# no longer exists (obsolete container runtimes, pre-1.16 labels). +# +# The dashboard is cAdvisor-only (container_* + machine_*). Panels marked +# "requires: machine_* / container_fs_*" render on real data only when the +# target cAdvisor scrape emits the node machine metrics and the root cgroup +# (id="/") filesystem series; otherwise they are an honest field/data gap. + +schema_version: 1 +gnet_id: 315 +gnet_revision: 3 +dashboard_title: "Kubernetes cluster monitoring (via Prometheus)" +maintainer: "community" + +panels: + # --- Network I/O pressure ----------------------------------------------- + - title: "Network I/O pressure" + fidelity: PERFECT + notes: "sum(rate(container_network_receive/transmit_bytes_total)) butterfly; engine emits Sent as the negated transmit rate." + + # --- Total usage (cluster KPIs) ----------------------------------------- + - title: "Cluster memory usage" + fidelity: APPROXIMATE + notes: "sum(container_memory_working_set_bytes{id=/})/sum(machine_memory_bytes)*100. requires: machine_memory_bytes + root-cgroup (id=/) series. Cross-metric ratio — PERFECT under native PROMQL." + - title: "Cluster CPU usage (1m avg)" + fidelity: APPROXIMATE + notes: "sum(rate(container_cpu_usage_seconds_total{id=/}))/sum(machine_cpu_cores)*100. requires: machine_cpu_cores + id=/ series." + - title: "Cluster filesystem usage" + fidelity: APPROXIMATE + notes: "container_fs_usage_bytes/container_fs_limit_bytes*100 with device=~/dev/[sv]d.. filter. requires: container_fs_* with a matching device label (the classic Grafana N/A when the device regex matches nothing)." + - title: "Used" # memory + fidelity: APPROXIMATE + notes: "sum(container_memory_working_set_bytes{id=/}). requires: root-cgroup id=/ series." + - title: "Total" # memory + fidelity: APPROXIMATE + notes: "sum(machine_memory_bytes). requires: machine_memory_bytes." + - title: "Used" # cpu (cores) + fidelity: APPROXIMATE + notes: "sum(rate(container_cpu_usage_seconds_total{id=/})). requires: id=/ series." + - title: "Total" # cpu (cores) + fidelity: APPROXIMATE + notes: "sum(machine_cpu_cores). requires: machine_cpu_cores." + - title: "Used" # filesystem + fidelity: APPROXIMATE + notes: "sum(container_fs_usage_bytes{device,id=/}). requires: container_fs_usage_bytes." + - title: "Total" # filesystem + fidelity: APPROXIMATE + notes: "sum(container_fs_limit_bytes{device,id=/}). requires: container_fs_limit_bytes." + + # --- Per-pod workload (the core, renders on modern cAdvisor) ------------- + - title: "Pods CPU usage (1m avg)" + fidelity: PERFECT + notes: "sum(rate(container_cpu_usage_seconds_total)) by pod. Curated ES|QL override groups by labels.pod directly: the native PROMQL path rewrites the metric but leaves the Lens breakdown accessor on the pre-rewrite `pod_name`, which fails as 'invalid column' after pod_name -> labels.pod. Explicit ES|QL keeps query output and breakdown aligned and excludes the root cgroup (id=/) series (no pod label)." + - title: "Pods memory usage" + fidelity: PERFECT + notes: "sum(container_memory_working_set_bytes) by pod. Same curated ES|QL breakdown-alignment override as Pods CPU." + - title: "Pods network I/O (1m avg)" + fidelity: PERFECT + notes: "receive positive, transmit negative, by pod. Curated ES|QL names the series Received/Sent; the native butterfly fusion otherwise labels transmit as Value B once a pod breakdown is present." + + # --- Per-container workload --------------------------------------------- + - title: "Containers CPU usage (1m avg)" + fidelity: APPROXIMATE + notes: "Curated ES|QL keeps the k8s pod/container series (excludes the POD pause container); dropped the obsolete docker (name!~^k8s_) and rkt (rkt_container_name) targets." + - title: "Containers memory usage" + fidelity: APPROXIMATE + notes: "As above; per pod/container working set." + - title: "Containers network I/O (1m avg)" + fidelity: APPROXIMATE + notes: "As above; received/sent by pod. Docker/rkt series dropped." + + # --- Obsolete-runtime and pre-1.16 panels (honest gaps) ----------------- + - title: "System services CPU usage (1m avg)" + fidelity: GAP + notes: "systemd_service_name is a pre-labelmap cAdvisor convention absent on modern k8s. Curated ES|QL override filters on an impossible container value so the panel degrades to an honest empty (data_gap) instead of a Lens 'invalid column' error on the non-existent systemd_service_name breakdown." + - title: "System services memory usage" + fidelity: GAP + notes: "systemd_service_name absent on modern k8s; same impossible-filter override as System services CPU so the panel renders empty rather than erroring." + - title: "All processes CPU usage (1m avg)" + fidelity: APPROXIMATE + notes: "Groups by the cAdvisor cgroup `id` (excludes the root cgroup id=/). Curated ES|QL keeps the Lens breakdown on labels.id. Renders per-cgroup series when the scrape emits `id` on container metrics." + - title: "All processes memory usage" + fidelity: APPROXIMATE + notes: "As above; grouped by cgroup id." + - title: "All processes network I/O (1m avg)" + fidelity: APPROXIMATE + notes: "As above; grouped by cgroup id with Received/Sent series names (native butterfly would emit Value B)." + +summary: + total_panels: 21 + perfect: 4 + approximate: 15 + gap: 2 + notes: > + The dashboard's per-pod/per-container workload panels (the operational core) + render on any modern cAdvisor scrape after the pre-1.16 label bridge. The + cluster-total KPI strip requires the node machine_* metrics and the root + cgroup (id="/") + container_fs_* series. The systemd and docker/rkt runtime + series are permanent gaps: those container runtimes no longer exist, so the + pack drops them rather than fabricate a single misleading aggregate. diff --git a/observability_migration/adapters/source/grafana/curated_packs/grafana_315_kubernetes_cadvisor/pack.yaml b/observability_migration/adapters/source/grafana/curated_packs/grafana_315_kubernetes_cadvisor/pack.yaml new file mode 100644 index 00000000..d25d654a --- /dev/null +++ b/observability_migration/adapters/source/grafana/curated_packs/grafana_315_kubernetes_cadvisor/pack.yaml @@ -0,0 +1,208 @@ +# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one or more contributor license agreements. +# SPDX-License-Identifier: Elastic-2.0 +# +# Curated pack — Grafana dashboard 315 +# https://grafana.com/grafana/dashboards/315-kubernetes-cluster-monitoring-via-prometheus/ +# +# Source: community "Kubernetes cluster monitoring (via Prometheus)". +# Revision 3. schemaVersion 12 (old `rows[]` layout), 21 panels across 13 rows. +# Metric family: cAdvisor container_* + machine_* (NOT kube-state-metrics). +# +# This dashboard was authored against a pre-1.16 cAdvisor lineage, so its PromQL +# uses label names and container-runtime conventions that modern Kubernetes / +# the Elastic cAdvisor integration no longer emit: +# - `pod_name` / `container_name` -> modern cAdvisor is `pod` / `container`. +# - `kubernetes_io_hostname` -> a per-node relabel that no longer exists; +# cAdvisor container series carry no node +# label at all, so the $Node control is +# inert and is dropped. +# - `systemd_service_name`, `rkt_container_name`, `name!~"^k8s_.*"` (docker) -> +# obsolete container runtimes. Those series +# are dropped as honest gaps (see +# fidelity_manifest.yaml); we never fake a +# single aggregate in their place. +# +# Engine vs pack split (do not duplicate engine work here): +# Engine: rate() counter handling, gauge sum, unary-minus butterfly net I/O, +# rows[] -> Kibana sections, singlestat reducers, control synthesis. +# Pack: bridge the pre-1.16 label names (label_rewrites, including cgroup +# `id` → `labels.id`), strip the dead selector labels so the good +# container panels are not filtered to empty (ignored_labels), +# classify counters/gauges (metric_kinds), keep only the k8s series +# on the multi-runtime panels, name Pods/All-processes network +# Received/Sent (native butterfly otherwise emits Value B), and +# group All-processes panels by cgroup id (query_overrides). + +query: + metrics_dataset_filter: "prometheus" + + # --- counter / gauge classification ----------------------------------- + metric_kinds: + container_cpu_usage_seconds_total: counter + container_network_receive_bytes_total: counter + container_network_transmit_bytes_total: counter + container_memory_working_set_bytes: gauge + container_fs_usage_bytes: gauge + container_fs_limit_bytes: gauge + machine_cpu_cores: gauge + machine_memory_bytes: gauge + + # --- pre-1.16 cAdvisor label renames ---------------------------------- + # The dashboard groups by / matches on the old names; rewrite to the modern + # cAdvisor labels the Elastic integration ingests. + label_rewrites: + pod_name: labels.pod + container_name: labels.container + namespace: labels.namespace + pod: labels.pod + container: labels.container + id: labels.id + + # --- dead selector labels --------------------------------------------- + # These appear only as filters on the good container panels; on modern data + # they do not exist, so without ignoring them every WHERE clause filters to + # empty. Ignoring drops the matcher (and the inert $Node selector) so the + # pod/container panels resolve. `systemd_service_name` / `rkt_container_name` + # are intentionally NOT ignored: those panels must degrade to an honest empty + # (data_gap), not collapse into one misleading aggregate line. + ignored_labels: + - kubernetes_io_hostname + - image + - name + + label_candidates: + pod: + - labels.pod + - labels.pod_name + container: + - labels.container + - labels.container_name + namespace: + - labels.namespace + id: + - labels.id + +panel: + query_overrides: + # Multi-runtime panels ship k8s + docker + rkt targets. On modern cAdvisor + # only the k8s (pod/container) series exist; keep just that series and + # disclose the dropped docker/rkt runtimes as an approximation. + - title_match: "Containers CPU usage (1m avg)" + approximation_note: "Dropped obsolete docker (name!~^k8s_) and rkt (rkt_container_name) runtime series; modern cAdvisor only emits the k8s pod/container series." + esql_query: | + TS metrics-* + | WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend + | WHERE {{metric:container_cpu_usage_seconds_total:counter}} IS NOT NULL AND {{label:container}} IS NOT NULL AND {{label:container}} != "POD" + | STATS value = SUM(RATE({{metric:container_cpu_usage_seconds_total:counter}})) BY time_bucket = TBUCKET(20, ?_tstart, ?_tend), {{label:pod}}, {{label:container}} + | KEEP time_bucket, `labels.pod`, `labels.container`, value + | SORT time_bucket ASC + - title_match: "Containers memory usage" + approximation_note: "Dropped obsolete docker and rkt runtime series; modern cAdvisor only emits the k8s pod/container series." + esql_query: | + TS metrics-* + | WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend + | WHERE {{metric:container_memory_working_set_bytes:gauge}} IS NOT NULL AND {{label:container}} IS NOT NULL AND {{label:container}} != "POD" + | STATS value = SUM(LAST_OVER_TIME({{metric:container_memory_working_set_bytes:gauge}})) BY time_bucket = TBUCKET(20, ?_tstart, ?_tend), {{label:pod}}, {{label:container}} + | KEEP time_bucket, `labels.pod`, `labels.container`, value + | SORT time_bucket ASC + - title_match: "Containers network I/O (1m avg)" + approximation_note: "Dropped obsolete docker and rkt runtime series; only the k8s pod series remain (received positive, sent negative)." + esql_query: | + TS metrics-* + | WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend + | WHERE {{metric:container_network_receive_bytes_total:counter}} IS NOT NULL OR {{metric:container_network_transmit_bytes_total:counter}} IS NOT NULL + | STATS Received = SUM(RATE({{metric:container_network_receive_bytes_total:counter}})), Sent = 0 - SUM(RATE({{metric:container_network_transmit_bytes_total:counter}})) BY time_bucket = TBUCKET(20, ?_tstart, ?_tend), {{label:pod}} + | KEEP time_bucket, `labels.pod`, Received, Sent + | SORT time_bucket ASC + # Engine butterfly fusion names the transmit series `value_B` / "Value B" + # when a label breakdown is present. Name Received/Sent like the Containers + # network override so the Lens legend matches Grafana's -> / <- series. + - title_match: "Pods network I/O (1m avg)" + esql_query: | + TS metrics-* + | WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend + | WHERE {{label:pod}} IS NOT NULL AND ({{metric:container_network_receive_bytes_total:counter}} IS NOT NULL OR {{metric:container_network_transmit_bytes_total:counter}} IS NOT NULL) + | STATS Received = SUM(RATE({{metric:container_network_receive_bytes_total:counter}})), Sent = 0 - SUM(RATE({{metric:container_network_transmit_bytes_total:counter}})) BY time_bucket = TBUCKET(20, ?_tstart, ?_tend), {{label:pod}} + | KEEP time_bucket, `labels.pod`, Received, Sent + | SORT time_bucket ASC + # All-processes panels group by the cAdvisor cgroup `id`. Explicit ES|QL + # keeps the Lens breakdown on `labels.id` (the native PROMQL path leaves + # it bound to the pre-rewrite `id`) and drops the root cgroup (id="/"). + - title_match: "All processes CPU usage (1m avg)" + esql_query: | + TS metrics-* + | WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend + | WHERE {{metric:container_cpu_usage_seconds_total:counter}} IS NOT NULL AND {{label:id}} IS NOT NULL AND {{label:id}} != "/" + | STATS value = SUM(RATE({{metric:container_cpu_usage_seconds_total:counter}})) BY time_bucket = TBUCKET(20, ?_tstart, ?_tend), {{label:id}} + | KEEP time_bucket, `labels.id`, value + | SORT time_bucket ASC + - title_match: "All processes memory usage" + esql_query: | + TS metrics-* + | WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend + | WHERE {{metric:container_memory_working_set_bytes:gauge}} IS NOT NULL AND {{label:id}} IS NOT NULL AND {{label:id}} != "/" + | STATS value = SUM(LAST_OVER_TIME({{metric:container_memory_working_set_bytes:gauge}})) BY time_bucket = TBUCKET(20, ?_tstart, ?_tend), {{label:id}} + | KEEP time_bucket, `labels.id`, value + | SORT time_bucket ASC + - title_match: "All processes network I/O (1m avg)" + esql_query: | + TS metrics-* + | WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend + | WHERE {{label:id}} IS NOT NULL AND {{label:id}} != "/" AND ({{metric:container_network_receive_bytes_total:counter}} IS NOT NULL OR {{metric:container_network_transmit_bytes_total:counter}} IS NOT NULL) + | STATS Received = SUM(RATE({{metric:container_network_receive_bytes_total:counter}})), Sent = 0 - SUM(RATE({{metric:container_network_transmit_bytes_total:counter}})) BY time_bucket = TBUCKET(20, ?_tstart, ?_tend), {{label:id}} + | KEEP time_bucket, `labels.id`, Received, Sent + | SORT time_bucket ASC + # Per-pod panels group by the pre-1.16 `pod_name` label. The engine's native + # PROMQL DSL rewrites the metric name but leaves the Lens breakdown accessor + # bound to the pre-rewrite `pod_name`, so after label_rewrites the compiled + # column is `labels.pod` and Lens fails with "invalid column". Emit explicit + # ES|QL so the query output column and the Lens breakdown agree (same pattern + # the 12485 pack uses for its by-label legends). `{{label:pod}} IS NOT NULL` + # drops the root cgroup (id="/") series which carries no pod label. + - title_match: "Pods CPU usage (1m avg)" + esql_query: | + TS metrics-* + | WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend + | WHERE {{metric:container_cpu_usage_seconds_total:counter}} IS NOT NULL AND {{label:pod}} IS NOT NULL + | STATS value = SUM(RATE({{metric:container_cpu_usage_seconds_total:counter}})) BY time_bucket = TBUCKET(20, ?_tstart, ?_tend), {{label:pod}} + | KEEP time_bucket, `labels.pod`, value + | SORT time_bucket ASC + - title_match: "Pods memory usage" + esql_query: | + TS metrics-* + | WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend + | WHERE {{metric:container_memory_working_set_bytes:gauge}} IS NOT NULL AND {{label:pod}} IS NOT NULL + | STATS value = SUM(LAST_OVER_TIME({{metric:container_memory_working_set_bytes:gauge}})) BY time_bucket = TBUCKET(20, ?_tstart, ?_tend), {{label:pod}} + | KEEP time_bucket, `labels.pod`, value + | SORT time_bucket ASC + # systemd system-service panels group by `systemd_service_name`, a pre-1.16 + # cAdvisor relabel artifact that modern cAdvisor never emits. Rather than let + # the native breakdown reference a non-existent column (Lens "invalid + # column"), emit a valid query with an impossible container filter so the + # panel degrades to an honest empty (data_gap) instead of erroring. + - title_match: "System services CPU usage (1m avg)" + approximation_note: "systemd system-service series (systemd_service_name) are a pre-1.16 cAdvisor relabel artifact; modern cAdvisor does not emit them, so this panel is an honest empty (data_gap)." + esql_query: | + TS metrics-* + | WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend + | WHERE {{metric:container_cpu_usage_seconds_total:counter}} IS NOT NULL AND {{label:container}} == "__systemd_service__" + | STATS value = SUM(RATE({{metric:container_cpu_usage_seconds_total:counter}})) BY time_bucket = TBUCKET(20, ?_tstart, ?_tend), {{label:container}} + | KEEP time_bucket, `labels.container`, value + | SORT time_bucket ASC + - title_match: "System services memory usage" + approximation_note: "systemd system-service series are a pre-1.16 cAdvisor relabel artifact; modern cAdvisor does not emit them, so this panel is an honest empty (data_gap)." + esql_query: | + TS metrics-* + | WHERE @timestamp >= ?_tstart AND @timestamp <= ?_tend + | WHERE {{metric:container_memory_working_set_bytes:gauge}} IS NOT NULL AND {{label:container}} == "__systemd_service__" + | STATS value = SUM(LAST_OVER_TIME({{metric:container_memory_working_set_bytes:gauge}})) BY time_bucket = TBUCKET(20, ?_tstart, ?_tend), {{label:container}} + | KEEP time_bucket, `labels.container`, value + | SORT time_bucket ASC + + layout_overrides: + # $Node is the only variable and cannot bind (cAdvisor container series carry + # no node label); the engine drops it as inert. Nothing to move here beyond + # the rows[] -> section geometry the engine already produces. Kept as a + # placeholder anchor so the manifest and layout stay in one place. + - title_match: "Network I/O pressure" + size: {w: 48} diff --git a/observability_migration/adapters/source/grafana/curated_packs/registry.yaml b/observability_migration/adapters/source/grafana/curated_packs/registry.yaml index dee83d4f..02ee1709 100644 --- a/observability_migration/adapters/source/grafana/curated_packs/registry.yaml +++ b/observability_migration/adapters/source/grafana/curated_packs/registry.yaml @@ -127,3 +127,12 @@ packs: gnet_revision: 1 dashboard_sha256: "e14a35eac532db4f79837a293411edb20d04164ca59de17d73557d08637a4700" description: "postgres_exporter overview — v0.15 name remap (size_bytes/lag_seconds/stat_statements), _count gauge fixes, Instance/Database populate rewrite" + + - gnet_id: 315 + name: grafana_315_kubernetes_cadvisor + title_hint: "Kubernetes cluster monitoring (via Prometheus)" + tags_hint: ["kubernetes"] + path: grafana_315_kubernetes_cadvisor + gnet_revision: 3 + dashboard_sha256: "6fb5e045bc6d860f0f22ce7e145f4da04d2e25fd4fbf9fda29355cef6d63aeae" + description: "cAdvisor cluster monitoring — pre-1.16 label remap (pod_name/container_name → pod/container), obsolete rkt/docker/systemd series dropped as gaps, machine_*/container_fs_* KPIs" diff --git a/parity-rig/curated/grafana_763_redis_exporter/docker-compose.yml b/parity-rig/curated/grafana_763_redis_exporter/docker-compose.yml index bfa31417..73081078 100644 --- a/parity-rig/curated/grafana_763_redis_exporter/docker-compose.yml +++ b/parity-rig/curated/grafana_763_redis_exporter/docker-compose.yml @@ -384,6 +384,18 @@ services: - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.retention.time=1d + # Synthetic Kubernetes exporter (cAdvisor + kube-state-metrics + node_exporter) + # so the Kubernetes curated packs (Grafana 315 + 6417) validate on real data. + k8s_exporter: + image: python:3.11-slim + container_name: redis-rig-k8s-exporter + volumes: + - ./k8s_exporter.py:/app/k8s_exporter.py:ro + working_dir: /app + command: ["python", "k8s_exporter.py"] + networks: + - redis-rig + redis-scraper: image: python:3.11-slim container_name: redis-rig-scraper @@ -407,6 +419,8 @@ services: condition: service_started postgres_exporter: condition: service_started + k8s_exporter: + condition: service_started restart: unless-stopped grafana: diff --git a/parity-rig/curated/grafana_763_redis_exporter/k8s_exporter.py b/parity-rig/curated/grafana_763_redis_exporter/k8s_exporter.py new file mode 100644 index 00000000..6b18e97c --- /dev/null +++ b/parity-rig/curated/grafana_763_redis_exporter/k8s_exporter.py @@ -0,0 +1,237 @@ +#!/usr/bin/env python3 +# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one or more contributor license agreements. +# SPDX-License-Identifier: Elastic-2.0 +# +# Synthetic Kubernetes metrics exporter for the curated-pack validation rig. +# +# Emits a small, coherent cluster in modern shape so the two Kubernetes curated +# packs can be validated on real (rig-ingested) data: +# - Grafana 315 (cAdvisor): container_* + machine_* + container_fs_* with the +# modern `pod`/`container` labels and a root-cgroup `id="/"` series. +# - Grafana 6417 (kube-state-metrics + node_exporter): kube_* in the modern +# resource-split shape (kube_node_status_allocatable{resource=...}, etc.), +# plus node_filesystem_*_bytes. +# +# Counters are wall-clock monotonic so RATE()/DELTA() render on the rig. The +# `OutOfDisk` node condition is deliberately NOT emitted (removed in k8s 1.12) so +# the 6417 "Nodes Out of Disk" panel stays an honest empty gap. + +import time +from http.server import BaseHTTPRequestHandler, HTTPServer + +CLUSTER = "rig-cluster" +NODES = ["node-1", "node-2"] +# namespace -> [(pod, container)] +WORKLOADS = { + "default": [("web-0", "web"), ("web-1", "web"), ("cache-0", "redis")], + "staging": [("api-0", "api"), ("api-1", "api")], + "kube-system": [("coredns-0", "coredns"), ("kube-proxy-0", "kube-proxy")], +} +DEPLOYMENTS = { + "default": {"web": 2, "cache": 1}, + "staging": {"api": 2}, + "kube-system": {"coredns": 2}, +} + +_START = time.time() + + +def _node_for(idx: int) -> str: + return NODES[idx % len(NODES)] + + +def render() -> str: + now = time.time() + elapsed = max(1.0, now - _START) + L: list[str] = [] + + # ---- cAdvisor: machine_* (per node) --------------------------------- + L.append("# HELP machine_cpu_cores Number of CPU cores on the machine") + L.append("# TYPE machine_cpu_cores gauge") + L.append("# HELP machine_memory_bytes Amount of memory installed on the machine") + L.append("# TYPE machine_memory_bytes gauge") + for n in NODES: + L.append(f'machine_cpu_cores{{instance="{n}",node="{n}"}} 4') + L.append(f'machine_memory_bytes{{instance="{n}",node="{n}"}} {16 * 1024**3}') + + # ---- cAdvisor: root cgroup (id="/") node totals --------------------- + L.append("# HELP container_cpu_usage_seconds_total Cumulative cpu time consumed") + L.append("# TYPE container_cpu_usage_seconds_total counter") + L.append("# HELP container_memory_working_set_bytes Current working set") + L.append("# TYPE container_memory_working_set_bytes gauge") + L.append("# HELP container_network_receive_bytes_total Cumulative bytes received") + L.append("# TYPE container_network_receive_bytes_total counter") + L.append("# HELP container_network_transmit_bytes_total Cumulative bytes transmitted") + L.append("# TYPE container_network_transmit_bytes_total counter") + L.append("# HELP container_fs_usage_bytes Filesystem bytes consumed") + L.append("# TYPE container_fs_usage_bytes gauge") + L.append("# HELP container_fs_limit_bytes Filesystem capacity in bytes") + L.append("# TYPE container_fs_limit_bytes gauge") + + for ni, n in enumerate(NODES): + # Root cgroup totals used by the cluster-usage KPIs (id="/"). + root_cpu = elapsed * (1.2 + 0.3 * ni) + L.append( + f'container_cpu_usage_seconds_total{{id="/",instance="{n}",node="{n}"}} {root_cpu:.3f}' + ) + L.append( + f'container_memory_working_set_bytes{{id="/",instance="{n}",node="{n}"}} ' + f'{int((5.0 + ni) * 1024**3)}' + ) + L.append( + f'container_fs_usage_bytes{{id="/",device="/dev/sda1",instance="{n}"}} ' + f'{int((30 + 5 * ni) * 1024**3)}' + ) + L.append( + f'container_fs_limit_bytes{{id="/",device="/dev/sda1",instance="{n}"}} ' + f'{(100 * 1024**3)}' + ) + + # ---- cAdvisor: per pod/container series ----------------------------- + idx = 0 + for ns, pods in WORKLOADS.items(): + for pod, container in pods: + n = _node_for(idx) + idx += 1 + cgroup_id = f"/kubepods/{pod}/{container}" + base = ( + f'id="{cgroup_id}",namespace="{ns}",pod="{pod}",container="{container}",' + f'image="registry/{container}:latest",name="k8s_{container}_{pod}",instance="{n}"' + ) + cpu = elapsed * (0.05 + 0.02 * (idx % 5)) + L.append(f"container_cpu_usage_seconds_total{{{base}}} {cpu:.4f}") + L.append( + f"container_memory_working_set_bytes{{{base}}} " + f"{int((128 + 40 * (idx % 6)) * 1024**2)}" + ) + rx = elapsed * (2000 + 300 * (idx % 7)) + tx = elapsed * (1500 + 200 * (idx % 7)) + L.append(f"container_network_receive_bytes_total{{{base}}} {rx:.0f}") + L.append(f"container_network_transmit_bytes_total{{{base}}} {tx:.0f}") + + # ---- kube-state-metrics: nodes -------------------------------------- + L.append("# HELP kube_node_info Information about a cluster node") + L.append("# TYPE kube_node_info gauge") + L.append("# HELP kube_node_spec_unschedulable Whether a node can schedule new pods") + L.append("# TYPE kube_node_spec_unschedulable gauge") + L.append("# HELP kube_node_status_condition The condition of a cluster node") + L.append("# TYPE kube_node_status_condition gauge") + L.append("# HELP kube_node_status_allocatable Node resources allocatable for scheduling") + L.append("# TYPE kube_node_status_allocatable gauge") + L.append("# HELP kube_node_status_capacity Total node resource capacity") + L.append("# TYPE kube_node_status_capacity gauge") + for n in NODES: + L.append(f'kube_node_info{{node="{n}",cluster="{CLUSTER}"}} 1') + L.append(f'kube_node_spec_unschedulable{{node="{n}"}} 0') + # Ready condition present; OutOfDisk intentionally absent (removed in 1.12). + L.append(f'kube_node_status_condition{{node="{n}",condition="Ready",status="true"}} 1') + for resource, alloc, cap in (("pods", 110, 110), ("cpu", 4, 4), ("memory", 15 * 1024**3, 16 * 1024**3)): + L.append(f'kube_node_status_allocatable{{node="{n}",resource="{resource}"}} {alloc}') + L.append(f'kube_node_status_capacity{{node="{n}",resource="{resource}"}} {cap}') + + # ---- kube-state-metrics: pods --------------------------------------- + L.append("# HELP kube_pod_info Information about pod") + L.append("# TYPE kube_pod_info gauge") + L.append("# HELP kube_pod_status_phase The pods current phase") + L.append("# TYPE kube_pod_status_phase gauge") + L.append("# HELP kube_pod_container_status_running Whether the container is running") + L.append("# TYPE kube_pod_container_status_running gauge") + L.append("# HELP kube_pod_container_status_waiting Whether the container is waiting") + L.append("# TYPE kube_pod_container_status_waiting gauge") + L.append("# HELP kube_pod_container_status_terminated Whether the container is terminated") + L.append("# TYPE kube_pod_container_status_terminated gauge") + L.append("# HELP kube_pod_container_status_restarts_total Container restart count") + L.append("# TYPE kube_pod_container_status_restarts_total counter") + L.append("# HELP kube_pod_container_resource_requests Requested container resources") + L.append("# TYPE kube_pod_container_resource_requests gauge") + idx = 0 + for ns, pods in WORKLOADS.items(): + for pod, container in pods: + n = _node_for(idx) + phase = "Running" + idx += 1 + L.append(f'kube_pod_info{{namespace="{ns}",pod="{pod}",node="{n}"}} 1') + L.append(f'kube_pod_status_phase{{namespace="{ns}",pod="{pod}",phase="{phase}"}} 1') + L.append( + f'kube_pod_container_status_running{{namespace="{ns}",pod="{pod}",container="{container}"}} 1' + ) + L.append( + f'kube_pod_container_status_waiting{{namespace="{ns}",pod="{pod}",container="{container}"}} 0' + ) + L.append( + f'kube_pod_container_status_terminated{{namespace="{ns}",pod="{pod}",container="{container}"}} 0' + ) + restarts = int(elapsed / 600) + (idx % 3) + L.append( + f'kube_pod_container_status_restarts_total{{namespace="{ns}",pod="{pod}",container="{container}"}} {restarts}' + ) + L.append( + f'kube_pod_container_resource_requests{{namespace="{ns}",pod="{pod}",container="{container}",node="{n}",resource="cpu",unit="core"}} 0.25' + ) + L.append( + f'kube_pod_container_resource_requests{{namespace="{ns}",pod="{pod}",container="{container}",node="{n}",resource="memory",unit="byte"}} {int(0.5 * 1024**3)}' + ) + + # ---- kube-state-metrics: deployments -------------------------------- + L.append("# HELP kube_deployment_status_replicas The number of replicas per deployment") + L.append("# TYPE kube_deployment_status_replicas gauge") + L.append("# HELP kube_deployment_status_replicas_updated The number of updated replicas") + L.append("# TYPE kube_deployment_status_replicas_updated gauge") + L.append("# HELP kube_deployment_status_replicas_unavailable The number of unavailable replicas") + L.append("# TYPE kube_deployment_status_replicas_unavailable gauge") + for ns, deps in DEPLOYMENTS.items(): + for dep, replicas in deps.items(): + L.append(f'kube_deployment_status_replicas{{namespace="{ns}",deployment="{dep}"}} {replicas}') + L.append(f'kube_deployment_status_replicas_updated{{namespace="{ns}",deployment="{dep}"}} {replicas}') + L.append(f'kube_deployment_status_replicas_unavailable{{namespace="{ns}",deployment="{dep}"}} 0') + + # ---- kube-state-metrics: jobs --------------------------------------- + L.append("# HELP kube_job_status_succeeded The number of pods which reached Complete") + L.append("# TYPE kube_job_status_succeeded gauge") + L.append("# HELP kube_job_status_active The number of actively running pods") + L.append("# TYPE kube_job_status_active gauge") + L.append("# HELP kube_job_status_failed The number of pods which reached Failed") + L.append("# TYPE kube_job_status_failed gauge") + for ns in ("default", "kube-system"): + L.append(f'kube_job_status_succeeded{{namespace="{ns}",job_name="backup"}} 3') + L.append(f'kube_job_status_active{{namespace="{ns}",job_name="backup"}} 1') + L.append(f'kube_job_status_failed{{namespace="{ns}",job_name="backup"}} 0') + + # ---- node_exporter: filesystem (modern *_bytes names) --------------- + L.append("# HELP node_filesystem_size_bytes Filesystem size in bytes") + L.append("# TYPE node_filesystem_size_bytes gauge") + L.append("# HELP node_filesystem_free_bytes Filesystem free space in bytes") + L.append("# TYPE node_filesystem_free_bytes gauge") + for n in NODES: + labels = f'device="/dev/sda1",fstype="ext4",mountpoint="/",instance="{n}"' + L.append(f"node_filesystem_size_bytes{{{labels}}} {(100 * 1024**3)}") + L.append(f"node_filesystem_free_bytes{{{labels}}} {((60 - 5) * 1024**3)}") + + return "\n".join(L) + "\n" + + +class Handler(BaseHTTPRequestHandler): + def do_GET(self): + if self.path not in ("/metrics", "/"): + self.send_response(404) + self.end_headers() + return + body = render().encode("utf-8") + self.send_response(200) + self.send_header("Content-Type", "text/plain; version=0.0.4") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *_args): # silence access logs + return + + +def main() -> None: + server = HTTPServer(("0.0.0.0", 9288), Handler) + print("synthetic k8s exporter on :9288/metrics", flush=True) + server.serve_forever() + + +if __name__ == "__main__": + main() diff --git a/parity-rig/curated/grafana_763_redis_exporter/redis_scraper.py b/parity-rig/curated/grafana_763_redis_exporter/redis_scraper.py index 53f9516a..64e46bb4 100644 --- a/parity-rig/curated/grafana_763_redis_exporter/redis_scraper.py +++ b/parity-rig/curated/grafana_763_redis_exporter/redis_scraper.py @@ -55,6 +55,15 @@ "postgres:5432", {}, ), + # Synthetic Kubernetes exporter: cAdvisor (container_*/machine_*) for + # Grafana 315 and kube-state-metrics + node_exporter for Grafana 6417. + ( + "http://k8s_exporter:9288/metrics", + "k8s.prometheus", + "kube-state-metrics", + "ksm:8080", + {}, + ), ] _LABEL_RE = re.compile(r'(\w+)="([^"]*)"') diff --git a/tests/test_curated_packs.py b/tests/test_curated_packs.py index 7d785d08..7611661a 100644 --- a/tests/test_curated_packs.py +++ b/tests/test_curated_packs.py @@ -4182,3 +4182,318 @@ def test_panel_layout_override_loaded_from_pack_yaml_round_trip(): assert override["collapsed"] is False finally: os.unlink(tmp_path) + + +# --------------------------------------------------------------------------- +# Grafana 315 — Kubernetes cluster monitoring (cAdvisor) +# --------------------------------------------------------------------------- + + +def _resolve_315(): + dashboard = { + "gnetId": 315, + "title": "Kubernetes cluster monitoring (via Prometheus)", + "tags": ["kubernetes"], + } + resolved = resolve_pack_for_dashboard(dashboard, RulePackConfig()) + return resolved, SchemaResolver(resolved) + + +def test_315_registry_entry_present(): + entry = find_curated_pack(gnet_id=315, title="", tags=[]) + assert entry is not None + assert entry["name"] == "grafana_315_kubernetes_cadvisor" + assert entry["gnet_revision"] == 3 + + +def test_315_classifies_cadvisor_counters_and_gauges(): + resolved, _ = _resolve_315() + assert resolved.metric_kinds["container_cpu_usage_seconds_total"] == "counter" + assert resolved.metric_kinds["container_network_receive_bytes_total"] == "counter" + assert resolved.metric_kinds["container_memory_working_set_bytes"] == "gauge" + assert resolved.metric_kinds["machine_cpu_cores"] == "gauge" + + +def test_315_rewrites_pre_116_labels_and_ignores_dead_matchers(): + resolved, _ = _resolve_315() + assert resolved.label_rewrites["pod_name"] == "labels.pod" + assert resolved.label_rewrites["container_name"] == "labels.container" + assert resolved.label_rewrites["id"] == "labels.id" + assert "kubernetes_io_hostname" in resolved.ignored_labels + assert "image" in resolved.ignored_labels + # systemd/rkt labels must NOT be ignored (those panels degrade to empty). + assert "systemd_service_name" not in resolved.ignored_labels + assert "rkt_container_name" not in resolved.ignored_labels + + +def test_315_pods_cpu_panel_groups_by_pod_without_dead_matchers(): + resolved, resolver = _resolve_315() + panel = { + "id": 1, + "type": "graph", + "title": "Pods CPU usage (1m avg)", + "targets": [ + { + "expr": ( + 'sum (rate (container_cpu_usage_seconds_total' + '{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^$Node$"}[1m]))' + " by (pod_name)" + ), + "legendFormat": "{{ pod_name }}", + "refId": "A", + }, + ], + "gridPos": {"x": 0, "y": 0, "w": 12, "h": 7}, + } + yaml_panel, result = translate_panel( + panel, + datasource_index="metrics-*", + esql_index="metrics-*", + rule_pack=resolved, + resolver=resolver, + ) + assert result.status in {"migrated", "migrated_with_warnings"}, result.reasons + query = (yaml_panel.get("esql") or {}).get("query") or "" + assert "labels.pod" in query + assert "RATE(container_cpu_usage_seconds_total)" in query + # Dead selector labels must be stripped, not filtered on. + assert "kubernetes_io_hostname" not in query + assert 'name RLIKE' not in query + assert "image" not in query + + +def test_315_containers_override_keeps_k8s_series_and_discloses_drop(): + resolved, resolver = _resolve_315() + panel = { + "id": 2, + "type": "graph", + "title": "Containers CPU usage (1m avg)", + "targets": [ + { + "expr": ( + 'sum (rate (container_cpu_usage_seconds_total' + '{image!="",name=~"^k8s_.*",container_name!="POD"}[1m]))' + " by (container_name, pod_name)" + ), + "legendFormat": "pod: {{ pod_name }} | {{ container_name }}", + "refId": "A", + }, + { + "expr": ( + 'sum (rate (container_cpu_usage_seconds_total' + '{image!="",name!~"^k8s_.*"}[1m]))' + " by (kubernetes_io_hostname, name, image)" + ), + "legendFormat": "docker: {{ name }}", + "refId": "B", + }, + ], + "gridPos": {"x": 0, "y": 0, "w": 12, "h": 7}, + } + yaml_panel, result = translate_panel( + panel, + datasource_index="metrics-*", + esql_index="metrics-*", + rule_pack=resolved, + resolver=resolver, + ) + assert result.status == "migrated_with_warnings" + query = (yaml_panel.get("esql") or {}).get("query") or "" + assert "labels.pod" in query and "labels.container" in query + assert 'labels.container != "POD"' in query + # The dropped-runtime disclosure must surface as a warning. + assert any("docker" in r or "rkt" in r for r in result.reasons) + + +def test_315_system_services_panel_degrades_to_empty_not_aggregate(): + resolved, resolver = _resolve_315() + panel = { + "id": 3, + "type": "graph", + "title": "System services CPU usage (1m avg)", + "targets": [ + { + "expr": ( + 'sum (rate (container_cpu_usage_seconds_total' + '{systemd_service_name!="",kubernetes_io_hostname=~"^$Node$"}[1m]))' + " by (systemd_service_name)" + ), + "legendFormat": "{{ systemd_service_name }}", + "refId": "A", + }, + ], + "gridPos": {"x": 0, "y": 0, "w": 12, "h": 7}, + } + yaml_panel, result = translate_panel( + panel, + datasource_index="metrics-*", + esql_index="metrics-*", + rule_pack=resolved, + resolver=resolver, + ) + assert result.status in {"migrated", "migrated_with_warnings"} + query = (yaml_panel.get("esql") or {}).get("query") or "" + # Curated ES|QL override: systemd_service_name never exists on modern + # cAdvisor, so grouping/breaking down by it would fail in Lens ("invalid + # column"). The override filters on an impossible container value so the + # panel degrades to an honest empty (data_gap) with valid columns — it must + # NOT reference the non-existent systemd_service_name column, and must NOT + # collapse into a single misleading aggregate (a BY grouping is retained). + assert "systemd_service_name" not in query + assert '"__systemd_service__"' in query + assert "labels.container" in query + assert "RATE(container_cpu_usage_seconds_total)" in query + + +def test_315_pods_memory_override_groups_by_pod_last_over_time(): + resolved, resolver = _resolve_315() + panel = { + "id": 4, + "type": "graph", + "title": "Pods memory usage", + "targets": [ + { + "expr": ( + "sum (container_memory_working_set_bytes" + '{image!="",name=~"^k8s_.*",kubernetes_io_hostname=~"^$Node$"})' + " by (pod_name)" + ), + "legendFormat": "{{ pod_name }}", + "refId": "A", + }, + ], + "gridPos": {"x": 0, "y": 0, "w": 12, "h": 7}, + } + yaml_panel, result = translate_panel( + panel, + datasource_index="metrics-*", + esql_index="metrics-*", + rule_pack=resolved, + resolver=resolver, + ) + assert result.status in {"migrated", "migrated_with_warnings"} + query = (yaml_panel.get("esql") or {}).get("query") or "" + # Gauge → LAST_OVER_TIME (no illegal SUM(MAX(...)) nested aggregate), grouped + # by pod with the breakdown accessor aligned to the ES|QL output column. + assert "LAST_OVER_TIME(container_memory_working_set_bytes)" in query + assert "labels.pod" in query + # Root cgroup (id=/) has no pod label; the override excludes it. + assert "labels.pod IS NOT NULL" in query + + +def test_315_pods_panels_breakdown_accessor_matches_query_column(): + """Curated ES|QL keeps the Lens breakdown on labels.pod, not phantom pod_name. + + The native PROMQL DSL path leaves the breakdown accessor bound to the + pre-rewrite ``pod_name`` (Lens "invalid column" after pod_name -> + labels.pod). The override must emit ``labels.pod`` as an actual query + output column and never reference the bare ``pod_name``. + """ + resolved, resolver = _resolve_315() + for title, expr in ( + ( + "Pods CPU usage (1m avg)", + "sum (rate (container_cpu_usage_seconds_total[1m])) by (pod_name)", + ), + ( + "Pods memory usage", + "sum (container_memory_working_set_bytes) by (pod_name)", + ), + ): + panel = { + "id": 9, + "type": "graph", + "title": title, + "targets": [ + {"expr": expr, "legendFormat": "{{ pod_name }}", "refId": "A"} + ], + "gridPos": {"x": 0, "y": 0, "w": 12, "h": 7}, + } + yaml_panel, result = translate_panel( + panel, + datasource_index="metrics-*", + esql_index="metrics-*", + rule_pack=resolved, + resolver=resolver, + ) + assert result.status in {"migrated", "migrated_with_warnings"} + esql = yaml_panel.get("esql") or {} + query = esql.get("query") or "" + assert "`labels.pod`" in query, f"{title}: {query}" + # No bare pre-rewrite label token as a standalone identifier. + assert "pod_name" not in query, f"{title} leaked pod_name: {query}" + + +def test_315_pods_network_override_names_received_and_sent(): + resolved, resolver = _resolve_315() + panel = { + "id": 5, + "type": "graph", + "title": "Pods network I/O (1m avg)", + "targets": [ + { + "expr": ( + "sum (rate (container_network_receive_bytes_total" + '{image!="",name=~"^k8s_.*"}[1m])) by (pod_name)' + ), + "legendFormat": "-> {{ pod_name }}", + "refId": "A", + }, + { + "expr": ( + "- sum (rate (container_network_transmit_bytes_total" + '{image!="",name=~"^k8s_.*"}[1m])) by (pod_name)' + ), + "legendFormat": "<- {{ pod_name }}", + "refId": "B", + }, + ], + "gridPos": {"x": 0, "y": 0, "w": 12, "h": 7}, + } + yaml_panel, result = translate_panel( + panel, + datasource_index="metrics-*", + esql_index="metrics-*", + rule_pack=resolved, + resolver=resolver, + ) + assert result.status in {"migrated", "migrated_with_warnings"} + query = (yaml_panel.get("esql") or {}).get("query") or "" + assert "Received =" in query + assert "Sent =" in query + assert "value_B" not in query + assert "labels.pod" in query + + +def test_315_all_processes_override_groups_by_cgroup_id(): + resolved, resolver = _resolve_315() + panel = { + "id": 6, + "type": "graph", + "title": "All processes CPU usage (1m avg)", + "targets": [ + { + "expr": ( + "sum (rate (container_cpu_usage_seconds_total" + '{id!="/",kubernetes_io_hostname=~"^$Node$"}[1m])) by (id)' + ), + "legendFormat": "{{ id }}", + "refId": "A", + }, + ], + "gridPos": {"x": 0, "y": 0, "w": 12, "h": 7}, + } + yaml_panel, result = translate_panel( + panel, + datasource_index="metrics-*", + esql_index="metrics-*", + rule_pack=resolved, + resolver=resolver, + ) + assert result.status in {"migrated", "migrated_with_warnings"} + query = (yaml_panel.get("esql") or {}).get("query") or "" + assert "`labels.id`" in query + assert 'labels.id != "/"' in query + assert "labels.id IS NOT NULL" in query + +