Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions docs/design/curated-pack-1621-kubernetes-cadvisor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Curated Pack — Grafana 1621 "Kubernetes cluster monitoring (via Prometheus)"

> Design + living discoveries for the 1621 curated pack. Follows the general
> Curation Playbook in `curated-dashboard-packs-plan.md`. Stacks on 315
> (same dashboard family; 1621 is the all-partitions filesystem fork).

- Source: community **"Kubernetes cluster monitoring (via Prometheus)"**,
<https://grafana.com/grafana/dashboards/1621>
- gnetId **1621**, only revision **1**.
- canonical sha256 (rev 1) = `d714551536ca794e3088ac535e59b3d2f93dd705e3b60f84d6f3c42da40fe9d4`.
- Datasource: Prometheus — curated-pack eligible.
- Schema: **v12 (`rows[]`)**. 21 panels across 13 rows (same layout as 315).
- Metric family: **cAdvisor** (`container_*` + `machine_*`).
- Variables: `$Node` = `label_values(kubernetes_io_hostname)` (includeAll `.*`).

## Goal

Ship a curated pack so 1621 renders in Kibana against a modern cAdvisor scrape
with **every `/dev/*` filesystem partition aggregated** (the reason this
dashboard exists vs 315) and a working `$Node` control via `instance`.

## 1621 vs 315

| | 315 | 1621 |
|---|---|---|
| Cluster filesystem device matcher | `^/dev/[sv]d[a-z][1-9]$` | `^/dev/.*$` |
| `$Node` | dropped (ignored hostname) | rewritten to `instance`; multi-select populate from `machine_cpu_cores` |
| Title | identical | identical — 1621 is **gnetId-only** in the registry |

## Engine vs pack split

Same split as 315 for the pre-1.16 label bridge, k8s-only container series,
Received/Sent names, cgroup-id All-processes, and systemd honest empty. Pack
adds filesystem per-device `LAST_OVER_TIME` then `SUM`, Node bind on every
live panel, and a 48-col named Memory/CPU/Filesystem KPI strip.

## Fidelity

- **PERFECT**: Network I/O pressure; filesystem used/total/% (all `/dev/*`);
per-pod CPU/memory/network.
- **APPROXIMATE**: cluster memory/CPU ratios (need `machine_*` + `id="/"`);
Containers (docker/rkt dropped); All-processes (cgroup id).
- **GAP**: System services (`systemd_service_name`).

## Validation

Shared curated rig `k8s_exporter.py` now emits two root-cgroup devices
(`/dev/sda1`, `/dev/nvme0n1p1`) into
`metrics-k8s.prometheus-default`. 315's regex matches only sda1; 1621 sums
both.
55 changes: 55 additions & 0 deletions docs/design/curated-pack-747-kubernetes-pod-metrics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Curated Pack — Grafana 747 "Kubernetes Pod Metrics"

> Design + living discoveries for the 747 curated pack. Follows the general
> Curation Playbook in `curated-dashboard-packs-plan.md`. Stacks on 315/1621
> (cAdvisor) and 6417 (kube-state-metrics).

- Source: community **"Kubernetes Pod Metrics"**,
<https://grafana.com/grafana/dashboards/747-pod-metrics/>
- gnetId **747**, latest revision **2**.
- canonical sha256 (rev 2) = `47837bfea31e6156decd5c152379addfd9c3f9a9991a0cc96d764f91388b2d55`.
- Datasource: Prometheus — curated-pack eligible.
- Schema: **v12 (`rows[]`)**. 18 leaf panels.
- Metric family: **cAdvisor** (`container_*` + `machine_*`) **and**
**kube-state-metrics** (`kube_pod_info`, `kube_pod_status_phase`,
`kube_pod_container_info`, `kube_pod_container_status_restarts`).
- Variables: `$Node` (hostname), `$Pod` (`label_values(kube_pod_info, pod)`),
hidden `$Pod_ip` / `$phase` / `$container`.

## Goal

Ship a curated pack so 747 is a pod-scoped Kibana dashboard: working Pod/Node
controls, live IP/phase/container tiles instead of markdown `$var`
interpolation, and cAdvisor graphs that group by canonical `pod`.

## The core problem

| Dashboard convention | Modern reality | Pack handling |
|---|---|---|
| `pod_name` / `io_kubernetes_pod_name` | cAdvisor `pod` | `label_rewrites` + ES\|QL overrides |
| `kubernetes_io_hostname` | scrape `instance` | rewrite → `instance`; plugin populate |
| markdown `# $Pod_ip` (hidden var) | Kibana markdown does not interpolate controls | datatable of pod → IP; engine allows query_overrides on `text` panels |
| `kube_pod_container_status_restarts` | `*_total` counter | `metric_map` |
| `pod=~"$Pod.*$"` prefix | Grafana regex interpolation | exact `MV_CONTAINS` plus `pod IS NOT NULL` (root cgroup must not leak in) |
| CPU Total `format=bytes` + node-wide container rate | not capacity, not bytes | Kibana title **Node CPU** |
| `$Pod` single-select + All | hydrates to first pod | multi-select so first paint is All |

## Engine vs pack split

Pipeline already handles rate/gauge, `rows[]` → sections, hidden-variable
skip, and `^$Pod$` exact matchers. The pack carries the Heapster label
bridge, text→ES|QL overrides (skip the markdown early-return
when a curated `esql_query` is present), Node/Pod multi-select, Received/Sent
names, named Used/Total tiles, and `pod IS NOT NULL` so MV_CONTAINS cannot
pull in the root cgroup.

## Fidelity

- **PERFECT**: IP/status/container tiles, restarts, network pressure, pod
graphs, Memory/CPU used, Node CPU, working-set tile.
- **APPROXIMATE**: pod/node % ratios (cross-metric); All-processes (cgroup id).

## Validation

Shared curated rig `k8s_exporter.py` emits `pod_ip` on `kube_pod_info` and
`kube_pod_container_info` into `metrics-k8s.prometheus-default`.
25 changes: 25 additions & 0 deletions docs/sources/grafana.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,31 @@ non-existent breakdown column. The cluster-total KPI strip additionally needs th
node `machine_*` metrics and the root-cgroup (`id="/"`) + `container_fs_*` series
to populate.

The Kubernetes cluster-monitoring 1621 pack is the community fork of 315 whose
Cluster Filesystem usage sums every cAdvisor `device=~"^/dev/.*$"` partition
instead of 315's `^/dev/[sv]d[a-z][1-9]$` (which misses nvme and extra disks).
Filesystem KPIs `LAST_OVER_TIME` per device then `SUM`. Unlike 315, `$Node` is
kept: `kubernetes_io_hostname` rewrites to canonical `instance`, the plugin
populates from `label_values(machine_cpu_cores, instance)` and marks the control
multi-select so first paint is Grafana All. Duplicate Used/Total tiles are
renamed Memory/CPU/Filesystem used/total on a 48-col strip. Title fallback
cannot distinguish 1621 from 315 (identical grafana.com titles); detection is
gnetId-only.

The Kubernetes Pod Metrics (747) pack is pod-scoped cAdvisor plus
kube-state-metrics. Heapster `pod_name` / `io_kubernetes_pod_name` rewrite to
canonical `pod`; `$Node` follows the 741/1621 instance bridge. Hidden Grafana
variables (`$Pod_ip`, `$phase`, `$container`) only interpolated markdown
(`# $Pod_ip`); Kibana cannot interpolate those, so curated ES|QL datatables
list `pod` → `pod_ip` / `container`, and the phase tile keeps the 9628-style
metric legend. `$Pod` is multi-select on
`kube_pod_info`. Restarts map `kube_pod_container_status_restarts` → `*_total`.
Pod-scoped queries also require the canonical `pod` label so `MV_CONTAINS`
cannot include the root cgroup (`id="/"`) the way PromQL `pod=~` would reject
unlabeled series. Grafana's CPU Total tile queried node-wide container CPU rate with
`format=bytes`; Kibana titles it Node CPU and shows a number. Network
butterflies are named Received/Sent; All-processes panels group by cgroup `id`.

The Kubernetes Cluster (kube-state-metrics 6417) pack targets the KSM +
`node_exporter` family and was authored against an older lineage: `metric_map`
bridges the renamed names (`node_filesystem_size`/`_free` →
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one or more contributor license agreements.
# SPDX-License-Identifier: Elastic-2.0
#
# Fidelity manifest — Grafana dashboard 1621 (Kubernetes cluster monitoring via Prometheus)
# https://grafana.com/grafana/dashboards/1621
#
# Fork of 315 whose Cluster Filesystem usage sums every cAdvisor `/dev/*`
# partition instead of `/dev/[sv]d[a-z][1-9]`. `$Node` is kept (via instance)
# rather than dropped.

schema_version: 1
gnet_id: 1621
gnet_revision: 1
dashboard_title: "Kubernetes cluster monitoring (via Prometheus)"
maintainer: "community"

panels:
- title: "Network I/O pressure"
fidelity: PERFECT
notes: "sum(rate(receive/transmit)) butterfly named Received/Sent; bound to $Node via labels.instance."

- title: "Cluster memory usage"
fidelity: APPROXIMATE
notes: "root-cgroup working set / machine_memory_bytes. requires: machine_memory_bytes + id=/ series. Cross-metric ratio."
- title: "Cluster CPU usage (1m avg)"
fidelity: APPROXIMATE
notes: "root-cgroup CPU rate / machine_cpu_cores. requires: machine_cpu_cores + id=/ series."
- title: "Cluster filesystem usage"
fidelity: PERFECT
notes: "1621 differentiator vs 315: SUM of every device starting with /dev/ (nvme, extra disks included). Per-device LAST_OVER_TIME then SUM. requires: container_fs_* with a /dev/ device label."
- title: "Memory used"
fidelity: APPROXIMATE
notes: "sum(container_memory_working_set_bytes{id=/}). Kibana title renamed from duplicate Used."
- title: "Memory total"
fidelity: APPROXIMATE
notes: "sum(machine_memory_bytes)."
- title: "CPU used"
fidelity: APPROXIMATE
notes: "sum(rate(container_cpu_usage_seconds_total{id=/}))."
- title: "CPU total"
fidelity: APPROXIMATE
notes: "sum(machine_cpu_cores)."
- title: "Filesystem used"
fidelity: PERFECT
notes: "All /dev/* partitions (not 315's [sv]d[a-z][1-9] regex)."
- title: "Filesystem total"
fidelity: PERFECT
notes: "All /dev/* partition limits."

- title: "Pods CPU usage (1m avg)"
fidelity: PERFECT
notes: "Curated ES|QL groups by canonical pod; Node bound via instance."
- title: "Pods memory usage"
fidelity: PERFECT
notes: "LAST_OVER_TIME working set by pod."
- title: "Pods network I/O (1m avg)"
fidelity: PERFECT
notes: "Received/Sent by pod (native butterfly would emit Value B)."

- title: "Containers CPU usage (1m avg)"
fidelity: APPROXIMATE
notes: "k8s pod/container series only; docker/rkt dropped."
- title: "Containers memory usage"
fidelity: APPROXIMATE
notes: "As above."
- title: "Containers network I/O (1m avg)"
fidelity: APPROXIMATE
notes: "As above; Received/Sent by pod."

- title: "System services CPU usage (1m avg)"
fidelity: GAP
notes: "systemd_service_name absent on modern cAdvisor; impossible-filter override → honest empty."
- title: "System services memory usage"
fidelity: GAP
notes: "Same systemd gap."
- title: "All processes CPU usage (1m avg)"
fidelity: APPROXIMATE
notes: "Grouped by cgroup id excluding /."
- title: "All processes memory usage"
fidelity: APPROXIMATE
notes: "As above."
- title: "All processes network I/O (1m avg)"
fidelity: APPROXIMATE
notes: "As above; Received/Sent."

summary:
total_panels: 21
perfect: 7
approximate: 12
gap: 2
notes: >
Same cAdvisor lineage as 315, but filesystem KPIs aggregate every /dev/*
partition (the reason this dashboard exists) and $Node filters via
instance instead of being dropped. Duplicate Used/Total titles are renamed
Memory/CPU/Filesystem used/total on a 48-col Kibana KPI strip.
Loading