feat(grafana): add Kubernetes 3831 and 1471 curated packs

docs/design/curated-pack-1471-kubernetes-apps.md

@@ -0,0 +1,56 @@
+# Curated Pack — Grafana 1471 "Kubernetes App Metrics"
+
+> Design + living discoveries for the 1471 curated pack. Follows the general
+> Curation Playbook in `curated-dashboard-packs-plan.md`. Stacks on 315
+> (cAdvisor) and 741 (Deployment metrics).
+
+- Source: community **"Kubernetes App Metrics"**,
+  <https://grafana.com/grafana/dashboards/1471-kubernetes-apps/>
+- gnetId **1471**, only revision **1**.
+- canonical sha256 (rev 1) = `27552f4c9ba5ce3e43ae7c962c98c6979e87e1e4c2f4d463f26df62652cabc52`.
+- Datasource: Prometheus — curated-pack eligible.
+- Schema: **v14 (`rows[]`)**. 13 leaf panels.
+- Metric family: **cAdvisor** (`container_*`, `container_spec_*`) **and**
+  app HTTP (`http_requests_total`, `nginx_http_*`, `haproxy_backend_http_*`).
+- Variables: `$namespace` =
+  `label_values(container_memory_usage_bytes{…container_name!="POD"}, namespace)`;
+  `$container` =
+  `label_values(…{namespace=~"$namespace",container_name!="POD"}, container_name)`.
+  HTTP panels also filter `app="$container"` (app name == container name).
+
+## Goal
+
+Ship a curated pack so 1471 renders in Kibana against a modern cAdvisor scrape
+plus optional native/nginx/haproxy HTTP metrics, with working namespace and
+container controls.
+
+## The core problem
+
+| Dashboard convention | Modern reality | Pack handling |
+|---|---|---|
+| `container_name` / `pod_name` | cAdvisor `container` / `pod` | `label_rewrites` |
+| `kubernetes_io_hostname` | scrape `instance` | rewrite → `instance` |
+| HTTP `kubernetes_namespace` | `namespace` | rewrite → `namespace` |
+| `app="$container"` | app label equals container name | HTTP ES\|QL binds `?container` to `app` |
+| nginx grouped by `status`, native/haproxy by `code` | engine drops nginx | curated ES\|QL; per-doc `series` like `native \| 200` then `STATS rate BY time_bucket, series` so Lens matches Grafana legendFormat |
+| `histogram_quantile(*_bucket)` | no classic-histogram field | `PERCENTILE` of duration gauge |
+| per-pod `by (id, pod_name)` + limit/request lines | Lens XY one breakdown | group by `pod`; drop reference lines |
+
+## Engine vs pack split
+
+Pipeline already handles irate/rate, `rows[]` → sections, flattening
+single-panel legacy rows, and control synthesis. The pack carries the Heapster
+label bridge, populate rewrites, HTTP named series, ratio ES|QL, and
+`LAST_OVER_TIME` gauges (raw `SUM` of gauge docs over-counts).
+
+## Fidelity
+
+- **PERFECT**: request/error rates, pod/host counts, usage-vs-request/limit,
+  avg-per-pod and total CPU/memory.
+- **APPROXIMATE**: response-time percentiles (PERCENTILE vs histogram_quantile);
+  per-pod CPU/memory (reference lines dropped).
+
+## Validation
+
+Shared curated rig `k8s_exporter.py` emits cAdvisor spec/user/system + HTTP
+series (`app` = container name) into `metrics-k8s.prometheus-default`.

docs/design/curated-pack-3831-kubernetes-autoscaler.md

@@ -0,0 +1,48 @@
+# Curated Pack — Grafana 3831 "Kubernetes Cluster Autoscaler (via Prometheus)"
+
+> Design + living discoveries for the 3831 curated pack. Follows the general
+> Curation Playbook in `curated-dashboard-packs-plan.md`. Stacks on the
+> Kubernetes 315 / 6417 / 741 / 8171 packs.
+
+- Source: community **"Kubernetes Cluster Autoscaler (via Prometheus)"**,
+  <https://grafana.com/grafana/dashboards/3831-autoscaler/>
+- gnetId **3831**, only revision **1**.
+- canonical sha256 (rev 1) = `fbfc5ff33d138f2a449d4d28b0c0dcd09d351f08a2c78b91e3455d00e2b0b597`.
+- Datasource: Prometheus — curated-pack eligible.
+- Schema: **v14 (`rows[]`)**. 10 leaf panels. No template variables.
+- Metric family: **cluster-autoscaler** `/metrics`
+  (`cluster_autoscaler_nodes_count`, `*_last_activity`, `*_total` counters).
+
+## Goal
+
+Ship a curated pack so 3831 renders in Kibana against a cluster-autoscaler
+scrape. Force ES|QL for every panel (native PROMQL `LAST(value, step)` is
+empty in Lens on prometheus_native ingest) and polish KPI titles for Kibana
+tiles.
+
+## Engine vs pack split
+
+Pipeline already handles `rows[]` → sections, flattening Grafana's placeholder
+"New row", and singlestat reducers. The pack pins metric kinds, emits ES|QL
+for `time()-last_activity` (`DATE_DIFF`), named Activity/Autoscaling series,
+and the ready/total + scaled_up−scaled_down KPIs.
+
+Grafana plots `*_total` counters with bare `sum()` (no `rate()`). Kibana
+matches that with `LAST_OVER_TIME`, not `RATE`.
+
+Grafana's placeholder row `"New row"` is flattened to top-level panels.
+Those must sit at layout `y: 2` (after Info `y=0` and Activity `y=1`) or
+they overlap the Info KPIs. Nodes available is `ready/total * 100` so the
+Kibana number+% tile shows 100%, not 1.00%.
+
+## Fidelity
+
+- **PERFECT**: Total nodes, Nodes available, last-activity tiles, Pod/Node/
+  Autoscaling activity graphs.
+- **APPROXIMATE**: Safe to autoscale (0/1, no Yes/No value map), shortened
+  Unscheduled pods / Net scaled nodes titles.
+
+## Validation
+
+Shared curated rig `k8s_exporter.py` emits `cluster_autoscaler_*` into
+`metrics-k8s.prometheus-default`.

docs/sources/grafana.md

@@ -280,6 +280,32 @@ ES|QL (`time_bucket` + `labels.device`, `$server` bound) because native PROMQL
 "No results found". `$server` stays
 `label_values(node_boot_time_seconds, instance)`.
 
+The Kubernetes Cluster Autoscaler (3831) pack targets cluster-autoscaler
+`/metrics`. Grafana plots several `*_total` counters with bare `sum()` (no
+`rate()`), so Kibana keeps `LAST_OVER_TIME` cumulative totals rather than
+`RATE`. Every panel is curated ES|QL because native PROMQL `LAST(value, step)`
+is empty in Lens on prometheus_native ingest. `time()-last_activity` becomes
+`DATE_DIFF` seconds. Nodes available is `ready/total * 100` (Grafana percent
+0–100) so the Kibana number+% tile shows 100%, not 1%. The placeholder
+Grafana row `"New row"` flattens to top-level panels placed at `y: 2` after
+the Info and Activity sections. KPI chrome titles are shortened for Kibana
+tiles (Safe to autoscale, Unscheduled pods, Since scale-down/autoscale, Net
+scaled nodes); the 0/1 safe-to-scale gauge has no Yes/No value map.
+
+The Kubernetes App Metrics (1471) pack is a pre-1.16 cAdvisor + app-HTTP mix.
+Heapster labels (`container_name`, `pod_name`, `kubernetes_io_hostname`) and
+HTTP `kubernetes_namespace` rewrite to canonical `container` / `pod` /
+`instance` / `namespace`. `$container` is used both as the cAdvisor container
+and as HTTP `app` (dashboard convention: `app` equals container name). The
+pack plugin rewrites the populate queries off `container_name`. Request rate
+restores the nginx series the engine drops (status vs code grouping) as
+Grafana-style `native | 200` / `nginx | 500` / `haproxy | 2xx` series (one
+Lens breakdown, not a cartesian Native-by-code legend). nginx/haproxy metrics are
+`live_optional`. Per-pod graphs group by `pod` rather than `(id, pod_name)`
+and drop Grafana's limit/request reference lines (Lens XY has one breakdown).
+Response-time panels approximate `histogram_quantile` with `PERCENTILE` of
+the duration gauge.
+
 Each pack is registered in `curated_packs/registry.yaml` with a
 `gnet_revision` and `dashboard_sha256` — maintainer-verified provenance pins
 recording the exact grafana.com revision the pack authors read, re-checkable

observability_migration/adapters/source/grafana/curated_packs/grafana_1471_kubernetes_app_metrics/fidelity_manifest.yaml

@@ -0,0 +1,71 @@
+# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one or more contributor license agreements.
+# SPDX-License-Identifier: Elastic-2.0
+#
+# Fidelity manifest — Grafana dashboard 1471 (Kubernetes App Metrics)
+# https://grafana.com/grafana/dashboards/1471-kubernetes-apps/
+#
+# PERFECT     = same information as Grafana on modern cAdvisor + app HTTP scrape.
+# APPROXIMATE = documented delta (histogram_quantile, dropped reference lines).
+
+# Field-profile portable: pack.yaml authors canonical names; the resolver
+# namespaces per --field-profile. Verified by the cross-profile leakage gate.
+
+schema_version: 1
+gnet_id: 1471
+gnet_revision: 1
+dashboard_title: "Kubernetes App Metrics"
+maintainer: "community"
+
+panels:
+  - title: "Request rate"
+    fidelity: PERFECT
+    notes: "irate of native/nginx/haproxy request counters. Nginx restored (engine dropped it for status vs code grouping). One rate series per source+status (native | 200 / nginx | 500 / haproxy | 2xx) so Lens breaks down on `series` instead of a cartesian source x code legend; $container binds app, $namespace binds namespace."
+  - title: "Error rate"
+    fidelity: PERFECT
+    notes: "5xx / total per native (code=~5xx, handler!=prometheus), nginx (status=~5xx), haproxy (code=5xx) as named series."
+  - title: "Response time percentiles"
+    fidelity: APPROXIMATE
+    notes: "Grafana histogram_quantile on *_bucket; Kibana PERCENTILE of the duration gauge (t-digest scrape samples)."
+  - title: "Number of pods"
+    fidelity: PERFECT
+    notes: "COUNT_DISTINCT(pod) / COUNT_DISTINCT(instance) replacing nested count(count()) by pod_name / kubernetes_io_hostname."
+  - title: "Cpu usage (relative to request)"
+    fidelity: PERFECT
+    notes: "irate(cpu_system|user) / (cpu_shares/1024) as system/user series."
+  - title: "Cpu usage (relative to limit)"
+    fidelity: PERFECT
+    notes: "irate(cpu_usage) / (quota/period). Curated ES|QL; native PROMQL LAST step is empty in Lens."
+  - title: "Memory usage (relative to limit)"
+    fidelity: PERFECT
+    notes: "working memory_usage / spec_memory_limit LAST_OVER_TIME ratio."
+  - title: "Cpu usage (per pod)"
+    fidelity: APPROXIMATE
+    notes: "Per-pod IRATE by labels.pod. Grafana also plotted limit/request reference lines; Lens XY has one breakdown so those lines are dropped."
+  - title: "Memory usage (per pod)"
+    fidelity: APPROXIMATE
+    notes: "Per-pod LAST_OVER_TIME by labels.pod. Limit reference line dropped for the same Lens breakdown reason."
+  - title: "Cpu usage (avg per pod)"
+    fidelity: PERFECT
+    notes: "total CPU / COUNT_DISTINCT(pod) plus per-pod-average limit (quota/period) and request (shares/1024)."
+  - title: "Memory usage (avg per pod)"
+    fidelity: PERFECT
+    notes: "SUM(memory)/COUNT_DISTINCT(pod) and SUM(limit)/COUNT_DISTINCT(pod)."
+  - title: "Cpu usage (total)"
+    fidelity: PERFECT
+    notes: "IRATE cpu plus limit=quota/period and request=shares/1024 as named series."
+  - title: "Memory usage (total)"
+    fidelity: PERFECT
+    notes: "LAST_OVER_TIME memory vs spec limit."
+
+summary:
+  total_panels: 13
+  perfect: 10
+  approximate: 3
+  gap: 0
+  notes: >
+    $namespace / $container populate from container_memory_usage_bytes with
+    canonical container/namespace labels (plugin rewrites container_name).
+    HTTP panels bind $container to app (dashboard convention: app == container
+    name) and $namespace after kubernetes_namespace → namespace. nginx/haproxy
+    series are live_optional. Detailed usage rows stay collapsed like Grafana;
+    Request rate, Error rate, latency, pod count, and usage-vs-request stay open.