Skip to content
17 changes: 17 additions & 0 deletions docs/design/curated-dashboard-packs-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,23 @@ tests/

## Key Discoveries (update as you go)

**PostgreSQL Exporter 12485 (2026-08-31) — see `curated-pack-12485-postgresql-exporter.md`:**
- The shared curated rig (`parity-rig/curated/grafana_763_redis_exporter/`) already
runs a real `prometheuscommunity/postgres-exporter:v0.15.0` + load generator into
`metrics-postgres.prometheus-default`. New postgres packs validate here — no
throwaway rig. Enable `--collector.stat_statements` + `--collector.postmaster`
(+ the `pg_stat_statements` extension) on the rig's postgres services when a pack
needs those series; redis/other exporters are untouched.
- Read the **real** exporter (`curl :9187/metrics`) before writing `metric_map`:
12485 was authored against an older lineage, so `pg_database_size` →
`pg_database_size_bytes`, `pg_replication_lag` → `pg_replication_lag_seconds`,
`pg_stat_statements_calls` → `_calls_total`, `pg_stat_statements_total_time_seconds`
→ `_seconds_total`. `pg_stat_activity_count` / `pg_locks_count` are **gauges**
despite the `_count` suffix — force them in `metric_kinds`.
- Live result: 35 panels, 0 Red / 0 not-feasible, uploaded; render audit 0
render_error (32/32 rendered); both controls populate + bind (`?Instance` ×57,
`?Database` ×23). 14114 re-validated on the same rig: 6/6, render PASS, no regression.

**Dashboard investigation (2026-07-29):**
- Dashboard **12776** ("Redis" by the Redis org): uses `redis-datasource` plugin (NOT Prometheus). Queries are Redis commands (`INFO`, `CLIENT LIST`, `SLOWLOG GET`) with empty `expr` fields. `infer_query_language()` returns `"unknown"` for empty queries → panels land as not_feasible. **Not the right target for a PromQL curated pack.** The 12776 readme on grafana.com links to two Prometheus-based alternatives (see below). Future work: a separate ES|QL-injection curated pack variant for `redis-datasource` type dashboards mapped to Elastic Redis integration fields.
- Dashboard **763** ("Redis Dashboard for Prometheus Redis Exporter 1.x" by oliver006): uses `prometheus` datasource. 13 panels, all PromQL using `redis_exporter` metrics. gnetId=763, revision=6 (latest, 2024-02-17, 181,727 downloads). **This is our first target.**
Expand Down
10 changes: 9 additions & 1 deletion docs/design/curated-dashboard-packs.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,15 @@ panel:
status_override: migrated # migrated | migrated_with_warnings (default: migrated)
```

**Merge semantics:** User pack overrides win by `title_match`. If both the curated pack and the user `--rules-file` declare an override for the same panel title, the user's query wins. Overrides with different titles are merged (both apply).
Optional `section_match` scopes the override to a Grafana row whose title
casefolds to (or starts with) that string, the same way layout overrides
distinguish Global vs Database duplicate titles. Layout matching uses that
source title even when a layout override later renames the section.

**Merge semantics:** User pack overrides win by `(title_match, section_match)`.
If both the curated pack and the user `--rules-file` declare an override for
the same panel title and section, the user's query wins. Overrides with
different titles or sections are merged (both apply).

**ES|QL shape constraints:** The query must produce a shape that `_native_esql_panel_spec` can parse for the target Kibana panel type:
- `metric` / `gauge` panels: a `STATS` query with exactly one metric column and no `BY` clause. The simplest form is an inline division: `STATS value = MAX(...) / MAX(...)`.
Expand Down
163 changes: 163 additions & 0 deletions docs/design/curated-pack-12485-postgresql-exporter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
# Curated Pack — Grafana 12485 "PostgreSQL Exporter"

> Design + living discoveries for the 12485 curated pack. Follows the general
> Curation Playbook in `curated-dashboard-packs-plan.md`. 14114 (PostgreSQL
> Exporter Quickstart) already ships a validated pack; this doc is the net-new
> 12485 work plus a live re-validation pass of 14114.

- Source: Grafana Labs / community **"PostgreSQL Exporter"**,
<https://grafana.com/grafana/dashboards/12485-postgresql-exporter/>
- gnetId **12485**, only revision is **1** (2020-06-17).
- canonical sha256 (rev 1) = `e14a35eac532db4f79837a293411edb20d04164ca59de17d73557d08637a4700`
(matches `parity-rig/benchmark/community_corpus.json`).
- Datasource: Prometheus (`__inputs[0].pluginId == prometheus`) — curated-pack eligible.
- Panels: ~37 across two sections — **Global Statistics** and **Database: $Database**
(the DB section repeats the same metric families scoped by `datname`).
- Controls: `Instance`, `Database`, `Interval` (Grafana interval var).

## Goal

Ship a curated pack so 12485 renders in Kibana with the same information as
Grafana (or better) against a real `postgres_exporter` scrape ingested in the
Elastic `prometheus_native` layout (`metrics.* + labels.*`), then prove it with
render + interaction audits and a side-by-side.

## Engine vs pack split

The general pipeline already handles rate/gauge translation, five-target
fusion, and control synthesis. The pack only carries what 12485 needs beyond
that, verified against the **real** exporter (not guessed):

### Empirical exporter findings (prometheuscommunity/postgres-exporter v0.15.0)

Read directly from the live rig exporter (`curl :9187/metrics`). The dashboard
was authored against an older exporter lineage, so several names differ from
what v0.15.0 actually emits:

| Dashboard PromQL name | Real exporter field | Kind | Note |
|---|---|---|---|
| `pg_database_size` | `pg_database_size_bytes` | gauge | rename |
| `pg_replication_lag` | `pg_replication_lag_seconds` | gauge | rename; 0 on a standalone primary |
| `pg_stat_statements_calls` | `pg_stat_statements_calls_total` | counter | needs `--collector.stat_statements` + extension |
| `pg_stat_statements_total_time_seconds` | `pg_stat_statements_seconds_total` | counter | same |
| `pg_postmaster_start_time_seconds` | *(same)* | gauge | needs `--collector.postmaster` (off by default) |
| `pg_stat_activity_count` | *(same)* | **gauge** | `_count` suffix would mislead offline heuristic → force gauge |
| `pg_locks_count` | *(same)* | **gauge** | same |
| `pg_settings_shared_buffers_bytes` | *(same)* | gauge | present as-is |
| `pg_stat_database_*` (xact/tup/blk_time/deadlocks/temp_files/blks_*) | *(same)* | counter | exporter `# TYPE` = counter |

`stat_statements` and `postmaster` collectors are **off by default** in v0.15.0,
and `pg_stat_statements` requires `shared_preload_libraries` + `CREATE EXTENSION`.
The rig (`parity-rig/curated/grafana_763_redis_exporter/`) was extended to enable
all three so Query rate / Average query runtime / Uptime render on real data.
On a target cluster that does not run these, those panels are an honest
`field_gap`/`data_gap`, not a translator bug.

### Pack rules

- **`metric_kinds`** — force `pg_stat_activity_count` + `pg_locks_count` +
`pg_stat_database_numbackends` to `gauge`; assert counters for the rated
`pg_stat_database_*` and mapped `pg_stat_statements_*` targets; gauges for the
`pg_settings_*` / size / lag / start-time series.
- **`metric_map`** — the four renames above (targets emitted verbatim under `metrics.`).
- **`label_rewrites` / `label_candidates`** — `instance`→`labels.instance`,
`datname`/`db`→`labels.datname`, `job`→`labels.job`, `state`→`labels.state`,
`mode`→`labels.mode`.
- **`controls.field_overrides`** — `Instance`/`instance`→`labels.instance`,
`Database`/`database`/`datname`→`labels.datname` (both cases, since
`resolve_control_field` matches the variable name exactly).
- **`plugin.py`** — rewrite `Instance` populate
`label_values({job="postgres-exporter"}, instance)` → `label_values(pg_up, instance)`
(the `postgres-exporter` job filter never matches Elastic labels); rewrite
`Database` populate `label_values(datname)` →
`label_values(pg_stat_database_numbackends, datname)` (needs a per-db metric
anchor); drop the `Interval` control if it lands as an inert control.

### Fidelity

- **PERFECT**: rate panels (Transactions, Tuples, Deadlocks, Temp files, I/O
time, Transaction rate, Query rate), gauge stats (Version, Max/Shared
buffers, Active clients, Connections by state/db, Locks by state, DB size,
Replication lag, Numbackends).
- **APPROXIMATE** (PERFECT under native PROMQL, documented delta in ES|QL):
Shared Buffer Hits, Commit Ratio, Connections used, PostgreSQL Uptime
(`time() - start_time`). Average query runtime is a curated last-non-null
`rate/rate` override so the KPI does not render N/A on the incomplete
window-edge `delta/delta` bucket. Add per-panel ES|QL `query_overrides`
only where a panel would otherwise `render_error` or empty-state.

## Validation gates (UI testing)

1. Migrate + upload to Kibana (`prometheus_native`, `--esql-index` = data view).
2. Render audit — 0 `render_error`; any `field_gap`/`data_gap` documented in
`fidelity_manifest.yaml`.
3. Interaction audit — `Instance` + `Database` controls rewrite panel queries.
4. Side-by-side vs the provisioned Grafana 12485 in a clean Kibana view session.

## Task checklist

- [x] registry.yaml entry (12485, rev 1, sha above)
- [x] pack.yaml + plugin.py + fidelity_manifest.yaml
- [x] offline fixture tests; `typecheck` green (own files `ruff`-clean)
- [x] rig: enable stat_statements + postmaster + extension
- [x] live: migrate + upload + render/interaction audit
- [x] re-validate 14114 on the same rig
- [x] docs: discoveries here + `docs/sources/grafana.md`

## Live validation results (2026-08-31, rig ES 9.5 + Kibana, real postgres_exporter)

**Migration** (`obs-migrate migrate --field-profile prometheus_native --es-url … --upload`):
35 panels — 18 migrated, 17 migrated_with_warnings, **0 requires-manual, 0
not-feasible; verification gate 18 Green / 17 Yellow / 0 Red; 35/35 ES|QL
queries validated; uploaded**. 23 panels native-PROMQL (oracle-verifiable), 12
ES|QL. Curated pack auto-fired on gnetId 12485.

**Render audit** (`render_audit_driver --elements`, headless Chrome vs the live
upload): **all 32 panel elements rendered; 0 render_error, 0 error markers, 0
console/server errors.** Emitted viz types are correct (graphs → `xy`,
singlestats → `metric`, ratios → `gauge`). The `warn` status is benign: the
element audit's per-element chart-kind heuristic (XY panels expose a secondary
metric element) and three duplicate panels inside the collapsed `Database:`
row (they render on expand).
Comment thread
giorgi-imerlishvili-elastic marked this conversation as resolved.
Outdated

**Interaction**: both controls populate from live ES —
`Instance` → `['.*', 'postgres:5432']`, `Database` → `['.*', 'postgres',
'rigdb', 'template0', 'template1']` — and panels bind the params (`?Instance`
×57, `?Database` ×23), e.g. `max(metrics.pg_replication_lag_seconds{instance=~?Instance})`.

**Visual**: every panel renders real data (Uptime 16.57 min via the postmaster
collector; Query rate 25.5 & Avg runtime 2.45 ms via pg_stat_statements; Total
DB size 62.80 MB via the `_bytes` rename; gauges + xy time-series all correct).
UI polish (2026-09-01): I/O legends are Read/Write; ratio gauges keep chrome
titles; Global KPI strip fills 48 cols; Database section is a hole-free 3+2
KPI grid plus 24+24 graph pairs; Locks by state is a stacked bar with the
legend on the right; Deadlocks / temp files legend by database name instead of
a leftover `deadlocks`/`temp_files` series; Replication
lag spans the full row. Average query runtime skips the incomplete last
`delta/delta` bucket (native PROMQL `LAST` → Kibana duration N/A) and shows
last-non-null seconds-per-call (~2.4 ms on the rig).

**14114 re-validation** (same rig, single input): curated pack fired, **6/6
migrated, 6 Green / 0 Red, render audit PASS (6/6 rendered, 0 errors)** — no
regression, no changes needed.
Comment thread
giorgi-imerlishvili-elastic marked this conversation as resolved.
Outdated

## Discoveries

- The curated Redis rig (`parity-rig/curated/grafana_763_redis_exporter/`) is a
**shared multi-exporter rig** that already runs a real
`prometheuscommunity/postgres-exporter:v0.15.0` + a load generator, ingested
via `redis_scraper.py` into `metrics-postgres.prometheus-default`
(`prometheus_native`). New postgres packs validate here rather than in a
throwaway rig; the scraper honours each exporter's `# TYPE`, so counters land
counter-typed in ES.
- The rig exporter's default collectors do NOT emit
`pg_postmaster_start_time_seconds` or any `pg_stat_statements_*`. Enabling
`--collector.postmaster` + `--collector.stat_statements` (with
`shared_preload_libraries=pg_stat_statements` + `CREATE EXTENSION`) surfaced
`pg_postmaster_start_time_seconds` (gauge), `pg_stat_statements_calls_total`
and `pg_stat_statements_seconds_total` (counters, labelled by `datname`,
`queryid`, `user`).
- `metric_map` targets are emitted **verbatim** (the field-profile prefix is
not prepended), so map targets must include the `metrics.` prefix; non-mapped
gauges are emitted bare offline (`pg_stat_activity_count`, not
`metrics.pg_stat_activity_count`).
45 changes: 44 additions & 1 deletion docs/sources/grafana.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,17 @@ picker; use it when a pinned window (commonly 24h hourly bars) renders empty
in Lens on mixed `metrics-*` even though `_query` returns rows.
`panel.layout_overrides` can also set `title` to rename a section or leaf
panel after translation (Grafana's empty first row becomes Kibana
"Section 1"; a pack can rename it to "Overview"). Grafana 5 singlestat
"Section 1"; a pack can rename it to "Overview"), `section_match` so a
duplicate title in Global vs Database gets independent geometry
(`section_match` is the Grafana row title, the same string `query_overrides`
use, captured before any section rename), `hide_title`
to keep metric/gauge chrome titles visible, `kibana_type_override` /
`xy_mode` to pick the Lens chart (stacked bar for composition-over-time,
line for rates) without replacing the query, and `legend_position` to move an
XY legend (`right` for a long categorical breakdown that does not fit under
the plot). `query_overrides` accept the same `section_match` so a duplicated
Global vs Database title can get different ES|QL (for example Global
deadlocks must not take `?Database`). Grafana 5 singlestat
panels store units on the panel root (`format: bytes` / `s` / `percent`);
those map to Lens bytes, duration, and `%` formats. Helm-flavored community
dashboards (PostgreSQL Database 9628) may also ship a pack `plugin.py` that
Expand All @@ -146,6 +156,39 @@ rewrites Instance from Prometheus `up{job=~"postgres.*"}` to
`label_values(pg_up, instance)` — Elastic prometheus_native scrapes store
exporter health as `pg_up`, not scrape `up` — and drops the unused `$job`
control so native PROMQL panels are not left with an empty Instance param.
The pack also restretches QPS to the Rows height and lays the four remaining
graphs as a 24+24 grid so the short Grafana singlestat does not leave a hole,
and replaces the mixin's `{{__name__}}` connections legend (which GROKs to
`(null)` under native PROMQL) with a per-`(instance, datname)` ES|QL series
composited into one Lens XY breakdown.
The PostgreSQL Exporter (12485) pack targets the same exporter family but was
authored against an older `postgres_exporter` lineage, so its `metric_map`
bridges four names that changed in `prometheuscommunity/postgres-exporter`
v0.15 (`pg_database_size` → `pg_database_size_bytes`, `pg_replication_lag` →
`pg_replication_lag_seconds`, `pg_stat_statements_calls` →
`pg_stat_statements_calls_total`, `pg_stat_statements_total_time_seconds` →
`pg_stat_statements_seconds_total`), and its `metric_kinds` force
`pg_stat_activity_count` / `pg_locks_count` / `pg_stat_database_numbackends` to
`gauge` (the `_count` suffix would otherwise make the offline heuristic
`rate()` a gauge). Its plugin repopulates the `Instance` control from
`label_values(pg_up, instance)` (the source `up{job="postgres-exporter"}` job
filter never matches an Elastic scrape) and anchors the bare
`label_values(datname)` `Database` control on `pg_stat_database_numbackends`,
and the `Interval` Grafana interval variable is dropped rather than emitted as
an inert control. Duplicate Global/Database panel titles are laid out with
`section_match` so the Database header is a hole-free 3+2 KPI grid and the
composition panels (connections by state, locks by mode) render as stacked
bars with the lock-mode legend on the right. Grafana's duplicated `blk_read_time` legend on I/O Read/Write time is
replaced with explicit Read/Write series. Deadlocks and temporary files legend
by `datname` instead of a leftover metric name. Average query runtime uses last-non-null
`rate(seconds_total)/rate(calls_total)` instead of native PROMQL
`LAST(delta/delta)` — the incomplete window-edge bucket is often 0/0, which
Kibana's duration formatter renders as N/A even while Query rate is populated
(Grafana's own singlestat also maps a null current value to "N/A"). `pg_stat_statements` / `pg_postmaster_start_time_seconds`
panels (Query rate, Average query runtime, Uptime) only show data when the
target exporter runs the `stat_statements` + `postmaster` collectors and the
`pg_stat_statements` extension is installed; otherwise they degrade to an
honest field/data gap.

Each pack is registered in `curated_packs/registry.yaml` with a
`gnet_revision` and `dashboard_sha256` — maintainer-verified provenance pins
Expand Down
Loading
Loading