feat(grafana): MySQL 7362 and PostgreSQL 9628/14114 curated packs

docs/command-contract.md

@@ -871,7 +871,8 @@ customization:
   optional embedded `query.metric_map` (overridden by `--metric-map-file` on
   duplicate keys). When a dashboard carries a `gnetId` that matches a bundled
   curated pack (e.g. Redis 763 / 11835 / 14091, Redis Enterprise 18405, Redis
-  Cloud 18406, Node Exporter Full 1860), the
+  Cloud 18406, Node Exporter Full 1860, MySQL Overview 7362, PostgreSQL
+  Database 9628, PostgreSQL Exporter Quickstart 14114), the
   pack is merged in automatically beneath the user `--rules-file` so the user
   always wins on collision. Pass `--no-curated-packs` to skip all curated packs
   and use only the base rule pack.
@@ -907,7 +908,9 @@ Two further gates still push panels to ES|QL even when the target supports
    limits can still degrade individual panels to ES|QL even on Kibana 9.5+.
 2. **Curated pack / construct limits** — pack overrides and unsupported PromQL
    shapes can still emit ES|QL. Pass `--no-curated-packs` to isolate the core
-   translator.
+   translator. Same-metric Grafana fallbacks (`rate(M[$interval]) or irate(M[5m])`)
+   collapse to the left operand before the native path, matching ES|QL
+   translation. True set-union `or` / `and` / `unless` still degrades.
 
 Construct-level unsupported cases can still degrade or require manual review.
 Datadog accepts `--translation-mode` for CLI parity, but it is a no-op because

docs/sources/grafana.md

@@ -122,7 +122,27 @@ field-caps discovery (`--es-url`) to resolve the metric's actual field name;
 without it they no-op rather than guess. Metrics listed in the pack's
 `live_optional_metrics` that field-caps proved absent are stripped from the
 hand-written override so the rest of the panel can still render; those
-omissions are not reported as pack gaps.
+omissions are not reported as pack gaps. `kibana_type_override` forces the
+Lens chart type when the curated query shape does not match the Grafana panel
+(for example a stacked CPU graph emitted as overlay lines). `drop_time_from`
+strips Grafana's panel `timeFrom` so the override follows the dashboard time
+picker; use it when a pinned window (commonly 24h hourly bars) renders empty
+in Lens on mixed `metrics-*` even though `_query` returns rows.
+`panel.layout_overrides` can also set `title` to rename a section or leaf
+panel after translation (Grafana's empty first row becomes Kibana
+"Section 1"; a pack can rename it to "Overview"). Grafana 5 singlestat
+panels store units on the panel root (`format: bytes` / `s` / `percent`);
+those map to Lens bytes, duration, and `%` formats. Helm-flavored community
+dashboards (PostgreSQL Database 9628) may also ship a pack `plugin.py` that
+rewrites `query_result()` Instance variables to `label_values()` so Kibana
+still gets a populate query after the unused `release` / `namespace` cascade
+is dropped. Native PROMQL also strips `ignored_labels` matchers (so a Helm
+`release` filter cannot bind a kernel `release` field from mixed `metrics-*`
+and empty the panel). The PostgreSQL Exporter Quickstart (14114) plugin
+rewrites Instance from Prometheus `up{job=~"postgres.*"}` to
+`label_values(pg_up, instance)` — Elastic prometheus_native scrapes store
+exporter health as `pg_up`, not scrape `up` — and drops the unused `$job`
+control so native PROMQL panels are not left with an empty Instance param.
 
 The console pipeline is **5 stages**, not 7: `[1/5] Extracting dashboards`,
 `[2/5] Translating dashboards`, `[3/5] Verification-packet ES|QL validation`,

observability_migration/adapters/source/datadog/generate.py

@@ -971,12 +971,11 @@ def _build_esql_panel(
                 for field in keep_fields
             ]
         else:
-            if widget.widget_type == "change" and "value" not in metrics:
+            if widget.widget_type == "change":
                 # A change widget ranks by the computed delta `value` (the SORT
-                # key), which STATS-based inference misses because it is an EVAL
-                # alias. Surface it as the leading metric so the ranked delta —
-                # the whole point of the widget — is actually displayed.
-                metrics = ["value", *metrics]
+                # key). Keep it as the leading metric even when shape inference
+                # already includes the EVAL alias among the STATS columns.
+                metrics = ["value", *[m for m in metrics if m != "value"]]
             if metrics:
                 esql_block["metrics"] = [
                     _metric_config(widget, result, m)

observability_migration/adapters/source/grafana/curated_packs/grafana_14114_postgres_exporter_quickstart/fidelity_manifest.yaml

@@ -0,0 +1,52 @@
+# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one or more contributor license agreements.
+# SPDX-License-Identifier: Elastic-2.0
+#
+# Fidelity manifest — Grafana dashboard 14114 (PostgreSQL Exporter Quickstart)
+# https://grafana.com/grafana/dashboards/14114-postgres-overview/
+#
+# PERFECT = same information as Grafana on postgres_exporter.
+# APPROXIMATE = documented delta (lifetime cache-hit ratio, includeAll All).
+
+schema_version: 1
+gnet_id: 14114
+gnet_revision: 1
+dashboard_title: "PostgreSQL Exporter Quickstart and Dashboard"
+maintainer: "Grafana Labs"
+
+panels:
+  - title: "Rows"
+    fidelity: PERFECT
+    notes: "irate() of tup_fetched/returned/inserted/updated/deleted; ES|QL fusion of five targets."
+  - title: "QPS"
+    fidelity: PERFECT
+    notes: "Native PROMQL sum(irate(commit)) + sum(irate(rollback))."
+  - title: "Buffers"
+    fidelity: PERFECT
+    notes: >
+      Source names pg_stat_bgwriter_buffers_* without OpenMetrics _total.
+      Pack metric_map (and live suffix alias) retargets to *_total on
+      postgres_exporter v0.15. ES|QL IRATE of the five series.
+  - title: "Conflicts/Deadlocks"
+    fidelity: PERFECT
+    notes: "rate() of deadlocks and conflicts counters."
+  - title: "Cache hit ratio"
+    fidelity: APPROXIMATE
+    notes: "Grafana divides lifetime blks_hit / (blks_read + blks_hit) without rate(). Native PROMQL preserves that lifetime ratio."
+  - title: "Number of active connections"
+    fidelity: APPROXIMATE
+    notes: >
+      Native PROMQL of pg_stat_database_numbackends returns the gauge series.
+      Grafana legend {{__name__}} GROKs native PROMQL _timeseries, which does
+      not carry __name__, so Kibana shows "(null)" in the legend. The line is
+      the connection count.
+
+summary:
+  total_panels: 6
+  perfect: 4
+  approximate: 2
+  not_feasible: 0
+  overall_fidelity: HIGH
+  known_gaps:
+    - "Grafana $job is unused in panel queries; pack drops the Kibana control"
+    - "Instance populate query rewritten from Prometheus up to pg_up"
+    - "instance/db includeAll All is a single-select .* token in Kibana"

observability_migration/adapters/source/grafana/curated_packs/grafana_14114_postgres_exporter_quickstart/pack.yaml

@@ -0,0 +1,88 @@
+# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one or more contributor license agreements.
+# SPDX-License-Identifier: Elastic-2.0
+#
+# Curated pack — Grafana dashboard 14114
+# https://grafana.com/grafana/dashboards/14114-postgres-overview/
+#
+# Source: Grafana Labs "PostgreSQL Exporter Quickstart and Dashboard"
+#         (postgres_exporter mixin / Prometheus scrape)
+# Revision 1. Panels: 6 (5 graph + 1 singlestat).
+#
+# Engine vs pack split (do not duplicate engine work here):
+#   Engine: OpenMetrics ``_total`` suffix alias against live field-caps
+#           (Buffers without ``--es-url`` still needs the pack map below);
+#           native PROMQL for QPS / cache-hit / numbackends; ES|QL fusion
+#           for multi-target Rows / Buffers / Conflicts.
+#   Pack:   mixin Instance is ``label_values(up{job=~"postgres.*"}, instance)``
+#           but Elastic prometheus_native scrapes store postgres health as
+#           ``pg_up``, not Prometheus ``up`` (that series is redis-only in
+#           mixed ``metrics-*``). plugin.py rewrites Instance to
+#           ``label_values(pg_up, instance)`` and drops the unused ``job``
+#           control (no panel binds ``$job``). ``metric_map`` renames the
+#           five pre-OpenMetrics bgwriter counters so offline ES|QL matches
+#           postgres_exporter v0.15.
+
+query:
+  metrics_dataset_filter: "prometheus"
+
+  label_rewrites:
+    instance: labels.instance
+    job: labels.job
+    datname: labels.datname
+    db: labels.datname
+
+  metric_kinds:
+    pg_stat_database_tup_fetched: counter
+    pg_stat_database_tup_returned: counter
+    pg_stat_database_tup_inserted: counter
+    pg_stat_database_tup_updated: counter
+    pg_stat_database_tup_deleted: counter
+    pg_stat_database_xact_commit: counter
+    pg_stat_database_xact_rollback: counter
+    pg_stat_database_deadlocks: counter
+    pg_stat_database_conflicts: counter
+    pg_stat_database_blks_hit: counter
+    pg_stat_database_blks_read: counter
+    pg_stat_bgwriter_buffers_alloc: counter
+    pg_stat_bgwriter_buffers_backend: counter
+    pg_stat_bgwriter_buffers_backend_fsync: counter
+    pg_stat_bgwriter_buffers_clean: counter
+    pg_stat_bgwriter_buffers_checkpoint: counter
+    pg_stat_bgwriter_buffers_alloc_total: counter
+    pg_stat_bgwriter_buffers_backend_total: counter
+    pg_stat_bgwriter_buffers_backend_fsync_total: counter
+    pg_stat_bgwriter_buffers_clean_total: counter
+    pg_stat_bgwriter_buffers_checkpoint_total: counter
+    pg_stat_database_numbackends: gauge
+    pg_up: gauge
+
+  # metric_map targets are emitted VERBATIM (the field profile prefix is NOT
+  # prepended). See docs/command-contract.md, "metric_map targets are verbatim".
+  metric_map:
+    pg_stat_bgwriter_buffers_alloc: metrics.pg_stat_bgwriter_buffers_alloc_total
+    pg_stat_bgwriter_buffers_backend: metrics.pg_stat_bgwriter_buffers_backend_total
+    pg_stat_bgwriter_buffers_backend_fsync: metrics.pg_stat_bgwriter_buffers_backend_fsync_total
+    pg_stat_bgwriter_buffers_clean: metrics.pg_stat_bgwriter_buffers_clean_total
+    pg_stat_bgwriter_buffers_checkpoint: metrics.pg_stat_bgwriter_buffers_checkpoint_total
+
+  label_candidates:
+    instance:
+      - labels.instance
+      - service.instance.id
+      - host.name
+    job:
+      - labels.job
+      - service.name
+    datname:
+      - labels.datname
+      - datname
+    db:
+      - labels.datname
+      - datname
+
+controls:
+  field_overrides:
+    instance: labels.instance
+    db: labels.datname
+    datname: labels.datname
+    job: labels.job

observability_migration/adapters/source/grafana/curated_packs/grafana_14114_postgres_exporter_quickstart/plugin.py

@@ -0,0 +1,36 @@
+# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one or more contributor license agreements.
+# SPDX-License-Identifier: Elastic-2.0
+
+"""Grafana 14114 (PostgreSQL Exporter Quickstart) curated pack plugin.
+
+Revision 1's Instance variable is ``label_values(up{job=~"postgres.*"}, instance)``.
+Prometheus ``up`` is not stored for postgres_exporter on typical Elastic
+prometheus_native scrapes (``pg_up`` is). The unused ``job`` template never
+appears in panel PromQL, so emitting it only adds an incompatible default
+(``postgres`` vs ``postgres_exporter``).
+"""
+
+
+_PACK_NAME = "grafana_14114_postgres_exporter_quickstart"
+
+
+def register(api):
+    @api["variable_translators"].register("grafana_14114_pg_up_instance", priority=5)
+    def rewrite_instance_and_drop_job(context):
+        pack = getattr(context, "rule_pack", None)
+        if getattr(pack, "_curated_pack_name", "") != _PACK_NAME:
+            return None
+        variable = context.variable or {}
+        name = str(variable.get("name") or "")
+        query_text = context.query_text or str(variable.get("query") or "")
+        compact = query_text.replace(" ", "").lower()
+        if name == "job":
+            context.handled = True
+            return f"skipped unused job variable {name}"
+        if name == "instance" and "label_values(up" in compact:
+            rewritten = "label_values(pg_up, instance)"
+            context.query_text = rewritten
+            context.variable = dict(variable)
+            context.variable["query"] = rewritten
+            return None
+        return None

observability_migration/adapters/source/grafana/curated_packs/grafana_7362_mysql_overview/fidelity_manifest.yaml

@@ -0,0 +1,123 @@
+# Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one or more contributor license agreements.
+# SPDX-License-Identifier: Elastic-2.0
+#
+# Fidelity manifest — Grafana dashboard 7362 (MySQL Overview)
+# https://grafana.com/grafana/dashboards/7362-mysql-overview/
+#
+# PERFECT = same information as Grafana on mysqld_exporter + node_exporter.
+# APPROXIMATE = documented delta (MySQL 8 replacements, $interval, host split).
+
+schema_version: 1
+gnet_id: 7362
+gnet_revision: 5
+dashboard_title: "MySQL Overview"
+
+panels:
+  - title: "MySQL Uptime"
+    fidelity: PERFECT
+    notes: "Native PROMQL LAST of mysql_global_status_uptime. Grafana 5 singlestat format s maps to a duration tile."
+  - title: "Current QPS"
+    fidelity: APPROXIMATE
+    notes: "rate() or irate() prefers rate; Grafana $interval is inlined to TBUCKET. Requires untyped queries pinned as counter."
+  - title: "InnoDB Buffer Pool Size"
+    fidelity: PERFECT
+    notes: "Native PROMQL LAST of innodb_buffer_pool_size. Grafana 5 singlestat format bytes maps to a bytes tile."
+  - title: "Buffer Pool Size of Total RAM"
+    fidelity: APPROXIMATE
+    notes: "Cross-instance ratio when mysqld and node_exporter use different instance labels. TS LAST_OVER_TIME of both gauges over the dashboard picker (FROM over mixed metrics-* timed out as N/A in Lens). Grafana 5 singlestat format percent maps to a % suffix."
+  - title: "MySQL Connections"
+    fidelity: PERFECT
+    notes: ""
+  - title: "MySQL Client Thread Activity"
+    fidelity: APPROXIMATE
+    notes: "max_over_time($interval) or 5m prefers the left window."
+  - title: "MySQL Questions"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair; $interval inlined."
+  - title: "MySQL Thread Cache"
+    fidelity: APPROXIMATE
+    notes: "threads_created rate requires counter typing."
+  - title: "MySQL Temporary Objects"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair; $interval inlined."
+  - title: "MySQL Select Types"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair; $interval inlined."
+  - title: "MySQL Sorts"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair; $interval inlined."
+  - title: "MySQL Slow Queries"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair; $interval inlined."
+  - title: "MySQL Aborted Connections"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair; $interval inlined."
+  - title: "MySQL Table Locks"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair; $interval inlined."
+  - title: "MySQL Network Traffic"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair; $interval inlined."
+  - title: "MySQL Network Usage Hourly"
+    fidelity: APPROXIMATE
+    notes: "Grafana pins a 24h panel window with 1h increase() bars. Kibana follows the dashboard time picker (mixed metrics-* Lens 24h override rendered empty despite rows). RATE over TBUCKET(20)."
+  - title: "MySQL Internal Memory Overview"
+    fidelity: APPROXIMATE
+    notes: "Query cache / additional mem pool / TokuDB series are live_optional and omitted on MySQL 8."
+  - title: "Top Command Counters"
+    fidelity: APPROXIMATE
+    notes: "topk(rate or irate) prefers rate; $interval inlined."
+  - title: "Top Command Counters Hourly"
+    fidelity: APPROXIMATE
+    notes: "Grafana 24h/1h increase() bars; Kibana follows the dashboard picker with RATE by labels.command. No hard top-k cut."
+  - title: "MySQL Handlers"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair; $interval inlined."
+  - title: "MySQL Transaction Handlers"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair; $interval inlined."
+  - title: "Process States"
+    fidelity: PERFECT
+    notes: "Pack maps mysql_info_schema_threads → mysql_info_schema_processlist_threads and groups by labels.state."
+  - title: "Top Process States Hourly"
+    fidelity: APPROXIMATE
+    notes: "Grafana 24h topk(5, avg_over_time[1h]); Kibana follows the dashboard picker with AVG_OVER_TIME by state and no hard top-5 cut."
+  - title: "MySQL Query Cache Memory"
+    fidelity: APPROXIMATE
+    notes: "Query cache removed in MySQL 8. Kibana panel title is InnoDB Buffer Pool Pages; shows InnoDB buffer-pool data/free/dirty pages instead of an empty cache gauge."
+  - title: "MySQL Query Cache Activity"
+    fidelity: APPROXIMATE
+    notes: "Query cache removed in MySQL 8. Kibana panel title is InnoDB Buffer Pool Activity; shows InnoDB buffer-pool hit ratio plus data read/write rates."
+  - title: "MySQL File Openings"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair; $interval inlined."
+  - title: "MySQL Open Files"
+    fidelity: PERFECT
+    notes: ""
+  - title: "MySQL Table Open Cache Status"
+    fidelity: APPROXIMATE
+    notes: "rate/irate pair plus hit-ratio; $interval inlined."
+  - title: "MySQL Open Tables"
+    fidelity: PERFECT
+    notes: ""
+  - title: "MySQL Table Definition Cache"
+    fidelity: APPROXIMATE
+    notes: "opened_table_definitions rate requires counter typing."
+  - title: "I/O Activity"
+    fidelity: APPROXIMATE
+    notes: "Host filter dropped: mysql_up instance does not match node_exporter instance. Shows node_exporter pgpgin/pgpgout."
+  - title: "Memory Distribution"
+    fidelity: APPROXIMATE
+    notes: "Host filter dropped for the same instance split. Uses node_exporter 0.16 *_bytes names."
+  - title: "CPU Usage / Load"
+    fidelity: APPROXIMATE
+    notes: "Grafana per-mode stacked clamp_max(rate or irate)*100 plus Load on a second axis is not_feasible. Kibana shows overall non-idle busy % and load1 as overlay lines, without the mysql host filter. Load 1m is on the right axis without a % suffix."
+  - title: "Disk Latency"
+    fidelity: APPROXIMATE
+    notes: "Linux read/write latency ratio without AWS RDS fallback; host filter dropped."
+  - title: "Network Traffic"
+    fidelity: APPROXIMATE
+    notes: "node_exporter receive/transmit rates without RDS OSMetrics fallback; host filter dropped. Distinct from MySQL Network Traffic."
+  - title: "Swap Activity"
+    fidelity: APPROXIMATE
+    notes: "Host filter dropped; pswpin/pswpout * 4096."