Add mz-release-signoff agent skill (#38355)

### Motivation

Release verification is a weekly rotation across seven Grafana
dashboards, and almost all of its cost is rediscovery rather than
judgement. Each verifier works out again which environments run the
candidate, when each stack was upgraded, which panels matter, and which
apparent signals are artifacts of the upgrade itself. This records that
method once so a run starts from the measurement.

### Description

An area-agnostic spine plus one reference per dashboard.

The spine covers finding the candidate version and the upgrade
boundaries, selecting the environment set, choosing a window that
reaches the previous upgrade, batching many metrics into one range
query, and the rules for separating a regression from drift. Environment
selection differs by stack: production pins the canary environments by
name, because they share the released version with customer environments
for part of the week and a version filter loses them, while staging can
join on the release-candidate version.

The judgement rules are the part worth reviewing. Every upgrade restarts
the process, which resets memory downward and flatters the new version,
so levels have to be compared at equal age since restart. Zero-downtime
upgrades run two generations side by side, so every sum doubles for one
bucket and that bucket has to be excluded. Arrangement gauges are
bimodal and only their base level is comparable. Fleet CPU on staging
moved by a factor of two over three weeks, so a few percent across a
boundary carries no information on its own.

Each reference names the metrics with their types and labels, then gives
one `Hazards and invariants` section. Each entry there leads with the
property that holds at any fleet size and gives the measurement it came
from inline, because splitting the two produced near-verbatim
duplication in every reference.

### Guarding against metric drift

A reference that names a metric the product no longer exports is worse
than no reference, because the sweep reads the empty result as a healthy
zero. Two mechanisms cover that, split by when they run.

At run time, `scripts/catalog-diff.sh` reads `doc/user/data/metrics.yml`
at the release tags on either side of a sign-off boundary. The catalog
is checked in, so it is tagged with each release, and the working tree's
copy describes `main` rather than the build under test. Diffing the two
tags separates an empty panel that is a finding from a metric that
arrived or departed with the release.

At commit time, `ci/test/lint-skill-metrics.sh` resolves every `mz_*`
name in the skill against the catalog and fails on any that neither
resolves nor carries a documented exemption. A rename therefore breaks
in the commit that renames rather than during a sign-off weeks later.
Matching is not a string comparison: a `metric!` built with `format!` is
catalogued with its placeholders globbed, and histograms are catalogued
as their expanded families, so both forms have to be resolved through
patterns.

`scripts/` also holds the batched-query builder that step 5 needs and
the panel-query slicer that step 4 refers to.

### No recorded baselines

An earlier revision carried per-area baseline tables. They are gone,
because they duplicated work the method already does and would need
syncing with the product forever. The before-window of a run is derived
in the same query as the after-window, which makes it the only baseline
guaranteed to describe the same fleet, so a figure recorded months
earlier is redundant the day it is written. Absolute fleet sums also
move whenever an environment is created, deleted, or resized, and a
stale reference value is worse than none because it invites a comparison
the reader should not make.

What survives is the part that does not decay. Invariants such as a
healthy `mz_source_progress` series contributing exactly 1000 to its
rate, so the panel is a series count in disguise; `environmentd` never
swapping; upsert state being structurally absent from staging, so that
dashboard can only be verified from the production sandbox; rehydration
latency being a last-value staircase with one step per upgrade; and
`environmentd_needs_update` acting as the rollout clock. Alongside
those, a coarse dated order-of-magnitude block per area, kept only to
catch a mis-scoped selector such as a missing `container="clusterd"`,
and explicitly not for comparison.

### Verification

Documentation only. Every claim in the references was measured against
the live dashboards rather than estimated, and the skill was then run
end to end against the `v26.39.0-rc.3` boundary across production canary
and staging in us-east-1 and eu-west-1. That run is what surfaced the
persist recheck above and the two dashboard defects added to the list.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: antiguru

.agents/skills/README.md

@@ -20,6 +20,7 @@ explicitly with `$<skill-name>` in Codex and `/<skill-name>` in Claude Code.
 | **mz-pr-review** | Reviewing code changes | Local code review against Materialize standards for tests, style, error messages, architecture, and polish |
 | **mz-debug-ci** | CI failures, red builds, Buildkite issues | Investigates Buildkite failures using `gh` and `bk` CLI tools to identify root causes |
 | **mz-dbt-release** | Cutting a new `dbt-materialize` PyPI release | Bumps the version in `__version__.py` + `setup.py`, dates the `Unreleased` CHANGELOG entry, opens the release PR |
+| **mz-release-signoff** | Verifying a release candidate on the Grafana dashboards | Compares metrics across the release boundary per area and region. Metric references with measured baselines for compute, adapter, sources and sinks, persist, and the release-health and networking dashboards |
 
 ## Performance
 

.agents/skills/mz-release-signoff/SKILL.md

@@ -0,0 +1,98 @@
+# Adapter metric reference
+
+Dashboard `environmentd-health`, UID `mR1Kg1d4z`. All metrics come from `environmentd`, one process per environment, so there is no cluster or replica dimension and no system-cluster split. During a zero-downtime upgrade two `environmentd` pods run at once and every sum doubles for one bucket.
+
+## Variables
+
+This dashboard names its environment variable `$env`, not `$namespace`, and also exposes `$pod`. The release bot sets `var-namespace` and `var-env` on every link for exactly this reason. When reproducing a panel expression, substitute `$env` with the namespace regex.
+
+## Roster
+
+### Durable catalog
+
+| Metric | Type | Notes |
+|---|---|---|
+| `mz_catalog_transaction_commit_latency_seconds_{bucket,count,sum}` | histogram | Catalog write latency. |
+| `mz_catalog_transaction_commits`, `mz_catalog_transactions_started` | counter | Commit volume, very low in steady state. |
+| `mz_catalog_sync_latency_seconds_{bucket,count,sum}` | histogram | Catalog read-side sync latency. |
+| `mz_catalog_syncs` | counter | Sync volume. |
+| `mz_catalog_collection_entries` | gauge, by `collection` | Catalog size. Grows monotonically with catalog contents, so a slow rise is expected and a fall is worth explaining. |
+| `mz_catalog_snapshot_latency_seconds_*`, `mz_catalog_snapshot_seconds_*`, `mz_catalog_snapshots_taken` | histogram, counter | Snapshots are taken at boot, so these are only non-empty around a restart. |
+| `mz_catalog_transact_seconds_*`, `mz_catalog_transact_phase_seconds_*` | histogram | Finer breakdown than the commit latency histogram. |
+| `mz_catalog_allocate_id_seconds_*` | histogram | ID allocation. |
+| `mz_catalog_snapshot_cache`, `mz_catalog_snapshot_consolidations`, `mz_catalog_snapshot_max_entries` | counter, gauge | Snapshot cache behaviour. |
+| `mz_catalog_arc_strong_count`, `mz_catalog_arc_weak_count` | gauge | Catalog handle counts, useful for leak hunting. |
+| `v2_mz_catalog_items` | gauge | Item count, an alternative to collection entries. |
+
+### Adapter and coordinator
+
+| Metric | Type | Notes |
+|---|---|---|
+| `v2_mz_envd_up` | gauge | One per healthy `environmentd`. The cheapest liveness check, and it should equal the environment count. |
+| `mz_start_time_environmentd` | gauge, milliseconds | Startup duration. Divide by 1000. Resets only on restart, so it reports the current generation's boot. |
+| `mz_connection_status` | counter, by `status` and `source` | Upstream connection outcomes. |
+| `mz_determine_timestamp` | counter, by `respond_immediately` and `isolation_level` | Timestamp selection volume. A fall in `respond_immediately` means more queries are waiting on a timestamp. |
+| `mz_time_to_first_row_seconds_bucket` | histogram, by `isolation_level` | The label values are `strict serializable` and `serializable`, with a space. Their distributions differ by roughly a factor of five, so never aggregate across the label. |
+| `mz_row_set_finishing_seconds_{bucket,sum,count}` | histogram | Row-set finishing. The panel title notes the top bucket is 16s. |
+| `mz_linearize_message_seconds_bucket` | histogram, by `immediately_handled` | Read linearization. Sub-millisecond in steady state. |
+| `mz_slow_message_handling_{bucket,sum,count}` | histogram, by `message_kind` | Coordinator message handling. The `_sum` rate is the coordinator's busy time and the best single coordinator-load signal. |
+| `mz_coord_queue_busy_seconds_{bucket,count}` | histogram | Coordinator queue. The `> 1s` panel subtracts the `le="1"` bucket increase from the count increase. |
+| `mz_append_table_duration_seconds_{bucket,sum,count}` | histogram | Table append latency. |
+| `mz_query_total` | counter, by `session_type` and `statement_type` | `session_type` is `user` or `system`. System queries dominate by an order of magnitude. |
+| `mz_active_sessions` | gauge, by `session_type` | Session counts. |
+
+### HTTP and auth
+
+| Metric | Type | Notes |
+|---|---|---|
+| `mz_http_requests_total` | counter, by `path` and `status` | Panels split webhook paths (`/api/webhook.*`) from the rest, because webhook volume swamps everything else. |
+| `mz_http_request_duration_seconds_bucket` | histogram | Same split. |
+| `mz_auth_request_count` | counter, by `path`, `status`, `mz_context_org_name` | Frontegg. Status values are strings such as `401 Unauthorized`, not bare codes. |
+| `mz_auth_request_duration_seconds_bucket` | histogram, by `path` | |
+| `mz_auth_refresh_tasks_active` | gauge | |
+
+### Infrastructure
+
+Container metrics select `container="environmentd"`. CPU percent divides by `container_spec_cpu_quota / container_spec_cpu_period`, and memory percent by `container_spec_memory_limit_bytes`.
+
+`crdb_dedicated_sys_cpu_combined_percent_normalized` and its `_maximum` variant carry no namespace label. They describe the whole regional CockroachDB cluster, so they are shared across all environments in the region and cannot be attributed to the release under test.
+
+## Broken panels, as of 2026-08-19
+
+These render empty and are not evidence of a healthy system. Verified against the metric catalogue in production us-east-1.
+
+* `Avg Transaction Commit Latency` and `Avg Transaction Commit Latency by Env` divide `mz_catalog_transaction_commit_latency_seconds` by `mz_catalog_transaction_commits`. The first name does not exist, because the metric is a histogram exposing only `_bucket`, `_count`, and `_sum`. Use `rate(..._sum) / rate(..._count)` instead.
+* `Avg Sync Latency` and `Avg Sync Latency By Env` have the same defect for `mz_catalog_sync_latency_seconds`.
+* `Stash (CRDB) Query Latencies` reads `mz_query_latency_bucket`, which no longer exists. The catalog moved off the stash, so the panel has no replacement.
+* `Swap Usage (bytes)` plots `container_spec_swap_limit_bytes` as its limit series. The real name is `container_spec_memory_swap_limit_bytes`. The usage series is fine.
+
+## Hazards and invariants
+
+Each entry states a property that holds at any fleet size, followed by the measurement it came from. The property is what survives a release. The measurement is dated, describes whatever fleet existed when it was taken, and is recorded only so the property is not mistaken for a guess.
+
+**Two p99 panels are pinned by bucket resolution.** `p99 Slow Coordinator Messages` reported 0.0001276 s across eight days in staging with five-digit stability, and `Coordinator Table Append Latencies` p99 reported 0.12673 s in production with the same rigidity, both while the underlying counters advanced normally. The quantile is landing inside one wide bucket, so interpolation returns the bucket boundary and the panel cannot move. Read `rate(_sum) / rate(_count)` instead for these two, and treat an implausibly constant quantile anywhere as a bucket artifact rather than as stability.
+
+**`v2_mz_envd_up` should equal the environment count.** It is the cheapest liveness check on the dashboard.
+
+**`environmentd` does not swap.** Swap usage measured exactly zero across the whole window in production canary, so a non-zero reading is itself the finding.
+
+**Catalog collection entries grow monotonically with catalog contents.** A fall needs explaining.
+
+**Catalog snapshot latency is recorded at boot only.** `mz_catalog_snapshot_latency_seconds` records once at boot, so an average over a steady-state window divides by zero and reads NaN, while an upgrade bucket reads non-NaN. Neither is a finding.
+
+**Serializable and strict serializable time-to-first-row are not comparable.** In production canary the p99 ran about 1.2 s for strict serializable and about 6.5 s for serializable over the same window, roughly a factor of five. Never aggregate across the `isolation_level` label, and compare each against its own history.
+
+**System queries outnumber user queries by an order of magnitude.** The gap is far larger in staging, where most environments are idle apart from introspection, so staging user-query numbers are not a workload signal.
+
+**Coordinator busy time is the best single coordinator-load signal.** It is the `_sum` rate of `mz_slow_message_handling`. The message rate alone hides how expensive each message was.
+
+**`crdb_dedicated_*` metrics describe the whole regional CockroachDB cluster.** They cannot be attributed to the release under test.
+
+## Order of magnitude
+
+Recorded 2026-08 for scope-checking only. Derive the real baseline from your own before-window.
+
+* `environmentd` CPU is a fraction of a core per environment, and working set is low single-digit GB per environment.
+* Coordinator message rate is thousands per second per environment, and coordinator busy time a few hundredths of a second per second.
+* Linearize and coordinator-message quantiles are sub-millisecond. Time to first row is order one second for strict serializable and several seconds for serializable.
+* Catalog commits are rare, well under one per second, so their latency average is noisy by construction.

.agents/skills/mz-release-signoff/references/adapter.md

@@ -0,0 +1,100 @@
+# Compute metric reference
+
+Dashboard `compute-overview`, UID `f248986d-81c6-42a6-817b-00cd7759d808`. The rows that matter for sign-off are `By process` and `By cluster`, each inspected twice, once across all clusters and once restricted to system clusters. The `By replica`, `By worker`, and `By collection` rows require a narrowed variable and belong to the drill-down in step 7 of the skill, not to the sweep.
+
+## Selectors
+
+Process-level panels read cAdvisor and kube-state metrics and select on the pod name:
+
+```
+pod=~".*cluster-<cluster_id>-replica-<replica_id>-.+", container="clusterd"
+```
+
+Cluster-level panels split into two families with different label names for the same concepts.
+
+| Family | Cluster label | Replica label |
+|---|---|---|
+| Controller, protocol, peeks, and `v2_mz_*` replica metrics | `instance_id` | `replica_id` |
+| `mz_arrangement_*`, `mz_dataflow_replica_*`, `mz_subscribe_*`, `mz_cluster_*` | `cluster_environmentd_materialize_cloud_cluster_id` | `cluster_environmentd_materialize_cloud_replica_id` |
+
+System clusters are `s` followed by digits in either family, and `.*cluster-s[0-9]+-replica-.*` in the pod name.
+
+## Roster
+
+### By process
+
+| Metric | Type | Notes |
+|---|---|---|
+| `container_cpu_usage_seconds_total` | counter | Needs `cpu="total"`. Divide by `container_spec_cpu_quota` for percent of limit. |
+| `container_memory_working_set_bytes` | gauge | Resident plus active file. The primary memory signal. |
+| `container_memory_rss` | gauge | Tracks working set closely; a divergence between them is itself a finding. |
+| `container_memory_swap` | gauge | Limit is `container_spec_memory_swap_limit_bytes`. Grows with process age on swap-enabled nodes. |
+| `mz_memory_limiter_memory_usage_bytes` | gauge | Memory plus swap, the quantity the limiter enforces. Divide by `mz_memory_limiter_memory_limit_bytes`. |
+| `mz_metrics_libc_ru_maxrss_bytes` | gauge, monotone per process | Peak resident set. Resets to zero on restart, so it measures the current generation only. |
+| `kubelet_volume_stats_used_bytes` | gauge | Scratch disk. Limit is `kubelet_volume_stats_capacity_bytes`. |
+| `container_start_time_seconds` | gauge | Source of the uptime panel, and the cheapest way to establish restart age. |
+| `kube_pod_container_status_restarts_total` | counter | Resets when a pod is replaced, which every upgrade does. |
+| `kube_pod_container_status_last_terminated_exitcode` | gauge | `== 137` is an OOM kill. Counting matching series beats summing restart increases. |
+| `container_network_receive_bytes_total`, `container_network_transmit_bytes_total` | counter | Per pod, with no `container` label. |
+| `mz_metrics_libc_ru_minflt_total`, `mz_metrics_libc_ru_majflt_total` | counter | Major faults spike hard during rehydration and while paging in from swap. |
+| `mz_metrics_libc_ru_utime_seconds_total`, `mz_metrics_libc_ru_stime_seconds_total` | counter | Process-reported CPU. Cross-checks the cAdvisor CPU series, and the two should move together. |
+
+### By cluster
+
+| Metric | Type | Notes |
+|---|---|---|
+| `mz_compute_controller_replica_count`, `_collection_count`, `_peek_count`, `_subscribe_count` | gauge | Controller-side inventory. Collection count tracks catalog growth, so a slow rise is expected. |
+| `mz_compute_controller_command_queue_size` | gauge | Sustained depth means the controller is not draining. |
+| `mz_compute_controller_response_send_count`, `_recv_count` | counter | Response queue depth, as the difference of the two. There is no depth gauge, because the response channel is an `instrumented_unbounded_channel` (`src/ore/src/channel.rs`), which takes a send and a receive counter and exports nothing else. Contrast `command_queue_size`, a real gauge that the command path increments and decrements directly. |
+| `mz_compute_commands_total`, `mz_compute_responses_total` | counter | Protocol volume. Doubles for one bucket during a zero-downtime upgrade. |
+| `mz_compute_command_message_bytes_total`, `mz_compute_response_message_bytes_total` | counter | Protocol bytes. Worth checking when a change touches command encoding. |
+| `mz_compute_controller_history_command_count`, `_history_dataflow_count` | gauge | Controller-side command history, which should be reduced and not grow without bound. |
+| `mz_compute_replica_history_command_count`, `_history_dataflow_count` | gauge | Replica-side equivalent. |
+| `mz_compute_peeks_total` | counter | Label `result`. Successes are `rows` and `rows_stashed`; anything else is an error or a cancellation. |
+| `mz_compute_peek_duration_seconds_bucket` | histogram | Quantiles need `sum by (le)` after any namespace join. |
+| `v2_mz_dataflow_elapsed_seconds_total` | counter | Compute time. Strongly workload-shaped, and often the noisiest series on the dashboard. |
+| `mz_arrangement_maintenance_seconds_total` | counter | Merge and compaction work. Uses the `cluster_environmentd_*` labels. |
+| `v2_mz_compute_replica_park_duration_seconds_total` | counter | Time parked, so a fall here alongside a CPU rise localizes new work to the dataflow loop. |
+| `v2_mz_arrangement_count`, `_record_count`, `_batch_count`, `_size_bytes` | gauge | Bimodal, see hazards. |
+| `v2_mz_hint_group_size_potential_savings_bytes` | gauge | Missing group-size hints. Informational, and not release-sensitive. |
+| `v2_mz_orphan_dataflow_count` | gauge | Non-zero means a dataflow outlived its collection, which is a genuine bug signal. |
+| `v2_mz_dataflow_error_count` | gauge | Erroring dataflows. Some staging environments carry a permanent floor. |
+| `mz_cluster_handle_command_duration_seconds_bucket` | histogram | Replica-side command handling latency. |
+| `mz_dataflow_replica_expiration_timestamp_seconds`, `_expiration_remaining_seconds` | gauge | Replica expiration. Panels filter `> 0` and `!= 0` because the metric is exported as zero when unset. |
+| `mz_subscribe_snapshots_skipped_total` | counter | Subscribe snapshot optimization hit rate. The panel appends `> 0` to hide inactive replicas. |
+
+## Hazards and invariants
+
+Each entry states a property that holds at any fleet size, followed by the measurement it came from. The property is what survives a release. The measurement is dated, describes whatever fleet existed when it was taken, and is recorded only so the property is not mistaken for a guess.
+
+**OOM kills, dataflow errors, and orphan dataflows are absent rather than zero when healthy.** Production canary returned no series at all for `v2_mz_dataflow_error_count` and `v2_mz_orphan_dataflow_count` across a full week. That is the healthy case, and it is indistinguishable from a renamed metric unless the metric is confirmed to exist elsewhere.
+
+**`v2_mz_orphan_dataflow_count` above zero is always a bug, never a load effect.**
+
+**Working set falls at every upgrade.** In production canary us-east-1 the sum fell from about 290 GB to about 236 GB at the v26.37.0 rollout with no change in the code that mattered, purely because arrangements were rebuilt fresh. Judge memory by the slope within a release, not the step across one.
+
+**Arrangement gauges are bimodal.** `v2_mz_arrangement_record_count` and `v2_mz_arrangement_size_bytes` swing by a factor of three to ten as periodic dataflows rebuild. In August 2026 staging us-east-1 alternated between 0.72e9 and 2.86e9 records with no release involvement. Compare low state against low state, because spike heights are not comparable.
+
+**Peak resident set and swap are restart-sensitive.** Both reset or decay at an upgrade, so a level drop across the boundary is the restart and not the release. Peak resident set therefore describes the current generation only.
+
+**Arrangement maintenance ramps after a restart.** Measured at 0.020 s/s one day after an upgrade and 0.030 s/s three days later on the same release, so an apparent increase across a boundary can be nothing more than a difference in age.
+
+**Compute time and park time are complementary.** A CPU rise with a park fall localizes new work to the dataflow loop, while a CPU rise with park flat points outside it.
+
+**Response queue depth is a difference of two counters, so it only holds while neither has reset.** Both reset when a pod is replaced, which every upgrade does, and a scrape that catches one reset and not the other yields a wild value. Staging us-east-1 read -761 in one bucket and +5371 in another over a window where every other bucket sat within one of zero. Read a single implausible bucket as a reset artifact, and judge the panel by whether it returns to zero rather than by any one sample.
+
+## Known noise classes
+
+Some environments are unhealthy independently of the release, and their contribution is constant across the boundary rather than absent. Check for these first, because they can dominate a fleet aggregate.
+
+* Staging carries persistently crashlooping replicas. Their restart rate is high and flat, and flat means not release-related.
+* Some staging environments carry permanently erroring dataflows, likewise flat.
+* Where such an environment masks everything else, exclude it with the dashboard variables, as the panel instructions suggest.
+
+## Order of magnitude
+
+Recorded 2026-08 for scope-checking a query, not for comparison. If a result sits an order of magnitude away from these, suspect a mis-scoped selector rather than a regression. Derive the actual baseline from the before-window of your own run.
+
+* Production canary, two environments: a few cores of clusterd CPU, hundreds of GB of working set summed, single-digit peeks per second.
+* Staging, about fifteen environments: a few cores, low hundreds of GB, tens of peeks per second.
+* System clusters account for most of staging's dataflow time and roughly a quarter of production canary's CPU.