Add mz-release-signoff agent skill

[antiguru]

Motivation

Release verification is a weekly rotation across seven Grafana dashboards, and almost all of its cost is rediscovery rather than judgement. Each verifier works out again which environments run the candidate, when each stack was upgraded, which panels matter, and which apparent signals are artifacts of the upgrade itself. This records that method once so a run starts from the measurement.

Description

An area-agnostic spine plus one reference per dashboard.

The spine covers finding the candidate version and the upgrade boundaries, selecting the environment set, choosing a window that reaches the previous upgrade, batching many metrics into one range query, and the rules for separating a regression from drift. Environment selection differs by stack: production pins the canary environments by name, because they share the released version with customer environments for part of the week and a version filter loses them, while staging can join on the release-candidate version.

The judgement rules are the part worth reviewing. Every upgrade restarts the process, which resets memory downward and flatters the new version, so levels have to be compared at equal age since restart. Zero-downtime upgrades run two generations side by side, so every sum doubles for one bucket and that bucket has to be excluded. Arrangement gauges are bimodal and only their base level is comparable. Fleet CPU on staging moved by a factor of two over three weeks, so a few percent across a boundary carries no information on its own.

Each reference names the metrics with their types and labels, then gives one Hazards and invariants section. Each entry there leads with the property that holds at any fleet size and gives the measurement it came from inline, because splitting the two produced near-verbatim duplication in every reference.

Guarding against metric drift

A reference that names a metric the product no longer exports is worse than no reference, because the sweep reads the empty result as a healthy zero. Two mechanisms cover that, split by when they run.

At run time, scripts/catalog-diff.sh reads doc/user/data/metrics.yml at the release tags on either side of a sign-off boundary. The catalog is checked in, so it is tagged with each release, and the working tree's copy describes main rather than the build under test. Diffing the two tags separates an empty panel that is a finding from a metric that arrived or departed with the release.

At commit time, ci/test/lint-skill-metrics.sh resolves every mz_* name in the skill against the catalog and fails on any that neither resolves nor carries a documented exemption. A rename therefore breaks in the commit that renames rather than during a sign-off weeks later. Matching is not a string comparison: a metric! built with format! is catalogued with its placeholders globbed, and histograms are catalogued as their expanded families, so both forms have to be resolved through patterns.

scripts/ also holds the batched-query builder that step 5 needs and the panel-query slicer that step 4 refers to.

No recorded baselines

An earlier revision carried per-area baseline tables. They are gone, because they duplicated work the method already does and would need syncing with the product forever. The before-window of a run is derived in the same query as the after-window, which makes it the only baseline guaranteed to describe the same fleet, so a figure recorded months earlier is redundant the day it is written. Absolute fleet sums also move whenever an environment is created, deleted, or resized, and a stale reference value is worse than none because it invites a comparison the reader should not make.

What survives is the part that does not decay. Invariants such as a healthy mz_source_progress series contributing exactly 1000 to its rate, so the panel is a series count in disguise; environmentd never swapping; upsert state being structurally absent from staging, so that dashboard can only be verified from the production sandbox; rehydration latency being a last-value staircase with one step per upgrade; and environmentd_needs_update acting as the rollout clock. Alongside those, a coarse dated order-of-magnitude block per area, kept only to catch a mis-scoped selector such as a missing container="clusterd", and explicitly not for comparison.

Dashboard defects found, not fixed here

Recorded in the references so they are not rediscovered:

• Four expressions on environmentd-health render empty and therefore read as healthy. Two divide a histogram by a counter using a bare metric name that does not exist, one reads mz_query_latency_bucket, and one plots container_spec_swap_limit_bytes instead of container_spec_memory_swap_limit_bytes. mz_query_latency was real: added in adapter: Add more metrics #22049 and deleted in Delete stash files #26647 along with the stash.
• Two of the sixteen arms on the persist should be small panel resolve nowhere, so the panel carries two permanent zeros and reads healthier than it is. mz_persist_columnar_validation_count duplicates a series that mz_persist_columnar_op_count{op="validation", result="invalid"} already carries on the same panel, and mz_txn_placeholder_schema_apply has no counterpart. Neither name has ever appeared in this repository.
• The compute dashboard's scratch-disk panels plot kubelet_volume_stats_used_bytes, which resolves to five series in a single namespace in both production and staging us-east-1, none of them a clusterd volume. Those panels are blank in both stacks.
• Two adapter p99 panels are pinned by bucket resolution and cannot move while their counters advance.
• The storage-overview sign-off panel points at a canary environment list in MaterializeInc/cloud that no longer exists. The live list is in MaterializeInc/release.

Open finding for Persist

On staging us-east-1, compare-and-set mismatches per persist command stepped from 0.0061 to 0.0099 in the first bucket after the v26.38.0-rc.1 upgrade, held there for four days, then fell to 0.0085 with rc.3. The ratio is normalized by command volume, the step lands inside the upgrade bucket and is flat on both sides, and production canary shows no equivalent. No failure counter or latency moved, so it is recorded as something to recheck next release rather than as a blocker. This is the one place dated numbers remain in full, because it is a claim about a specific moment rather than a reference value.

Rechecked at the v26.39.0-rc.3 boundary. The elevation is permanent rather than transient: staging us-east-1 sits at 0.0089 and never returned to 0.0061, while the v26.39 boundary itself is flat in that fleet.

Verification

Documentation only. Every claim in the references was measured against the live dashboards rather than estimated, and the skill was then run end to end against the v26.39.0-rc.3 boundary across production canary and staging in us-east-1 and eu-west-1. That run is what surfaced the persist recheck above and the two dashboard defects added to the list.

🤖 Generated with Claude Code

[bosconi]

Looks good! I will get it running automatically.

[antiguru]

Thanks for the review!

[def-]

QA LLM Review (Post Merge)

@antiguru — an automated review of commit a6535c218c found the following potential MEDIUM+ issue(s) after this PR was merged.

1. MEDIUM -- lint_metrics.py stops checking a file after its first fenced code block

.agents/skills/mz-release-signoff/scripts/lint_metrics.py:80

BACKTICKED = re.compile(r"([^]+)") pairs backticks sequentially over the whole file, so a triple-backtick fence flips the parity of every backtick after it: prose becomes "inline code" and real code spans become separators. In the three files that contain a fence (SKILL.md, compute.md, sources-and-sinks.md`) essentially no metric name is checked, so a rename in them passes CI silently. That is the failure the lint exists to prevent, and it is the reason finding 2 slipped through.

Details

Mutation test, one name renamed to a nonexistent one per file:

file	fenced blocks	lint result
persist.md	no	rc=1, offender named
adapter.md	no	rc=1, offender named
reference-dashboards.md	no	(all 19 names checked)
compute.md	yes	rc=0, silent
sources-and-sinks.md	yes	rc=0, silent
SKILL.md	yes	rc=0, silent

Overall the lint resolves 96 of the 166 mz_* names in the skill. compute.md contributes 0 of 26 and sources-and-sinks.md 2 of 40. You can see the desync directly in the token stream for sources-and-sinks.md, where the extracted "code spans" after the first fence are ', ' and ' | counter | Upstream read volume. |\n| ' — the separators between the real code spans, not the spans.

Fix: pull fenced blocks out first and scan them as their own tokens.

FENCED = re.compile(r"^```.*?^```", re.S | re.M)

def referenced():
    """Yield (name, file) for every mz_* token inside backticks in the skill."""
    for path in sorted(SKILL.rglob("*.md")):
        text = path.read_text()
        for token in FENCED.findall(text) + BACKTICKED.findall(FENCED.sub("\n", text)):
            ...

With that change the lint reports exactly three names: mz_storage_upsert_state_rocksdb_autospill_in_use (finding 2), plus mz_metrics_libc_ru_maxrss and mz_query_latency, which the references already document as dead panel arms and which therefore need allowlist entries next to the mz_query_latency_bucket one that is already there. Note that the if "*" in token: continue skip gets coarse when a token is a whole fence, so it is worth applying that test per whitespace-separated word rather than per token.

2. MEDIUM -- Two roster entries name metrics the product does not export, with no "dead" marker

.agents/skills/mz-release-signoff/references/sources-and-sinks.md:59

mz_storage_upsert_state_rocksdb_autospill_in_use and mz_persist_gc_skipped sit in roster tables as live metrics with reading instructions, but neither is exported. A sweep that queries them gets an empty result, which the skill itself names as the most dangerous reading on any dashboard.

Details

mz_storage_upsert_state_rocksdb_autospill_in_use was deleted along with the autospill upsert backend in 18fb1f796d; the surviving mz_storage_upsert_state_* metrics are rehydration_latency, rehydration_total, and rehydration_updates, and autospill appears nowhere in src/ or in doc/user/data/metrics.yml. Its row still explains how to read the percentage panel ("divides sum by count, so it reports the fraction of workers spilling"), so a verifier following the reference concludes no workers are spilling rather than that the concept is gone. The Container Max RSS Memory Usage and Stash (CRDB) Query Latencies entries a few lines away are marked as dead; this one is not, which is what makes it misleading rather than merely stale.

.agents/skills/mz-release-signoff/references/persist.md:35 lists mz_persist_gc_seconds, _started, _finished, _noop, _skipped, _merged_reqs. There is no mz_persist_gc_skipped; the catalog holds _finished, _merged_reqs, _noop, _seconds, _started, _step_seconds. The "already done" case is _noop (src/persist-client/src/internal/metrics.rs:1013), and _skipped exists only as mz_persist_compaction_skipped.

Worth folding into finding 1's fix: bare-suffix continuations such as `_skipped` and `_rehydration_updates` are how most roster rows abbreviate, and they carry no mz_ prefix, so CANDIDATE never sees them and the lint cannot check them even once the fence bug is fixed. Expanding a `mz_foo_bar`, `_baz` sequence against the preceding name's prefix would bring roughly another 40 names under the guard, including this one.