diff --git a/.agents/skills/README.md b/.agents/skills/README.md index 988e7cb1f469e..5943f4b662402 100644 --- a/.agents/skills/README.md +++ b/.agents/skills/README.md @@ -20,6 +20,7 @@ explicitly with `$` in Codex and `/` in Claude Code. | **mz-pr-review** | Reviewing code changes | Local code review against Materialize standards for tests, style, error messages, architecture, and polish | | **mz-debug-ci** | CI failures, red builds, Buildkite issues | Investigates Buildkite failures using `gh` and `bk` CLI tools to identify root causes | | **mz-dbt-release** | Cutting a new `dbt-materialize` PyPI release | Bumps the version in `__version__.py` + `setup.py`, dates the `Unreleased` CHANGELOG entry, opens the release PR | +| **mz-release-signoff** | Verifying a release candidate on the Grafana dashboards | Compares metrics across the release boundary per area and region. Metric references with measured baselines for compute, adapter, sources and sinks, persist, and the release-health and networking dashboards | ## Performance diff --git a/.agents/skills/mz-release-signoff/SKILL.md b/.agents/skills/mz-release-signoff/SKILL.md new file mode 100644 index 0000000000000..b8aec0715ca49 --- /dev/null +++ b/.agents/skills/mz-release-signoff/SKILL.md @@ -0,0 +1,293 @@ +--- +name: mz-release-signoff +description: > + Verify a release candidate on the Grafana dashboards and sign off in #release. + Trigger: "verify the release", "sign off on the release", "release + verification", "check the canary", "does this release look healthy", a pasted + grafana.dev.materialize.com dashboard link, or a release-bot "please verify + the vX.Y.Z" request. Also when asked to compare metrics across a release + boundary. +argument-hint: +--- + +Verify a release candidate by comparing metrics across the release boundary, then report the outcome. + +The release bot posts a request in the `#release` Slack channel naming the version and linking one dashboard per area. Each area is verified by that area's team. This skill covers the shared method plus per-area metric references. + +## Prerequisites + +This workflow needs the Grafana MCP server against `grafana.dev.materialize.com`. Verify it answers before starting: + +``` +mcp__grafana__list_datasources with type "prometheus" +``` + +Reading the bot request also needs the Slack MCP server, but a user-pasted dashboard link or version string is enough to proceed without it. + +## Read the panel instructions, then apply this skill + +Two dashboards carry their own `Signing off on Releases` text panel, and they say different things. Read both at the start of every run, because they are maintained separately from this skill: + +``` +mcp__grafana__get_dashboard_property + uid: f248986d-81c6-42a6-817b-00cd7759d808 # compute + jsonPath: $.panels[?(@.type=="text")].options.content + +mcp__grafana__get_dashboard_property + uid: e6dc7745-7d35-4968-a23d-689883a984bb # storage-overview + jsonPath: $.panels[?(@.type=="text")].options.content +``` + +If a panel and this skill disagree, the panel wins on *what* to inspect and this skill wins on *how* to measure it. Report the disagreement so one of them gets fixed. + +## Every dashboard names its variables differently + +This is the single most common source of a wasted query. There is no shared convention. + +| Dashboard | Environment variable | Others | +|---|---|---| +| `compute-overview` | `namespace` | `version`, `organization`, `cluster_id`, `replica_id`, `worker_id`, `collection_id` | +| `storage-overview` | `env` | `cluster`, `replica`, `object_id`, `pod` | +| `storage-upsert-sources` | `namespace` | `pod`, `source` | +| `persist` | `env` | `pod` | +| `environmentd-health` | `env` | `pod` | +| `release-health` | `organization`, an organization id, not a namespace | `mz_cluster`, `version` | +| `networking` | `namespace` | `pod`, `tenant` | + +The release bot sets both `var-namespace` and `var-env` on every link for this reason. + +## Step 1: Establish the version and the boundaries + +Every judgement in this workflow is a before/after comparison, so the upgrade times must be pinned first. Ask Prometheus which version each environment runs over the past 10 days: + +```promql +count by (mz_version) (group by (namespace, mz_version) (v2_mz_compute_cluster_status{mz_version!~".*-dev.*"})) +``` + +Run it as a range query with a 6h step. The `-dev` exclusion drops personal development environments, which run arbitrary old builds and only add noise. + +Two facts come out of this. The release under test is the highest `vX.Y.0-rc.N` present, and the boundary per stack is the step where the previous version disappears and it appears. Expect two boundaries for the same release, because a later `rc` usually supersedes an earlier one, and both are the new release for sign-off purposes. + +A second, sharper boundary marker is the process count doubling. Zero-downtime upgrades run the old and new generation side by side, so any per-pod series count roughly doubles for one bucket: + +```promql +count(avg_over_time(container_memory_working_set_bytes{pod=~".*cluster-.*-replica-.*", container="clusterd"}[6h])) +``` + +Use those doubling buckets as the boundary, and exclude them from both the before and after samples. They contain two full fleets and will corrupt any sum. + +## Step 2: Choose the namespace set + +The dashboards' `version` variable does not filter panels directly. It narrows the `organization` variable, which narrows `namespace`, and the panels filter on `namespace`. Setting `version` to the new release therefore means "the environments that run the new release now", and the panels then show those same environments on both sides of the boundary. Reproduce that selection explicitly rather than relying on the variable chain. + +Production and staging need different selectors. In production only the canary environments run a release candidate, and they share the plain released version with every customer environment for part of the week, so a version-based filter loses them. Derive their namespaces once and pin them by name: + +```promql +group by (namespace, mz_context_org_name) (v2_mz_compute_cluster_status{mz_version=~".*"}) +``` + +The canary organizations are `Materialize Production Sandbox` and `Materialize Production Analytics`. Which regions carry which has changed over time, so always derive rather than assume, and note that the bot's links and Prometheus have disagreed on this. Sources and Sinks historically inspected only the sandbox environment. + +The canonical canary list lives in `MaterializeInc/release`, in `templates/issue.md`, as the `--environment` arguments to `bin/deploy upgrade production`. The `storage-overview` sign-off panel points instead at `MaterializeInc/cloud/.github/ISSUE_TEMPLATE/03-release.md`, which no longer exists. + +In staging every environment runs a release candidate, so a version join both selects the right set and excludes development environments: + +```promql +sum(rate([6h]) * on(namespace) group_left() + group by (namespace) (v2_mz_compute_cluster_status{mz_version=~".*-rc[.].*"})) +``` + +Escaping note: write `-rc[.]` rather than `-rc\\.` so the expression survives JSON encoding unchanged. + +## Step 3: Choose the time window + +The bot's links default to roughly `now-2d`, resolved to an absolute timestamp at post time. That is too short. The window must contain a clean stretch of the previous release, the boundary, and a clean stretch of the new release, which in practice means seven days or more. Look back far enough to include the previous upgrade, so that the previous boundary is available as a calibration reference. + +A window of `Sun 12:00 UTC` through the current day at `12:00 UTC` with a 6h step has worked well. The 6h step averages away diurnal structure while leaving enough points to see a step. + +Sample the three phases separately, and never straddle a boundary or a doubling bucket: + +* Previous release, steady state. +* New release under the earlier `rc`, steady state. +* New release under the current `rc`. + +## Step 4: Derive the metric roster + +Take the roster from the dashboard itself so it cannot drift out of date: + +``` +mcp__grafana__get_dashboard_property uid: jsonPath: $.panels[*].title +mcp__grafana__get_dashboard_panel_queries uid: +``` + +The panel titles are cheap and give the row structure. The panel queries are not. On the compute dashboard that call returns about 62 KB across 165 panels, which overflows the tool result and is written to a file instead. Slice that file with a script and extract only metric names and label selectors. Never read it whole, and never paste it into the conversation. + +Row panels appear in the title list but not in the query list, so the two are offset. Match them by title, not by index. + +`scripts/panel-metrics.py` does the slicing. Take the roster from the dashboard, but resolve every name against the metrics catalog for the release under test before trusting it, as described in *When a metric has gone missing* below. A dashboard panel outlives the metric it plots, so the panel list is a superset of what the build actually exports. + +## Step 5: Measure + +Batch many metrics into one range query by tagging each aggregate with a synthetic label and combining with `or`. This turns twenty tool calls into one: + +```promql +label_replace(sum(rate([6h])), "m", "1_name", "", "") +or label_replace(sum(avg_over_time([6h])), "m", "2_name", "", "") +``` + +Prefix the tags so the result order is stable and readable. Use `rate(x[6h])` for counters and `avg_over_time(x[6h])` for gauges, both matching the step so buckets do not overlap. + +`scripts/build-range-query.py` writes these expressions from a metric list, because twenty of them by hand is where typos live. It also encodes the one thing that differs between stacks, namely the staging release-candidate join against the pinned production canary namespaces, so the same roster can be run against either without rewriting the selector. + +Run each area twice. Once across all clusters, and once restricted to the system clusters, which the panel instructions call out because a system-cluster regression is easy to lose in the noise of user clusters. System clusters are `instance_id=~"s[0-9]+"` for the controller and replica metrics, and `pod=~".*cluster-s[0-9]+-replica-.*"` for the container metrics. + +## Step 6: Judge + +There are no thresholds, so the discipline is in ruling out the confounders before believing a signal. + +**Calibrate before calling anything a regression.** Fleet composition and workload drift produce steps of the same size as most real regressions, so pull three weeks of the metric and look at its natural spread before believing a step. Do this live rather than against a recorded figure, because the spread itself changes as the fleet does. When this was first measured, staging fleet clusterd CPU ranged from 2.83 to 5.55 cores over three weeks, which put a 3% step across the boundary far inside the noise; the useful part of that observation is its size, roughly a factor of two, not the numbers. + +**Compare at equal post-restart age.** Every upgrade restarts `clusterd`, and a fresh process holds less memory than one that has been running for days. Comparing the pre-upgrade level against the post-upgrade level therefore flatters the new release, and comparing a post-upgrade level against a mid-week pre-upgrade level exaggerates a regression. Sample both sides at a similar age since restart, and treat a monotonic climb within one release as more informative than any level difference across the boundary. + +**Read the base level of bimodal metrics.** Arrangement record counts and sizes swing by a factor of three or more as periodic dataflows rebuild. Compare the low state against the low state; spike heights are not comparable. + +**Discount pre-existing noise.** Some staging environments crashloop or carry permanently erroring dataflows. In August 2026 staging us-east-1 sustained roughly 90 `clusterd` restarts per 6h and staging eu-west-1 carried 50 to 400 dataflow errors continuously, both flat across the boundary. Flat means not release-related. The panel instructions suggest filtering such environments out with the dashboard variables, which is worth doing when they mask everything else. + +**Separate a signal from its location.** A fleet-wide step and one environment moving the fleet total are different findings. Split them with a per-namespace ratio of the two windows: + +```promql +sort_desc(sum by (namespace) (rate([15h])) + / sum by (namespace) (rate([42h] offset 126h))) +``` + +Set the instant query's end time to the end of the new window, and pick the offset so the second window lands in the previous release. + +## Step 7: Drill deeper + +When a signal survives Step 6, tighten the aggregation one level at a time. Each level costs one query and narrows the search: + +1. Fleet total, which establishes that something moved. +2. Per namespace, which finds the environments involved. +3. Per cluster, via `instance_id`, which separates system from user clusters. +4. Per replica, via `replica_id`, which distinguishes a replica-local effect from a cluster-wide one. +5. Per worker, via `worker_id`, which exposes skew across workers of one replica. +6. Per collection, via `collection_id`, which names the dataflow. + +Below the metric layer, hand off rather than guess. Use `mz-profile` for CPU and memory attribution inside a process, `mz-query-tracing` for the latency breakdown of a statement, and the Polar Signals MCP server for on-CPU profiles of a running environment. If the signal implicates a specific change, `mz-debug-ci` covers finding the responsible build. + +## Step 8: Report + +State the verdict first, then the method, then the findings. The method matters because the reader has to judge whether the comparison was fair: name the version, the boundaries, the window, the namespace selection, and which regions and rows were covered. Say explicitly what was skipped and why. + +Separate release-blocking findings from notes. A note is something worth a second look next release that does not survive Step 6 as a regression. Give each note its magnitude and the reason it is not conclusive, so the next person can compare against it rather than rediscover it. + +Sign-off happens in the bot's thread. As of August 2026 there is a proposal to sign off by reacting to each team's message rather than replying, which was not yet confirmed; check the thread's convention before posting, and never post to Slack without the user asking. + +## Areas + +The bot links one dashboard per area. All are on `grafana.dev.materialize.com`. + +| Area | Dashboard | UID | Reference | +|---|---|---|---| +| Compute | `compute-overview` | `f248986d-81c6-42a6-817b-00cd7759d808` | `references/compute.md` | +| Sources and Sinks | `storage-overview` | `e6dc7745-7d35-4968-a23d-689883a984bb` | `references/sources-and-sinks.md` | +| Sources and Sinks | `storage-upsert-sources` | `ac2de0ab-4a35-48b5-93aa-7e645569debb` | `references/sources-and-sinks.md` | +| Persist | `persist` | `m3U1U6ZVk` | `references/persist.md` | +| Adapter | `environmentd-health` | `mR1Kg1d4z` | `references/adapter.md` | +| Reference | `release-health` | `zKe0K0N4z` | `references/reference-dashboards.md` | +| Reference | `networking` | `bHQE8bN4k` | `references/reference-dashboards.md` | + +Each reference names the metrics, their types and labels, the invariants that hold at any fleet size, and the hazards specific to that area. Read the one for the area you are verifying before running a single query. The `Storage` label the bot still uses refers to work now split between Sources and Sinks and Persist. + +Sweep sizes differ by an order of magnitude. Compute has about 60 sweep-relevant panels, adapter about 45, storage-overview about 55, and persist 375 panel targets over roughly 230 metrics. For persist, the dashboard's own `should be small` panel defines the sweep; see its reference. + +The bot links production only, but staging is a larger and earlier sample of the same release, and Adapter verification has covered staging in practice. Prefer running both. + +Datasource UIDs, which the dashboards take as the `datasource` variable: + +| Stack | Region | UID | +|---|---|---| +| Production | us-east-1 | `2K85O21Vz` | +| Production | eu-west-1 | `E0J0O2J4k` | +| Production | us-west-2 | `ee2e6227-dc2d-4ca5-bb58-ca826fd6d614` | +| Staging | us-east-1 | `Ks85Oh14z` | +| Staging | eu-west-1 | `JKT0Oh1Vk` | +| Staging | us-west-2 | `c979f3ee-16d0-44f1-9f24-9e208e0326d9` | + +Staging us-west-2 held no compute environments in August 2026, so `v2_mz_compute_cluster_status` returns nothing there. Confirm it is still empty rather than reporting a region as clean. + +## Characterizing an area + +All seven dashboards have a reference, written from a sweep of v26.38.0-rc.3 against v26.37.0 in August 2026 across production canary and staging in us-east-1, with compute additionally covering production eu-west-1, production us-west-2, and staging eu-west-1. + +To characterize a new dashboard, or to refresh one, run Steps 1 through 6 against it for one release and record in `references/.md` what the run taught you: + +* Each metric with its type, the labels that select cluster and replica, and what it means. +* Which metrics are bimodal, restart-sensitive, or absent when zero. +* Hazards and invariants, in one section rather than two. Each entry leads with the property that holds at any fleet size, such as one counter equalling the difference of two others, a gauge whose only meaningful aggregate is a series count, or a metric that is structurally absent in one stack, and then gives the measurement it came from. Splitting these across two sections produced near-verbatim duplication in every reference, because almost every hazard is an invariant with a number attached. +* Label naming inconsistencies, duplicate-series hazards, and any panel expression whose filters are not what they appear to be. +* Known noise classes, meaning the environments that are unhealthy independently of any release and whose flat contribution can dominate a fleet aggregate. + +Record invariants, not levels. A recorded level is stale the week after it is written, because environments are created, deleted, and resized continuously, and a stale reference value is worse than none: it invites a comparison the reader should not make. The comparison that matters is always derived in-run, since the before-window of your own query is the only baseline guaranteed to describe the same fleet as the after-window. + +Coarse order-of-magnitude figures are worth keeping for one narrow purpose: catching a mis-scoped selector, for example a missing `container="clusterd"` that inflates a result tenfold. Keep them dated, keep them to one significant figure, and say plainly that they are not for comparison. + +## When a metric has gone missing + +A reference that names a metric the build no longer exports is worse than no reference, because the sweep reads the empty result as a healthy zero. Two things make this checkable, and both have to be applied to the right version. + +`doc/user/data/metrics.yml` is the generated catalog of `metric!` invocations in the Rust tree. It is checked in, so it is tagged along with each release, and the working tree's copy describes `main`, which is ahead of whatever you are verifying. Reading that copy during a sign-off answers a question you did not ask. Read the catalog at the two release tags instead and diff them: + +``` +scripts/catalog-diff.sh v26.38.0 v26.39.0-rc.3 +``` + +That turns "is this empty panel a regression or a rename?" into a lookup. A name in the removed list explains an after-window that went empty, and it is a documentation fix rather than a finding. A name in the added list explains an empty before-window, and comparing across the boundary on it is meaningless. + +**Catalog membership is not a string match.** A `metric!` whose name is built with `format!` is catalogued with its placeholders globbed, so `mz_persist_user_bytes` and `mz_persist_compaction_goodbytes` are both covered by the single entry `mz_persist_*_bytes`, and histograms are catalogued as their expanded `_bucket`, `_count`, and `_sum` families rather than under the base name. Grepping the catalog for a literal name will therefore report a live metric as missing. Match against the patterns, as `scripts/lint_metrics.py` does. + +**The catalog covers only this repository's Rust tree.** It holds no `v2_mz_*`, which the promsql exporter derives from SQL, no `container_*`, `kube_*`, or `kubelet_*` from cAdvisor and kube-state-metrics, and no cloud-side names such as `mz_envd_up` or `mz_external_*`. Those families are confirmed against Prometheus with `list_prometheus_metric_names`, and they are where the sweep has actually been bitten: the compute dashboard's scratch-disk panels plot `kubelet_volume_stats_used_bytes`, which resolves to five series in a single namespace in both production and staging us-east-1 and none of them a `clusterd` volume, so those panels render blank in both stacks. + +The diff survives that blind spot, because both sides share it and a dynamically named metric is missing from both catalogs and so never appears in either list. The diff therefore never reports a rename that did not happen. It can still miss one, so an empty result that the diff does not explain is not yet cleared. Confirm it against Prometheus with `list_prometheus_metric_names` before reporting the metric as zero. + +When a name resolves in neither the catalog nor Prometheus, the panel that plots it is dead, and the reference should record that rather than the metric. `git log -S --all -- src/` settles which kind of dead it is, and the two kinds read differently in a report. `mz_query_latency` was real, added in #22049 and deleted in #26647 along with the stash, so the `environmentd-health` panel that still plots its `_bucket` family has been empty since that deletion. `mz_persist_columnar_validation_count` and `mz_txn_placeholder_schema_apply` have never appeared in this repository at all, yet both are live arms of the persist dashboard's `should be small` panel, and the working spelling of the first sits beside it on the same panel as `mz_persist_columnar_op_count` with `op="validation"` and `result="invalid"`. + +`ci/test/lint-skill-metrics.sh` guards the other direction, so the references cannot rot silently between releases. It resolves every `mz_*` name in this skill against the working tree's catalog, patterns included, and fails on any that neither resolves nor appears in `scripts/metrics-allowlist.txt`. The allowlist is the point: it carries one line per name the catalog does not cover, with the reason, so adding to it is a deliberate act and a genuinely renamed metric still fails. It also fails on an allowlist entry that has started resolving, so the exemptions cannot outlive their reason. A failure is an instruction to update the skill, never to suppress the lint. + +## Traps + +**A gauge divided by a limit can fail on duplicate series.** `container_spec_memory_limit_bytes` is exported once per node-label set, and labels such as `karpenter_sh_initialized` flip during node lifecycle, which yields two series for one pod and a `many-to-many matching not allowed` error. Collapse both sides first: + +```promql +max by (namespace, pod, container) (avg_over_time(container_memory_working_set_bytes{...}[6h])) +/ on (namespace, pod, container) +max by (namespace, pod, container) (avg_over_time(container_spec_memory_limit_bytes{...}[6h])) +``` + +**An absent series is not the same as a healthy zero.** Error and orphan counters are only exported when non-zero, so an empty result reads as clean when it can also mean the metric was renamed. Confirm the metric exists somewhere in the window before reporting zero. + +**Label names are not consistent across metrics.** Most compute metrics carry `instance_id` and `replica_id`, but the arrangement maintenance metric carries `cluster_environmentd_materialize_cloud_cluster_id` and `cluster_environmentd_materialize_cloud_replica_id` instead. Copy selectors from the panel expressions rather than writing them from memory. + +**Some panel filters are variable-substitution artifacts.** Several compute panels append `instance_id!="$cluster_id"`, which exists to blank a series when a single cluster is selected and is not a semantic filter. Reproducing it in an aggregate query is unnecessary. + +**Counter resets hide inside long rate windows.** Pods are replaced at every upgrade, so restart counters reset. Prefer counting the pods whose last termination matched a condition over summing increases across a boundary: + +```promql +count(kube_pod_container_status_last_terminated_exitcode{pod=~".*cluster-.*-replica-.*", container="clusterd"} == 137) +``` + +**Which exit code matters depends on the dashboard.** The compute dashboard looks for 137, an OOM kill. The `release-health` non-clusterd restart panel joins against `!= 166`, treating 166 as an expected termination. Do not carry one dashboard's convention into another. + +**An empty panel is the most dangerous reading on any dashboard.** Metrics get renamed and panels do not follow, so the panel renders blank and looks like health. This was not hypothetical: the sweep found four dead expressions on `environmentd-health` and two on `storage-overview`, all silently empty. Before reporting any metric as zero, confirm the name still exists: + +``` +mcp__grafana__list_prometheus_metric_names datasourceUid: ... regex: mz_catalog.* +``` + +The per-area references list the dead panels found so far. Check for new ones whenever a panel that should have data does not. + +**Some odd-looking expressions are deliberate and must be preserved.** Two idioms recur. An `or` between two aggregations, as on `storage-overview`'s controller protocol panels, straddles a metric rename so the panel keeps working across the version boundary; imitate it rather than deleting the dead arm. A trailing `^0`, as in `networking`'s egress panels, raises a series to the power zero to yield 1 for every series that exists, making it a set-membership filter rather than arithmetic. + +**An implausibly constant quantile is a bucket artifact, not stability.** A `histogram_quantile` landing inside one wide bucket returns the bucket boundary and cannot move, which reads as a rock-steady latency. Two adapter panels do this. When a quantile is stable to five significant figures while its counters advance, switch to `rate(_sum) / rate(_count)`. + +**Counting series is sometimes the measurement.** Several metrics are per-entity gauges whose only useful aggregate is a series count: `mz_persist_shard_upper` for shards, `mz_balancer_metadata_seconds` for balancers, `mz_persist_metadata_seconds` by `version` for persist client builds. Similarly `mz_source_progress` is a millisecond frontier timestamp, so a healthy series contributes exactly 1000 to its rate and the panel's absolute value is really a series count in disguise. diff --git a/.agents/skills/mz-release-signoff/references/adapter.md b/.agents/skills/mz-release-signoff/references/adapter.md new file mode 100644 index 0000000000000..4a3979c01a6fa --- /dev/null +++ b/.agents/skills/mz-release-signoff/references/adapter.md @@ -0,0 +1,98 @@ +# Adapter metric reference + +Dashboard `environmentd-health`, UID `mR1Kg1d4z`. All metrics come from `environmentd`, one process per environment, so there is no cluster or replica dimension and no system-cluster split. During a zero-downtime upgrade two `environmentd` pods run at once and every sum doubles for one bucket. + +## Variables + +This dashboard names its environment variable `$env`, not `$namespace`, and also exposes `$pod`. The release bot sets `var-namespace` and `var-env` on every link for exactly this reason. When reproducing a panel expression, substitute `$env` with the namespace regex. + +## Roster + +### Durable catalog + +| Metric | Type | Notes | +|---|---|---| +| `mz_catalog_transaction_commit_latency_seconds_{bucket,count,sum}` | histogram | Catalog write latency. | +| `mz_catalog_transaction_commits`, `mz_catalog_transactions_started` | counter | Commit volume, very low in steady state. | +| `mz_catalog_sync_latency_seconds_{bucket,count,sum}` | histogram | Catalog read-side sync latency. | +| `mz_catalog_syncs` | counter | Sync volume. | +| `mz_catalog_collection_entries` | gauge, by `collection` | Catalog size. Grows monotonically with catalog contents, so a slow rise is expected and a fall is worth explaining. | +| `mz_catalog_snapshot_latency_seconds_*`, `mz_catalog_snapshot_seconds_*`, `mz_catalog_snapshots_taken` | histogram, counter | Snapshots are taken at boot, so these are only non-empty around a restart. | +| `mz_catalog_transact_seconds_*`, `mz_catalog_transact_phase_seconds_*` | histogram | Finer breakdown than the commit latency histogram. | +| `mz_catalog_allocate_id_seconds_*` | histogram | ID allocation. | +| `mz_catalog_snapshot_cache`, `mz_catalog_snapshot_consolidations`, `mz_catalog_snapshot_max_entries` | counter, gauge | Snapshot cache behaviour. | +| `mz_catalog_arc_strong_count`, `mz_catalog_arc_weak_count` | gauge | Catalog handle counts, useful for leak hunting. | +| `v2_mz_catalog_items` | gauge | Item count, an alternative to collection entries. | + +### Adapter and coordinator + +| Metric | Type | Notes | +|---|---|---| +| `v2_mz_envd_up` | gauge | One per healthy `environmentd`. The cheapest liveness check, and it should equal the environment count. | +| `mz_start_time_environmentd` | gauge, milliseconds | Startup duration. Divide by 1000. Resets only on restart, so it reports the current generation's boot. | +| `mz_connection_status` | counter, by `status` and `source` | Upstream connection outcomes. | +| `mz_determine_timestamp` | counter, by `respond_immediately` and `isolation_level` | Timestamp selection volume. A fall in `respond_immediately` means more queries are waiting on a timestamp. | +| `mz_time_to_first_row_seconds_bucket` | histogram, by `isolation_level` | The label values are `strict serializable` and `serializable`, with a space. Their distributions differ by roughly a factor of five, so never aggregate across the label. | +| `mz_row_set_finishing_seconds_{bucket,sum,count}` | histogram | Row-set finishing. The panel title notes the top bucket is 16s. | +| `mz_linearize_message_seconds_bucket` | histogram, by `immediately_handled` | Read linearization. Sub-millisecond in steady state. | +| `mz_slow_message_handling_{bucket,sum,count}` | histogram, by `message_kind` | Coordinator message handling. The `_sum` rate is the coordinator's busy time and the best single coordinator-load signal. | +| `mz_coord_queue_busy_seconds_{bucket,count}` | histogram | Coordinator queue. The `> 1s` panel subtracts the `le="1"` bucket increase from the count increase. | +| `mz_append_table_duration_seconds_{bucket,sum,count}` | histogram | Table append latency. | +| `mz_query_total` | counter, by `session_type` and `statement_type` | `session_type` is `user` or `system`. System queries dominate by an order of magnitude. | +| `mz_active_sessions` | gauge, by `session_type` | Session counts. | + +### HTTP and auth + +| Metric | Type | Notes | +|---|---|---| +| `mz_http_requests_total` | counter, by `path` and `status` | Panels split webhook paths (`/api/webhook.*`) from the rest, because webhook volume swamps everything else. | +| `mz_http_request_duration_seconds_bucket` | histogram | Same split. | +| `mz_auth_request_count` | counter, by `path`, `status`, `mz_context_org_name` | Frontegg. Status values are strings such as `401 Unauthorized`, not bare codes. | +| `mz_auth_request_duration_seconds_bucket` | histogram, by `path` | | +| `mz_auth_refresh_tasks_active` | gauge | | + +### Infrastructure + +Container metrics select `container="environmentd"`. CPU percent divides by `container_spec_cpu_quota / container_spec_cpu_period`, and memory percent by `container_spec_memory_limit_bytes`. + +`crdb_dedicated_sys_cpu_combined_percent_normalized` and its `_maximum` variant carry no namespace label. They describe the whole regional CockroachDB cluster, so they are shared across all environments in the region and cannot be attributed to the release under test. + +## Broken panels, as of 2026-08-19 + +These render empty and are not evidence of a healthy system. Verified against the metric catalogue in production us-east-1. + +* `Avg Transaction Commit Latency` and `Avg Transaction Commit Latency by Env` divide `mz_catalog_transaction_commit_latency_seconds` by `mz_catalog_transaction_commits`. The first name does not exist, because the metric is a histogram exposing only `_bucket`, `_count`, and `_sum`. Use `rate(..._sum) / rate(..._count)` instead. +* `Avg Sync Latency` and `Avg Sync Latency By Env` have the same defect for `mz_catalog_sync_latency_seconds`. +* `Stash (CRDB) Query Latencies` reads `mz_query_latency_bucket`, which no longer exists. The catalog moved off the stash, so the panel has no replacement. +* `Swap Usage (bytes)` plots `container_spec_swap_limit_bytes` as its limit series. The real name is `container_spec_memory_swap_limit_bytes`. The usage series is fine. + +## Hazards and invariants + +Each entry states a property that holds at any fleet size, followed by the measurement it came from. The property is what survives a release. The measurement is dated, describes whatever fleet existed when it was taken, and is recorded only so the property is not mistaken for a guess. + +**Two p99 panels are pinned by bucket resolution.** `p99 Slow Coordinator Messages` reported 0.0001276 s across eight days in staging with five-digit stability, and `Coordinator Table Append Latencies` p99 reported 0.12673 s in production with the same rigidity, both while the underlying counters advanced normally. The quantile is landing inside one wide bucket, so interpolation returns the bucket boundary and the panel cannot move. Read `rate(_sum) / rate(_count)` instead for these two, and treat an implausibly constant quantile anywhere as a bucket artifact rather than as stability. + +**`v2_mz_envd_up` should equal the environment count.** It is the cheapest liveness check on the dashboard. + +**`environmentd` does not swap.** Swap usage measured exactly zero across the whole window in production canary, so a non-zero reading is itself the finding. + +**Catalog collection entries grow monotonically with catalog contents.** A fall needs explaining. + +**Catalog snapshot latency is recorded at boot only.** `mz_catalog_snapshot_latency_seconds` records once at boot, so an average over a steady-state window divides by zero and reads NaN, while an upgrade bucket reads non-NaN. Neither is a finding. + +**Serializable and strict serializable time-to-first-row are not comparable.** In production canary the p99 ran about 1.2 s for strict serializable and about 6.5 s for serializable over the same window, roughly a factor of five. Never aggregate across the `isolation_level` label, and compare each against its own history. + +**System queries outnumber user queries by an order of magnitude.** The gap is far larger in staging, where most environments are idle apart from introspection, so staging user-query numbers are not a workload signal. + +**Coordinator busy time is the best single coordinator-load signal.** It is the `_sum` rate of `mz_slow_message_handling`. The message rate alone hides how expensive each message was. + +**`crdb_dedicated_*` metrics describe the whole regional CockroachDB cluster.** They cannot be attributed to the release under test. + +## Order of magnitude + +Recorded 2026-08 for scope-checking only. Derive the real baseline from your own before-window. + +* `environmentd` CPU is a fraction of a core per environment, and working set is low single-digit GB per environment. +* Coordinator message rate is thousands per second per environment, and coordinator busy time a few hundredths of a second per second. +* Linearize and coordinator-message quantiles are sub-millisecond. Time to first row is order one second for strict serializable and several seconds for serializable. +* Catalog commits are rare, well under one per second, so their latency average is noisy by construction. diff --git a/.agents/skills/mz-release-signoff/references/compute.md b/.agents/skills/mz-release-signoff/references/compute.md new file mode 100644 index 0000000000000..95a3ed5997250 --- /dev/null +++ b/.agents/skills/mz-release-signoff/references/compute.md @@ -0,0 +1,100 @@ +# Compute metric reference + +Dashboard `compute-overview`, UID `f248986d-81c6-42a6-817b-00cd7759d808`. The rows that matter for sign-off are `By process` and `By cluster`, each inspected twice, once across all clusters and once restricted to system clusters. The `By replica`, `By worker`, and `By collection` rows require a narrowed variable and belong to the drill-down in step 7 of the skill, not to the sweep. + +## Selectors + +Process-level panels read cAdvisor and kube-state metrics and select on the pod name: + +``` +pod=~".*cluster--replica--.+", container="clusterd" +``` + +Cluster-level panels split into two families with different label names for the same concepts. + +| Family | Cluster label | Replica label | +|---|---|---| +| Controller, protocol, peeks, and `v2_mz_*` replica metrics | `instance_id` | `replica_id` | +| `mz_arrangement_*`, `mz_dataflow_replica_*`, `mz_subscribe_*`, `mz_cluster_*` | `cluster_environmentd_materialize_cloud_cluster_id` | `cluster_environmentd_materialize_cloud_replica_id` | + +System clusters are `s` followed by digits in either family, and `.*cluster-s[0-9]+-replica-.*` in the pod name. + +## Roster + +### By process + +| Metric | Type | Notes | +|---|---|---| +| `container_cpu_usage_seconds_total` | counter | Needs `cpu="total"`. Divide by `container_spec_cpu_quota` for percent of limit. | +| `container_memory_working_set_bytes` | gauge | Resident plus active file. The primary memory signal. | +| `container_memory_rss` | gauge | Tracks working set closely; a divergence between them is itself a finding. | +| `container_memory_swap` | gauge | Limit is `container_spec_memory_swap_limit_bytes`. Grows with process age on swap-enabled nodes. | +| `mz_memory_limiter_memory_usage_bytes` | gauge | Memory plus swap, the quantity the limiter enforces. Divide by `mz_memory_limiter_memory_limit_bytes`. | +| `mz_metrics_libc_ru_maxrss_bytes` | gauge, monotone per process | Peak resident set. Resets to zero on restart, so it measures the current generation only. | +| `kubelet_volume_stats_used_bytes` | gauge | Scratch disk. Limit is `kubelet_volume_stats_capacity_bytes`. | +| `container_start_time_seconds` | gauge | Source of the uptime panel, and the cheapest way to establish restart age. | +| `kube_pod_container_status_restarts_total` | counter | Resets when a pod is replaced, which every upgrade does. | +| `kube_pod_container_status_last_terminated_exitcode` | gauge | `== 137` is an OOM kill. Counting matching series beats summing restart increases. | +| `container_network_receive_bytes_total`, `container_network_transmit_bytes_total` | counter | Per pod, with no `container` label. | +| `mz_metrics_libc_ru_minflt_total`, `mz_metrics_libc_ru_majflt_total` | counter | Major faults spike hard during rehydration and while paging in from swap. | +| `mz_metrics_libc_ru_utime_seconds_total`, `mz_metrics_libc_ru_stime_seconds_total` | counter | Process-reported CPU. Cross-checks the cAdvisor CPU series, and the two should move together. | + +### By cluster + +| Metric | Type | Notes | +|---|---|---| +| `mz_compute_controller_replica_count`, `_collection_count`, `_peek_count`, `_subscribe_count` | gauge | Controller-side inventory. Collection count tracks catalog growth, so a slow rise is expected. | +| `mz_compute_controller_command_queue_size` | gauge | Sustained depth means the controller is not draining. | +| `mz_compute_controller_response_send_count`, `_recv_count` | counter | Response queue depth, as the difference of the two. There is no depth gauge, because the response channel is an `instrumented_unbounded_channel` (`src/ore/src/channel.rs`), which takes a send and a receive counter and exports nothing else. Contrast `command_queue_size`, a real gauge that the command path increments and decrements directly. | +| `mz_compute_commands_total`, `mz_compute_responses_total` | counter | Protocol volume. Doubles for one bucket during a zero-downtime upgrade. | +| `mz_compute_command_message_bytes_total`, `mz_compute_response_message_bytes_total` | counter | Protocol bytes. Worth checking when a change touches command encoding. | +| `mz_compute_controller_history_command_count`, `_history_dataflow_count` | gauge | Controller-side command history, which should be reduced and not grow without bound. | +| `mz_compute_replica_history_command_count`, `_history_dataflow_count` | gauge | Replica-side equivalent. | +| `mz_compute_peeks_total` | counter | Label `result`. Successes are `rows` and `rows_stashed`; anything else is an error or a cancellation. | +| `mz_compute_peek_duration_seconds_bucket` | histogram | Quantiles need `sum by (le)` after any namespace join. | +| `v2_mz_dataflow_elapsed_seconds_total` | counter | Compute time. Strongly workload-shaped, and often the noisiest series on the dashboard. | +| `mz_arrangement_maintenance_seconds_total` | counter | Merge and compaction work. Uses the `cluster_environmentd_*` labels. | +| `v2_mz_compute_replica_park_duration_seconds_total` | counter | Time parked, so a fall here alongside a CPU rise localizes new work to the dataflow loop. | +| `v2_mz_arrangement_count`, `_record_count`, `_batch_count`, `_size_bytes` | gauge | Bimodal, see hazards. | +| `v2_mz_hint_group_size_potential_savings_bytes` | gauge | Missing group-size hints. Informational, and not release-sensitive. | +| `v2_mz_orphan_dataflow_count` | gauge | Non-zero means a dataflow outlived its collection, which is a genuine bug signal. | +| `v2_mz_dataflow_error_count` | gauge | Erroring dataflows. Some staging environments carry a permanent floor. | +| `mz_cluster_handle_command_duration_seconds_bucket` | histogram | Replica-side command handling latency. | +| `mz_dataflow_replica_expiration_timestamp_seconds`, `_expiration_remaining_seconds` | gauge | Replica expiration. Panels filter `> 0` and `!= 0` because the metric is exported as zero when unset. | +| `mz_subscribe_snapshots_skipped_total` | counter | Subscribe snapshot optimization hit rate. The panel appends `> 0` to hide inactive replicas. | + +## Hazards and invariants + +Each entry states a property that holds at any fleet size, followed by the measurement it came from. The property is what survives a release. The measurement is dated, describes whatever fleet existed when it was taken, and is recorded only so the property is not mistaken for a guess. + +**OOM kills, dataflow errors, and orphan dataflows are absent rather than zero when healthy.** Production canary returned no series at all for `v2_mz_dataflow_error_count` and `v2_mz_orphan_dataflow_count` across a full week. That is the healthy case, and it is indistinguishable from a renamed metric unless the metric is confirmed to exist elsewhere. + +**`v2_mz_orphan_dataflow_count` above zero is always a bug, never a load effect.** + +**Working set falls at every upgrade.** In production canary us-east-1 the sum fell from about 290 GB to about 236 GB at the v26.37.0 rollout with no change in the code that mattered, purely because arrangements were rebuilt fresh. Judge memory by the slope within a release, not the step across one. + +**Arrangement gauges are bimodal.** `v2_mz_arrangement_record_count` and `v2_mz_arrangement_size_bytes` swing by a factor of three to ten as periodic dataflows rebuild. In August 2026 staging us-east-1 alternated between 0.72e9 and 2.86e9 records with no release involvement. Compare low state against low state, because spike heights are not comparable. + +**Peak resident set and swap are restart-sensitive.** Both reset or decay at an upgrade, so a level drop across the boundary is the restart and not the release. Peak resident set therefore describes the current generation only. + +**Arrangement maintenance ramps after a restart.** Measured at 0.020 s/s one day after an upgrade and 0.030 s/s three days later on the same release, so an apparent increase across a boundary can be nothing more than a difference in age. + +**Compute time and park time are complementary.** A CPU rise with a park fall localizes new work to the dataflow loop, while a CPU rise with park flat points outside it. + +**Response queue depth is a difference of two counters, so it only holds while neither has reset.** Both reset when a pod is replaced, which every upgrade does, and a scrape that catches one reset and not the other yields a wild value. Staging us-east-1 read -761 in one bucket and +5371 in another over a window where every other bucket sat within one of zero. Read a single implausible bucket as a reset artifact, and judge the panel by whether it returns to zero rather than by any one sample. + +## Known noise classes + +Some environments are unhealthy independently of the release, and their contribution is constant across the boundary rather than absent. Check for these first, because they can dominate a fleet aggregate. + +* Staging carries persistently crashlooping replicas. Their restart rate is high and flat, and flat means not release-related. +* Some staging environments carry permanently erroring dataflows, likewise flat. +* Where such an environment masks everything else, exclude it with the dashboard variables, as the panel instructions suggest. + +## Order of magnitude + +Recorded 2026-08 for scope-checking a query, not for comparison. If a result sits an order of magnitude away from these, suspect a mis-scoped selector rather than a regression. Derive the actual baseline from the before-window of your own run. + +* Production canary, two environments: a few cores of clusterd CPU, hundreds of GB of working set summed, single-digit peeks per second. +* Staging, about fifteen environments: a few cores, low hundreds of GB, tens of peeks per second. +* System clusters account for most of staging's dataflow time and roughly a quarter of production canary's CPU. diff --git a/.agents/skills/mz-release-signoff/references/persist.md b/.agents/skills/mz-release-signoff/references/persist.md new file mode 100644 index 0000000000000..3c17f59e9a984 --- /dev/null +++ b/.agents/skills/mz-release-signoff/references/persist.md @@ -0,0 +1,87 @@ +# Persist metric reference + +Dashboard `persist`, UID `m3U1U6ZVk`. It carries 375 panel targets across roughly 230 distinct `mz_persist_*` metrics, which is far more than a release sweep should touch. The dashboard solves this itself: the `Top-Level` row includes a panel literally titled `should be small`, whose series are a curated list of counters that ought to sit near zero. That panel plus the throughput, timings, and cmds panels beside it are the sweep. Everything else, in particular the `By Shard` and `Compaction state` rows, is drill-down. + +## Variables + +`$env` for the namespace and `$pod`, matching the adapter dashboard rather than the compute one. + +Persist reports its own build independently of `mz_version`: `mz_persist_metadata_seconds` carries a `version` label, and the `# processes by version` panel counts by it. This is a useful cross-check that a rollout actually reached the persist clients. + +## The sweep subset + +### Should be small + +Every one of these is a counter whose rate should be at or near zero. Treat any of them departing from its measured baseline as a finding, because unlike the throughput metrics they have no legitimate reason to grow with load. + +`mz_persist_blob_failures`, `mz_persist_consensus_failures`, `mz_persist_state_update_state_slow_path`, `mz_persist_lease_timeout_read`, `mz_persist_compaction_noop`, `mz_persist_compaction_failed`, `mz_persist_compaction_dropped`, `mz_persist_external_blob_delete_noop_count`, `mz_persist_external_failed_count`, `mz_persist_cmd_failed_count`, `mz_persist_pushdown_parts_mismatched_stats_count`, `mz_persist_schema_cache_fetch_state_count`, `mz_persist_shard_unconsolidated_snapshot`, and `mz_persist_columnar_op_count{op="validation", result="invalid"}`. + +Two of the panel's sixteen arms are dead and are deliberately left out of the list above, because an arm that cannot resolve contributes a permanent zero and makes the panel look healthier than it is. `mz_persist_columnar_validation_count{result="invalid"}` duplicates the columnar validation series that `mz_persist_columnar_op_count{op="validation", result="invalid"}` already carries, and only the second resolves. `mz_txn_placeholder_schema_apply` resolves nowhere. Neither name has ever appeared in the Materialize source tree, in any release from v26.36.0 through v26.39.0-rc.3 or in the history behind them, so both are panel-authored rather than renamed. Sweep the fourteen that resolve and treat the panel's own count as wrong by two. + +Worth adding to the sweep even though the panel omits them: `mz_persist_compaction_timed_out`, `mz_persist_s3_operation_timeouts`, `mz_persist_pubsub_client_grpc_error_count`, `mz_txn_op_errored_count`, and `mz_txn_op_retry_count`. + +### Volume and timings + +| Metric | Type | Notes | +|---|---|---| +| `mz_persist_user_bytes`, `mz_persist_user_goodbytes` | counter | Physical and logical write volume. Goodbytes exceeds bytes because bytes are encoded and compressed. | +| `mz_persist_cmd_started_count`, `_succeeded_count`, `_failed_count` | counter | State machine command volume. | +| `mz_persist_cmd_cas_mismatch_count` | counter | Compare-and-set contention on shard state. Superlinear in writer concurrency, so it is the most sensitive contention signal on the dashboard. Always normalize by `cmd_started_count`. | +| `mz_persist_cmd_seconds` | counter of seconds | Time in state machine commands. | +| `mz_persist_encode_seconds`, `mz_persist_decode_seconds` | counter of seconds | Codec cost. Decode rises sharply during rehydration. | +| `mz_persist_external_seconds`, `_started_count`, `_succeeded_count`, `_failed_count` | counter | Blob and consensus calls. External time dominates every other timing series. | +| `mz_persist_external_rtt_latency`, `mz_persist_external_op_latency_bucket` | gauge, histogram | Per-operation latency, split by `op`. | +| `mz_persist_compaction_seconds`, `_requested`, `_applied`, `_bytes`, `_goodbytes` | counter | Compaction. Requested minus applied equals the noop and dropped counts. | +| `mz_persist_gc_seconds`, `_started`, `_finished`, `_noop`, `_skipped`, `_merged_reqs` | counter | Garbage collection. | +| `mz_persist_retry_retries_count`, `_started_count`, `_finished_count`, `_sleep_seconds` | counter | The panel excludes `op="next_listen_batch"`, which has its own panel because it retries by design. | +| `mz_persist_shard_upper` | gauge, per shard | Counting distinct `shard` labels gives the shard count. | +| `mz_persist_read_batch_part_bytes`, `_count` | counter, by `op` | The `op="unindexed"` slice is the `unindexed reads` panel. Rises by roughly 3x during rehydration. | +| `mz_persist_pushdown_parts_filtered_count`, `_fetched_count`, `_inline_count`, `_faked_count` and their `_bytes` variants | counter | The ratio panels divide one by the sum of all four. | +| `mz_persist_blob_cache_hits_bytes`, `_hits_blobs`, `_evictions` | counter | | +| `mz_txn_batch_unapplied_count`, `_unapplied_min_ts`, `_commit_count`, `_commit_bytes` | gauge, counter | Txn-shard backlog. | +| `mz_txn_op_started_count`, `_succeeded_count`, `_retry_count`, `_errored_count` | counter, by `op` | | + +### Everything else + +The remaining rows are for drill-down once the sweep flags something: `By Shard` for per-shard state, `Compaction` and `Compaction state` for compaction internals, `GC`, `External`, `Retries`, `Codec`, `Audit`, `Postgres/Consensus`, `PubSub Server` and `PubSub Client`, `Schema/Structured`, `Rehydration`, and `Txns`. Per-shard panels are keyed by `shard` and often by `name`, which is how a source or collection is joined to its shard, for example `mz_persist_shard_update_count{name="$source"}`. + +## Hazards and invariants + +Each entry states a property that holds at any fleet size, followed by the measurement it came from. The property is what survives a release. The measurement is dated, describes whatever fleet existed when it was taken, and is recorded only so the property is not mistaken for a guess. + +**Every metric in the `should be small` list has no legitimate reason to grow with load.** Any of them departing from zero is a finding rather than a scaling effect. + +**CAS mismatch must be normalized by `mz_persist_cmd_started_count`.** The raw rate moves with command volume, which moves with fleet size, while the normalized ratio is stable to three significant figures within a release. That stability is what makes it an unusually sharp instrument. + +**Compaction requested minus applied equals the noop plus dropped counts.** If it does not, one of the three is broken. + +**The panel named `compaction write amp` is a compression ratio, not an amplification.** It computes `mz_persist_compaction_bytes / mz_persist_compaction_goodbytes`, and production canary measured 0.14 to 0.24. Values below one are the healthy case and mean compaction is compressing. A rise toward one is the bad direction, which is the opposite of what the name suggests. + +**`mz_persist_pushdown_parts_faked_*` needs the `or up * 0` guard.** The dashboard writes `rate(...) or (up{...} * 0)` because the metric is absent when nothing is faked, and without the guard the whole ratio goes empty rather than to zero. Imitate this whenever a ratio's numerator can vanish. + +**Two writers contend during a zero-downtime upgrade.** CAS mismatch rose roughly six-fold in production canary during each upgrade bucket, from about 0.4 to about 2.5 per second, then returned. This is the two generations writing the same shards and is expected. + +**GC and decode move in opposite directions during an upgrade.** `mz_persist_gc_finished` dips, because a restarting process stops collecting, while `mz_persist_decode_seconds` and unindexed read bytes spike, because state is being refetched. Both are upgrade artifacts and neither is a finding. + +**Persist reports its own build independently of `mz_version`.** `mz_persist_metadata_seconds` carries a `version` label, and the `# processes by version` panel counts by it. Use it to confirm a rollout reached the persist clients. + +**Counting distinct `shard` labels on `mz_persist_shard_upper` is how the shard count is obtained.** There is no shard-count gauge. + +## Order of magnitude + +Recorded 2026-08 for scope-checking only. + +* External time dominates every other timing series, by roughly a factor of two over command time and an order of magnitude over compaction. +* Command rates are hundreds per second for a handful of environments and low thousands across a staging fleet. +* Shard counts are order a thousand per stack. +* Normalized CAS mismatch is order 1e-3 in a two-environment canary and order 1e-2 in a dense staging fleet, so the two stacks are not comparable to each other. + +## Open finding from the characterization run + +Staging us-east-1 stepped on CAS mismatch per persist command exactly at the release boundary, and the step is not explained by volume or fleet composition. The ratio held 0.00608 to 0.00629 for the three days before, jumped to 0.00990 in the first bucket after the v26.38.0-rc.1 upgrade, held between 0.00988 and 0.01009 for the next four days, then fell to 0.00853 with rc.3 and held there. Commands started stepped alongside it from 1345 to 1417 per second, about 5%. + +Three things make this worth a Persist owner's attention rather than dismissal. The step is normalized per command, so it survives the calibration test in the skill's step 6. It is sharp, landing inside the upgrade bucket and flat on both sides, which drift cannot produce. And rc.3 recovered about a third of it, which points at a specific change rather than at the environment. + +Against it being a release blocker: no failure counter moved, command and external latency were flat, and production canary showed no step at all with its normalized ratio slightly falling, from 0.00076 to 0.00070. The plausible reading is more frequent state writes raising contention superlinearly in a dense multi-environment stack, which staging is and the two-environment canary is not. + +Recheck this ratio next release. If it stays elevated, the question for Persist is which v26.38 change increased state-write frequency by about 5%. diff --git a/.agents/skills/mz-release-signoff/references/reference-dashboards.md b/.agents/skills/mz-release-signoff/references/reference-dashboards.md new file mode 100644 index 0000000000000..dae69f38ec85b --- /dev/null +++ b/.agents/skills/mz-release-signoff/references/reference-dashboards.md @@ -0,0 +1,117 @@ +# Reference dashboards metric reference + +Two dashboards that the release bot files under `Reference`: `release-health`, UID `zKe0K0N4z`, and `networking`, UID `bHQE8bN4k`. Neither belongs to a database team, and neither is organized around comparing one release against another. They answer a different question: is the fleet available, and did the rollout itself proceed. Use them to establish that the deploy landed and nothing is on fire, then use the team dashboards to look for regressions. + +## Scoping differs from every other dashboard + +`release-health` keys on `$organization`, an organization id rather than a namespace, and constructs namespaces from it as `environment-$organization-0` or `environment-${organization}-.`. It also takes `$mz_cluster`, the EKS cluster, and `$version`. Many of its panels are deliberately fleet-wide with no environment filter at all, because they measure regional availability. + +`networking` keys on `$namespace`, `$pod`, and `$tenant`. Most of its panels are node-level or cluster-level rather than environment-level. + +Nearly every `release-health` panel excludes end-to-end test organizations with `mz_context_org_type!="e2e_test"`. Reproduce that exclusion or the availability percentages will be wrong. + +## Roster: release-health + +### Rollout progress + +These come from the cloud control plane, not from Materialize, and carry no namespace label. + +| Metric | Type | Notes | +|---|---|---| +| `running_deploys` | gauge, by `deploy_id`, `mz_cloud_stack_name`, `state` | One series per in-flight deploy. | +| `running_environment_rollouts` | gauge, by `deploy_id`, `state`, `organization_id` | Panels exclude `state="Deprovisioned"`, and the per-environment view filters `state="RollingOut"`. | +| `running_node_replacements` | gauge, by `deploy_id`, `state`, `node_id` | Panels exclude `NoChanges` and `Deleted`. | +| `environmentd_needs_update` | gauge | Count of `environmentd` instances pending a restart. The cleanest single indicator of rollout progress: it jumps to the fleet size when a deploy starts and returns to zero when it finishes. | + +### Availability + +| Metric | Type | Notes | +|---|---|---| +| `v2_mz_can_connect` | gauge, 0 or 1 | Drives `Minutes of SLA downtime`. | +| `v2_mz_views_query_successful` | gauge, 0 or 1 | Drives `Minutes of SHOW VIEWS downtime`. | +| `mz_external_envd_up` | gauge, 0 or 1 | Reachability from outside the VPC, averaged by `mz_aws_region`. | +| `v2_mz_envd_up` | gauge | Also on the adapter dashboard. | +| `mz_envd_up` | gauge, with `mz_version` | Used only by the CPU-throttling panel, which compares the set of organizations up an hour ago against the set up now via `unless`. | + +The downtime panels compute `sum_over_time((1 - max by (org) (metric))[$__range:])`. That counts samples where the metric was zero, so the result is only "minutes" if the scrape interval is one minute. Read it as a sample count and convert deliberately. + +### Restarts and crashloops + +| Metric | Notes | +|---|---| +| `kube_pod_container_status_restarts_total` | Split into `container!~"clusterd"` and `container="clusterd"` panels, each divided by 100 in the display. | +| `kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}` | The direct crashloop indicator. | +| `kube_pod_container_status_last_terminated_exitcode` | The non-clusterd panel joins against `!= 166`, so exit code 166 is treated as an expected termination here. Note the contrast with the compute dashboard, which looks for 137, an OOM kill. | +| `kube_pod_created` | Used by `New clusterd restarts` to bound pod age. | + +`New clusterd restarts` is the most intricate expression on the dashboard. It selects pods younger than 86400 seconds that have restarted, strips the generation suffix into a `pod_base` label, and joins that against the same `pod_base` in an older generation which did not restart in the last 12 hours. The effect is to surface restarts that are new to the current generation, which is exactly the release question. It is also fragile, because it depends on the pod name matching `.*-(cluster-.*)-gen-([0-9]+)-[0-9]+$`. + +### Rehydration + +| Metric | Notes | +|---|---| +| `mz_dataflow_initial_output_duration_seconds` | Present once a dataflow has produced output. `sgn()` of it over the count of series gives the hydrated fraction. | +| `mz_compute_collection_count` | Carries a `hydrated` label. The by-generation panel extracts `gen` from `environmentd_materialize_cloud_service_id` with the pattern `.*-replica-.*-gen-(.*)`, which is how the new generation's hydration is separated from the old one's. | + +This is the only place hydration progress is broken out per generation, which makes it the right dashboard for answering whether the new version hydrated as fast as the old one. + +### External dependencies + +`crdb_dedicated_sys_cpu_combined_percent_normalized` and `crdb_dedicated_capacity_used / crdb_dedicated_capacity` describe the regional CockroachDB cluster and cannot be attributed to one environment. `mz_parameter_frontend_last_cse_time_seconds` and `_last_sse_time_seconds` are LaunchDarkly sync freshness, compared as `timestamp(metric) - metric` against 60 or 600 seconds, so they measure staleness rather than a value. `mz_connection_status` by `status` and `source` covers upstream connection health. `cilium_bpf_map_pressure` and `container_tasks_state{state="running"}` cover the data plane. + +## Roster: networking + +| Metric | Type | Notes | +|---|---|---| +| `mz_balancer_connection_status` | counter, by `status` and `source` | `source` is `pgwire` or `https`; `status` is `success` or `error`. | +| `mz_balancer_connection_active` | gauge, by `source` | | +| `mz_balancer_metadata_seconds` | gauge | Counting series gives the balancer count, the same trick persist uses. | +| `mz_balancer_tenant_connection_active`, `_rx`, `_tx` | gauge, counter, by `tenant` | Per-tenant balancer traffic. | +| `mz_auth_session_request_count` | counter, by `existing_session` | Values `new`, `active`, `pending`. The panel titles explain the balancerd session cache semantics in full and are worth reading once. | +| `mz_external_calls_count` | counter, by `status`, `connection_type`, `job` | The uptime-checker panels filter `job="external-uptime-checker"`. | +| `mz_external_long_lived_connection_count` | counter, by `status` | | +| `mz_cloud_egress_check_reachability_count` | counter, by `k8s_app`, `result` | Synthetic egress reachability. | +| `cilium_drop_count_total` | counter, by `direction`, `reason` | The panel excludes system clusters with `mz_cluster!~".*-sys"`. | +| `cilium_bpf_map_pressure`, `cilium_bpf_map_ops_total` | gauge, counter, by `map_name` | | +| `cilium_endpoint_regeneration_time_stats_seconds_bucket` | histogram, by `scope` | | +| `cilium_datapath_signals_handled_total` | counter, by `signal` | Panel excludes `status="muted"`. | +| `cilium_node_connectivity_latency_seconds` | gauge, by `protocol` | | +| `container_network_receive_errors_total`, `_transmit_errors_total`, `_receive_packets_dropped_total`, `_transmit_packets_dropped_total` | counter | Pod and node network faults. | +| `node_network_receive_bytes_total`, `_transmit_bytes_total` | counter, `device="eth0"` | Egress gateway traffic. | +| `node_netstat_Tcp_InErrs`, `node_netstat_TcpExt_TCPTimeouts` | counter | Node TCP errors and timeouts. | +| `machine_memory_bytes` | gauge, `workload="materialize-egress"` | | + +Three panels hardcode organization ids in `materialize_cloud_organization_id=~"3b1aeb7c-...|b65cc970-..."`, one staging environment and the production analytics canary. These will rot when the canary set changes, and they are not driven by the dashboard variables. + +## Hazards and invariants + +Each entry states a property that holds at any fleet size, followed by the measurement it came from. The property is what survives a release. The measurement is dated, describes whatever fleet existed when it was taken, and is recorded only so the property is not mistaken for a guess. + +**The egress panels use `^0` as a set-membership filter.** Expressions such as `... * on(node) group_left (workload) (sum by (node, workload) (rate(...)) ^0)` raise the right side to the power zero, which yields 1 for every series that exists and drops nodes where it does not. It is an intersection filter, not arithmetic, so do not simplify it away. + +**Availability metrics are booleans averaged into percentages.** A single environment failing for one scrape moves the fleet percentage by a fraction of a percent, which is why the numbers read as 99.98 rather than 99. Convert a dip into affected environment-samples before deciding whether it matters. + +**Downtime panels count samples, not minutes.** They count samples where the metric was zero, so their unit is only minutes if the scrape interval is one minute. + +**Crashloop series are absent when there are none.** Both the clusterd and non-clusterd crashloop queries returned no series at all across the whole window in production us-east-1. Absent is the healthy case and must not be reported as "checked and zero" without confirming the metric exists elsewhere. + +**`environmentd_needs_update` is the rollout clock.** It jumps to the fleet size when a deploy starts and returns to zero when it finishes. It sat at zero except for one bucket at 74.4 during the production rollout, and under 1.0 during canary upgrades. If it is non-zero when you begin, the rollout you are evaluating has not completed and nothing else on the dashboard means what you think it does. + +**Balancer errors are not rare.** Production us-east-1 sustained 2.9 to 5.3 errors per second against 5.2 to 21.3 successes, an error fraction around a half, flat across the release boundary. Whatever `status="error"` counts on the balancer, it is a normal part of steady state here, so only a change in the ratio is informative. + +**Exit code 166 is treated as an expected termination on this dashboard,** in contrast to 137 as an OOM kill on the compute dashboard. Do not carry one dashboard's convention into the other. + +**`crdb_dedicated_*` metrics are regional and shared across all environments.** They cannot be attributed to the release. + +**`mz_balancer_metadata_seconds` has no useful value of its own.** Counting its series gives the balancer count. + +## Order of magnitude + +Recorded 2026-08 for scope-checking only. + +* Availability metrics sit at 100% and dip to 99.98% for a single bucket during a fleet rollout. +* Non-clusterd restarts, external call failures, and dropped packets are at or near zero, with isolated single-bucket blips. +* CRDB CPU runs around half its normalized capacity, and disk under 5%. +* Cilium drop rates and pod network error rates are single-digit to low tens per second region-wide, and noisy. Treat them as a floor to compare against, not a threshold. + +Staging was not measured for these two dashboards. Their availability and control-plane metrics describe a fleet nobody is paged for, so a staging baseline would carry little weight. The rollout-progress metrics are worth checking there only to confirm a deploy finished. diff --git a/.agents/skills/mz-release-signoff/references/sources-and-sinks.md b/.agents/skills/mz-release-signoff/references/sources-and-sinks.md new file mode 100644 index 0000000000000..a61d1ecbd2524 --- /dev/null +++ b/.agents/skills/mz-release-signoff/references/sources-and-sinks.md @@ -0,0 +1,122 @@ +# Sources and Sinks metric reference + +Two dashboards. `storage-overview`, UID `e6dc7745-7d35-4968-a23d-689883a984bb`, is the sweep. `storage-upsert-sources`, UID `ac2de0ab-4a35-48b5-93aa-7e645569debb`, is a drill-down that requires a specific pod and source, and its own panel titles say so. Run the sweep first and only open the upsert dashboard when a source-level signal needs explaining. + +## Variables + +The two dashboards disagree on naming, and neither matches the compute dashboard. + +| Dashboard | Environment variable | Others | +|---|---|---| +| `storage-overview` | `$env` | `$cluster`, `$replica`, `$object_id`, `$pod` | +| `storage-upsert-sources` | `$namespace` | `$pod`, `$source` | + +Within `storage-overview` the environment matcher is itself inconsistent: some panels use `namespace=~"$env"` and others `namespace=~".*$env.*"`. The behaviour is the same for a full namespace value, so this is untidy rather than wrong. + +## This dashboard has its own sign-off panel + +`storage-overview` carries a `Signing off on Releases` text panel with instructions that differ from the compute one. Read it each run. As of 2026-08-19 it says to check all three `cloud-staging` datasources plus the production sandbox environment on all three `cloud-production` datasources, to include at least one day before the canary release, and to look for discrepancies in source and sink counts, new replicas with short uptime, regressed memory and CPU, and drastic changes in source, sink, and command statistics. + +Two corrections to that panel. It points at `MaterializeInc/cloud/.github/ISSUE_TEMPLATE/03-release.md` for the canary environment list, and that file no longer exists; the canonical list now lives in `MaterializeInc/release/templates/issue.md`, in the `bin/deploy upgrade production` invocation. It also names only the sandbox environment, while the release bot links both sandbox and analytics for us-east-1, and Prometheus shows both running the release candidate. + +## Roster + +### Storage objects + +| Metric | Type | Notes | +|---|---|---| +| `v2_mz_storage_objects` | gauge | Labels `id`, `type` (`source` or `sink`), `object_type`, `envelope_type`, `networking_type`, `cluster_id`, `replica_id`. `replica_id="none"` means the object has no replicas, which is how the active panels filter. | + +Counting these needs the dashboard's inner aggregation, or multi-replica objects are counted once per replica: + +```promql +sum(group by (id, namespace) (v2_mz_storage_objects{namespace=~"...", type="source"})) +``` + +The panel comments attribute this guard to a past double-counting bug, so keep it even when it looks redundant. + +### Source statistics + +| Metric | Type | Notes | +|---|---|---| +| `mz_source_messages_received`, `mz_source_bytes_received` | counter | Upstream read volume. | +| `mz_source_updates_staged`, `mz_source_updates_committed` | counter | Persist sink progress. Staged above committed is normal transiently, and a persistent gap is workload-dependent rather than automatically a fault. | +| `mz_source_offset_known`, `mz_source_offset_committed` | gauge | Upstream lag. Summing across sources gives a crude fleet lag proxy only. | +| `mz_source_progress` | gauge, milliseconds | A frontier timestamp, not a counter. See hazards. | +| `mz_source_snapshot_records_known`, `mz_source_snapshot_records_staged` | gauge | Snapshot progress, the ratio being the percentage panel. | +| `mz_source_rehydration_latency_ms` | gauge | An alternative to the upsert-specific rehydration metric, per the panel comment. | + +Label naming is not consistent: the rate panels filter `parent_source_id` while the gauge panels filter `source_id`. Copy whichever the panel uses. + +### Upsert and envelope state + +| Metric | Type | Notes | +|---|---|---| +| `mz_source_records_indexed`, `mz_source_bytes_indexed` | gauge | Envelope state size. Their ratio is the average record size panel. | +| `mz_storage_upsert_deletes_total`, `mz_source_envelope_state_tombstones` | counter, gauge | | +| `mz_storage_upsert_state_rehydration_latency` | gauge, milliseconds | Last rehydration duration. See hazards. | +| `mz_storage_upsert_state_rehydration_total`, `_rehydration_updates` | counter | Records and updates replayed. Their ratio exposes retraction-heavy state. | +| `mz_storage_upsert_state_rocksdb_autospill_in_use` | gauge, one per worker | The percentage panel divides `sum` by `count`, so it reports the fraction of workers spilling. | +| `mz_storage_upsert_backpressure_emitted_bytes`, `_retired_bytes`, `_last_backpressured_bytes` | counter, gauge | Backpressure. | +| `mz_storage_upsert_merge_snapshot_latency_bucket` | histogram | | +| `mz_storage_upsert_merge_snapshot_updates_total`, `_inserts_total`, `_deletes_total` | counter | | +| `mz_storage_rocksdb_multi_get_latency_bucket`, `_multi_put_latency_bucket` | histogram | Panels plot p95, p99, and p999. | +| `mz_storage_rocksdb_multi_get_count_total`, `_size_total`, `_result_bytes_total`, `_result_count_total`, and the `multi_put` equivalents | counter | Batch sizes are the size over the count. | +| `mz_persist_shard_update_count` | gauge, by `shard` and `name` | Filtered by `name="$source"`, so it joins a source to its persist shard. | + +### Sink statistics + +| Metric | Type | Notes | +|---|---|---| +| `mz_sink_messages_staged`, `mz_sink_messages_committed` | counter | | +| `mz_sink_bytes_staged`, `mz_sink_bytes_committed` | counter | | +| `mz_sink_oustanding_progress_records` | gauge | The metric name carries a spelling error, `oustanding`. `mz_sink_consumed_progress_records` also exists and is not on the dashboard. | +| `mz_sink_partition_count` | gauge | The panel aggregates with `max` and no namespace filter beyond the grouping. | + +### Shard finalization + +`mz_shard_finalization_outstanding` and `mz_shard_finalization_pending_commit` are gauges; `mz_shard_finalization_op_started`, `_op_succeeded`, and `_op_failed` are counters. The started and succeeded panels wrap `rate` in `max by (namespace, pod)`, which is unusual but harmless. + +### Controller protocol + +The two panels each carry an `or` fallback that straddles a metric rename: + +```promql +sum by (namespace) (rate(mz_storage_messages_sent_bytes_count{...}[...])) +or +sum by (namespace) (rate(mz_storage_commands_total{...}[...])) +``` + +Only the second arm resolves today. `mz_storage_messages_sent_bytes_count`, `_received_bytes_count`, `_sent_bytes_sum`, and `_received_bytes_sum` no longer exist. The live names are `mz_storage_commands_total`, `mz_storage_responses_total`, `mz_storage_command_message_bytes_total`, and `mz_storage_response_message_bytes_total`. Do not read the `or` as a bug; it is a deliberate hedge so the panel keeps working across the version boundary, and it is worth imitating. + +### Basic health + +This row selects every container in the matched pods, via `container!="POD", container!=""` and a `$pod` prefix, rather than just `clusterd`. It is therefore not comparable to the compute dashboard's process row, which pins `container="clusterd"`. + +`Container Max RSS Memory Usage` has two series. The `mz_metrics_libc_ru_maxrss` one, multiplied by 1024 to convert from KiB, refers to a metric that no longer exists; the `mz_metrics_libc_ru_maxrss_bytes` one works. + +## Hazards and invariants + +Each entry states a property that holds at any fleet size, followed by the measurement it came from. The property is what survives a release. The measurement is dated, describes whatever fleet existed when it was taken, and is recorded only so the property is not mistaken for a guess. + +**`mz_source_progress` is a frontier timestamp, so its rate is a health check, not a throughput.** The metric holds a millisecond epoch, so a healthy advancing frontier yields a rate of exactly 1000 per series. Production canary measured a flat 87000, meaning 87 series advancing at wall-clock rate. A value below the expected multiple of 1000 means some frontiers are stalled, and the multi-hundred-million spikes seen in every upgrade bucket are frontier reinitialization, not progress. Judge this panel by counting series, never by the absolute number. + +**Rehydration latency is a last-value gauge.** `mz_storage_upsert_state_rehydration_latency` holds the most recent rehydration's duration and does not decay, so it forms a staircase with one step per upgrade. That makes it the most useful single release signal here, because each step is a direct measurement of the new version rehydrating. A flat line between upgrades carries no information. + +**Upsert state is absent from staging.** `mz_source_records_indexed` and `mz_source_bytes_indexed` summed to exactly zero across staging us-east-1 for the whole window, and the RocksDB and rehydration series were missing entirely. The upsert dashboard can only be verified from the production sandbox, which is why the panel instructions emphasize it. + +**Staged and committed update rates differ by environment class.** Production canary measured them equal at about 145 per second. Staging measured 9.59 staged against 6.60 committed, a persistent 45% gap that was stable across the boundary. Compare each stack against its own history rather than expecting the two counters to track. + +**Object counts must be aggregated with the inner `group by (id, ...)` guard.** Without it, multi-replica objects are counted once per replica. + +**Shard finalization failures should be zero.** Outstanding and pending counts spike at upgrades and then drain. + +**The version panel is known broken.** `storage-overview` titles its version panel `Materialize Version (currently broken?)`. Use the compute dashboard's version panel, or the query in the skill's step 1, to establish boundaries. + +## Order of magnitude + +Recorded 2026-08 for scope-checking only. + +* Production canary carries tens of sources and a handful of sinks. Staging carries a similar number of sources and very few sinks. +* Upstream read rates are order a hundred messages per second in production canary and order ten in staging. +* Envelope state in the production sandbox is order a billion records and a hundred GB. Single-digit GB means the selector is wrong. diff --git a/.agents/skills/mz-release-signoff/scripts/build-range-query.py b/.agents/skills/mz-release-signoff/scripts/build-range-query.py new file mode 100644 index 0000000000000..5686b29e9545d --- /dev/null +++ b/.agents/skills/mz-release-signoff/scripts/build-range-query.py @@ -0,0 +1,118 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +"""Build one batched PromQL range query from a roster of metrics. + +Step 5 of the skill batches many metrics into a single query by tagging each +aggregate with a synthetic label and combining with `or`. Twenty of those +written by hand is where typos live, and a typo here reads as a healthy zero. + +The roster is a text file, one metric per line: + + c cmd_started mz_persist_cmd_started_count + c columnar_invalid mz_persist_columnar_op_count op="validation",result="invalid" + g arr_records v2_mz_arrangement_record_count + +The first field is the type, `c` for a counter, which becomes `rate`, or `g` +for a gauge, which becomes `avg_over_time`. The second is the tag that labels +the result series, and it is prefixed with its position so the output order is +stable. The third is the metric. Anything after that is an extra selector. +Blank lines and `#` comments are ignored. + +The stacks differ in exactly one way, which is how the fleet is selected, so +that is the only thing `--stack` changes. Staging joins against the +release-candidate version, which also drops the development environments +sharing the stack. Production pins the canary namespaces by name, because they +run the plain released version alongside customer environments for part of the +week and a version filter loses them. + + $ build-range-query.py --stack staging roster.txt + $ build-range-query.py --stack prod --namespaces environment-aaa-0,environment-bbb-0 roster.txt +""" + +import argparse +import sys + +RC_JOIN = ( + "and on(namespace) group by (namespace) " + '(v2_mz_compute_cluster_status{{mz_version=~".*-rc[.].*"}})' +) + + +def parse_roster(text): + """Yield (kind, tag, metric, selector) for each roster entry.""" + for lineno, raw in enumerate(text.splitlines(), 1): + line = raw.split("#", 1)[0].strip() + if not line: + continue + parts = line.split(None, 3) + if len(parts) < 3: + raise SystemExit( + f"roster line {lineno}: expected `kind tag metric [selector]`, got {raw!r}" + ) + kind, tag, metric = parts[0], parts[1], parts[2] + if kind not in ("c", "g"): + raise SystemExit( + f"roster line {lineno}: kind must be `c` or `g`, got {kind!r}" + ) + yield kind, tag, metric, parts[3] if len(parts) > 3 else "" + + +def aggregate(kind, metric, selector, window, stack, namespaces): + selectors = [selector] if selector else [] + if stack == "prod": + # A regex alternation keeps the canary set readable and keeps the + # expression valid when only one namespace is pinned. + alternation = "|".join(namespaces) + selectors.append(f'namespace=~"{alternation}"') + inner = f"{metric}{{{','.join(selectors)}}}" if selectors else metric + fn = ( + f"rate({inner}[{window}])" + if kind == "c" + else f"avg_over_time({inner}[{window}])" + ) + if stack == "prod": + return f"sum({fn})" + return f"sum({fn} {RC_JOIN.format()})" + + +def main(): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("roster", help="roster file, or - for stdin") + parser.add_argument("--stack", choices=("staging", "prod"), required=True) + parser.add_argument( + "--namespaces", + default="", + help="comma-separated canary namespaces, required for --stack prod", + ) + parser.add_argument( + "--window", + default="6h", + help="rate and avg_over_time window, which must match the query step (default 6h)", + ) + args = parser.parse_args() + + namespaces = [n.strip() for n in args.namespaces.split(",") if n.strip()] + if args.stack == "prod" and not namespaces: + parser.error("--stack prod requires --namespaces") + + text = sys.stdin.read() if args.roster == "-" else open(args.roster).read() + + parts = [] + for i, (kind, tag, metric, selector) in enumerate(parse_roster(text), 1): + agg = aggregate(kind, metric, selector, args.window, args.stack, namespaces) + parts.append(f'label_replace({agg}, "m", "{i:02d}_{tag}", "", "")') + + if not parts: + raise SystemExit("roster is empty") + print(" or ".join(parts)) + + +if __name__ == "__main__": + main() diff --git a/.agents/skills/mz-release-signoff/scripts/catalog-diff.sh b/.agents/skills/mz-release-signoff/scripts/catalog-diff.sh new file mode 100755 index 0000000000000..beb03e43dae06 --- /dev/null +++ b/.agents/skills/mz-release-signoff/scripts/catalog-diff.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash + +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +# catalog-diff.sh — which metrics changed between two releases. +# +# doc/user/data/metrics.yml is checked in, so it is tagged along with each +# release. Reading it at the two tags on either side of a sign-off boundary +# answers whether an empty panel is a real signal or a metric that arrived or +# departed with the release. The working tree's copy describes main, which is +# ahead of whatever is being verified, so do not read that one during a run. +# +# $ .agents/skills/mz-release-signoff/scripts/catalog-diff.sh v26.38.0 v26.39.0-rc.3 +# +# Covers only metrics registered by `metric!` in the Rust tree. It says nothing +# about v2_mz_*, container_*, kube_* or kubelet_*, which are exported by the +# promsql exporter, cAdvisor, and kube-state-metrics respectively. Confirm +# those against Prometheus instead. + +set -euo pipefail + +# sort and comm must agree on collation, and the ambient locale is not C on +# every machine that runs this. +export LC_ALL=C + +if [[ $# -ne 2 ]]; then + echo "usage: $(basename "$0") " >&2 + exit 2 +fi + +before=$1 +after=$2 + +# comm compares under the shell's collation, so sort here rather than relying +# on the extractor's own ordering. +names() { + git show "$1:doc/user/data/metrics.yml" \ + | python3 "$(dirname "$0")/catalog_names.py" \ + | LC_ALL=C sort +} + +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT + +names "$before" > "$tmp/before" +names "$after" > "$tmp/after" + +printf '%s: %s metrics\n' "$before" "$(wc -l < "$tmp/before")" +printf '%s: %s metrics\n\n' "$after" "$(wc -l < "$tmp/after")" + +echo "Added in $after (a before-window on these is empty by construction):" +comm -13 "$tmp/before" "$tmp/after" | sed 's/^/ + /' +echo + +echo "Removed in $after (an after-window on these is empty by construction):" +comm -23 "$tmp/before" "$tmp/after" | sed 's/^/ - /' diff --git a/.agents/skills/mz-release-signoff/scripts/catalog_names.py b/.agents/skills/mz-release-signoff/scripts/catalog_names.py new file mode 100644 index 0000000000000..24c2d232a5f74 --- /dev/null +++ b/.agents/skills/mz-release-signoff/scripts/catalog_names.py @@ -0,0 +1,34 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +"""Read a metrics catalog on stdin and print its metric names, sorted. + +The catalog is doc/user/data/metrics.yml, generated by bin/gen-metrics-catalog. +Parsing it with a regex rather than a YAML library keeps this dependency-free, +which matters because it runs against arbitrary git refs during a sign-off. +""" + +import re +import sys + +NAME = re.compile(r"^- name: '?(.+?)'?$") + + +def main() -> None: + names = set() + for line in sys.stdin: + match = NAME.match(line.rstrip("\n")) + if match: + names.add(match.group(1)) + for name in sorted(names): + print(name) + + +if __name__ == "__main__": + main() diff --git a/.agents/skills/mz-release-signoff/scripts/lint_metrics.py b/.agents/skills/mz-release-signoff/scripts/lint_metrics.py new file mode 100644 index 0000000000000..64d73fdc19728 --- /dev/null +++ b/.agents/skills/mz-release-signoff/scripts/lint_metrics.py @@ -0,0 +1,130 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +"""Check that every `mz_*` metric named by the skill resolves in the catalog. + +Run from the repository root. Exits non-zero and names the offenders on failure. +""" + +import fnmatch +import pathlib +import re +import sys + +SKILL = pathlib.Path(".agents/skills/mz-release-signoff") +CATALOG = pathlib.Path("doc/user/data/metrics.yml") +ALLOWLIST = SKILL / "scripts" / "metrics-allowlist.txt" + +CATALOG_NAME = re.compile(r"^- name: '?(.+?)'?$") +BACKTICKED = re.compile(r"`([^`]+)`") +CANDIDATE = re.compile(r"\b(mz_[a-z0-9_]+)\b") + +# Histograms and summaries are catalogued as their expanded families, so a +# reference naming the base is correct and must resolve through any suffix. +SUFFIXES = ("_bucket", "_count", "_sum") + + +def catalog_names(): + """Return (exact names, glob patterns) from the catalog. + + A `metric!` whose name is built with `format!` is catalogued with its + placeholders globbed, for example `mz_persist_*_bytes`, so the catalog is + a mix of literal names and patterns and membership is not a set lookup. + """ + exact, globs = set(), set() + for line in CATALOG.read_text().splitlines(): + match = CATALOG_NAME.match(line) + if match: + name = match.group(1) + (globs if "*" in name else exact).add(name) + # A base name is resolvable when any member of its family is catalogued. + bases = { + name.rsplit("_", 1)[0] + for name in exact + if name.rsplit("_", 1)[-1] in ("bucket", "count", "sum") + } + return exact | bases, globs + + +def resolves(name, exact, globs): + if name in exact: + return True + # Try the histogram and summary suffixes against the patterns too, so a + # reference naming the base of a globbed family still resolves. + candidates = [name] + [name + suffix for suffix in SUFFIXES] + return any( + fnmatch.fnmatchcase(candidate, pattern) + for candidate in candidates + for pattern in globs + ) + + +def allowlisted(): + names = set() + for line in ALLOWLIST.read_text().splitlines(): + line = line.split("#", 1)[0].strip() + if line: + names.add(line) + return names + + +def referenced(): + """Yield (name, file) for every mz_* token inside backticks in the skill.""" + for path in sorted(SKILL.rglob("*.md")): + for token in BACKTICKED.findall(path.read_text()): + # A prefix wildcard such as `mz_persist_*` names a family, not a + # metric, and its stem is not itself catalogued. Nothing here can + # be checked, so skip the whole token. + if "*" in token: + continue + for name in CANDIDATE.findall(token): + # Brace expansion such as `mz_foo_{sum,count}` leaves a + # trailing underscore. Resolve the base, which the catalog + # holds as an expanded family. + yield name.rstrip("_"), path + + +def main() -> int: + if not CATALOG.exists(): + print(f"{CATALOG} not found; run from the repository root", file=sys.stderr) + return 2 + + exact, globs = catalog_names() + allowed = allowlisted() + + unresolved = {} + for name, path in referenced(): + if name in allowed or resolves(name, exact, globs): + continue + unresolved.setdefault(name, set()).add(str(path)) + + stale = sorted(n for n in allowed if resolves(n, exact, globs)) + if stale: + print("Allowlisted names that now resolve in the catalog; remove them:") + for name in stale: + print(f" {name}") + print() + + if unresolved: + print( + "Metric names in the skill that resolve in neither the catalog nor the allowlist:" + ) + for name in sorted(unresolved): + print(f" {name} ({', '.join(sorted(unresolved[name]))})") + print() + print("Either the metric was renamed, in which case fix the reference, or the") + print( + f"catalog cannot see it, in which case add it to {ALLOWLIST} with a reason." + ) + + return 1 if (unresolved or stale) else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.agents/skills/mz-release-signoff/scripts/metrics-allowlist.txt b/.agents/skills/mz-release-signoff/scripts/metrics-allowlist.txt new file mode 100644 index 0000000000000..002e5096671fc --- /dev/null +++ b/.agents/skills/mz-release-signoff/scripts/metrics-allowlist.txt @@ -0,0 +1,61 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +# Metric names used by the mz-release-signoff skill that doc/user/data/metrics.yml +# does not list. One name per line, grouped by why. +# +# This file is the point of the lint. A name here is a deliberate exemption with +# a stated reason. A name that is neither resolvable in the catalog nor listed +# here fails CI, which is what catches a metric that was renamed or removed. +# +# The catalog does cover names built with `format!`, storing them with their +# placeholders globbed, for example `mz_persist_*_bytes`. The lint matches those +# patterns, so a dynamically named metric needs no entry here. +# +# Before adding a name, establish which group it belongs to. If it belongs to +# none of them, the reference is wrong and the fix is to correct the reference. + +# --- Exported by the promsql exporter from SQL, not registered in Rust --- +# Every v2_mz_* metric. The lint only considers mz_* names, so the prefix is +# skipped wholesale and nothing is listed here. + +# --- Cloud-side, exported outside this repository --- +mz_envd_up +mz_external_envd_up +mz_external_calls_count +mz_external_long_lived_connection_count +mz_cloud_egress_check_reachability_count +mz_dataflow_initial_output_duration_seconds +mz_storage_messages_sent_bytes_count + +# --- Labels and identifiers that merely look like metric names --- +mz_version +mz_cluster +mz_aws_region +mz_cloud_stack_name +mz_context_org_name +mz_context_org_type +mz_catalog + +# --- Dead dashboard panels, documented as defects rather than as metrics --- +# Each is named in a reference because a live dashboard panel plots it and so +# renders empty, which reads as health. The reference records the defect. Remove +# the entry when the panel is fixed. +# +# mz_persist_columnar_validation_count and mz_txn_placeholder_schema_apply are +# both arms of the persist dashboard's `should be small` panel. Neither name has +# ever appeared in this repository. The working spelling sits beside the first of +# them on the same panel, as mz_persist_columnar_op_count with op="validation" +# and result="invalid". +# +# mz_query_latency did exist. It was added in #22049 and deleted in #26647 with +# the stash, and the environmentd-health panel still plots its _bucket family. +mz_persist_columnar_validation_count +mz_txn_placeholder_schema_apply +mz_query_latency_bucket diff --git a/.agents/skills/mz-release-signoff/scripts/panel-metrics.py b/.agents/skills/mz-release-signoff/scripts/panel-metrics.py new file mode 100644 index 0000000000000..abc1ab12c0f7b --- /dev/null +++ b/.agents/skills/mz-release-signoff/scripts/panel-metrics.py @@ -0,0 +1,95 @@ +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +"""Extract metric names and label selectors from a dashboard panel-query dump. + +`mcp__grafana__get_dashboard_panel_queries` returns about 62 KB across 165 +panels for the compute dashboard, which overflows the tool result and is +written to a file instead. That file must never be read whole or pasted into +the conversation. Slice it with this: + + $ panel-metrics.py panels.json # metric -> panels that use it + $ panel-metrics.py panels.json --selectors # also show label selectors + $ panel-metrics.py panels.json --names-only # bare names, for a roster + +The output is a superset of what the build exports, because a panel outlives +the metric it plots. Resolve the names against the catalog for the release +under test before trusting them. +""" + +import argparse +import collections +import json +import re +import sys + +# A PromQL metric name, optionally followed by a label selector. Excludes +# names immediately preceded by a word character so that `foo_bucket` inside +# an already-matched token is not matched again. +METRIC = re.compile( + r"(?" + query = item.get("query") or item.get("processedQuery") or item.get("expr") + if query: + yield title, query + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("dump", help="JSON file written by get_dashboard_panel_queries") + parser.add_argument( + "--selectors", action="store_true", help="show label selectors too" + ) + parser.add_argument( + "--names-only", action="store_true", help="print bare metric names" + ) + args = parser.parse_args() + + with open(args.dump) as f: + doc = json.load(f) + + uses = collections.defaultdict(set) + selectors = collections.defaultdict(set) + for title, query in panels(doc): + for name, selector in METRIC.findall(query): + if name in NOT_METRICS: + continue + uses[name].add(title) + if selector: + selectors[name].add(selector) + + if not uses: + print("no metric names found; check the dump shape", file=sys.stderr) + return 1 + + for name in sorted(uses): + if args.names_only: + print(name) + continue + print(f"{name} ({len(uses[name])} panels)") + if args.selectors: + for selector in sorted(selectors[name]): + print(f" {selector}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/ci/test/lint-docs.sh b/ci/test/lint-docs.sh index 0ea773b709bbe..5646dda8f0d68 100755 --- a/ci/test/lint-docs.sh +++ b/ci/test/lint-docs.sh @@ -22,5 +22,6 @@ echo "" > ci/www/public/index.html try htmltest -s ci/www/public -c doc/user/.htmltest.yml try ci/test/lint-docs-catalog.sh try ci/test/lint-metrics-catalog.sh +try ci/test/lint-skill-metrics.sh try_status_report diff --git a/ci/test/lint-skill-metrics.sh b/ci/test/lint-skill-metrics.sh new file mode 100755 index 0000000000000..d07dcc40a8b95 --- /dev/null +++ b/ci/test/lint-skill-metrics.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash + +# Copyright Materialize, Inc. and contributors. All rights reserved. +# +# Use of this software is governed by the Business Source License +# included in the LICENSE file at the root of this repository. +# +# As of the Change Date specified in that file, in accordance with +# the Business Source License, use of this software will be governed +# by the Apache License, Version 2.0. + +# Checks that every `mz_*` metric named by the mz-release-signoff skill still +# resolves against the generated metrics catalog, so the skill's references +# cannot rot silently when a metric is renamed. A reference that names a metric +# the product no longer exports is worse than no reference, because a release +# verifier reads the resulting empty query as a healthy zero. +# +# Names the catalog structurally cannot see are exempted one at a time in +# .agents/skills/mz-release-signoff/scripts/metrics-allowlist.txt, with a reason. +# +# Example usages: +# +# $ ci/test/lint-skill-metrics.sh + +set -euo pipefail + +. misc/shlib/shlib.bash + +ci_uncollapsed_heading "Linting mz-release-signoff metric names" + +try python3 .agents/skills/mz-release-signoff/scripts/lint_metrics.py + +try_status_report