console: show real source status on the Objects page

[leedqin]

Motivation

Sources on the maintained-objects (Objects) page were described in replica-hydration terms, which misrepresents them three ways:

• The status column showed Not Hydrated / Hydrating / Running buckets derived from mz_hydration_statuses. For a source, hydrated only tracks whether rehydration_latency is set, so a healthy source could read Not Hydrated (Slack: #console, Aug 19 thread on what "Not Hydrated" means).
• A source could show 99% snapshot progress next to 1s freshness because completion was judged by the staged/known ratio, whose denominator is an estimate that can sit below 100% forever.
• Webhook sources have no hydration rows at all, so they showed no status.

Separately, the source detail panel gave no way to see the ingest cluster's resources while diagnosing a lagging source.

Changes

One commit per concern, reviewable in order:

1. Source ingestion status on the Objects page. The status column, filter, and detail-panel badge show the source's ingestion status (snapshotting, running, paused, stalled, ...) via ConnectorStatusPill, matching the Sources page. Non-source objects keep the hydration buckets. The filter derives from the same function the cell renders, so it can never offer a value the cell doesn't show. Status rides on the existing hydration-aggregate subscribe: the aggregate now FULL JOINs mz_source_statuses (so webhook sources get a row) and carries bool_and(snapshot_committed) from mz_source_statistics.

2. Snapshot display keys off snapshot_committed. The diagnostics card uses the authoritative boolean instead of the staged/known ratio, and now renders only when it has something diagnostic to say: a status error, or an in-progress snapshot (with a per-source-type note that counts are estimates, shared with the Sources page via snapshotEstimateNote). Steady-state lifecycle facts (rehydration time, messages received) already live on the source details page, so the card no longer duplicates them.

3. Source cluster metrics on the Freshness tab. The detail panel shows the ingest cluster's replica memory/CPU at the point selected on the freshness graph, including replicas dropped in the window (reuses the critical-path cluster metrics via a new shared ClusterReplicaMetrics component; falls back to each replica's latest bucket when the anchor bucket predates a replica swap).

4. Feed hardening. Subsources and progress collections are excluded from the aggregate (the UI hides them, and they multiply the feed's cardinality several times over), and the snapshot_committed aggregate is restricted to statistics rows from live replicas (keeping null-replica_id rows, which is how webhook sources report), so a dropped replica's stale row cannot pin a committed source back to snapshotting.

Tests

Extended hydrationAggregate.test.sql.ts for the FULL JOIN, bool_and(snapshot_committed), subsource/progress exclusion, and live-replica semantics, and MaintainedObjects.test.tsx for the source status column and filter.

Closes CNS-97
Closes CNS-82
Closes CNS-85

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- bool_and(snapshot_committed) includes dropped replicas, pinning healthy sources to "Snapshotting"

console/src/api/materialize/maintained-objects/hydrationAggregate.ts:51

snapshotCommittedBySource aggregates mz_source_statistics with no filter on extant replicas, but that relation keeps per-replica rows for up to storage_statistics_retention_duration (default 24h) after a replica is dropped. One stale snapshot_committed = false row makes bool_and false forever, so a source that has long since finished its snapshot renders as Snapshotting (spinner pill) on the Objects page and shows a permanent "Snapshot progress" card, for a day.

Details

Reachable scenario, and a routine one: a Postgres/MySQL/SQL Server source's initial snapshot is slow, so the operator resizes the ingest cluster (or drops to REPLICATION FACTOR 0 and back) mid-snapshot. The dropped replica's row is stuck at false (its shard upper never advanced past MIN). The replacement replica re-snapshots and eventually reports true. bool_and(false, true) = false.

Evidence that these rows persist: drop_replica in src/storage-controller/src/lib.rs:626 writes paused status updates but never touches source_statistics; the only removal paths are collection drop (lib.rs:1998) and the inactivity retain in src/storage-controller/src/statistics.rs:135, bounded by STATISTICS_RETENTION_DURATION (src/storage-types/src/dyncfgs.rs:430, one day). Every other console consumer of this relation filters explicitly — sourceStatisticsQueryWithReplicaId joins mz_cluster_replicas and requires r.id is not null or ss.replica_id is null (console/src/api/materialize/source/sourceStatistics.ts:37-40), and buildFirstReplicaSourceStatisticsTable inner-joins mz_cluster_replicas with the comment "we only want statistics from active ones" (console/src/api/materialize/expressionBuilders.ts:189). This new aggregate is the only one that doesn't, and bool_and is maximally sensitive to a single stale false.

Note the effect is worse than the pre-PR display it replaces: instead of a wrong-but-static hydration bucket, the page asserts an in-progress snapshot that is not happening, and the Status filter classifies the source as snapshotting, so filtering to Running hides it.

Fix: apply the same extant-replica predicate the other consumers use, keeping replica_id IS NULL rows so webhook sources (which report replica_id = NULL, snapshot_committed = true) are unaffected.

const snapshotCommittedBySource = queryBuilder
  .selectFrom("mz_source_statistics as ss")
  // Rows for dropped replicas linger for up to a day, so a replica replaced
  // mid-snapshot would hold `bool_and` at false long after the snapshot lands.
  .leftJoin("mz_cluster_replicas as r", "r.id", "ss.replica_id")
  .where((eb) =>
    eb.or([eb("r.id", "is not", null), eb("ss.replica_id", "is", null)]),
  )
  .select((eb) => [
    "ss.id",
    sql<boolean | null>`bool_and(${eb.ref("ss.snapshot_committed")})`.as(
      "snapshotCommitted",
    ),
  ])
  .groupBy("ss.id");

2. MEDIUM -- Anchor-bucket fallback reports present-time replica metrics as the selected past timestamp's

console/src/platform/maintained-objects/CriticalPathClusterMetrics.tsx:357

When no replica has a metrics bucket at the selected anchor, replicas now falls back to each replica's latest bucket in the window. Buckets are sparse (only where samples exist) and useClusterBucketsInWindow passes no end date, so the fallback is the current bucket — presented under a header that reads "Cluster metrics for X at 3:45:12 PM". The state it is meant to describe, "the cluster had no replicas at the anchor", is exactly the state that now renders as healthy live metrics.

Details

Concretely: lock the freshness graph on a bucket during which the ingest/compute cluster sat at REPLICATION FACTOR 0 (or before the current replica was created). atAnchor is empty, so the panel shows the dropped replica's last-seen numbers plus the new replica's current numbers, both labelled at the anchor time. DroppedReplicasFooter also goes silent, because lockedReplicas is now identical to replicasInWindow, so dropped is always empty (line 376-379) — the "N replicas dropped in this window" callout disappears precisely when it is the answer. Before this change the panel said "Inactive — replication factor 0", which was correct. An operator asking "why was this lagging at 3:45?" is shown evidence that the cluster was fine.

The live case the fallback targets (current partial bucket not yet populated) is a real bug, but it only applies when timestamp === null. Gating the fallback on that, rather than on atAnchor.length === 0, keeps both behaviours correct.

3. LOW -- Status column sorts sources by hydration ratio while displaying ingestion status

console/src/platform/maintained-objects/MaintainedObjects.tsx:355

The status column's accessor is still hydratedReplicas / totalReplicas, but the cell now renders ConnectorStatusPill for sources. Sorting by Status therefore orders sources by a value the column no longer shows: a stalled source whose replica reports hydrated (ratio 1) sorts among Hydrated objects, a snapshotting one (ratio 0) among Not Hydrated, and webhook sources (no hydration rows, ratio null) sort last regardless of status. Sorting to group broken objects together no longer does that. statusBucketForRow already computes the displayed bucket and could drive the accessor.

[def-]

QA LLM Review

1. MEDIUM -- Subsources are not excluded from the feed; they re-enter through the hydration side of the FULL JOIN

console/src/api/materialize/maintained-objects/hydrationAggregate.ts:59

The new type not in ('subsource', 'progress') predicate only filters the status side of the FULL JOIN, but subsources have their own rows in mz_hydration_statuses, so each one still produces a feed row (now with a null sourceStatus). Only progress collections are actually removed, one per source, so the "multiply the feed's cardinality several times over" cardinality the change targets is left in place.

Details

mz_hydration_statuses builds its source rows from sources_with_clusters AS (SELECT id, cluster_id FROM mz_catalog.mz_sources WHERE cluster_id IS NOT NULL AND type != 'webhook') (src/catalog/src/builtin/mz_internal.rs:7310). Subsources are ingestion exports and mz_sources.cluster_id resolves theirs from the parent source through of_source_id (src/catalog/src/builtin/builtin.rs:394-403); the imperative code this materialized view replaced said so directly, "Ingestion exports don't have their own cluster, but run on their ingestion's cluster". So a legacy Postgres or MySQL source with N tables contributes N subsource rows to h, and coalesce(h.object_id, ss.id) emits every one of them. Progress collections are the case that does work: DataSourceDesc::Progress keeps its own cluster id, which is None, so mz_sources.cluster_id is NULL and they never reach mz_hydration_statuses.

The accompanying test cannot catch this. hydrationAggregate.test.sql.ts:200 seeds only statuses for its subsource fixture and leaves mz_hydration_statuses empty, which is the one shape where the FULL JOIN does drop the row. Adding hydration: "('u2', 'r1', true)" to that fixture reproduces the leak.

There is no user-visible change today, because queries.ts:178 already drops rows whose sourceType is subsource. The cost is the intended cardinality reduction not happening, plus the doc comment at line 41 asserting an invariant the query does not hold. Filtering h with the same predicate (or anti-joining h.object_id against subsource ids) before the join would make the comment true.

[leedqin]

Thanks — all four findings addressed:

• bool_and over dropped replicas: fixed in c0ce012 (pushed shortly before the review ran against the earlier commits) with exactly the suggested extant-replica predicate, keeping null-replica_id rows for webhooks; SQL tests cover the stale-replica and null-replica cases.
• Subsources re-entering via the hydration side: right — only progress collections were actually removed. Fixed in 152c477: the exclusion now applies to both sides of the FULL JOIN, and the SQL test seeds the hydration-side row that reproduced the leak.
• Anchor fallback impersonating historical metrics: fixed in 152c477, with one deviation from the suggested timestamp === null gate: the empty-current-bucket case also occurs with a non-null auto anchor (the peak-lag bucket lands on the current bucket right after a replica swap — this is the live case that motivated the fallback). The gate is therefore "anchor within two buckets of now": recent anchors may fall back, historical anchors keep the honest inactive state and the dropped-replicas callout.
• Status column sort: the accessor now sorts by the displayed bucket via statusBucketForRow, so sorting groups what the cell shows.

[def-]

QA LLM Review

1. MEDIUM -- Empty anchor bucket makes the footer report live replicas as "dropped in this window"

console/src/platform/maintained-objects/CriticalPathClusterMetrics.tsx:360

Gating the fallback on anchorIsRecent means a historical anchor with no metrics bucket now yields replicas = [], and DroppedReplicasFooter computes its dropped set by subtracting that empty list from every replica in the window. So the panel renders "Inactive — replication factor 0" and, directly underneath, "1 replica dropped in this window: r1 (400cc) · Last seen 4:15:07 PM" naming the cluster's currently-running replica.

Details

dropped is windowReplicas.filter((r) => !lockedIds.has(r.replicaId)) (line 294), and lockedReplicas is the same replicas array (line 383). replicasInWindow is built from bs.slice(-1) over every replica with a bucket in the window, and the window extends to now because useClusterBucketsInWindow passes no end date, so live replicas are always in it. With lockedIds empty every one of them is labelled dropped. This path was unreachable at c0ce012c6e, where the fallback kept replicas equal to replicasInWindow and dropped was therefore always empty.

Reaching it is the panel's own diagnostic flow: the default anchor is the object's peak-lag bucket, and a cluster sitting at REPLICATION FACTOR 0, a replica gap during a non-graceful resize, or a crash-looping replica is both a lag cause and a bucket with no metrics. Under a 24h lookback bucketSizeMs is 24 minutes, so anything older than 48 minutes takes this branch.

ReplicaMetricsList already has the right signal: it marks rows dropped via liveReplicaIds from useAllClusters (line 268). Intersecting dropped with !liveReplicaIds.has(...) makes the footer state a fact about the cluster rather than about which replicas happen to have a bucket at the anchor, and also fixes the case where a replica created after the anchor is listed as dropped.