persist: audit and cut per-shard metric cardinality

.agents/skills/mz-release-signoff/SKILL.md

@@ -290,4 +290,4 @@ The per-area references list the dead panels found so far. Check for new ones wh
 
 **An implausibly constant quantile is a bucket artifact, not stability.** A `histogram_quantile` landing inside one wide bucket returns the bucket boundary and cannot move, which reads as a rock-steady latency. Two adapter panels do this. When a quantile is stable to five significant figures while its counters advance, switch to `rate(_sum) / rate(_count)`.
 
-**Counting series is sometimes the measurement.** Several metrics are per-entity gauges whose only useful aggregate is a series count: `mz_persist_shard_upper` for shards, `mz_balancer_metadata_seconds` for balancers, `mz_persist_metadata_seconds` by `version` for persist client builds. Similarly `mz_source_progress` is a millisecond frontier timestamp, so a healthy series contributes exactly 1000 to its rate and the panel's absolute value is really a series count in disguise.
+**Counting series is sometimes the measurement.** Several metrics are per-entity gauges whose only useful aggregate is a series count: `mz_balancer_metadata_seconds` for balancers, `mz_persist_metadata_seconds` by `version` for persist client builds. Similarly `mz_source_progress` is a millisecond frontier timestamp, so a healthy series contributes exactly 1000 to its rate and the panel's absolute value is really a series count in disguise.

.agents/skills/mz-release-signoff/references/persist.md

@@ -34,7 +34,7 @@ Worth adding to the sweep even though the panel omits them: `mz_persist_compacti
 | `mz_persist_compaction_seconds`, `_requested`, `_applied`, `_bytes`, `_goodbytes` | counter | Compaction. Requested minus applied equals the noop and dropped counts. |
 | `mz_persist_gc_seconds`, `_started`, `_finished`, `_noop`, `_skipped`, `_merged_reqs` | counter | Garbage collection. |
 | `mz_persist_retry_retries_count`, `_started_count`, `_finished_count`, `_sleep_seconds` | counter | The panel excludes `op="next_listen_batch"`, which has its own panel because it retries by design. |
-| `mz_persist_shard_upper` | gauge, per shard | Counting distinct `shard` labels gives the shard count. |
+| `mz_persist_shard_count` | gauge, per process | Shards the process holds a handle to. environmentd holds one per collection, so its value is the environment's shard count. |
 | `mz_persist_read_batch_part_bytes`, `_count` | counter, by `op` | The `op="unindexed"` slice is the `unindexed reads` panel. Rises by roughly 3x during rehydration. |
 | `mz_persist_pushdown_parts_filtered_count`, `_fetched_count`, `_inline_count`, `_faked_count` and their `_bytes` variants | counter | The ratio panels divide one by the sum of all four. |
 | `mz_persist_blob_cache_hits_bytes`, `_hits_blobs`, `_evictions` | counter | |
@@ -65,7 +65,7 @@ Each entry states a property that holds at any fleet size, followed by the measu
 
 **Persist reports its own build independently of `mz_version`.** `mz_persist_metadata_seconds` carries a `version` label, and the `# processes by version` panel counts by it. Use it to confirm a rollout reached the persist clients.
 
-**Counting distinct `shard` labels on `mz_persist_shard_upper` is how the shard count is obtained.** There is no shard-count gauge.
+**`mz_persist_shard_count` on the environmentd pod is the shard count.** Summing it across pods overcounts, because clusterd pods hold handles to the same shards.
 
 ## Order of magnitude
 

doc/user/data/metrics.yml

@@ -1646,24 +1646,15 @@ metrics:
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
 - name: mz_persist_backpressure_emitted_bytes
-  help: A counter with the number of emitted bytes.
-  labels:
-  - name
-  - shard
+  help: bytes emitted by backpressure operators
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
 - name: mz_persist_backpressure_last_backpressured_bytes
-  help: The last count of bytes we are waiting to be retired in the operator. This cannot be directly compared to `retired_bytes`, but CAN indicate that backpressure is happening.
-  labels:
-  - name
-  - shard
+  help: sum over backpressure operators of the inflight bytes each last stalled on; not comparable to retired bytes, but nonzero growth indicates backpressure is happening
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
 - name: mz_persist_backpressure_retired_bytes
-  help: A counter with the number of bytes retired by downstream processing.
-  labels:
-  - name
-  - shard
+  help: bytes retired by processing downstream of backpressure operators
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
 - name: mz_persist_blob_cache_evictions
@@ -2198,6 +2189,18 @@ metrics:
   help: count of grpc errors received
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
+- name: mz_persist_pubsub_client_receiver_diff_applied
+  help: number of diffs received via pubsub that applied
+  source: src/persist-client/src/internal/metrics.rs
+  visibility: internal
+- name: mz_persist_pubsub_client_receiver_diff_not_applied_out_of_order
+  help: number of diffs received via pubsub that did not apply due to out-of-order delivery
+  source: src/persist-client/src/internal/metrics.rs
+  visibility: internal
+- name: mz_persist_pubsub_client_receiver_diff_not_applied_stale
+  help: number of diffs received via pubsub that did not apply due to staleness
+  source: src/persist-client/src/internal/metrics.rs
+  visibility: internal
 - name: mz_persist_pubsub_client_receiver_state_push_diff_fast_path
   help: count fast-path state push_diff calls
   source: src/persist-client/src/internal/metrics.rs
@@ -2476,14 +2479,6 @@ metrics:
   - shard
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
-- name: mz_persist_shard_batch_part_version_bytes
-  help: total bytes in batch parts by shard and version
-  labels:
-  - name
-  - shard
-  - version
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
 - name: mz_persist_shard_batch_part_version_count
   help: count of batch parts by shard and version
   labels:
@@ -2573,20 +2568,6 @@ metrics:
   - shard
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
-- name: mz_persist_shard_inline_backpressure_count
-  help: count of CaA attempts retried because of inline backpressure
-  labels:
-  - name
-  - shard
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
-- name: mz_persist_shard_inline_part_bytes
-  help: total size of parts inline in shard metadata
-  labels:
-  - name
-  - shard
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
 - name: mz_persist_shard_inline_part_count
   help: count of parts inline in shard metadata
   labels:
@@ -2601,48 +2582,13 @@ metrics:
   - shard
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
-- name: mz_persist_shard_live_writers
-  help: number of writers that have recently appended updates to this shard
-  labels:
-  - name
-  - shard
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
 - name: mz_persist_shard_noncompact_batches
   help: number of batches in the shard that aren't compact and have no ongoing compaction
   labels:
   - name
   - shard
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
-- name: mz_persist_shard_pubsub_diff_applied
-  help: number of diffs received via pubsub that applied
-  labels:
-  - name
-  - shard
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
-- name: mz_persist_shard_pubsub_diff_not_applied_out_of_order
-  help: number of diffs received via pubsub that did not apply due to out-of-order delivery
-  labels:
-  - name
-  - shard
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
-- name: mz_persist_shard_pubsub_diff_not_applied_stale
-  help: number of diffs received via pubsub that did not apply due to staleness
-  labels:
-  - name
-  - shard
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
-- name: mz_persist_shard_rewrite_part_count
-  help: count of batch parts with rewrites by shard
-  labels:
-  - name
-  - shard
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
 - name: mz_persist_shard_rollup_count
   help: count of rollups by shard
   labels:
@@ -2657,13 +2603,6 @@ metrics:
   - shard
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
-- name: mz_persist_shard_schema_registry_version_count
-  help: count of versions in the schema registry
-  labels:
-  - name
-  - shard
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
 - name: mz_persist_shard_seqnos_held
   help: maximum count of gc-ineligible states by shard
   labels:
@@ -2678,27 +2617,13 @@ metrics:
   - shard
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
-- name: mz_persist_shard_since
-  help: since by shard
-  labels:
-  - name
-  - shard
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
 - name: mz_persist_shard_spine_batch_count
   help: count of spine batches by shard
   labels:
   - name
   - shard
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
-- name: mz_persist_shard_stale_version
-  help: indicates whether the current version of the shard is less than the current version of the code
-  labels:
-  - name
-  - shard
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
 - name: mz_persist_shard_unconsolidated_snapshot
   help: in snapshot_and_read, the number of times consolidating the raw data wasn't enough to produce consolidated output
   labels:
@@ -2713,13 +2638,6 @@ metrics:
   - shard
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
-- name: mz_persist_shard_upper
-  help: upper by shard
-  labels:
-  - name
-  - shard
-  source: src/persist-client/src/internal/metrics.rs
-  visibility: internal
 - name: mz_persist_shard_usage_current_state_batches_bytes
   help: data in batches/parts referenced by current version of state
   labels:
@@ -2783,6 +2701,10 @@ metrics:
   - worker_id
   source: src/persist-client/src/internal/metrics.rs
   visibility: internal
+- name: mz_persist_stale_shard_count
+  help: count of shards on this process whose persisted state version is behind this process's build version; per-process, so summing across processes counts (shard, process) pairs, not distinct shards
+  source: src/persist-client/src/internal/metrics.rs
+  visibility: internal
 - name: mz_persist_state_apply_spine_fast_path
   help: count of spine diff applications that hit the fast path
   source: src/persist-client/src/internal/metrics.rs

src/persist-client/src/cache.rs

@@ -415,17 +415,23 @@ where
                     "applied pushed diff {}. seqno {} -> {}.",
                     state.shard_id, seqno_before, state.seqno
                 );
-                self.shard_metrics.pubsub_push_diff_applied.inc();
+                self.metrics.pubsub_client.receiver.diff_applied.inc();
             } else {
                 debug!(
                     "failed to apply pushed diff {}. seqno {} vs diff {}",
                     state.shard_id, seqno_before, diff.seqno
                 );
                 if diff.seqno <= seqno_before {
-                    self.shard_metrics.pubsub_push_diff_not_applied_stale.inc();
+                    self.metrics
+                        .pubsub_client
+                        .receiver
+                        .diff_not_applied_stale
+                        .inc();
                 } else {
-                    self.shard_metrics
-                        .pubsub_push_diff_not_applied_out_of_order
+                    self.metrics
+                        .pubsub_client
+                        .receiver
+                        .diff_not_applied_out_of_order
                         .inc();
                 }
             }
@@ -627,6 +633,9 @@ pub(crate) struct LockingTypedState<K, V, T, D> {
     notifier: StateWatchNotifier<T>,
     cfg: Arc<PersistConfig>,
     metrics: Arc<Metrics>,
+    // Retained only to keep this shard's per-shard series registered for as long
+    // as the state is cached; nothing reads it through this handle anymore. Don't
+    // drop it as "unused" without moving that lifetime guarantee elsewhere.
     shard_metrics: Arc<ShardMetrics>,
     update_semaphore: AwaitableState<Option<tokio::time::Instant>>,
     /// A [SchemaCacheMaps<K, V>], but stored as an Any so the `: Codec` bounds