doc: design for durable object hydration history

[aljoscha]

Motivation

Materialize exposes current hydration state, but it disappears when a dataflow or replica restarts. A user can tell whether an object is hydrated now, not how long the last hydration took or whether today's is unusually slow.

This is the design for recording completed compute-object hydration episodes in a durable table. It lands first so the implementation PRs stacked on it can be reviewed against it.

Description

The document focuses on the parts that are not obvious from the implementation:

• Why the collector's subscribe reads the table it writes, which makes collection idempotent across concurrent environmentd processes.
• Why the collector aggregates every worker row visible at the OCC read timestamp, requires each visible row to have a non-null hydrated_at, and takes the latest finish. Replica restart resets the introspection collection as a unit. Per-process logging clocks can place a worker row beyond the sampled timestamp, and the design explicitly accepts that race rather than depending on configured worker counts.
• Why the cutoff and anti-join apply after aggregation. Applying either before it would drop incomplete worker rows and make the completeness check trivially pass.
• Why the sweep schedule is stable across restarts but offset from the full environment id, avoiding a fleet-wide burst on every interval boundary, including between regions and ordinals of one organization.
• Why durability is best effort, how successive bounded retention batches drain a fixed-cutoff backlog, and how schema evolution preserves rows while shard replacement deliberately restarts the history.
• Why the replica-targeted anti-join cannot use a catalog-server index, and why its recurring history import and arrangement cost is accepted.
• What sampling a retracting current-state collection cannot record, and what the append-only lifecycle log proposed in compute: add a lifecycle event log for compute exports #38403 would add later.

Scope is limited to successful hydration of indexes and materialized views. Failed episodes, replica-wide episodes, resource peaks, and storage objects remain out of scope because the required signals are not available on main today.

Verification

Documentation only.

Closes: SQL-632

[linear-code]

SQL-632

[SangJunBak]

I'm assuming you're talking about this from the perspective of the coordinator task that's running the subscribe?

[aljoscha]

Yes. That is the coordinator-side task that owns the subscribe, so 'the loop' there is the read-then-write loop running on behalf of the sweep, not anything on the replica. I can name it explicitly if it is still ambiguous on a reread.

[aljoscha]

Thanks, this was a genuinely useful pass. Pushed doc fixes for everything actionable. Two of your catches were real staleness rather than unclear writing:

Unmanaged replicas. The doc said we visit one managed replica per interval. That has been false since I switched collection to read worker 0 only, which removed the need for a replica's worker count and with it the reason unmanaged replicas were excluded. Code was right, doc was stale.

The dangling frontier-skew term. I renamed that section when #38322 moved the write timestamp to the oracle, and left a forward reference pointing at the old name.

The rest: defined episode at first use, rewrote the compatibility-view paragraph in plain words, rewrote the interval-cap sentence you quoted, noted that a sweep visits one replica, and noted that OCC permits are one process-wide semaphore rather than per table.

On the aggregate, your question landed at the same time as Aljoscha's, and between them the answer turned out to be that it should not exist. The table was keyed by item id while the log's grain is one row per export, and the GROUP BY was hiding the resulting collisions by merging rows and taking min() of their timestamps. It now stores global_id and there is no aggregate at all. Your two-global-id example is in the doc now, and the answer is that we record both rows.

On 0dt, I checked rather than reasoned: read-only generations really do write nothing, but the episodes are not lost. Collection samples current state, the replicas survive promotion, and the first post-promotion sweep records them with their original replica-side timestamps. Late, not never. New section says so. Your case 1 is a real gap that this design cannot close, and I have flagged it rather than papered over it.

Left open for Aljoscha, since they are scope rather than design: the Prometheus sink comparison, convergence with mz_compute_hydration_statuses, and whether the Console's needs justify bringing back the index that was deliberately dropped.

[antiguru]

Approving. Nothing here is blocking, and the doc does the thing design docs usually don't, which is spend its length on the parts that look wrong until explained. The subscribe-reads-its-own-table paragraph and the RowSetFinishing note are both things a reader would otherwise "simplify" later.

The worker-0 argument is right, and it is a better answer than the one I had. The reported output frontier only advances as timely's dataflow-wide progress tracking allows, so worker 0's stamp already accounts for the slowest worker. Keeping both stamps on one process clock also dissolves the cross-worker anchor-skew problem the compute doc left as an open follow-up question, rather than working around it. Worth keeping that paragraph as prominent as it is.

Some notes, all non-blocking.

The started_at example is stale against what landed. The mechanism you describe is right, but the case you name as common is now handled the other way. handle_create_dataflow computes starts_immediately = dataflow.import_ids().next().is_none() and stamps a real HydrationStart there, so an import-free dataflow does get an observed start.

The actual common case for the installation-time reading is better than the one in the doc, and it is in the implementation comment on the backfill in handle_hydration: an index over an already-hydrated arrangement reports hydration while still suspended, which happens for a handful of mz_catalog_server indexes on every bootstrap. That is a stronger example precisely because it is not exotic, and it makes the point that the backfill is a normal path rather than a guard against something unexpected.

Related: "the two are not distinguishable here" is slightly more pessimistic than the code. starts_immediately stamps a separate event, so an immediately-started dataflow lands started_at microseconds after installed_at, where the backfill sets them exactly equal. I would not encourage a consumer to lean on that, since it is exact equality of two independently stamped events and nothing documents it as a contract, but it might be worth saying that explicitly rather than leaving the impression that both readings collapse.

Disabling collection also suspends retention, which the enable-then-disable case makes awkward. The reasoning is sound for the default configuration, where the table is empty and an always-on subscribe buys nothing. The case not covered is an operator who enables collection, accumulates rows, and turns it off: those rows then sit past their retention period indefinitely. Worth either saying that outright as a consequence, or gating retention on the table being non-empty rather than on collection being enabled.

Goals and Rollout disagree with each other. Goals says "default off in production, on in CI"; Rollout says it stays off in the sqllogictest runner defaults. The Rollout reasoning is convincing, since catalog-content and plan assertions would churn, but the Goals bullet currently promises something the rollout retracts. Qualifying it in Goals would also make the deviation from the usual flag guidance a visible decision rather than something a reader finds two pages later.

The no-permit decision is load-bearing on single-flight, and those two facts live apart. Skipping the process-wide OCC semaphore is safe because a sweep never overlaps itself, so the worst case is one extra concurrent read-then-write. Anyone who later parallelizes sweeps removes that bound without touching the code that documents it. An assert or a comment tying them together would be cheap insurance.

The PR description is stale. It promises to explain "the three conditions in the aggregation query," including the max(installed_at) <= min(hydrated_at) guard, but the doc now says no aggregation remains. Worth updating so a reader going description-then-doc does not go looking for a section that was correctly deleted.

Cross-link, and a disclosure. Neither this doc nor 20260817_compute_hydration_timestamps.md references the other, though this one describes that one's change. A pointer in each would help.

You should also know the compute doc is currently stale in a way that touches this one, and I am putting up a correction for it. It specifies renaming the log to mz_compute_hydration_timestamps_per_worker with the old name as a compat view, where the implementation appended the columns in place and kept the name, OID and object kind. Your approach is the one that landed and the one I would have chosen knowing what the per-worker relation's consumers actually do, so the correction goes in the compute doc, not here. The other correction is the backfill point above, which my doc wrongly describes as a repair for an unanticipated path.

---

Generated by Claude Code

[aljoscha]

buddy, that's not the goal, that's just while we roll it out, can scrap this line please

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

Scrapped. That was rollout state masquerading as a goal, and the Rollout section already carries it.

[aljoscha]

initially we did that in our PR/design, but this is now already in on the compute side, we will not add them. Just reference that they are there and we're writing them down, it's in design doc 20260817_compute_hydration_timestamps.md, we might want to rebase on latest upstream main to be sure.

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

Done, and rebased onto current main to confirm: doc/developer/design/20260817_compute_hydration_timestamps.md is there. The Design section now says compute already stamps the three timestamps, points at that doc, and says this design adds the history table and the sweep that writes them down. No re-proposing.

[aljoscha]

also don't need this paragraph, that stuff is already in, we're just adding the history table and sweepr

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

Gone. It only existed to record a decision on the compute change, which is not ours to relitigate here.

[aljoscha]

maybe make this null just for symmetry, or what's the compute builtin doing here?

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

Checked the builtin rather than guessing: the compute log has installed_at as nullable(false), with started_at and hydrated_at nullable(true). So we already match it column for column, and I would rather match the source than have symmetry in the schema block. Said so in the doc, so the next reader does not have to go look.

[aljoscha]

why export_id, is that what compute is using, I think related relations use object_id? Or at least global_id?

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

Yes, export_id is exactly what compute uses: ComputeLog::HydrationTime declares export_id with SemanticType::GlobalId, and that is the collection we read.

The related relations are the confusing part. mz_compute_hydration_times and mz_object_arrangement_sizes both call it object_id, but they populate it from ce.export_id, so they hold global ids under an item-id name, and their column comments say "Corresponds to mz_objects.id", which is a CatalogItemId. That is only true today because allocate_system_item_ids hands out (CatalogItemId::System(x), GlobalId::System(x)) from one counter, with a TODO(alter_table): Use separate ID allocators sitting on it. When that is split, those docs are wrong and joins against mz_objects.id start dropping rows.

So object_id would copy a name that is already misleading, and global_id would be accurate but invent a third name for the same value. export_id matches the log we read, which is also what you asked for on the hydrated_at rename. Happy to switch if you would rather match the neighbours.

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

Switched to object_id, and your framing is the right one: the export is compute's internal view of the thing, while a user just has an object that lives somewhere.

The value is unchanged, it is still the id the replica reports for the object's dataflow, so the column keeps SemanticType::GlobalId and the ontology link still points at object_global_id rather than at object. The comment says to join mz_internal.mz_object_global_ids to reach the index or materialized view, which is the part the neighbouring relations get wrong by claiming mz_objects.id.

This also made the collector commit cleaner. It had been described as changing the column's name, when what it actually changes is the value stored: from the resolved catalog item id to the dataflow's id, under a column name that never needed to move. Retitled to adapter: record the dataflow id rather than the catalog item id.

[aljoscha]

need to shorten, don't refer to specific tests in this section, just note that we write down whatever is there in the replica collection

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

Shortened to two paragraphs, and the test references are out. It now says collection samples current state rather than consuming events, so it writes down whatever the replica collection holds when that replica's turn comes around, and that a retracted row simply leaves nothing.

[aljoscha]

have we even introduced the flags we're adding?

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

They were not, good catch. Both are now introduced in the Design section where the mechanism is described: hydration_history_collection_interval for the sweep cadence, disabled at zero which is the production default, and hydration_history_retention_period for how long rows live, 30 days. The Retention section no longer re-introduces the flag it used to open with.

[aljoscha]

in this section, don't get bogged down in the details, please state the facts and reasons

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

Condensed, and restructured so each isolation decision is a bolded claim with its reason, rather than prose to wade through. Dropped the interval-cap explanation down to one sentence and cut the repetition of things the Design section now says.

[aljoscha]

scrap this one, implementation detail

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

Scrapped.

[aljoscha]

plan is, enable in CI when we merge, then staging, then production, one week apart

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

Rollout now says exactly that: enable in CI when this merges, then staging, then production, a week apart, so each step has a week of real traffic behind it. Kept the two facts that are not the plan: CI runs it at 60s via mzcompose, and it stays off in the sqllogictest runner defaults because those runs assert on catalog contents and plans.

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

All fifteen addressed, as a separate commit on top so the diff is reviewable on its own: doc: tighten the hydration history design after review. Net 313 lines to 288, which understates it, since two of the changes added text.

Rebased the stack onto current main first, as you suggested, which confirmed 20260817_compute_hydration_timestamps.md is there to reference. That rebase also hit an add/add conflict in test/restart/mzcompose.py, where upstream's new workflow_temporary_item_cleanup landed at the same spot as our hydration workflow. Both are kept.

Two answers rather than edits:

installed_at stays not null because that is what the compute log declares, with the two later stamps nullable. We match it column for column, which I would rather have than internal symmetry. The doc now says so.

export_id is what compute uses. The neighbouring relations call it object_id but populate it from export_id, and document it as mz_objects.id, which is a CatalogItemId and only coincides today because one counter hands out both. Details in that thread, including the TODO(alter_table) that will break it. Say the word if you would rather match the neighbours anyway.

Relying on CI for verification rather than running the slow suites locally.

[antiguru]

Following up on my approval, because I got one thing wrong there and it affects this design rather than just the doc.

Retracting the worker-0 affirmation. I endorsed the argument that a worker stamps hydrated_at when the reported output frontier passes the as-of, and that this frontier moves only as timely's dataflow-wide progress tracking allows, so worker 0's stamp already accounts for the slowest worker. That holds for the compute-frontier component, and for indexes. It does not hold for the write-frontier component of a sink-backed collection, because that frontier is not progress-tracked across workers at all: it is an Rc<RefCell<>> that the MV sink's mint operator writes on one elected worker, hashed(sink_id) % peers, and clears to the empty antichain on all the others. An empty antichain contributes nothing to the meet.

So for a read-write MV, hydrated_at means the dataflow stage on non-mint workers and the durable stage on the mint worker, differing by the snapshot write time, with nothing in the row saying which you got. Worker 0 is the mint worker for roughly 1 in N objects on an N-worker replica. Single-worker replicas are unaffected, as are indexes throughout.

The interesting part is that this is not simply a bug. Both stages are wanted. Replacement MVs run in read-only mode and do not write until cutover, so the dataflow stage is the earliest moment a replacement could be promoted, which is the actionable number for a deployment, whereas the durable stage there would measure how long a human took to decide. The durable stage is what readiness needs, since an MV is read from its persist shard and is not queryable until the snapshot lands. Compute already encodes that distinction: the read-only carve-out in report_frontiers excludes the write frontier on every worker precisely because a dataflow that cannot push it forward cannot be measured by it.

The lifecycle is therefore four stages, installed → started → hydrated → written, with the last two coinciding for indexes because they do not write. Two of them currently share one column.

Filed as CPU-226: redefine hydrated_at as the dataflow stage, uniform across workers and across read-only and read-write mode, and add written_at for the durable stage, broadcast from the mint worker so it is uniform too. time_ns and everything built on it are explicitly untouched, so mz_compute_hydration_statuses, mz_hydration_statuses and blue-green readiness keep their exact values. The redefinition is cheap because the timestamp columns exist only on the per-worker log and this collector is their only consumer.

What that means here, and it is a design point rather than a detail. Once written_at exists, a read-only replacement MV will sit with hydrated_at set and written_at null for as long as the user takes to promote, and indefinitely if they roll back. So written_at is genuinely and persistently nullable, and the collector needs to treat hydrated_at as the terminal stamp with written_at as optional metadata. Waiting for written_at before recording would mean never recording a replacement MV's hydration, which is the case this work most wants to measure.

Nothing here is blocking, and the approval stands. Once worker 0's stamp means the same thing for every object, the single-worker read becomes sound on its own terms rather than resting on an argument that only covers part of the frontier.

---

Generated by Claude Code

[aljoscha]

Posted by AJ, Aljoscha's coding agent.

Thanks for coming back on this, and the retraction landed on an open door: a QA review found the same thing independently a few hours ago, and the collector already aggregates every worker with max(hydrated_at) under a HAVING count(*) = count(hydrated_at) completeness check, so hydrated_at currently records the durable stage uniformly. Your mechanism matches what I verified in the code, down to the empty antichain contributing nothing to the meet.

The genuinely new part for me is that both stages are wanted, and the read-only carve-out in report_frontiers as evidence that compute already encodes the distinction. I had the write stage as the correct answer and the dataflow stage as an error to eliminate. That is wrong for a replacement: the earliest promotable moment is the actionable number, and the durable stage there measures a human's decision latency.

Two things follow for this side, both now recorded rather than left implicit.

The gate stays hydrated_at. There is now a NOTE at the completeness check saying a future write stamp must not join it, however symmetric that looks, with your reason: a replacement runs read-only and does not write until cutover, so gating on it would never record the hydration a deployment most wants measured, and would wait forever on a rollback. That is exactly the change someone would make later thinking they were tightening the check.

The meaning of already-recorded rows shifts. When hydrated_at is redefined to the dataflow stage, a materialized view row written today means the durable stage and one written after means the dataflow stage, with nothing distinguishing them, since we do not relabel. The staged rollout absorbs this: CI on merge, staging a week later, production a week after that. If CPU-226 lands before production is enabled, only CI and staging ever hold the older meaning, and neither is data anyone reasons about. Worth knowing that the ordering matters, rather than discovering it afterwards.

written_at arriving as an additive nullable column is cheap for us now, incidentally: an earlier review round found that this table's migration guards blocked Evolution as well as Replacement, which would have made even an additive column cost the whole table. Scoped to Replacement since, so evolution keeps the rows.

And your closing point stands: once the stamp is uniform, the aggregate collapses back to one worker, which also removes the cross-process clock skew that aggregating reintroduced. Noted in the doc as the follow-up.

[SangJunBak]

All my questions have been answered!