doc: correct the hydration timestamps design against what shipped - #38384
Merged
Conversation
Two places where the merged design no longer describes main. The per-worker log was not renamed. The columns were appended to mz_compute_hydration_times_per_worker in place, keeping its name, OID and object kind, so there is no compat view and `SELECT *` against it returns a wider row. That is the better trade and the doc now argues for it: the rename bought only the `SELECT *` width on an unstable mz_introspection relation, while every consumer of that relation selects by name. The one positional consumer, arrangement_sizes_snapshot, reads the aggregate mz_compute_hydration_times, which neither approach touches. Success criterion 7 is narrowed to match, since the log did change shape. The started_at backfill in handle_hydration is a normal path, not a repair for something unanticipated, and the doc's reasoning for calling it one was wrong. Having no imports is sufficient to start immediately but not necessary in order to hydrate early: a dataflow that does import can still see its output frontier pass the as-of while suspended, when the arrangement it imports is already hydrated. A handful of mz_catalog_server indexes do that on every bootstrap. So the backfill needs no diagnostic, and the doc says why it exists instead. Also records the consequence for consumers, that a backfilled started_at equals installed_at exactly where a creation-time one is a separate event microseconds later, and that the difference is an artifact rather than a contract worth reading a zero queueing interval through. Ref: SQL-632 Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ZVCMBSLdxzGus78ZKWhZz
aljoscha
approved these changes
Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The design doc merged in #38238 no longer describes
mainin two places. Both surfaced while reviewing #38344, which builds on this change and describes it, so the two docs currently disagree with each other and one of them disagrees with the code.Ref: SQL-632
Doc only.
Description
The per-worker log was not renamed. The doc specifies renaming it to
mz_compute_hydration_timestamps_per_workerwith the old name left behind as a projecting view. What shipped appends the columns tomz_compute_hydration_times_per_workerin place, keeping its name, OID and object kind, so there is no view andSELECT *against it returns a wider row.The shipped behaviour is the better trade and the doc now argues for it rather than against it. The rename bought only the
SELECT *width on an unstablemz_introspectionrelation, and cost a new OID, a view, and the golden churn from both. Every consumer of that relation selects columns by name: the introspection subscribe,mz-debug, and the goldens, which churn either way. The one consumer that decodes positionally,arrangement_sizes_snapshot, reads the aggregatemz_internal.mz_compute_hydration_times, which neither approach would have touched. Success criterion 7 is narrowed accordingly, since the log did change shape and the original wording promised it would not.The
started_atbackfill is a normal path, not a repair. The doc claims the creation-time andhandle_schedulestamps cover every way a dataflow can begin computing without aSchedule, so a remaining NULL at hydration is unanticipated and worth a diagnostic. That reasoning is wrong, and the implementation comment says why: having no imports is sufficient to start immediately, but it is not necessary in order to hydrate early. A dataflow that does import can still see its output frontier pass the as-of while suspended, when the arrangement it imports is already hydrated, and a handful ofmz_catalog_serverindexes do exactly that on every bootstrap.So the backfill needs no diagnostic, and the doc now explains what it is for instead of treating it as a guard. It also records the consequence for consumers: a backfilled
started_atequalsinstalled_atexactly, where a creation-time one is a separate event a few microseconds later, and that difference is an artifact of stamping rather than a contract anyone should read a zero queueing interval through.Verification
Documentation only. No behaviour described here changes, and no code is touched: the corrections bring the document in line with what already shipped.
Generated by Claude Code