Skip to content

compute: stamp dataflow installation and hydration start - #38345

Closed
aljoscha wants to merge 0 commit into
aljoscha/hydration-01-designfrom
aljoscha/hydration-02-compute
Closed

compute: stamp dataflow installation and hydration start#38345
aljoscha wants to merge 0 commit into
aljoscha/hydration-01-designfrom
aljoscha/hydration-02-compute

Conversation

@aljoscha

@aljoscha aljoscha commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

NOTE: This is temporary, Moritz has the actual PR for the compute-side implementation.

Stacked on #38344.

This PR is expected to be dropped. It duplicates CPU-210, whose implementation is #38246, and exists only so the rest of the stack has the columns it needs while that lands. Once #38246 merges, this rung goes away and the stack rebases onto it. Review effort belongs on #38246, not here.

One semantic difference should be settled before that happens, see the note at the end.

Motivation

mz_introspection.mz_compute_hydration_times_per_worker reports only how long hydration took. That is enough to show a duration, but not to place it in time. A durable history needs a stable key for an episode, and telling a queued dataflow from a slow one needs to know when work actually began.

Design doc: 20260817_durable_object_hydration_history.md

Description

Adds installed_at, started_at, and hydrated_at to the log, all stamped by the replica.

installed_at is taken when the export is created. It is stable across an environmentd restart, which is what makes it usable as part of an episode's identity later in the stack.

started_at is only reported if the replica observed a start before it reported completion. An import-free dataflow is never suspended, so its Schedule can arrive after it has already hydrated, and stamping that late arrival would invent an interval nobody measured. NULL therefore means "the start was not observed", and in practice index exports report a start while materialized view exports frequently do not. Why that is so is worth running down before started_at is presented to users as a queueing signal, and it is noted as such in the design doc.

The change is additive. The log keeps its name, OID, and object kind, so its generated per-replica index and every relation built on it are unaffected, and the existing time_ns column keeps its meaning and values. A rename plus a compatibility view was considered, since the compute half of this project proposes it, but naming that relation is a decision for that change, not this one.

Verification

Existing hydration coverage (test/testdrive/hydration-status.td) exercises the stamping paths, since every hydrating dataflow now goes through them. The catalog goldens for the widened relation are updated.

This release will add installed_at, started_at, and hydrated_at columns to mz_introspection.mz_compute_hydration_times_per_worker.

Note on started_at when no start is observed

This PR and #38246 disagree here. #38246 backfills started_at from installed_at when the replica observed no start, so that installed_at <= started_at <= hydrated_at holds over all non-NULL values. This PR leaves started_at NULL, on the grounds that a backfilled start is a value nobody measured, and that a zero-length queueing interval is then indistinguishable from an unknown one.

Since #38246 is the version that lands, the durable history downstream will record a zero-length queueing interval for import-free dataflows rather than an absent one. That is defensible, it just wants to be deliberate, so flagging it here rather than quietly resolving it.

Ref: CPU-210

@linear-code

linear-code Bot commented Aug 19, 2026

Copy link
Copy Markdown

SQL-632

CPU-210

@aljoscha
aljoscha force-pushed the aljoscha/hydration-01-design branch from bd6ddd9 to 4e535a5 Compare August 19, 2026 16:20
@aljoscha
aljoscha force-pushed the aljoscha/hydration-02-compute branch from ce8b603 to 8b6d7be Compare August 19, 2026 16:20
@aljoscha
aljoscha force-pushed the aljoscha/hydration-01-design branch from 4e535a5 to 20e9707 Compare August 19, 2026 16:31
@aljoscha
aljoscha force-pushed the aljoscha/hydration-02-compute branch from 8b6d7be to 703a394 Compare August 19, 2026 16:31
@aljoscha
aljoscha force-pushed the aljoscha/hydration-01-design branch 2 times, most recently from 20e9707 to 18f47c4 Compare August 20, 2026 11:10
@aljoscha
aljoscha force-pushed the aljoscha/hydration-02-compute branch from 703a394 to 0518a86 Compare August 20, 2026 11:11
@aljoscha
aljoscha force-pushed the aljoscha/hydration-01-design branch from 18f47c4 to be177e3 Compare August 20, 2026 12:08
@aljoscha
aljoscha force-pushed the aljoscha/hydration-02-compute branch from 0518a86 to 175c7f8 Compare August 20, 2026 12:08
@aljoscha
aljoscha force-pushed the aljoscha/hydration-01-design branch from be177e3 to 5d97756 Compare August 20, 2026 12:39
@aljoscha
aljoscha force-pushed the aljoscha/hydration-02-compute branch from 175c7f8 to 3b51105 Compare August 20, 2026 12:39
@aljoscha
aljoscha force-pushed the aljoscha/hydration-01-design branch from 5d97756 to 7286ea7 Compare August 20, 2026 13:11
@aljoscha
aljoscha force-pushed the aljoscha/hydration-02-compute branch from 3b51105 to 7071722 Compare August 20, 2026 13:11
@aljoscha
aljoscha force-pushed the aljoscha/hydration-01-design branch from 7286ea7 to d530d17 Compare August 20, 2026 13:39
@aljoscha
aljoscha force-pushed the aljoscha/hydration-02-compute branch from 7071722 to 1d3ebf6 Compare August 20, 2026 13:39
@aljoscha
aljoscha force-pushed the aljoscha/hydration-01-design branch from d530d17 to 3405a66 Compare August 20, 2026 13:46
@aljoscha
aljoscha force-pushed the aljoscha/hydration-02-compute branch from 1d3ebf6 to 87ffaa7 Compare August 20, 2026 13:46
@aljoscha
aljoscha force-pushed the aljoscha/hydration-01-design branch from 3405a66 to 5fc9f21 Compare August 20, 2026 14:59
@aljoscha aljoscha closed this Aug 20, 2026
@aljoscha
aljoscha force-pushed the aljoscha/hydration-02-compute branch from 87ffaa7 to 5fc9f21 Compare August 20, 2026 15:00
@aljoscha

Copy link
Copy Markdown
Contributor Author

Closing: superseded by #38246, which merged and put the same three columns on main with the same names, order and nullability.

This rung existed only so the rest of the stack had the columns while that landed, and I said at the top it was expected to be dropped. The branch is now identical to its base, so this PR is empty.

One semantic difference is worth carrying forward, since #38246's version is the one that shipped. It backfills started_at from installed_at when the replica observed no start, where this PR left it NULL. So in the durable history a queueing interval of exactly zero now means "no start was observed" rather than "started immediately", and the two are not distinguishable. I have updated the column comment, the user-facing docs and the design doc in #38346 and #38344 to say that rather than promising a NULL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant