compute: stamp dataflow installation and hydration start - #38345
Conversation
bd6ddd9 to
4e535a5
Compare
ce8b603 to
8b6d7be
Compare
4e535a5 to
20e9707
Compare
8b6d7be to
703a394
Compare
20e9707 to
18f47c4
Compare
703a394 to
0518a86
Compare
18f47c4 to
be177e3
Compare
0518a86 to
175c7f8
Compare
be177e3 to
5d97756
Compare
175c7f8 to
3b51105
Compare
5d97756 to
7286ea7
Compare
3b51105 to
7071722
Compare
7286ea7 to
d530d17
Compare
7071722 to
1d3ebf6
Compare
d530d17 to
3405a66
Compare
1d3ebf6 to
87ffaa7
Compare
3405a66 to
5fc9f21
Compare
87ffaa7 to
5fc9f21
Compare
|
Closing: superseded by #38246, which merged and put the same three columns on main with the same names, order and nullability. This rung existed only so the rest of the stack had the columns while that landed, and I said at the top it was expected to be dropped. The branch is now identical to its base, so this PR is empty. One semantic difference is worth carrying forward, since #38246's version is the one that shipped. It backfills |
NOTE: This is temporary, Moritz has the actual PR for the compute-side implementation.
Stacked on #38344.
Motivation
mz_introspection.mz_compute_hydration_times_per_workerreports only how long hydration took. That is enough to show a duration, but not to place it in time. A durable history needs a stable key for an episode, and telling a queued dataflow from a slow one needs to know when work actually began.Design doc:
20260817_durable_object_hydration_history.mdDescription
Adds
installed_at,started_at, andhydrated_atto the log, all stamped by the replica.installed_atis taken when the export is created. It is stable across an environmentd restart, which is what makes it usable as part of an episode's identity later in the stack.started_atis only reported if the replica observed a start before it reported completion. An import-free dataflow is never suspended, so itsSchedulecan arrive after it has already hydrated, and stamping that late arrival would invent an interval nobody measured. NULL therefore means "the start was not observed", and in practice index exports report a start while materialized view exports frequently do not. Why that is so is worth running down beforestarted_atis presented to users as a queueing signal, and it is noted as such in the design doc.The change is additive. The log keeps its name, OID, and object kind, so its generated per-replica index and every relation built on it are unaffected, and the existing
time_nscolumn keeps its meaning and values. A rename plus a compatibility view was considered, since the compute half of this project proposes it, but naming that relation is a decision for that change, not this one.Verification
Existing hydration coverage (
test/testdrive/hydration-status.td) exercises the stamping paths, since every hydrating dataflow now goes through them. The catalog goldens for the widened relation are updated.This release will add
installed_at,started_at, andhydrated_atcolumns tomz_introspection.mz_compute_hydration_times_per_worker.Note on
started_atwhen no start is observedThis PR and #38246 disagree here. #38246 backfills
started_atfrominstalled_atwhen the replica observed no start, so thatinstalled_at <= started_at <= hydrated_atholds over all non-NULL values. This PR leavesstarted_atNULL, on the grounds that a backfilled start is a value nobody measured, and that a zero-length queueing interval is then indistinguishable from an unknown one.Since #38246 is the version that lands, the durable history downstream will record a zero-length queueing interval for import-free dataflows rather than an absent one. That is defensible, it just wants to be deliberate, so flagging it here rather than quietly resolving it.
Ref: CPU-210