Skip to content

Commit 85511ce

Browse files
committed
doc: record where the compute lifecycle work leaves this design
Compute is adding an append-only lifecycle log for the same stages (#38403) rather than more timestamp columns, and it leaves `mz_compute_hydration_times_per_worker` untouched. So nothing this collector has recorded changes meaning, and there is no ordering constraint between that work and enabling collection here. Notes what moving onto that log would buy, and the one piece of guidance that outlives the current shape: recording gates on hydration, not on a write stage, because a replacement runs read-only until cutover and would otherwise never be recorded at all. Also notes that the event log carries the dataflow's as-of, which this table does not, and without which a duration does not say how much work was done. Ref: SQL-644
1 parent 9fddd93 commit 85511ce

1 file changed

Lines changed: 18 additions & 1 deletion

File tree

doc/developer/design/20260817_durable_object_hydration_history.md

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,21 @@ hydrating is skipped and picked up by a later sweep. The cutoff and the anti-joi
136136
apply to the aggregate's output, since as `WHERE` clauses either one would drop the
137137
unfinished rows and make that check trivially true.
138138

139+
Compute is adding an append-only lifecycle log for the same stages, currently
140+
proposed in #38403: one row per export, worker and event, with a reason and the
141+
dataflow's as-of. It leaves `mz_compute_hydration_times_per_worker` alone, so
142+
everything above keeps its meaning and nothing recorded here is relabelled.
143+
144+
That log is the signal this design says is missing, and moving onto it later buys
145+
three things at once: stages that mean the same thing on every worker, a reason
146+
that distinguishes a replacement waiting for cutover from an index that will never
147+
write, and the as-of, without which an interval says nothing about how much work
148+
was done, since a replacement with a far behind as-of does far more of it in the
149+
same wall-clock time. Recording should keep gating on hydration when that happens.
150+
A replacement runs read-only and does not write until cutover, so gating on a write
151+
stage would never record the hydration a deployment most wants to measure, and
152+
would wait forever on a rollback.
153+
139154
Two consequences. An interval can span workers in different processes, each
140155
anchoring its logging clock at its own `SystemTime`, so skew inflates a duration.
141156
Nothing rejects a row for looking inconsistent, deliberately, because a guard on
@@ -204,7 +219,9 @@ Sweeps never overlap and each visits one replica, which bounds compute load and
204219
keeps the collector from contending with itself in the serialized timestamped-write
205220
path. Fires align to interval boundaries and each sleep is capped, so lowering a
206221
long interval takes effect within the cap rather than after the old interval
207-
elapses.
222+
elapses. The grid is offset by an amount seeded from the organization id, since the
223+
interval is one fleet-wide setting and an unshifted grid would have every
224+
environment sweep at the same instant.
208225

209226
**Background mutations take no OCC write permit.** The permits are one semaphore
210227
shared by every read-then-write in the process, not one per table. A session's wait

0 commit comments

Comments
 (0)