adapter: let the oracle choose the read-then-write write timestamp by aljoscha · Pull Request #38322 · MaterializeInc/materialize

aljoscha · 2026-08-18T13:57:44Z

Stacked on #38320. Makes the far-future write timestamp correct rather than merely refused. Rebased onto main now that #37924 has landed.

The conflation

The OCC loop wrote at the frontier its subscribe reported. A frontier certifies what the loop has a complete view of. Choosing which timestamp to write at is a separate decision, and the two coincide only because the target table is usually what pins the frontier. Nothing in the code stated that dependency, and the comment on the conflict arm was the same constraint seen from the other side, satisfied by pulling the target down to the frontier.

They come apart when another input pins it. A materialized view with a REFRESH option settles until its next refresh, so a selection over one reports a frontier hours or days out while the target table's upper is still near the clock.

The scheme

Target. T = peek_write_ts + 1, the smallest value commit_timestamped accepts. A conflict hands back next_eligible_timestamp, adopted verbatim.
Readiness. F >= T. A progress message at F certifies completeness strictly below F, which is the snapshot at T - 1.
Payload. Diffs with t < T, strictly. A diff at T is concurrent with the write and waits for a later target.
T > as_of, because the snapshot arrives at as_of and has to be in the payload. ensure_read_linearized(as_of) guarantees it by leaving the oracle at or above as_of.

Readiness gives T <= F, not equality. Where F is above T the payload excludes diffs in [T, F), and for a selection reading the target table those say the table moved past T, so the compare-and-append refuses and the loop retries higher. Persist arbitrates, not the frontier. This path produces that window itself: the committer appends at the target and applies it to the oracle only afterwards, so while one write sits in between, a second statement's target is a step behind the table's upper. Retries converge because both refusals name a strictly higher timestamp and the loop waits for the frontier to certify each one.

Zero rows stop waiting on a frontier

NoRowsMatched linearized against the frontier the emptiness was concluded at, which is what made a settled-frontier statement park for days. The rule is now stated once, on OccOutcome::NoRowsMatched.empty_as_of: the answer holds as of the timestamp our view is complete through, and the caller brings the oracle up to that. It is empty_as_of(min(frontier, target)), whichever certifies less. For the common UPDATE ... WHERE <no match> that is the statement's own as_of, linearized before the subscribe started, so the answer costs no group commit at all.

That removes the commit a zero-row statement used to need: group commits per statement fall from about 1.09 to 0.13-0.37, some 75 fewer over ManySmallUpdates' 100 statements. What that is worth in wallclock depends on what a commit costs. Comparing the same two images, the scenario runs 0.73s against 1.34s locally and 0.761s against 0.755s on the CI benchmark agents, so the feature benchmark cannot see this improvement. See the comment below for the full measurement.

Also

OccState splits into a consolidated payload (below the target) and timestamped pending (at or after it), which subsumes max_data_ts and the guard that refused to write when a diff at or after the target had been accumulated. byte_size spans both halves, so max_result_size still measures everything accumulated.

Net effect: no timestamp this path writes at, and no wait it performs, is governed by an input's frontier. Every one is governed by the oracle, which is clamped to the clock. That is also what makes the path immune to a frontier that is wrong rather than merely far out, which is how #38260 presented.

Tests

test_refresh_mv_write_commits_near_wall_clock: the far-future case from adapter: refuse a write timestamp past the write timeline's bound #38320 succeeds, commits 3 rows, leaves the oracle near the clock, and the timeline still takes writes.
test_serializable_read_sees_own_refresh_mv_write: a serializable session reads back its own refresh-MV-sourced write. This is the anomaly the old behavior produced, invisible to strict-serializable tests because those block instead.
workflow_test_occ_zero_row_write_linearization is reworked so that it still witnesses what it claims. Its winner is now a blind INSERT into a table the selection is gated on, rather than an OCC DELETE on the target. A timestamped winner leaves the oracle untouched while parked, so the UPDATE would pick the winner's own timestamp, submit, and queue behind it on the group committer, and its answer would only come back after the winner applied. A blind winner allocates before parking, so the UPDATE's target clears it, the answer is reached with no submission, and the linearization wait is observable again.
Unit tests on the fold, where the off-by-ones live.

Reviewer notes

One behavior change: a statement whose frontier lags the oracle ends in StatementTimeout rather than ReadThenWriteContention, since it waits for the frontier instead of burning retries, and holds its OCC permit while it waits. The design doc describes this as the intended behavior now.

A diff that arrives at or after the chosen target is not in the statement's answer, which is ordinary concurrency rather than a change of behavior. Diffs below the target are written as before.

Two things left undone on purpose:

UpperConflict reports next_eligible = target + 1 when InvalidUppers already carries the real upper. Plumbing it through would converge in one retry instead of k. k is 1 or 2 in practice.

aljoscha · 2026-08-18T15:40:27Z

 //! are frontier-independent, so the caller of the loop submits them right after
 //! it.
 //!
+//! ## The frontier certifies, the oracle chooses


this explanation needs to be quite a bit more concise, and focus on our local invariants and stuff

Cut from 50 lines to 25, and reorganised around the local invariants: the three the loop holds (F >= T before submitting, payload t < T strictly, T > as_of) as a bullet list, then one paragraph on why F bounds neither T nor the table upper, then two sentences on the far-ahead input.

Dropped from it: the worked example of the append-then-apply_write window, the DELETE ... IN (SELECT ... FROM mv) shape, and the retry-convergence argument. They live in run_occ_loop and in the design doc instead.

aljoscha · 2026-08-18T15:43:21Z

+        // by `statement_timeout`.
        self.ensure_read_linearized(&timeline, as_of).await?;

+        // The OCC loop derives its write target from this oracle, and reaching


more concise please

13 lines to 7. It now says only what the ordering buys, that it leaves the oracle at or above as_of so the target clears as_of and the payload holds the snapshot, plus one sentence that a far-future as_of parks here because the data does not exist yet.

aljoscha · 2026-08-18T15:46:05Z

+            // write might later have to treat as concurrent.
+            let fold_target = write_target.unwrap_or(min_target);
+
+            // Whether this iteration attempts a write.


please make shorter and more concise

The three blocks around the loop are down from 24 lines to 13 in total. The retry invariant is 6 lines instead of 16, the fold-target note is 3 instead of 4, and the bypass explanation keeps only the two things a reader needs: waiting for a message can wait forever when an input has settled, and the arm cannot spin because every attempt costs an AttemptWrite round trip and a retry_count.

Follow-up to MaterializeInc#38322, so currently based on top of it

aljoscha · 2026-08-19T12:45:36Z

+    /// step above its write timestamp, so hitting the bound means the oracle has
+    /// run away from the wall clock rather than that the statement asked for
+    /// anything unusual. Committing there would advance the oracle further, and
+    /// the oracle is monotone and durable, so every later write and


feels like we're duplicating this description in a few too many places, make a sweep, make sure we adhere to our commenting guidelines, please

Swept. The explanation now lives in one place, write_ts_upper_bound, and the other three sites state their local fact and point at it:

AdapterError::ReadThenWriteTimestampTooFarAhead: what it means plus "nothing was appended", then the pointer. The one thing it adds is that this commit changes the producer, the oracle rather than an input frontier.

commit_timestamped: "Committing here would apply the target to the oracle below, which is what makes it stick. See write_ts_upper_bound."

WriteResult::TimestampTooFarAhead: unchanged one-liner.

That is 21 lines down to 7 across the three.

aljoscha · 2026-08-19T12:49:11Z

+//! retries converge because both refusals name a timestamp above the one they
+//! rejected.
+//!
+//! Taking `T` from the oracle is also what makes a far-ahead input ordinary: a


probably don't need this last paragraph, and can tighten up the rest as well

Paragraph gone. I kept one sentence of it in the opener, because without it the section says what the rule is and never says why the frontier is the wrong source: "Taking T from F instead would let a settled input, whose frontier is legitimately days out, carry the timeline with it." Say the word and that goes too, the commit message carries the same argument.

aljoscha · 2026-08-19T12:49:48Z

+    ///
+    /// `None` means there is nothing to linearize against. Either the
+    /// subscribe ran to completion, so the emptiness rests on no frontier at
+    /// all, or the answer speaks about `as_of`, which the caller linearized


don't know what this "speaks about as_of" means here,

That language is gone everywhere. It was doing two jobs badly, and the field name was the root of it.

observed_ts is now empty_as_of, and it says what it is: the timestamp the emptiness holds at. The caller brings the oracle up to it. Nothing "speaks about" anything.

aljoscha · 2026-08-19T12:50:38Z

-    /// Diffs from a selection that reads no persisted state. The subscribe ran
-    /// to completion, so they are frontier-independent and the caller chooses
-    /// whether to submit them now or buffer them into the transaction.
+    /// Diffs no frontier can change, from a subscribe that ran to completion.


speculating about callers and other places in the code, so should stick to our facts and invariants

Cut. The variant doc states the contract and the reason it exists, and the caller-side detail is gone.

aljoscha · 2026-08-19T12:53:00Z

+        // data does not exist yet, so no choice of write timestamp avoids that.
        self.ensure_read_linearized(&timeline, as_of).await?;

+        // The loop takes its write target from this oracle, and reaching one takes


pleaes make more concise and to the point

13 lines to 4:

// Ordering is load-bearing: this leaves the oracle at or above `as_of`, // which is what makes the loop target clear `as_of` and so include the // snapshot. A far-future `as_of` parks here until the clock arrives, // bounded by `statement_timeout`.

aljoscha · 2026-08-19T12:53:21Z

                    }
-                    None => Ok(response),
+                    None => {
+                        // The subscribe closed cleanly, so the emptiness


we sure this is true?

No, and thank you for pulling on it. Two things wrong with it.

ensure_read_linearized reads read_ts before it nudges, and returns straight away when the oracle is already past the timestamp. The common zero-row case reports as_of, which the pre-read linearization already covered, so it does not park at all. And when it does have to wait, it asks the group committer for a commit rather than waiting for a keepalive, so the wait is a commit round trip, not up to a default_timestamp_interval.

Now:

// The wait is a no-op where the oracle is already past `empty_as_of`, // which is the common `WHERE <no match>`, and otherwise costs the group // commit that `ensure_read_linearized` asks for. Either way the // subscribe handle is gone, so the permit guards nothing and holding it // would throttle unrelated writes.

The design doc had inherited the same wrong cost model in its benchmark section, so I corrected that too.

aljoscha · 2026-08-19T12:54:24Z

+    /// Semantically this is a SELECT at `target - 1` followed by an INSERT at
+    /// `target`, where `target` comes from `write_oracle` and the subscribe's
+    /// frontier certifies that the payload is the complete set of diffs below
+    /// it. Because we hold no write lock, a concurrent writer may bump the


no need to talk about locks here, please just describe the facts and invariants, if any

Locks gone, and the rest is now the contract rather than a narration: where target comes from, what the frontier certifies, what a conflict returns, and the two obligations on the caller. 24 lines to 16.

aljoscha · 2026-08-19T12:55:46Z

+
+        // Retry invariant: `state` keeps every row the subscribe ever sent, and
+        // folding by target means the payload is what the query returns as of
+        // `target - 1`, retractions cancelling the snapshot rows they replace.


talking about retractions and the snapshot is too specific here, just say we consolidate at that timestamp or something

Now: "the payload is the selection consolidated at target - 1, and the diffs at or above target are concurrent with the write, so a retry folds them in only once it raises the target." 6 lines to 3.

aljoscha · 2026-08-19T12:56:58Z

+            // write would have to treat as concurrent.
+            let fold_target = write_target.unwrap_or(min_target);
+
+            // A conflict can leave a target the frontier we already observed


this comment can be crisped up, pleass

all we're really doing is: are we already ready -> proceed to write, if not, wait, right? and then we need some argument for why we don't livelock or deadlock?

Yes, exactly that. Rewritten to your framing:

// Already certified for the target we hold? Write. Otherwise wait for // the next subscribe message. Waiting first would hang after a // conflict, since an input settled until its next refresh sends // nothing further and does not close the channel either. // // Termination: the write arm awaits a round trip and only a conflict // returns to this one, raising `retry_count` towards // `max_occ_retries`, so neither arm spins.

aljoscha · 2026-08-19T12:59:13Z

+                    // tracks the wall clock rather than any input's frontier.
+                    let peek_write_ts = oracle.peek_write_ts().await;
+                    let Some(chosen) = peek_write_ts.try_step_forward() else {
+                        // There is no timestamp above `Timestamp::MAX` to


a bit heavy on the comments here

Trimmed to the two facts worth keeping, that one step above the oracle write timestamp is the smallest value commit_timestamped accepts, and that a timeline at Timestamp::MAX is a broken environment rather than anything the statement did. 14 lines to 6.

aljoscha · 2026-08-19T12:59:43Z

-                            );
-                            continue;
-                        }
+                    // The payload has to contain the snapshot the subscribe


is our comment duplicating the logic and the assert message?

It was. The comment is now three lines that say only what the assert message does not: that the branch is unreachable while the caller holds up its end, and that we clamp so the payload rule survives if it ever is not.

aljoscha · 2026-08-19T13:00:53Z

+
+            if state.payload.is_empty() {
+                // Everything below `target` canceled out. Nothing to write,
+                // and the answer speaks about `target` exactly as a write


not sure I understand this "speaks about" language, and this comment in general. We need to have our logic explained at the root place where it matters, not all these match arms and ifs

This one turned into the real fix of the pass, so it is worth reading the diff rather than the reply.

You were right that the logic was scattered, and the reason was that the rule was not stated anywhere, so each site invented its own phrasing for it. The rule is: an answer holds as of the timestamp our view is complete through, and the caller brings the oracle up to that. Stated once, on OccOutcome::NoRowsMatched.empty_as_of, with one helper:

fn empty_as_of(complete_below: Timestamp) -> Timestamp

Under that reading the three producers stop being three rules. The channel-close case reports None, emptiness holding at every timestamp. The loop reports empty_as_of(target). And process_message reports empty_as_of(min(ts, fold_target)), which subsumes the two cases you flagged at line 1871 below, since for a first pass fold_target is as_of + 1 and that expression is exactly as_of. The special case for it is deleted.

aljoscha · 2026-08-19T13:49:15Z

+    /// rising targets each diff moves exactly once, so incremental folding
+    /// costs no more than one fold at the end and keeps memory at the
+    /// consolidated size in between.
+    fn fold_below(&mut self, target: Timestamp) {


is it a bit brittle that we have a field current_upper but also a fold that takes a target, and in our doc for OccState we refer to target for stuff. Better to make our contract more tight/self contained, I'd say

Agreed, and fixed by making the type own the split.

OccState no longer refers to a target it does not have. It has split, payload is the consolidated net strictly below it, pending is the rest, and fold_below raises it. Lowering it was a documented obligation on two call sites that derived the target independently, which is what you are calling brittle. It is now enforced in fold_below: a lower split soft panics and clamps to the previous one, so the payload contract holds either way.

current_upper stays, and its doc now says what it is on its own terms, the last progress timestamp, which certifies no diff will arrive below it.

aljoscha · 2026-08-19T13:49:48Z

+
+    /// Moves the diffs the predicate rejects into `payload`, consolidates both
+    /// halves, and recomputes `byte_size`.
+    fn fold(&mut self, stays_pending: impl Fn(Timestamp) -> bool) {


my intuition would have been that the predicate says what get's folded, not the other way rounnd

Mine too, in hindsight. The closure is gone. fold takes Option<Timestamp>, the split point, None meaning no split, and fold_below / fold_all are the two callers. No predicate to read in either direction.

aljoscha · 2026-08-19T13:51:42Z

-                        return ProcessResult::NoRowsMatched { observed_ts: ts };
+                    if ts > as_of && state.payload.is_empty() {
+                        if as_of.try_step_forward() == Some(fold_target) {
+                            // The payload is then exactly the net of the times at


why do we need these two cases and the lengthy description? It being this long points at maybe a smell here

It was a smell, and the two cases were one case wearing a disguise. See my reply at line 1576.

Both branches asked "at what timestamp does this emptiness hold". The first pass answers as_of, a raised target answers one below the target, and both are empty_as_of(min(ts, fold_target)). One branch now, guarded by state.is_empty() rather than payload.is_empty().

That guard also removes a behavior change I had had to flag in the PR description. Diffs pending above the target no longer short-circuit to "0 rows": the statement goes on to a write attempt and writes the ones that land below its target. Only a diff at or after the target stays out, which is just concurrency. 30 lines to 15, and one fewer thing to explain.

aljoscha · 2026-08-19T13:52:30Z

    The `group_commit_before_apply_write` failpoint holds the winning writer
    inside that window, the same one a second `environmentd` process opens on
    its own with no ordering against local Persist visibility.
+


we're adding a lot of prose without changing much, is that needed throughout this file?

Trimmed. The docstring keeps the premise it has to pin, that the parked winner puts the frontier past the oracle target so the first attempt loses and the answer comes from the conflict retry, and drops the restatement of the rules themselves. 17 lines to 7, and the two inline blocks 16 to 10.

Nothing else in the file grew, the rest of the diff there is the guard change and an assertion message.

ggevay

LGTM, one wording nit inline.

A Nightly is running on the branch: https://buildkite.com/materialize/nightly/builds/18137 — worth a look before merge. The feature benchmarks are the main interest: the common zero-row statement no longer pays a commit, so ManySmallUpdates should visibly improve.

ggevay · 2026-08-19T19:45:46Z

+    /// [`OccOutcome::NoRowsMatched`] must be linearized against its
+    /// `empty_as_of` before the response goes out.
+    ///
+    /// `write_oracle` is `None` only for a selection that pins no timeline. Such


nit: "write_oracle is None only for a selection that pins no timeline" (here and at the write_oracle binding above) reads against governing_timeline's own doc, which maps a pins-no-timeline read side (TimestampDependent) to EpochMilliseconds and reserves None for timestamp-independent selections. "Timestamp-independent" in both spots would remove the contradiction.

Right on both spots, and the contradiction is with a doc in the same file, which makes it worse. get_timeline maps TimestampDependent to Some(EpochMilliseconds) and only TimestampIndependent to None (timestamp_selection.rs:213-219), and governing_timeline's own doc uses "pins no timeline" as the label for the former. So my two spots called None the case that is actually Some. Both now say "timestamp-independent selection". Pushed.

aljoscha · 2026-08-20T08:43:21Z

I looked at the Nightly, and the feature benchmark does not show the improvement. It is worth writing down why, because the expectation was mine to begin with, from a paragraph in the design doc.

ManySmallUpdates, this branch against the merge base image (main.g00299a05):

ManySmallUpdates | wallclock       |   0.648 |   0.649 |  s | better:  0.2% faster
ManySmallUpdates | memory_mz       | 681.672 | 671.781 | MB | worse:   1.5% more
ManySmallUpdates | memory_clusterd |  74.668 |  93.449 | MB | better: 20.1% less

0.2% is nothing, and the run bounds how much it could have been. Individual measurements of a single build came in at 0.650, 0.657, 0.663, 0.672 and 0.679, a spread of about 4%, so anything under roughly 30ms is invisible here. Simulating the scenario's arithmetic, 100 statements over 10 rows with all matched rows collapsing to one value, gives 67 to 82 zero-row statements depending on seed. So the commit this removes is worth under about half a millisecond per statement in that setup, and the 3.5x that scenario carries is dominated by the rest of the per-statement work, above all a subscribe dataflow installed and torn down for every statement.

Things I checked before concluding that, because a null result is usually a broken measurement:

The OCC path was on for both sides. enable_adapter_frontend_occ_read_then_write is a VariableSystemParameter whose default is true from v26.36.0-dev, CI_SYSTEM_PARAMETERS was unset in that build (no seed line in the log), and resolving get_default_system_parameters for v26.39.0-dev returns true. Both images are v26.39.0-dev.
The comparison really is against the merge base, v26.39.0-dev.0--main.g00299a05e3c63fe26bdcb78788c485853bf6faf3, not against an older release that would have run the lock path.
The mechanism does fire for this shape. On the merge base the zero-row answer linearizes against the observed frontier, which is the table's upper and one step above the oracle's read timestamp, so it nudges a commit. Here it reports as_of, which the pre-read linearization already covered. test_zero_rows_report_the_answer_not_the_frontier pins the first-pass case at exactly as_of.

So the mechanism is real and the effect is below this benchmark's noise floor. I have corrected the design doc, which implied the residue largely disappears for this workload and invited exactly this expectation, and the PR description now says the same. If we want the scenario faster, the target is the per-statement subscribe, not this.

On the rest of that Nightly: 4 real failures, Security advisories and Miri test (full) both of which fail on main's recent nightlies too, and two job timeouts (MySQL CDC source-versioning migration 2, Orchestratord + upgrade chain). Nothing attributable to the branch.

— AJ, aljoscha's agent

aljoscha · 2026-08-20T09:17:23Z

Retracting my previous comment: it was wrong. The improvement is real, and it is large. I had drawn a conclusion from the Nightly alone, and the Nightly's baseline number is the thing that does not reproduce.

Measured locally on the two CI-built images, with the path forced on for both sides and the flag verified inside each running container:

baseline	this	other	ratio
`pr.g836d6e7d` (this PR's parent), 4 runs	0.778 / 0.847 / 0.590 / 0.823	1.347 / 1.351 / 1.330 / 1.355	1.6-2.3x faster
`main.g00299a05` (the Nightly's own baseline)	0.845	1.335	1.37x faster
control, path off on both sides	0.336	0.337	0.3%

No measurement on one side overlaps the other in any run, and the control pins both machine noise and "these two images are otherwise equivalent".

The part that convinces me it is the intended mechanism and not a machine artifact is the group-commit counter, mz_group_commit_catalog_upper_seconds_count against the UPDATE count. Without the change it is ~1.09 group commits per statement, meaning every statement gets one including the ~75% that match nothing. With it, 0.13-0.37, which is roughly the statements that do match rows. That is exactly the claim: an answer of "no rows" that holds at the statement's own as_of needs no commit, because the pre-read linearization already put as_of behind the oracle.

Both sides ran the OCC path in all of this: mz_occ_read_then_write_retry_count was observed 482-745 times per run on each side, and 0 times in the control.

What I cannot explain is the Nightly, which reported 0.648 against 0.649. I checked the things that would make it vacuous and they all came back the other way: no CI_SYSTEM_PARAMETERS on the build or the step, and rendering the harness's own service config for the ancestor tag yields enable_adapter_frontend_occ_read_then_write=true. Its baseline number is also hard to reconcile with the 3.5x regression this scenario carried when the OCC path landed, since 0.649 would put the pre-change path at parity with the lock path. My local run against that same ancestor image gives 1.335. So I would treat that Nightly's ManySmallUpdates row as unreliable rather than as evidence, and a fresh Nightly on the current branch is worth having before merge, since the branch has moved twice since 18137.

The design doc and the PR description now carry the measured numbers, including that the scenario is still ~2x the lock path, so the rest of its gap is the per-statement subscribe dataflow rather than anything this change touches.

Apologies for the noise of a wrong conclusion in between.

— AJ, aljoscha's agent

def- · 2026-08-20T10:26:26Z

QA LLM Review

1. MEDIUM -- zero-row linearization test now passes vacuously

test/cluster/mzcompose.py:4894

Dropping the conflicts > 0 guard makes every surviving attempt of
workflow_test_occ_zero_row_write_linearization the case the old guard existed to
exclude: the UPDATE's zero-row answer is now always reached after the parked winner
applied its write to the oracle, so assert after == 0 holds whether or not
frontend_read_then_write.rs:955 linearizes at all. The only test pinning the zero-row
linearization guarantee no longer distinguishes it.

Details

Let W be the winner DELETE's append timestamp. commit_timestamped does not touch
the oracle's write timestamp before appending (appends.rs, apply_write runs only
after the failpoint), so while the winner is parked peek_write_ts() == W - 1 and the
table's upper is W + 1 — exactly the "two steps above the oracle" the new docstring
describes.

The UPDATE therefore picks target = peek_write_ts + 1 = W. Its payload is folded
strictly below W, so the winner's retraction at W stays in pending and the
payload still holds the pre-delete diffs. Non-empty payload plus upper (W+1) >= target
means it submits a write. That write is a TableWriteCmd::TimestampedWrite on
group_committer_tx (coord/read_then_write.rs:195) — the same serial channel the
parked winner is sitting inside attempt_write_to_txns on. The refusal cannot be
answered until the 10s park ends and apply_write(W) has run.

So by the time the loop adopts next_eligible = W + 1, re-folds, and reports
empty_as_of = W, the oracle's read timestamp is already W. ensure_read_linearized(W)
returns immediately, and the following count(session, key) reads at >= W and sees 0
regardless. Delete the ensure_read_linearized call in the NoRowsMatched arm and this
test still passes.

This is not the old behaviour: at the merge base the loop concluded emptiness at the
subscribe's frontier (W + 1) without submitting anything, so the answer really was
reached inside the window and the wait was observable. That is also why the guard had to
go — under the new design the conflict is guaranteed, so conflicts > 0 skipped every
attempt. The guard removal is necessary; what it removed was the test's only witness.

The docstring's claim that the retry lands "still at a timestamp the oracle has not
applied" is the same mistake in prose.

To restore a witness the emptiness has to be concluded without a round trip through the
group committer, i.e. the winner's diff must land strictly below the UPDATE's first
target. A blind write does that, because write_to_txns calls oracle.write_ts()
before the failpoint: e.g. gate the selection on a second table
(UPDATE t SET v = v + 1 WHERE k = <key> AND NOT EXISTS (SELECT 1 FROM guard)) and make
the winner INSERT INTO guard VALUES (1). Then peek_write_ts == W, the target is
W + 1, the flip at W folds into an empty payload with no write attempt, and
ensure_read_linearized(W) parks until the winner applies — which is the property under
test.

aljoscha · 2026-08-20T10:50:12Z

Confirmed and fixed. The finding is right, and I checked the mechanism rather than the narrative: commit_timestamped only calls peek_write_ts (appends.rs:470) and apply_write runs after the failpoint (appends.rs:626-628), while the blind write_to_txns allocates at appends.rs:551, before it. That asymmetry is the whole difference, and the old winner was a DELETE, so it took the timestamped path and left the oracle untouched while parked. Everything downstream follows as described: the UPDATE picks W, submits, queues behind the parked winner on the group committer, and by the time the refusal comes back apply_write(W) has run. My docstring's claim that the retry lands "still at a timestamp the oracle has not applied" was wrong.

Reworked along the lines you suggested. The winner is now a blind INSERT INTO guard VALUES (<key>), and the selection is gated on it:

UPDATE t SET v = v + 1 WHERE k = <key> AND NOT EXISTS (SELECT 1 FROM guard WHERE g = <key>)

So peek_write_ts == W, the UPDATE's target is W + 1, the guard row at W folds straight into the payload and cancels it, and the answer is reached with no submission at all. ensure_read_linearized(W) then parks until the winner applies, which is the property under test.

The witness is now the guard row rather than the target row. before is a strict-serializable read that must see 0 guard rows, which is what places us inside the window, and the assertion after the UPDATE is that a strict-serializable read sees 1. Without the linearization the UPDATE returns while the oracle is below W, so that read lands below W too and sees 0, deterministically, for as long as the window is open. Removing the ensure_read_linearized call now fails the test rather than passing it.

Two things I added on top of the suggestion:

assert conflicts == 0, because "no submission" is what makes the rest non-vacuous. If a future change puts a round trip through the committer back in front of the answer, that assertion fails instead of the test quietly going vacuous again, which is the failure mode we just had.
A NOTE on the remaining premise. The UPDATE's frontier has to reach W + 1 even though the winner wrote a different table, which holds because a txns-shard append advances the readable upper of every registered table. If that ever stops holding, the UPDATE waits on its frontier rather than on the oracle and an attempt passes without witnessing anything. That is a vacuous pass rather than a false failure, but it is worth naming since it is the same shape of trap.

Also dropped the guard removal note from the PR description, since the guard is no longer what makes the conflict path unobservable, and the docstring now states why a timestamped winner cannot witness this at all.

— AJ, aljoscha's agent

…8320) Part of: [SQL-641](https://linear.app/materializeinc/issue/SQL-641) Safeguard for the write timestamp the frontend OCC read-then-write path chooses. Independent of #38260. #38322 stacks on this and makes the far-future case work rather than merely be refused. ## The hole `GroupCommitter::commit_timestamped` documented an obligation it never discharged: ``` /// * The wall-clock throttle. `target_timestamp` is the caller's to choose, /// and it must not run the write timeline ahead of the clock. ``` Nothing checked it. Every other write allocates from the oracle, `GREATEST(write_ts + 1, now())`, so it cannot jump. A caller-chosen target can, and the oracle is monotone and durable: * Under the default **strict serializable**, every later write and read on the timeline blocks until the wall clock catches up. * Under **serializable** it is worse. Those reads never consult the oracle (`needs_linearized_read_ts` is false for them), so they pick a timestamp near the clock and an acknowledged write stays invisible. * It **survives restart** through the catalog shard's upper, which boot re-applies to the oracle. The startup sleep that once waited for the clock is gone. The frontend path produces such a target today, no bug involved: it writes at the frontier its subscribe observed, and a selection over a `REFRESH` materialized view settles until its next refresh. With `REFRESH EVERY '15 seconds'` that is a ~6s oracle jump. With a coarser interval it is a stalled environment. ## What this does **Enforces the bound where it is stated.** A target above `write_ts_upper_bound(now)` is refused before the append, and the session turns `WriteResult::TimestampTooFarAhead` into a statement error. The ceiling is the same `now + 2 * 5000ms` that `check_runaway_write_ts` already measured against, now named. A backstop, not the fix: the worst outcome becomes one failed statement instead of a stalled timeline, for any caller of the timestamped-write API. **Leaves the detector logging.** `check_runaway_write_ts` stays an `error!`. With the refusal above in place, every way left to reach it is a runaway this process inherited, a durable one re-applied at boot or a backwards clock step, and the group committer's throttle is written to wait those out. Failing there would turn a stalled timeline into a crash loop, which is the opposite of what the boot check below decides. **Reports the restart channel.** Boot logs an error when the catalog upper is already past the bound. It does not refuse to start: the timeline is stalled either way, and a process that will not boot turns that into an outage plus a crash loop. ## Tests `test_far_future_write_timestamp_is_refused` builds a genuinely far-future target from a materialized view with two refreshes, one seconds out so the view becomes readable, one in the year 3000 so the target is deterministic rather than racing a refresh interval. It asserts the refusal, that the oracle did not move (checked before reading the target back, since a read cannot be served once the oracle is out there), and that the timeline still takes a write and a read. The error is `FEATURE_NOT_SUPPORTED` here because retrying does not help while the frontier stays out there. #38322 changes the producer set to "the oracle itself ran away" and moves it to `INTERNAL_ERROR` with a hint that matches.

aljoscha · 2026-08-20T10:59:02Z

 read timestamp to at least the write timestamp, so at write time it holds that
 `write_ts <= oracle_read_ts`. The linearization invariant is maintained.

+A statement that matches no rows has no write to ride on, and it still made a


one last thing, we need to make this and the next paragraph more concise, just describe what happens and why and why it's correct please, none of that "rides on" messaging

Cut to two paragraphs, and the "rides on" framing is gone from here and from the module doc, which had the same phrasing.

What is left is what happens (no write, so nothing advances the oracle for it, it reports the timestamp its view is complete through and waits for the oracle to reach that), why the common case is free (empty at as_of means complete through as_of, already linearized), and why it is correct (the answer describes the timestamp it reports, later reads land at or above it, and the hazard is only the other direction). 19 lines to 15.

— AJ, aljoscha's agent

aljoscha · 2026-08-20T10:59:51Z

-statement. The residue is the price of the linearization guarantee rather than a
-defect: correctness requires the oracle to advance, and only a commit advances
-it.
+roughly 90% of its 100 updates end up matching no rows. A zero-row answer has to


please cut most of this, but maybe add a short "Note: " section that explains the updated experiment results

Cut. The original paragraph is back, with one correction so it is not stating something false: a selection already empty at its as_of needs no commit at all, and the residue is the selection that empties later. 30 lines of mine down to 11.

The numbers moved into a short Note: after it: 1.6x to 2.3x on ManySmallUpdates, the control, group commits per statement falling from ~1.09 to 0.13-0.37, and the remaining ~2x against the lock path being the per-statement subscribe dataflow. The note says the figures above it were taken before these two changes, since that is the only reason the two sets of numbers can sit next to each other without contradicting.

— AJ, aljoscha's agent

aljoscha · 2026-08-20T11:32:47Z

+//! the table moved past `T`, so the compare-and-append refuses and the loop
+//! retries higher. Persist arbitrates the timestamp, not the frontier.
+//!
+//! ## A zero-row answer


this should closely match the design doc, or be quite a bit shorter and reference the design doc please

Shortened both sections and made the design doc the place the argument lives, since it did not actually hold the rationale I would have been pointing at. ### The OCC loop gains it: the target comes from the oracle rather than the frontier because a frontier certifies what the loop has a complete view of, which is a different question from which timestamp to write at, an input can legitimately sit far in the future (a REFRESH materialized view), the three invariants, and the note that frontier >= T does not make them equal so a refusal is the design.

The module doc keeps only what someone reading this file needs and points there:

"The frontier certifies, the oracle chooses": 19 lines to 16, and what is left is the three invariants, which are the off-by-ones the code turns on, plus the NOTE that a refusal is expected rather than a bug. The motivation moved out.

"A zero-row answer": 19 lines to 9, one paragraph with the rule and the reason the common case is free, pointing at ### Linearization for why the lower timestamp is not weaker.

The design doc is 20 lines longer, the module doc 22 shorter.

— AJ, aljoscha's agent

The OCC loop wrote at the frontier its subscribe reported, which conflates two jobs. A frontier certifies what the loop has a complete view of. Choosing the timestamp to write at is a separate decision, and the two coincide only because the target table is usually what pins the frontier. They come apart when another input does. A materialized view with a `REFRESH` option settles until its next refresh, so a selection over one reports a frontier hours or days ahead of the clock while the target table's upper is still near it. Writing there ratchets the timeline's oracle into the future, and the oracle is monotone and durable, so every later write and strict-serializable read blocks until the clock catches up. Under `serializable` the write is simply invisible. So the oracle chooses and the frontier certifies. The target `T` is one step above the oracle's write timestamp, the smallest value `commit_timestamped` accepts, and a conflict hands back the next eligible one. The loop writes at `T` once the frontier reaches `T`, with every diff from strictly below `T` as the payload and everything at or after it held back for a later target. Three boundaries carry the correctness. A progress message at `F` certifies completeness below `F`, so readiness is `F >= T`. The payload is `t < T` strictly, because a diff at `T` is concurrent with the write. And `T > as_of`, because the snapshot arrives at `as_of` and has to be in the payload, which the pre-read linearization guarantees by leaving the oracle at or above `as_of`. Readiness gives `T <= F`, not equality, so the payload can exclude diffs the subscribe already delivered in `[T, F)`. For a selection that reads the target table those say the table moved past `T`, the compare-and-append refuses, and the loop retries at the timestamp the refusal names. Persist arbitrates, not the frontier. This path produces that window itself: the committer appends at the target and applies it to the oracle only afterwards, so while one write sits in between, a second statement's target is a step behind the table's upper. Retries converge because both refusals name a strictly higher timestamp and the loop waits for the frontier to certify each one. A zero-row answer no longer waits on a frontier either. It reports the timestamp the emptiness holds as of, one below whichever of the frontier and the target certifies less. For the common `UPDATE ... WHERE <no match>` that is the statement's own `as_of`, already linearized before the subscribe started, so the answer costs no group commit, and the two cases the loop used to separate turn out to be the same rule. Tests: the far-future refresh-MV write commits near the clock instead of being refused, and a `serializable` session reads back its own such write, which pins the anomaly this closes. Unit tests cover the fold's boundaries, where the off-by-ones live. The zero-row linearization workflow now parks its winner with a blind write rather than a timestamped one, so the answer is reached without a round trip through the group committer and the wait for the oracle is observable at all.

aljoscha · 2026-08-20T12:06:44Z

The benchmark story, settled

Third and final word on ManySmallUpdates, since I have now been wrong about it in both directions on this PR.

The nightly on this branch reports 0.761s against 0.755s, no change, which reproduces the earlier nightly row rather than contradicting it. My local A/B reported a large win. Both are correct measurements, and I finally know why they differ.

What I checked and ruled out.

The flag was off in the nightly. No. The build has no CI_SYSTEM_PARAMETERS and the log has no random-seed line, so get_default_system_parameters applies the variable defaults, where this flag's default is true for >= v26.36.0-dev. Both sides ran the OCC path. Grepping the job log for the flag name finds nothing because the log never echoes system parameters, which proves nothing either way.
My local baseline was stale. It was: I had compared against PR adapter: refuse a write timestamp past the write timeline's bound #38320's head on the older main, while the nightly compares against gff62f31a, which also carries compute: derive time dependence from the imports a dataflow reads #38260 and compute: prune the source imports no export reads #38319. So I re-ran the nightly's exact comparison, --other-tag v26.39.0-dev.0--main.gff62f31a... against this branch's own published image, no rebuild. Result: 0.734s against 1.342s, 1.83x faster. Same images as the nightly, opposite outcome, so the baseline is not the explanation either.

What it actually is. The change removes the commit, and that part is environment independent: group commits per statement fall from about 1.09 to 0.13-0.37, roughly 75 fewer over the scenario's 100 statements. What the removal is worth in wallclock is whatever those commits cost. Locally 75 commits account for 0.6s, about 8ms each, a metadata store round trip on that disk. On the benchmark agents the same 75 commits vanish into run-to-run noise.

The part worth keeping. The feature benchmark cannot see this improvement, which also means it would not flag a future change that put the commit back. That is now recorded in the design doc's performance section along with both sets of numbers, rather than the single-environment "1.6x to 2.3x" claim I had put there, which was overclaiming a local result as a general one.

My process error, stated plainly: I measured wallclock first and the mechanism last. The group-commit counter is the environment-independent quantity and should have been the headline from the start.

— AJ, aljoscha's agent

Follow-up to MaterializeInc#38322, so currently based on top of it

aljoscha force-pushed the aljoscha/occ-oracle-write-ts branch from 407f1bf to 1006eaa Compare August 18, 2026 14:48

aljoscha marked this pull request as ready for review August 18, 2026 15:18

aljoscha requested a review from a team as a code owner August 18, 2026 15:18

aljoscha commented Aug 18, 2026

View reviewed changes

aljoscha marked this pull request as draft August 18, 2026 15:59

aljoscha force-pushed the aljoscha/occ-oracle-write-ts branch from 1006eaa to d914c9c Compare August 18, 2026 16:11

aljoscha mentioned this pull request Aug 18, 2026

adapter: refuse a write timestamp past the write timeline's bound #38320

Merged

def- mentioned this pull request Aug 19, 2026

tests: Extend to run into OCC timestamp failure #38341

Open

def- added a commit to def-/materialize that referenced this pull request Aug 19, 2026

tests: Extend to run into OCC timestamp failure

e816afb

Follow-up to MaterializeInc#38322, so currently based on top of it

def- added a commit to def-/materialize that referenced this pull request Aug 19, 2026

tests: Extend to run into OCC timestamp failure

d4a9ebd

Follow-up to MaterializeInc#38322, so currently based on top of it

aljoscha commented Aug 19, 2026

View reviewed changes

ggevay approved these changes Aug 19, 2026

View reviewed changes

aljoscha marked this pull request as ready for review August 20, 2026 08:10

aljoscha force-pushed the aljoscha/occ-oracle-write-ts branch 2 times, most recently from 1ff7c9b to 0c1723a Compare August 20, 2026 08:42

aljoscha force-pushed the aljoscha/occ-oracle-write-ts branch from 0c1723a to 7e2f679 Compare August 20, 2026 09:17

aljoscha force-pushed the aljoscha/occ-oracle-write-ts branch from 7e2f679 to bac6265 Compare August 20, 2026 10:18

aljoscha force-pushed the aljoscha/occ-oracle-write-ts branch from bac6265 to dfe4065 Compare August 20, 2026 10:49

aljoscha added ci-nightly PR CI control: also trigger Nightly and removed ci-nightly PR CI control: also trigger Nightly labels Aug 20, 2026

Base automatically changed from aljoscha/occ-write-ts-bound to main August 20, 2026 10:57

aljoscha force-pushed the aljoscha/occ-oracle-write-ts branch from dfe4065 to 85e32a0 Compare August 20, 2026 10:57

aljoscha commented Aug 20, 2026

View reviewed changes

aljoscha force-pushed the aljoscha/occ-oracle-write-ts branch from 85e32a0 to a52a661 Compare August 20, 2026 11:21

aljoscha commented Aug 20, 2026

View reviewed changes

aljoscha force-pushed the aljoscha/occ-oracle-write-ts branch from a52a661 to 08a0b7b Compare August 20, 2026 11:37

aljoscha force-pushed the aljoscha/occ-oracle-write-ts branch from 08a0b7b to f1ade3e Compare August 20, 2026 12:06

aljoscha merged commit 30e4b1d into main Aug 20, 2026
78 checks passed

aljoscha deleted the aljoscha/occ-oracle-write-ts branch August 20, 2026 12:27

aljoscha mentioned this pull request Aug 20, 2026

doc: design for durable object hydration history #38344

Open

def- added a commit to def-/materialize that referenced this pull request Aug 21, 2026

tests: Extend to run into OCC timestamp failure

3f17795

Follow-up to MaterializeInc#38322, so currently based on top of it

Uh oh!

Conversation

aljoscha commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The conflation

The scheme

Zero rows stop waiting on a frontier

Also

Tests

Reviewer notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

aljoscha commented Aug 18, 2026 •

edited

Loading

def- commented Aug 20, 2026 •

edited

Loading