compute: walk fast-path index peeks off the serving worker - #38242

Closed

antiguru wants to merge 120 commits into

MaterializeInc:mainfrom

antiguru:mh/peek-placement

Commits on Aug 21, 2026

compute: two-runtime read isolation (Arc-batch spines, arrangement sharing, interactive runtime)

antiguru
and
claude
committed
9dff86f
testdrive: bump ii_t4 arrangement-size bound for Arc-batch overhead

antiguru
and
claude
committed
c61bba9
compute: release input read holds when a deferred interactive dataflow is dropped

antiguru
and
claude
committed
ad75d81
parallel-benchmark: add TwoRuntimeReadIsolation scenario

antiguru
and
claude
committed
155ef3a
compute: wake interactive peeks waiting on a re-exported index's seal

antiguru
and
claude
committed
61abaf5
doc: design for two-runtime arrangement sharing capture-based lifecycle

antiguru
and
claude
committed
5db39ff
doc: two-runtime arrangement sharing lifecycle (protocol-split design)

antiguru
and
claude
committed
e63c50c
doc: drop the Import command, use shared imports by role

antiguru
and
claude
committed
494814c
doc: implementation plan for two-runtime arrangement sharing lifecycle

antiguru
and
claude
committed
f08c31c
compute: single-source the shared-trace replay feed from the stream frontier

antiguru
and
claude
committed
2da28b3
compute: pre-allocated publication points (placeholder plus adopt-in-place)

antiguru
and
claude
committed
534c424
compute: registry get-or-create for publication points

antiguru
and
claude
committed
3384d49
compute: close and evict never-adopted placeholders

antiguru
and
claude
committed
b0f1ce3
compute-client: route to interactive by the bounded-read predicate

antiguru
and
claude
committed
f6b87fe
compute: render interactive dataflows in arrival order with late-bound imports

antiguru
and
claude
committed
c2e765c
compute: assert since <= as_of on the interactive import path

antiguru
and
claude
committed
0f36b46
compute: remove the dead live import(), keep import_snapshot_at

antiguru
and
claude
committed
5f51d14
compute: harden interactive routing tripwire and document eviction oks-first invariant

antiguru
and
claude
committed
f0a0490
compute: document that shared-import cleanup relies on the controller read-hold discipline

antiguru
and
claude
committed
ce61a15
compute: fix broken intra-doc links flagged by lint-doc

antiguru
and
claude
committed
273df48
compute: remove the dead insert/publish registry API superseded by adopt

antiguru
and
claude
committed
6e4f834
compute: collapse the now single-variant PendingWork enum

antiguru
and
claude
committed
b26a6cf
parallel-benchmark: encode HydrationChurn queries for psycopg execute

antiguru
and
claude
committed
ecbf731
compute: seed shared-trace importers with the full snapshot, not a frontier-filtered one

antiguru
and
claude
committed
8be6613
sqllogictest: rewrite relations.slt golden for shared-arrangement operators

antiguru
and
claude
committed
3867c25
compute: fire the shared-trace seal signal from the sink, not an upstream tap

antiguru
and
claude
committed
00a8591
compute: rewrite relations golden and fix doc link after the seal-signal move

antiguru
and
claude
committed
9783c4e
compute: drop DD fork, drive publisher compaction from AllowCompaction

antiguru
and
claude
committed
f786665
parallel-benchmark: halve TwoRuntimeReadIsolation read rates

antiguru
and
claude
committed
1550c97
compute: fix interactive_import tests for AllowCompaction-driven publisher

antiguru
and
claude
committed
4e39de8
parallel-benchmark: lower TwoRuntimeReadIsolation read probe to 50/12

antiguru
and
claude
committed
c106b13
doc: consolidate two-runtime design into a single document

antiguru
and
claude
committed
f3de43f
doc: record interactive-runtime observability open items

antiguru
and
claude
committed
183fc2a
compute: remove enable_index_arrangement_sharing dyncfg

antiguru
and
claude
committed
01b45b6
compute-client: make transient_owner a BTreeSet

antiguru
and
claude
committed
3799b93
clusterd-test-driver: use t-prefixed transient id in two-runtime spec

antiguru
and
claude
committed
7a27fdd
compute: police the shared-trace cut and seed importers with the chain's coverage
antiguru
committed
5cc4d33
compute: cut the unexercised sharing machinery, arm the routing tripwire
antiguru
committed
e24ce62
storage: port the Iceberg sink's batch stash to Arc-backed batches
antiguru
committed
2f960df
doc: correct the two-runtime design record and record the arrangement-size doubling
antiguru
committed
02df0ee
compute: let interactive peeks use the peek stash
antiguru
committed
270ca2f
compute: walk fast-path index peeks off the serving worker
antiguru
committed
7b110ae
compute: document what the peek-placement flags actually change
antiguru
committed
381972a
doc: experimental evaluation for the read-placement matrix
antiguru
committed
7425f36
compute: scope the peek-placement flags to replicas, and plan the arms around it
antiguru
committed
04001db
compute: drop the shared-trace cut precondition, it does not hold
antiguru
committed
6c38d75
compute: drop live batches an importer's seed already covers
antiguru
committed
39cf278
doc: write down the protocol invariant the runtime split breaks
antiguru
committed
0d36c2b
compute-client: cap compaction at what in-flight interactive dataflows read
antiguru
committed
ce5db21
compute: fix the lint failures from the protocol work
antiguru
committed
11fd652
compute-client: never let capping regress a compaction frontier
antiguru
committed
f2d39c3
controller: shorten the interactive port name, Kubernetes rejects 19 characters
antiguru
committed
2ec910d
doc: record the staging results for E6 and E1

antiguru
and
claude
committed
31de976
compute: count index peek walks by substrate

antiguru
and
claude
committed
e735cc1
doc: record that peeks broadcast regardless of replica targeting

antiguru
and
claude
committed
9329b87
doc: arms needing independent load need a cluster each

antiguru
and
claude
committed
7bfc5e6
doc: E1 and E2 measured, the offload alone removes head-of-line blocking

antiguru
and
claude
committed
d5f0628
doc: plan to bring the persist stash to the offloaded walk

antiguru
and
claude
committed
9556581
doc: E7 measures temporary dataflows, the second runtime's remaining case

antiguru
and
claude
committed
474a17c
controller: scope enable_two_runtime_compute to replicas

antiguru
and
claude
committed
e06d7f6
compute: let an offloaded peek walk divert to the stash

antiguru
and
claude
committed
53b00b1
doc: reruns with the stash on, and a first look at disk contention

antiguru
and
claude
committed
fa9b1c3
doc: swap with a matched control shows no regression

antiguru
and
claude
committed
c3254c0
doc: E9 measures the console under load, latency and staleness apart

antiguru
and
claude
committed
3ba44c0
doc: hydration memory is a sawtooth that overshoots about fourfold

antiguru
and
claude
committed
6cb0882
doc: the observability itself is the result

antiguru
and
claude
committed
e0e11ac
doc: E11 measures the skewed point lookup, a real customer pattern

antiguru
and
claude
committed
958b427
doc: how the branch splits into landable pieces

antiguru
and
claude
committed
cb44957
compute: fix defects found by adversarial review

antiguru
and
claude
committed
26b282d
doc: design of record reflects the fork, and the open review findings

antiguru
and
claude
committed
43bb0e2
doc: systems framing, the incremental path, and the serving-layer constraint

antiguru
and
claude
committed
2680679
doc: equal peer counts are a soundness requirement, not a sizing choice

antiguru
and
claude
committed
e39101f
doc: what the platform isolates, and the QoS constraint on pinning

antiguru
and
claude
committed
2523ebe
doc: correct the CPU isolation story, the class is Burstable

antiguru
and
claude
committed
98bd503
doc: problems and mechanisms as the spine

antiguru
and
claude
committed
b957d1c
doc: correct the mechanism table after adversarial review

antiguru
and
claude
committed
2dc11a3
doc: E12 measures the operator-activation mechanism, and inverts two cells

antiguru
and
claude
committed
c58dda2
doc: add freshness as a mechanism, with predictions registered before measuring

antiguru
and
claude
committed
ce67fc6
doc: E13 measures the freshness dual, refuting one registered prediction

antiguru
and
claude
committed
5c909f1
doc: complete the mechanism table, and say what is still missing from it

antiguru
and
claude
committed
831fdf9
doc: SUBSCRIBE slow start, and the exclusion that makes it unmovable

antiguru
and
claude
committed
d09cc95
doc: the import shape is not the obstacle, the read hold is

antiguru
and
claude
committed
ce59f54
doc: the interactive read hold needs early release, not advancing

antiguru
and
claude
committed
3b996d1
doc: compaction feedback exists, and two frozen holds defeat it

antiguru
and
claude
committed
300d1c2
doc: separate the cause of the inert physical hold from how it manifests

antiguru
and
claude
committed
866c347
doc: the shared handle reports the request, not the grant

antiguru
and
claude
committed
1cb8f39
doc: localise the physical-hold defect to two divergences from TraceAgent

antiguru
and
claude
committed
198fd52
doc: capture where the design exploration stands, and mark the split plan stale

antiguru
and
claude
committed
eeb22c7
compute: hold back shared-trace compaction correctly

antiguru
and
claude
committed
dad8727
compute: make the interactive import's read hold downgradeable

antiguru
and
claude
committed
05c1615
compute-client: retire interactive holds on confirmation, not on the drop

antiguru
and
claude
committed
2fc8abb
doc: replace the TLA+ protocol sketch with a checked Lean 4 model

antiguru
and
claude
committed
afcb80c
compute: give the shared import a hold that outlives construction

antiguru
and
claude
committed
a336399
compute: fix three defects in the shared-trace hold machinery

antiguru
and
claude
committed
c34f402
doc: design and TLA+ model for read holds across two runtimes

antiguru
and
claude
committed
a5bd883
compute-client: add the hold commands, without their semantics

antiguru
and
claude
committed
720b70c
doc: record the read-hold implementation sequence

antiguru
and
claude
committed
2ea9ca6
compute: install and maintain command-acquired read holds
antiguru
committed
939099b
doc: record what steps 1 and 2 settled, and that G3 dissolves
antiguru
committed
dd1678f
compute-client: synthesize the hold commands and delete the cap
antiguru
committed
b106806
doc: bring read-holds.md up to what was built
antiguru
committed
15a10fa
doc: make the TLA+ model check what was built
antiguru
committed
3c6f3e8
compute: format the rebase conflict resolution
antiguru
committed
87075f8
clusterd: depend on mz-compute-client again
antiguru
committed
516c48e
compute: stop linking module docs to private items
antiguru
committed
93c9f53
compute: derive physical compaction from readers' cut floors

antiguru
and
claude
committed
9383dea
compute: model broadcast compaction in the holds spec

antiguru
and
claude
committed
357e12f
compute: design broadcast compaction, superseding the hold protocol

antiguru
and
claude
committed
f279fe5
compute: broadcast compaction and assert I1c

antiguru
and
claude
committed
4e5a1ed
compute: delete the command-acquired hold layer

antiguru
and
claude
committed
0c14e23
compute: remodel the holds spec around stream positions

antiguru
and
claude
committed
7e93173
compute: do not broadcast a transient collection's compaction

antiguru
and
claude
committed
fa56da4
compute: report the standing hold when an import is refused

antiguru
and
claude
committed
775ea1f
compute: rename the flag to enable_compute_interactive_runtime

antiguru
and
claude
committed
4fbddd4
design: park peek placement as an orthogonal problem

antiguru
and
claude
committed
b1b562f
design: move the design docs and models to their own PR

antiguru
and
claude
committed
15e8be3
compute: move the peek offload to its own branch

antiguru
and
claude
committed
0406c04
compute: reconcile with the merged ComputeRuntimeRole and error-distinct work
antiguru
committed
3e03995
compute: walk fast-path index peeks off the serving worker

antiguru
and
claude
committed
f73b5ce
design: move the peek-placement documents to the offload work
antiguru
committed
f8fc314

Uh oh!

Commits

Commits on Aug 21, 2026