Add buffer-managed-state design doc and swap-extent buffer pool prototype - #36973

Draft

DAlperin wants to merge 41 commits into

MaterializeInc:mainfrom

DAlperin:dov/swap-pool-prototype

Commits on Jul 6, 2026

design: buffer-managed dataflow state
DAlperin
committed
b3471f9
design: integrate buffer-managed state with differential's Chunk abstraction
DAlperin
committed
ec40649
design: incremental migration and swap coexistence
DAlperin
committed
b0af742
design: swap-backed extent store for filesystem-less nodes
DAlperin
committed
0472cb5
prototype: swap-extent buffer pool, layers 2 and 3
DAlperin
committed
f35e9da
column-pager: wire buffer pool behind dyncfgs for staging
DAlperin
committed
5bd34ef
column-pager: log spill mechanism resolution for debugging
DAlperin
committed
81b4402
column-pager: keep tiered singleton configured in pool mode; resolve stash pager at render
DAlperin
committed
f483d57
design: batched-seek cursors as the probe prefetch story
DAlperin
committed
479dece
pool: off-worker spill threads for eviction I/O
DAlperin
committed
7c829c0
compute-types: default pool spill threads to 2
DAlperin
committed
bf0ce66
pool: insert_with writes serialized columns directly into the slot
DAlperin
committed
9806605
pool: 64 GiB class reservations; degrade to heap on slot exhaustion
DAlperin
committed
4cb565c
pool: scope slot lifetime to residency, dropping the stable-address invariant
DAlperin
committed
6cbb931
pool: single-flight budget enforcement over a resident-only queue
DAlperin
committed
bff2b60
pool: size extents to the compressed payload, not lz4 worst case
DAlperin
committed
f4d307c
pool: add 4 and 8 MiB size classes for boundary-overshooting chunks
DAlperin
committed
a16b9bb
storage: drain upsert stash seal chunk-at-a-time
DAlperin
committed
cafb6d8
design: record staging prototype findings; add remote-extents direction
DAlperin
committed
2eb7bda
pool: cleanup pass from complexity review
DAlperin
committed
a7a6dfc
storage: tidy upsert drain seal path
DAlperin
committed
425d738
timely-util: collapse column pager config globals
DAlperin
committed
673321a
pool: rename elided frees to writes elided
DAlperin
committed
f252b04
timely-util: keep short merge-batcher chains resident
DAlperin
committed
a989db9
compute: rename pool spill-threads dyncfg to column_paged_batcher_spill_worker_count
DAlperin
committed
dff1ee4
pool: warm slot reuse and hugepage-backed large classes
DAlperin
committed
b16844f
pool: eager backing to compressed-but-resident off the workers
DAlperin
committed
850ee2c
pool: compressed-resident extent tier under an RSS target
DAlperin
committed
3626c85
pool: park eager-backing threads when nothing is backable
DAlperin
committed
fe3ea9a
ore/pool: comprehension pass from the simplicity review
DAlperin
committed
e51f8a0
design: update buffer-managed-state doc to match the implementation
DAlperin
committed
44d03ea
Patch differential-dataflow to the Chunk-API branch
DAlperin
committed
90425a1
timely-util: ColumnChunk, differential's Chunk over pooled columns
DAlperin
committed
87583d2
storage/upsert: move the v2 stash to differential's ChunkBatcher
DAlperin
committed
3f480c6
storage/upsert: arrange persist feedback as chunk batches; drain via UnloadChunk
DAlperin
committed
bb49c7b
ore/pool: retire the pinned read surface and the pager opt-in seam
DAlperin
committed
e0a793b
compute: move the last paged-batcher consumers to the chunk world
DAlperin
committed
5178bb6
Delete the pager: mz_ore::pager, ColumnPager, ColumnMergeBatcher
DAlperin
committed
4d5bcd7
chunk: review fixes — disjoint-merge fast path, bounded advance cuts, windowed drain probes
DAlperin
committed
bf2c858
design: update buffer-managed-state doc for the Chunk-API architecture
DAlperin
committed
67464b6
Branch CI: fmt, source-allowlist, and dyncfg-coverage lint fixes
DAlperin
committed
b1e94a4