Skip to content

compute: select the arrange merge batcher by dyncfg - #38454

Merged
antiguru merged 1 commit into
MaterializeInc:mainfrom
antiguru:batcher-selector
Aug 25, 2026
Merged

compute: select the arrange merge batcher by dyncfg#38454
antiguru merged 1 commit into
MaterializeInc:mainfrom
antiguru:batcher-selector

Conversation

@antiguru

@antiguru antiguru commented Aug 25, 2026

Copy link
Copy Markdown
Member

Compute arrange sites could already choose between the columnation merge batcher and a columnar one, but only in the form of the paged batcher, so the columnation-versus-columnar comparison came entangled with the pager and its spill budget. This change separates the two axes. enable_columnar_merge_batcher selects the resident columnar batcher: the same Column chains and the same builder as the paged arm, merged through ColumnMerger, with no pager and no spill accounting. The columnation-versus-columnar question can now be measured on its own, and the paged arm becomes an increment on a columnar baseline rather than the only way to reach one.

The three type parameters mz_arrange_core takes are one unit rather than three independent knobs. Both the chunker's container and the builder's input are pinned to Batcher::Output, so the chunker and the builder follow from the batcher and each call site has to spell out a whole arm per variant. ArrangementBatcher in mz_compute::extensions::arrange names the three arms and resolves them from the config set in one place, so the two arrange sites, ArrangeBy in render::context and JoinStage in render::join::linear_join, no longer each encode the precedence rule. enable_column_paged_batcher keeps precedence, since it asks for the same columnar chains and additionally routes them through the pager, which leaves that flag's meaning and its rollout untouched.

Col2ValColBatcher in mz_timely_util::columnar is the new alias. It wraps ColumnMerger, which already existed and was exercised only by benchmarks, so this is the first production path through it. It reuses RowRowColPagedBuilder, whose Input is Column rather than PagedColumn and which therefore never depended on paging. The Paged in that builder's name records where it came from rather than a restriction, and its documentation now says so; the rename is left out of this change to keep the diff on the flag.

The flag is off in production and on in the CI configuration, so sqllogictest, testdrive, and the optimizer goldens exercise the columnar arm before it earns a production default. Parallel workload varies it alongside the paged flag. Only the two arrange sites named above are covered: logging arrangements are constructed before the replica's configuration is known and stay on the columnation batcher, and reduce, top-k, and threshold build batches through a builder with no merge batcher at all, so they have no arm to select.

Release notes: No user-facing changes.

🤖 Generated with Claude Code

@antiguru
antiguru requested review from a team as code owners August 25, 2026 11:44
@antiguru
antiguru requested a review from DAlperin August 25, 2026 11:59
Compute arrange sites could already choose between the columnation merge
batcher and a columnar one, but only in the form of the paged batcher, so
the columnation-versus-columnar comparison came entangled with the pager
and its spill budget. Separate the two axes:
`enable_columnar_merge_batcher` selects the resident columnar batcher,
which holds the same `Column` chains and feeds the same builder as the
paged arm but merges through `ColumnMerger` with no pager and no spill
accounting.

The three type parameters `mz_arrange_core` takes are one unit rather
than three independent knobs, because the chunker's container and the
builder's input are both pinned to `Batcher::Output`. `ArrangementBatcher`
names the three resulting arms and resolves them from the config set in
one place, so neither arrange site encodes the precedence rule.
`enable_column_paged_batcher` keeps precedence: it asks for the same
columnar chains and additionally routes them through the pager, so its
existing meaning and rollout are unchanged.

`Col2ValColBatcher` wraps `ColumnMerger`, which until now existed only
for benchmarks. It reuses `RowRowColPagedBuilder`, whose `Input` is
`Column` rather than `PagedColumn` and which therefore never depended on
paging; that builder's documentation now says so, while the rename its
name invites is left out to keep this change on the flag.

The flag is off in production and on in the CI configuration so the
columnar arm is exercised before it earns a production default, and
parallel workload varies it alongside the paged flag.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@antiguru
antiguru merged commit 32dcad4 into MaterializeInc:main Aug 25, 2026
83 checks passed
@antiguru

Copy link
Copy Markdown
Member Author

Thanks for the review!

@antiguru
antiguru deleted the batcher-selector branch August 25, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants