compute: select the arrange merge batcher by dyncfg - #38454
Merged
Conversation
DAlperin
approved these changes
Aug 25, 2026
Compute arrange sites could already choose between the columnation merge batcher and a columnar one, but only in the form of the paged batcher, so the columnation-versus-columnar comparison came entangled with the pager and its spill budget. Separate the two axes: `enable_columnar_merge_batcher` selects the resident columnar batcher, which holds the same `Column` chains and feeds the same builder as the paged arm but merges through `ColumnMerger` with no pager and no spill accounting. The three type parameters `mz_arrange_core` takes are one unit rather than three independent knobs, because the chunker's container and the builder's input are both pinned to `Batcher::Output`. `ArrangementBatcher` names the three resulting arms and resolves them from the config set in one place, so neither arrange site encodes the precedence rule. `enable_column_paged_batcher` keeps precedence: it asks for the same columnar chains and additionally routes them through the pager, so its existing meaning and rollout are unchanged. `Col2ValColBatcher` wraps `ColumnMerger`, which until now existed only for benchmarks. It reuses `RowRowColPagedBuilder`, whose `Input` is `Column` rather than `PagedColumn` and which therefore never depended on paging; that builder's documentation now says so, while the rename its name invites is left out to keep this change on the flag. The flag is off in production and on in the CI configuration so the columnar arm is exercised before it earns a production default, and parallel workload varies it alongside the paged flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
antiguru
force-pushed
the
batcher-selector
branch
from
August 25, 2026 13:29
8407ec2 to
664988f
Compare
Member
Author
|
Thanks for the review! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compute arrange sites could already choose between the columnation merge batcher and a columnar one, but only in the form of the paged batcher, so the columnation-versus-columnar comparison came entangled with the pager and its spill budget. This change separates the two axes.
enable_columnar_merge_batcherselects the resident columnar batcher: the sameColumnchains and the same builder as the paged arm, merged throughColumnMerger, with no pager and no spill accounting. The columnation-versus-columnar question can now be measured on its own, and the paged arm becomes an increment on a columnar baseline rather than the only way to reach one.The three type parameters
mz_arrange_coretakes are one unit rather than three independent knobs. Both the chunker's container and the builder's input are pinned toBatcher::Output, so the chunker and the builder follow from the batcher and each call site has to spell out a whole arm per variant.ArrangementBatcherinmz_compute::extensions::arrangenames the three arms and resolves them from the config set in one place, so the two arrange sites,ArrangeByinrender::contextandJoinStageinrender::join::linear_join, no longer each encode the precedence rule.enable_column_paged_batcherkeeps precedence, since it asks for the same columnar chains and additionally routes them through the pager, which leaves that flag's meaning and its rollout untouched.Col2ValColBatcherinmz_timely_util::columnaris the new alias. It wrapsColumnMerger, which already existed and was exercised only by benchmarks, so this is the first production path through it. It reusesRowRowColPagedBuilder, whoseInputisColumnrather thanPagedColumnand which therefore never depended on paging. ThePagedin that builder's name records where it came from rather than a restriction, and its documentation now says so; the rename is left out of this change to keep the diff on the flag.The flag is off in production and on in the CI configuration, so sqllogictest, testdrive, and the optimizer goldens exercise the columnar arm before it earns a production default. Parallel workload varies it alongside the paged flag. Only the two arrange sites named above are covered: logging arrangements are constructed before the replica's configuration is known and stay on the columnation batcher, and reduce, top-k, and threshold build batches through a builder with no merge batcher at all, so they have no arm to select.
Release notes: No user-facing changes.
🤖 Generated with Claude Code