compute: consolidate operator output only while it recovers records - #38674
Open
frankmcsherry wants to merge 1 commit into
Open
compute: consolidate operator output only while it recovers records#38674frankmcsherry wants to merge 1 commit into
frankmcsherry wants to merge 1 commit into
Conversation
frankmcsherry
force-pushed
the
adaptive-consolidation
branch
2 times, most recently
from
September 5, 2026 00:23
e6373c4 to
c2bfe82
Compare
The reduce's key formation, explode_one, the MFP flat_map and FlatMap emit through a builder that sorts and consolidates every output chunk. That collapses repeated keys before the exchange when there are few distinct keys, and recovers nothing when there are many. Add a builder that measures what each consolidation recovers and passes chunks through unsorted while it stays under two percent, probing one chunk in thirty-two. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
frankmcsherry
force-pushed
the
adaptive-consolidation
branch
from
September 5, 2026 00:26
c2bfe82 to
f991ddd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Several render operators emit through differential's
ConsolidatingContainerBuilder, which sorts and consolidates every output chunk of about a thousand records before it leaves the operator. That collapses repeated(data, time)pairs before they are exchanged or arranged, a large saving when the data has few distinct keys. When it has many, the sort recovers nothing: aGROUP BYover 1.25M groups spread across 10M rows merges under one percent of each chunk, and the sort was the largest cost of the reduce's key formation and of the accumulator explosion.AdaptiveConsolidatingContainerBuilder(inmz-timely-util) keeps differential's buffering and measures what each consolidation recovers, as an exponential moving average in permille. While that stays below two percent it passes chunks through unsorted, consolidating one chunk in every thirty-two so a change in the data is noticed within that many chunks. Consumers never relied on the output being consolidated or ordered, so nothing downstream changes.It replaces the differential builder at four sites: the reduce's key formation,
explode_one(the accumulable reduce's accumulator explosion), the MFPflat_mapon collections, and theFlatMapoperator's Ok output. Error outputs and the join closures keep the differential builder.Hydration at 10M rows, one worker, median of three, on a local build with the other hydration PRs in flight applied:
SELECT k % g, count(*), sum(v) ... GROUP BY 1(1.25M groups)max(v)byk % gwithAGGREGATE INPUT GROUP SIZE = 8SELECT DISTINCT k, vThe downstream arrangements pay slightly more for unconsolidated input (ArrangeAccumulable +24ms); the sorts they no longer need paid far more.
Unit tests cover the three regimes: repeated keys stay consolidated, distinct keys stop the sort without losing records, and a probe resumes consolidation when keys start repeating.