compute: split rows by bytes when the arrangement key is a column prefix - #38665
Open
frankmcsherry wants to merge 1 commit into
Open
compute: split rows by bytes when the arrangement key is a column prefix#38665frankmcsherry wants to merge 1 commit into
frankmcsherry wants to merge 1 commit into
Conversation
`FormArrangementKey` turns each row into the arrangement's `(key, value)` pair by decoding every datum, evaluating the key expressions, and packing two fresh `Row`s datum by datum. When the key is the leading columns and the value the remaining columns, in order, key and value are two byte ranges of the encoded row and the only work needed is finding the boundary. `RowRef::split_at_datum(n)` splits a row after its first `n` datums into two `RowRef`s, decoding only those `n` datums. `arrange_collection` learns the arrangement's arity from the permutation the plan already carries and, when the key expressions are columns `0..k` in order and the thinning is `k..arity` in order, fills key and value from the two halves. Other shapes take the existing path. Hydrating an index on `(k)` over a 10M-row `(k, v)` table on one worker: the operator went from 0.59s to 0.29s and the hydration from 2.81s to 2.57s. A fact-to-dimension join with an index on its output, three such arrangements, went from 7.25s to 6.56s. An index on `SELECT DISTINCT k, v`, whose arrangement is not a prefix split, was unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
frankmcsherry
added a commit
to frankmcsherry/materialize
that referenced
this pull request
Sep 4, 2026
Split a row's bytes at a datum boundary without decoding past it. Also part of MaterializeInc#38665; whichever lands first carries it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
frankmcsherry
added a commit
to frankmcsherry/materialize
that referenced
this pull request
Sep 4, 2026
Split a row's bytes at a datum boundary without decoding past it. Also part of MaterializeInc#38665; whichever lands first carries it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
FormArrangementKeyturns each row into the arrangement's(key, value)pair by decoding every datum into aDatumVec, evaluating the key expressions, and packing two freshRows datum by datum. In the common case the key is the leading columns and the value is the remaining columns, in order (an index on(k)over(k, v), both inputs of an equi-join on the first column, the join's output arrangement). Then key and value are two byte ranges of the encoded row, and the only work needed is finding the boundary.Hydrating an index over a 10M-row table on one worker, this operator was 0.59s of 2.81s; in a fact-to-dimension join with an index on the output, two of them were 0.58s and 0.47s of 7.25s.
Change
RowRef::split_at_datum(n)splits a row after its firstndatums into twoRowRefs. Datums are encoded back to back, so both halves are complete row encodings; only the firstndatums are decoded, to find the boundary. It panics on fewer thanndatums.arrange_collectionreceives the arrangement's arity (the length of the permutation the plan already carries) and, when the key expressions are columns0..kin order and the thinning isk..arityin order, fills key and value from the two halves withextend_by_row_ref. Other shapes take the existing path.Measurements
One worker, 10M rows, three repetitions, same tree with and without the change:
FormArrangementKey(k)over(k, v)k, index on output(k)SELECT DISTINCT k, v(no prefix arrangement, control)Checked through fast-path lookups on one- and two-column prefix keys over a table with int, text, numeric, bool and jsonb columns, nulls, and a 300-byte string: every value half round-trips.
Testing
Unit tests for
split_at_datumcover every split point of a mixed-type row, including the empty halves, and the panic on a split past the end. Every sqllogictest with an index on a leading column exercises the render path.Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.Release notes
This release will not include user-visible changes.
🤖 Generated with Claude Code