Skip to content

compute: split rows by bytes when the arrangement key is a column prefix - #38665

Open
frankmcsherry wants to merge 1 commit into
MaterializeInc:mainfrom
frankmcsherry:arrange-prefix-split
Open

compute: split rows by bytes when the arrangement key is a column prefix#38665
frankmcsherry wants to merge 1 commit into
MaterializeInc:mainfrom
frankmcsherry:arrange-prefix-split

Conversation

@frankmcsherry

Copy link
Copy Markdown
Contributor

Motivation

FormArrangementKey turns each row into the arrangement's (key, value) pair by decoding every datum into a DatumVec, evaluating the key expressions, and packing two fresh Rows datum by datum. In the common case the key is the leading columns and the value is the remaining columns, in order (an index on (k) over (k, v), both inputs of an equi-join on the first column, the join's output arrangement). Then key and value are two byte ranges of the encoded row, and the only work needed is finding the boundary.

Hydrating an index over a 10M-row table on one worker, this operator was 0.59s of 2.81s; in a fact-to-dimension join with an index on the output, two of them were 0.58s and 0.47s of 7.25s.

Change

  • RowRef::split_at_datum(n) splits a row after its first n datums into two RowRefs. Datums are encoded back to back, so both halves are complete row encodings; only the first n datums are decoded, to find the boundary. It panics on fewer than n datums.
  • arrange_collection receives the arrangement's arity (the length of the permutation the plan already carries) and, when the key expressions are columns 0..k in order and the thinning is k..arity in order, fills key and value from the two halves with extend_by_row_ref. Other shapes take the existing path.

Measurements

One worker, 10M rows, three repetitions, same tree with and without the change:

shape before after FormArrangementKey
index on (k) over (k, v) 2.81s 2.57s 0.59s to 0.29s
fact join dim on k, index on output (k) 7.25s 6.56s 0.58s and 0.47s to 0.28s each
index on SELECT DISTINCT k, v (no prefix arrangement, control) 4.12s 4.12s

Checked through fast-path lookups on one- and two-column prefix keys over a table with int, text, numeric, bool and jsonb columns, nulls, and a 300-byte string: every value half round-trips.

Testing

Unit tests for split_at_datum cover every split point of a mixed-type row, including the empty halves, and the panic on a split past the end. Every sqllogictest with an index on a leading column exercises the render path.

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label.
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

Release notes

This release will not include user-visible changes.

🤖 Generated with Claude Code

`FormArrangementKey` turns each row into the arrangement's `(key, value)`
pair by decoding every datum, evaluating the key expressions, and packing
two fresh `Row`s datum by datum. When the key is the leading columns and
the value the remaining columns, in order, key and value are two byte
ranges of the encoded row and the only work needed is finding the
boundary.

`RowRef::split_at_datum(n)` splits a row after its first `n` datums into
two `RowRef`s, decoding only those `n` datums. `arrange_collection` learns
the arrangement's arity from the permutation the plan already carries and,
when the key expressions are columns `0..k` in order and the thinning is
`k..arity` in order, fills key and value from the two halves. Other shapes
take the existing path.

Hydrating an index on `(k)` over a 10M-row `(k, v)` table on one worker:
the operator went from 0.59s to 0.29s and the hydration from 2.81s to
2.57s. A fact-to-dimension join with an index on its output, three such
arrangements, went from 7.25s to 6.56s. An index on `SELECT DISTINCT k, v`,
whose arrangement is not a prefix split, was unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@frankmcsherry
frankmcsherry requested a review from a team as a code owner September 4, 2026 18:20
frankmcsherry added a commit to frankmcsherry/materialize that referenced this pull request Sep 4, 2026
Split a row's bytes at a datum boundary without decoding past it. Also part
of MaterializeInc#38665; whichever lands first carries it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
frankmcsherry added a commit to frankmcsherry/materialize that referenced this pull request Sep 4, 2026
Split a row's bytes at a datum boundary without decoding past it. Also part
of MaterializeInc#38665; whichever lands first carries it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant