compute: columnar Get/Mfp output via as_collection_core - #37763
Draft
antiguru wants to merge 1 commit into
Draft
Conversation
antiguru
force-pushed
the
columnar-p1-get-mfp-output
branch
2 times, most recently
from
July 22, 2026 08:41
8dab0e8 to
2e5920e
Compare
antiguru
force-pushed
the
columnar-p1-get-mfp-output
branch
from
July 22, 2026 16:24
2e5920e to
c420261
Compare
antiguru
changed the title
antiguru
force-pushed
the
columnar-p1-get-mfp-output
branch
from
July 22, 2026 17:50
c420261 to
47a405c
Compare
antiguru
force-pushed
the
columnar-p1-get-mfp-output
branch
2 times, most recently
from
August 19, 2026 13:17
0a3b022 to
570a36a
Compare
antiguru
force-pushed
the
columnar-p1-get-mfp-output
branch
from
August 20, 2026 08:49
570a36a to
b01857d
Compare
antiguru
force-pushed
the
columnar-p1-get-mfp-output
branch
from
August 20, 2026 09:13
b01857d to
176a1c1
Compare
Make `as_collection_core` build its ok output into a `ConsolidatingColumnBuilder` and return a `CollectionEdge::Columnar`, flipping the Get and Mfp producers to emit the columnar edge. This is the first producer flip: the columnar arm goes live for Get/Mfp-fed edges, which Wave 1 already made every consumer accept natively or decode only at a sanctioned leaf. Rework the identity fast-path so an unarranged trivial MFP hands the input edge straight through (a columnar producer stays columnar with no `ColumnarToVec` hop) instead of forcing `as_specific_collection` to decode it to `Vec`. A keyed identity still reads its arrangement as a `Vec` edge. `as_specific_collection` stays the row-based consumer leaf. `ConsolidatingColumnBuilder` folds within-batch duplicates, matching the row-based `ConsolidatingContainerBuilder` this replaced, and emits a `Column` instead of a `Vec`. It stages owned `(Row, T, Diff)` tuples to consolidate in place, so the producer gives its records owned. `mfp_plan.evaluate` already produces a fresh owned `Row` per result, so this is a move into staging, not a new allocation. The borrowed-push no-owned-Row pattern is a consumer optimization for reading an existing columnar batch; it never applied to a producer computing new rows. The internal fueled `flat_map` chain (`CollectionBundle::flat_map`, `ArrangementFlavor::flat_map`, `flat_map_core_fallible`, `PendingWork:: do_work`) carried a `PushInto<(D, T, Diff)>` bound that constrained the element type up front. The bound was never exercised by those bodies (the caller's `logic` performs the `give`), so relax it to `DCB: ContainerBuilder` and drop the now-unused `D`, letting the producer choose its own builder and push shape through the same fuel machinery. Sink and `ensure_collections` temporal bucketing decode the produced edge at their existing sanctioned leaves. Tests: a `Get -> ArrangeBy` end-to-end unit test asserting the producer emits columnar and the arrange input keeps the columnar passthrough; an identity-fast-path test asserting the input edge variant is preserved; and a within-batch consolidation test asserting duplicate output rows fold to a single record with summed diff. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
antiguru
force-pushed
the
columnar-p1-get-mfp-output
branch
from
September 6, 2026 18:23
176a1c1 to
489ab17
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First producer flip: Get/Mfp emit their output as the columnar edge via
as_collection_core, building aConsolidatingColumnBuilderso within-batch consolidation matches the priorConsolidatingContainerBuilder.Columnar dataflow-edge migration. Design doc:
doc/developer/design/20260720_columnar_dataflow_edges.md(#37744).Part of CPU-51.