compute: Emit dataflow import outputs as the columnar edge - #37786
Draft
antiguru wants to merge 3 commits into
Draft
compute: Emit dataflow import outputs as the columnar edge#37786antiguru wants to merge 3 commits into
antiguru wants to merge 3 commits into
Conversation
antiguru
force-pushed
the
columnar-p9-imports
branch
from
July 22, 2026 08:41
c6a1d6e to
862beda
Compare
antiguru
force-pushed
the
columnar-p9-imports
branch
from
July 22, 2026 16:24
862beda to
7fecbce
Compare
antiguru
force-pushed
the
columnar-p9-imports
branch
from
July 22, 2026 17:50
7fecbce to
8013d36
Compare
antiguru
force-pushed
the
columnar-p9-imports
branch
2 times, most recently
from
August 19, 2026 13:17
30395be to
7af78d1
Compare
antiguru
force-pushed
the
columnar-p9-imports
branch
from
August 20, 2026 08:49
7af78d1 to
0d72be7
Compare
antiguru
force-pushed
the
columnar-p9-imports
branch
from
August 20, 2026 09:13
0d72be7 to
22894de
Compare
Flip the three import producers to the columnar edge via a `vec_to_columnar` leaf-encode at the boundary: source imports in the recursive and non-recursive scopes, and the `SnapshotMode::Exclude` index import. These read row-shaped data from persist or a filtered trace, so encoding at the boundary is the accepted leaf-encode, symmetric to the join outputs and the reduce/join Vec-internal pattern. The batches are already consolidated and the prior `from_collections` was non-consolidating, so a plain `vec_to_columnar` (non-consolidating) is correct; no consolidating builder is added. The persist/trace decode machinery is untouched. `from_collections` -> `from_edge`; err stays `Vec`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The import boundary now encodes outputs to the columnar edge, inserting a VecToColumnar operator. Reflect the new conversion channel and the Vec->Column flip on the arrange input in the introspection golden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment-only, no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
antiguru
force-pushed
the
columnar-p9-imports
branch
from
September 6, 2026 18:23
22894de to
c30a407
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dataflow imports (persist sources, index/trace imports) emit the columnar edge via a
vec_to_columnarleaf-encode at the import output. Includes therelations.sltgolden update for the resultingVecToColumnarboundary operator.Columnar dataflow-edge migration. Design doc:
doc/developer/design/20260720_columnar_dataflow_edges.md(#37744).Part of CPU-51.