compute: Re-encode LetRec read-edges to the columnar edge - #37794
Draft
antiguru wants to merge 3 commits into
Draft
compute: Re-encode LetRec read-edges to the columnar edge#37794antiguru wants to merge 3 commits into
antiguru wants to merge 3 commits into
Conversation
antiguru
force-pushed
the
columnar-tp-letrec-reencode
branch
from
July 22, 2026 08:41
b045d8b to
53a7cb3
Compare
antiguru
force-pushed
the
columnar-tp-letrec-reencode
branch
from
July 22, 2026 16:25
53a7cb3 to
46a3139
Compare
antiguru
force-pushed
the
columnar-tp-letrec-reencode
branch
from
July 22, 2026 17:50
46a3139 to
e507d11
Compare
antiguru
force-pushed
the
columnar-tp-letrec-reencode
branch
from
August 19, 2026 12:01
e507d11 to
fb1e5f7
Compare
antiguru
force-pushed
the
columnar-tp-letrec-reencode
branch
2 times, most recently
from
August 20, 2026 08:49
b904adb to
fb2a7a1
Compare
antiguru
force-pushed
the
columnar-tp-letrec-reencode
branch
from
August 20, 2026 09:13
fb2a7a1 to
dd93b9e
Compare
A rec binding's collection edge was `Vec`: the in-loop feedback bundle and the outer-scope extraction both built the bundle from a `VecCollection`. Union reads its inputs' `.collection` edges directly, so an identity `Get` on a rec binding fed a `Vec` input into `concat_many`, the last source of a `Vec` edge into a Union. Re-encode the read-edge to columnar via `vec_to_columnar` at both sites so `Get`s on a rec binding see a columnar edge. The feedback `Variable` stays `Vec` (the recursive value still flows `Vec` through the loop); only the externally-visible collection is re-containered. `vec_to_columnar` is a stateless, timestamp-agnostic, non-consolidating pass-through, so in the iterative scope it behaves like any loop-body operator and does not alter the feedback frontier or fixed-point behavior. The value is already consolidated (`LetRecConsolidation`) before the read, so the re-encode just re-containers it. With this, a Union over a rec binding receives all-columnar inputs, and `from_collections` (the last `CollectionEdge::Vec` producer constructor for real data) has no callers, so remove it. Adds a `with_mutually_recursive` case whose recursive term is a bare identity `Get` placed directly as a `Union` input. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment-only, no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The rendered value was decoded twice, by two independent operators over the same stream: once where the binding is fed back, and again where it is extracted into the outer scope. Each decode allocates an owned row per record per iteration. Keep the first decode and reuse it. Also correct the comment above, which said the recursive value flows as `Vec` through the loop. It does not: the read edge is encoded for the readers and the value is decoded again where it is fed back, so each iteration crosses the container boundary twice.
antiguru
force-pushed
the
columnar-tp-letrec-reencode
branch
from
September 6, 2026 18:23
dd93b9e to
520c038
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-encode LetRec recursive-binding read-edges to the columnar edge; the feedback
VariablestaysVec. Removes the second mixed-variant edge source feedingconcat_many.Columnar dataflow-edge migration. Design doc:
doc/developer/design/20260720_columnar_dataflow_edges.md(#37744).Part of CPU-51.