compute: Re-encode LetRec read-edges to the columnar edge - #37794
compute: Re-encode LetRec read-edges to the columnar edge#37794antiguru wants to merge 5 commits into
Conversation
b045d8b to
53a7cb3
Compare
53a7cb3 to
46a3139
Compare
46a3139 to
e507d11
Compare
e507d11 to
fb1e5f7
Compare
b904adb to
fb2a7a1
Compare
fb2a7a1 to
dd93b9e
Compare
dd93b9e to
520c038
Compare
520c038 to
0faa280
Compare
0fb4ae1 to
5db06fc
Compare
QA LLM Review1. MEDIUM -- Rec-binding
|
|
Confirmed and fixed in The reasoning holds as written. Variables are installed before the rec loop and each binding's real bundle replaces its variable bundle at the end of its own iteration, so while Two details worth recording. Went with the check as suggested, naming the accumulator
Left the outer extraction site at Posted by Claude Code |
526eac3 to
1f30d0b
Compare
1f30d0b to
29bae81
Compare
A rec binding's collection edge was `Vec`: the in-loop feedback bundle and the outer-scope extraction both built the bundle from a `VecCollection`. Union reads its inputs' `.collection` edges directly, so an identity `Get` on a rec binding fed a `Vec` input into `concat_many`, the last source of a `Vec` edge into a Union. Re-encode the read-edge to columnar via `vec_to_columnar` at both sites so `Get`s on a rec binding see a columnar edge. The feedback `Variable` stays `Vec` (the recursive value still flows `Vec` through the loop); only the externally-visible collection is re-containered. `vec_to_columnar` is a stateless, timestamp-agnostic, non-consolidating pass-through, so in the iterative scope it behaves like any loop-body operator and does not alter the feedback frontier or fixed-point behavior. The value is already consolidated (`LetRecConsolidation`) before the read, so the re-encode just re-containers it. With this, a Union over a rec binding receives all-columnar inputs, and `from_collections` (the last `CollectionEdge::Vec` producer constructor for real data) has no callers, so remove it. Adds a `with_mutually_recursive` case whose recursive term is a bare identity `Get` placed directly as a `Union` input. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment-only, no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The rendered value was decoded twice, by two independent operators over the same stream: once where the binding is fed back, and again where it is extracted into the outer scope. Each decode allocates an owned row per record per iteration. Keep the first decode and reuse it. Also correct the comment above, which said the recursive value flows as `Vec` through the loop. It does not: the read edge is encoded for the readers and the value is decoded again where it is fed back, so each iteration crosses the container boundary twice.
A rec binding's `Variable` serves `Get`s only until the rec loop binds the real value, so its readers are exactly the values of `recs[0..=i]`. Building the columnar re-encode for a binding no such value reads leaves a `VecToColumnar` operator whose output has no targets, and it repacks that binding's whole collection into columnar buffers on every iteration and drops it. The first-rendered binding of a mutually recursive SCC hits this, since the other binding's value reads the already-bound bundle instead. Install the `Variable`-backed bundle only for the bindings whose `Variable` some value reads. The rest get their bundle when the rec loop inserts the rendered value, which is what every remaining `Get` on them resolves to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keep the boundary crossing and the frontier guarantee, drop the rest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
29bae81 to
3043cf2
Compare
Re-encode LetRec recursive-binding read-edges to the columnar edge; the feedback
VariablestaysVec. Removes the second mixed-variant edge source feedingconcat_many.Columnar dataflow-edge migration. Design doc:
doc/developer/design/20260720_columnar_dataflow_edges.md(#37744).Part of CPU-51.