Conversation
Add prompts.md, log.md, and log-detailed.md documenting the phased plan to convert rendering from row-first (Vec<(Row, T, Diff)>) to column-first representation using the columnar crate. The plan has 10 phases and 18 prompts, designed so each step compiles and passes tests independently. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ection field Add `ColumnarCollection<S, D, R>` type alias backed by `Column<(D, T, R)>` containers as the columnar equivalent of `VecCollection`. Add `columnar_collection` field to `CollectionBundle` (initially always None) with helper methods: `from_columnar_collections`, `columnar_collection()`, and `ensure_vec_collection()` escape hatch for unconverted operators. https://claude.ai/code/session_01M7GpFugaDcNYDrxPmKDVVh
Implement stream operators for converting between VecCollection and ColumnarCollection representations. vec_to_columnar uses ColumnBuilder for automatic ~2MB batch sizing; columnar_to_vec iterates columnar containers and emits owned tuples. Includes round-trip unit tests. https://claude.ai/code/session_01M7GpFugaDcNYDrxPmKDVVh
Claude/complete prompt task f rlg l
At the import boundary in render.rs, convert Vec-based collections from persist_source into columnar format using vec_to_columnar. Both Vec and columnar collections are stored in the CollectionBundle so downstream operators continue working unchanged via the Vec path while the columnar variant is available for future operator conversions. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Add negate_columnar() that negates diffs in columnar containers without unpacking rows. The Negate match arm now checks for a columnar collection first and produces columnar output, falling back to Vec path otherwise. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
When all Union inputs have columnar collections, concatenate them directly using differential_dataflow::collection::concatenate on Column containers. Falls back to Vec path when any input lacks columnar data. Consolidation round-trips through Vec when needed. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Convert the Constant match arm in render.rs to produce a columnar collection alongside the existing Vec collection, using vec_to_columnar. This ensures all source-like operators now populate the columnar_collection field for downstream operators to consume. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
…ompt 3.1) Add as_columnar_collection_core method that applies MFP row-at-a-time (delegating to as_collection_core) and converts results to columnar. Wire Get::Collection and Mfp arms in render_plan_expr to prefer the columnar path when input has a columnar collection. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
… 4.1) Convert render_flat_map to handle columnar collections at boundaries. When input has columnar, converts to Vec for row-at-a-time table function evaluation, then converts output back to columnar. Table functions are inherently row-at-a-time so no vectorization is attempted. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
When ensure_collections needs a Vec collection for arrangement creation and only columnar is available, convert columnar→Vec first via ensure_vec_collection(). This prevents panics when upstream operators produce columnar-only bundles. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Add ensure_vec_collection() guard in render_reduce so that columnar-only inputs are converted to Vec before the flat_map that extracts keys and values. Key extraction and aggregation remain row-at-a-time. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Add ensure_vec_collection() guard in render_topk so columnar-only inputs are converted to Vec before as_specific_collection. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Threshold operates entirely on arrangements via arrangement(&key), never accessing unarranged collections. No code changes needed — columnar-only bundles work correctly. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Add ensure_vec_collection() guard in render_join_inner's fallback path so columnar-only inputs are converted to Vec before as_specific_collection. Join stages continue to operate on arrangements unchanged. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Delta joins operate entirely on arrangements for input/join stages. Convert the Vec output to columnar when any input had a columnar collection, maintaining columnar flow through the dataflow graph. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Handle columnar-only bundles in export_sink by converting to Vec via ensure_vec_collection(). Sinks write to persist which requires Row-format data, so columnar→Vec conversion is the correct terminal behavior. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
…pt 9.1) Source operators (persist imports, Constant) now produce columnar-only bundles instead of maintaining both Vec and columnar. All downstream operators handle columnar-only via ensure_vec_collection() guards. Added tracing::debug! in ensure_vec_collection to track fallback usage. Fixed PassArrangements assertion to accept columnar collections. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Remove the Vec-based `collection` field entirely. Data now flows exclusively through `columnar_collection`. `from_collections` auto-converts Vec→columnar. Added `as_vec_collection()` for on-demand columnar→Vec conversion at operator boundaries. Removed `ensure_vec_collection` and all its callers. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
…0.1) Research findings on feasibility of column-of-datums arrangement spines: - Current DatumContainer stores rows as contiguous bytes with offset indexing - Columnar spines require schema propagation, new BatchContainer impls, modified merge/cursor logic, and vectorized eval (PR MaterializeInc#35464) as prereq - Recommended phased approach starting with vectorized MFP evaluation https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
- Use Rc::clone(&results) instead of results.clone() for Rc pointers - Replace vec![...] with array literal where Vec is unnecessary - Replace Iterator::zip (disallowed) with direct assert_eq https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
…atches New prompts 11.1–11.6 to eliminate columnar→Vec conversions by operating directly on &RowRef from columnar containers. Key insight: DatumVec's borrow_with already accepts &RowRef (the columnar Ref<'_, Row> type), so operators can process columnar data without materializing owned Rows. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
…rompt 11.1) Add a columnar path to CollectionBundle::flat_map that iterates the columnar container via into_index_iter(), passing &RowRef directly to borrow_with_limit. This eliminates the columnar→Vec conversion and avoids allocating owned Row values. Only timestamps and diffs are converted to owned (cheap scalar copies). https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
…tly (Prompt 11.2) Add as_specific_columnar_collection that returns the columnar collection without conversion when key is None. Optimize as_columnar_collection_core to detect identity MFPs and skip the columnar→Vec→columnar round-trip. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Identity MFPs return columnar directly (11.2). Non-identity MFPs iterate columnar via &RowRef (11.1) but output is Vec-based due to map_fallible Ok/Err split. Updated doc comment to reflect current state. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
render_reduce calls flat_map which now iterates &RowRef directly from columnar containers (11.1). No Vec conversion needed for key/value extraction. Verification only, no code changes. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
…t 11.5) Add a columnar path to render_flat_map that uses unary_fallible directly on Column<(Row, T, Diff)> containers. Iterates &RowRef without allocating owned Rows for expression evaluation. Changed drain_through_mfp to accept &RowRef. Vec fallback retained for arrangement key paths. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
…ctly (Prompt 11.6) Add arrange_columnar_collection that takes ColumnarCollection and iterates &RowRef from columnar containers for key/value expression evaluation, avoiding the columnar→Vec conversion. Wire ensure_collections to use it when identity MFP + no input_key + columnar available. The passthrough stream stays columnar throughout the arrangement loop. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Replace Columnar::into_owned with copy_from on reusable buffers in all columnar iteration loops. This avoids allocating new Row/Timestamp/Diff values each iteration, reusing the buffer's existing allocation instead. Affected operators: ColumnarToVec, NegateColumnar, ColumnarFlatMap, FlatMapStageColumnar, FormArrangementKeyColumnar. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
Change the flat_map closure signature from (DatumVecBorrow, T, Diff) to (DatumVecBorrow, &T, &Diff). This eliminates unnecessary clones in the columnar path (references to copy_from buffers are passed directly) and in the arrangement path (owned values from buffer.drain are passed by reference). Callers clone/copy only when they actually need ownership. https://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d
antiguru
force-pushed
the
columnar_rendering
branch
3 times, most recently
from
April 10, 2026 12:54
b20d40a to
eee765a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes the columnar rendering migration (Prompts 2.3 through 10.1), building on the foundation from the earlier PR (Prompts 0.1–2.2).
Changes by phase
as_columnar_collection_coremethod; wiredGet::CollectionandMfpto prefer columnar pathensure_collectionshandles columnar-only inputs for arrangement creationcollectionfield fromCollectionBundleentirely. Data flows exclusively throughcolumnar_collection.from_collectionsauto-converts Vec→columnar. Addedas_vec_collection()for on-demand conversion at operator boundaries. Net -50 lines.Key architectural decisions
as_vec_collection()converts columnar→Vec at operator boundaries (arrangements, sinks, etc.)from_collectionsauto-converts: Operators producing Vec output seamlessly convert to columnarDataflowErroris not suited for columnar layoutSkipped
DatumColumn/ColumnDatum/rows_to_columns/MfpPlan::evaluate_batch) which is not present in the codebaseTest plan
cargo check -p mz-computepassescargo check -p mz-compute --testspassescargo clippy -p mz-compute --all-targets— zero warningsbin/fmt(rustfmt) passesrender/columnar.rscover round-trip, negate, union, and constant conversionshttps://claude.ai/code/session_01JHo5sTCSGPW5NavNE2b49d