Draft: SPIRE materialization prune handoff from Task 122

reviews/task-122/002-spire-bounded-materialization-prune/artifacts/cargo-test-spire-diagnostics.log

@@ -0,0 +1,21 @@
+Command:
+
+`cargo test -p ecaz --lib --no-default-features --features pg18 collect_scan_placement_diagnostics_reports_candidate_truncation`
+
+Result:
+
+```text
+Finished `test` profile [unoptimized + debuginfo] target(s) in 0.24s
+Running unittests src/lib.rs (target/debug/deps/ecaz-cf818e07e808d9a6)
+
+running 1 test
+test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_reports_candidate_truncation ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2219 filtered out; finished in 0.05s
+```
+
+Build warnings:
+
+```text
+warning: ecaz@0.1.1: csrc/pg18_pgstat_shim.c included PostgreSQL 18 headers that emitted 6 unused-parameter warnings.
+```

reviews/task-122/002-spire-bounded-materialization-prune/artifacts/manifest.md

@@ -0,0 +1,14 @@
+# Artifact Manifest
+
+- Head SHA: `6efa5997de16463f03eb3ddd382a12b2ff00c389`
+- Task bucket: `reviews/task-122`
+- Packet path: `reviews/task-122/002-spire-bounded-materialization-prune`
+- Timestamp: 2026-06-26
+- Lane / fixture / storage format / rerank mode: SPIRE local unit validation; TurboQuant bounded candidate retention path.
+- Isolation: unit test only; no index/table benchmark run.
+- Commands:
+  - `cargo test -p ecaz --lib --no-default-features --features pg18 collect_scan_placement_diagnostics_reports_candidate_truncation`
+- Key result lines:
+  - `running 1 test`
+  - `test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_reports_candidate_truncation ... ok`
+  - `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2219 filtered out; finished in 0.05s`

reviews/task-122/002-spire-bounded-materialization-prune/request.md

@@ -0,0 +1,45 @@
+# Task 122 Packet 002: SPIRE Bounded Materialization Prune
+
+## Summary
+
+This is the first behavior slice for Task 122 Phase 2.
+
+SPIRE V2 leaf scans already score column payloads in a batch. After that batch
+score, bounded non-deduped scans can know the current heap threshold before
+materializing a row. This change skips row materialization when all of the
+following are true:
+
+- the candidate is visible from flags;
+- the score is finite;
+- there are no delete-delta vec_ids that require row materialization before
+  delete filtering;
+- candidate dedupe is disabled;
+- the bounded heap already has a minimum inner-product threshold;
+- the candidate inner product is strictly below that threshold.
+
+Skipped rows are reported through the existing truncated-candidate diagnostics,
+so candidate accounting remains visible without introducing a new SQL-visible
+diagnostic column in this slice.
+
+## Scope
+
+Touched file:
+
+- `src/am/ec_spire/scan/candidates.rs`
+
+This keeps the existing block scorer path intact: the column payload slab is
+still batch-scored before the pruning check.
+
+## Validation
+
+Passed:
+
+- `cargo test -p ecaz --lib --no-default-features --features pg18 collect_scan_placement_diagnostics_reports_candidate_truncation`
+
+See `artifacts/cargo-test-spire-diagnostics.log`.
+
+## Measurement
+
+No benchmark claim is made in this packet. The next required step is an
+`ecaz bench suite` A/B run for SPIRE bounded TQ scans at 10k/50k/100k before
+claiming latency or materialization wins.

reviews/task-122/003-spire-batched-materialization-prune/artifacts/cargo-test-spire-placement-diagnostics.log

@@ -0,0 +1,33 @@
+warning: ecaz@0.1.1: In file included from csrc/pg18_pgstat_shim.c:10:
+warning: ecaz@0.1.1: In file included from /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:19:
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:537:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:   537 | dlist_next_node(dlist_head *head, dlist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:547:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:   547 | dlist_prev_node(dlist_head *head, dlist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:1054:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:  1054 | slist_next_node(slist_head *head, slist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: In file included from csrc/pg18_pgstat_shim.c:10:
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:875:58: warning: unused parameter 'size' [-Wunused-parameter]
+warning: ecaz@0.1.1:   875 | pgstat_cmp_hash_key(const void *a, const void *b, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                          ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:875:70: warning: unused parameter 'arg' [-Wunused-parameter]
+warning: ecaz@0.1.1:   875 | pgstat_cmp_hash_key(const void *a, const void *b, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                                      ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:882:56: warning: unused parameter 'arg' [-Wunused-parameter]
+warning: ecaz@0.1.1:   882 | pgstat_hash_hash_key(const void *d, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                        ^
+warning: ecaz@0.1.1: 6 warnings generated.
+    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.09s
+     Running unittests src/lib.rs (target/debug/deps/ecaz-cf818e07e808d9a6)
+
+running 4 tests
+test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_reports_candidate_truncation ... ok
+test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_reports_boundary_dedupe_and_winners ... ok
+test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_skips_degraded_unavailable_leaf ... ok
+test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_counts_routed_store_rows ... ok
+
+test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 2216 filtered out; finished in 0.04s
+

reviews/task-122/003-spire-batched-materialization-prune/artifacts/cargo-test-spire-truncation.log

@@ -0,0 +1,30 @@
+warning: ecaz@0.1.1: In file included from csrc/pg18_pgstat_shim.c:10:
+warning: ecaz@0.1.1: In file included from /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:19:
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:537:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:   537 | dlist_next_node(dlist_head *head, dlist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:547:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:   547 | dlist_prev_node(dlist_head *head, dlist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:1054:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:  1054 | slist_next_node(slist_head *head, slist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: In file included from csrc/pg18_pgstat_shim.c:10:
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:875:58: warning: unused parameter 'size' [-Wunused-parameter]
+warning: ecaz@0.1.1:   875 | pgstat_cmp_hash_key(const void *a, const void *b, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                          ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:875:70: warning: unused parameter 'arg' [-Wunused-parameter]
+warning: ecaz@0.1.1:   875 | pgstat_cmp_hash_key(const void *a, const void *b, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                                      ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:882:56: warning: unused parameter 'arg' [-Wunused-parameter]
+warning: ecaz@0.1.1:   882 | pgstat_hash_hash_key(const void *d, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                        ^
+warning: ecaz@0.1.1: 6 warnings generated.
+    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.10s
+     Running unittests src/lib.rs (target/debug/deps/ecaz-cf818e07e808d9a6)
+
+running 1 test
+test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_reports_candidate_truncation ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2219 filtered out; finished in 0.05s
+

reviews/task-122/003-spire-batched-materialization-prune/artifacts/manifest.md

@@ -0,0 +1,20 @@
+# Artifact Manifest
+
+- Head SHA: `fe9ff76b7c5c1379aa7b9ffb213e0865708d3917`
+- Task bucket: `reviews/task-122`
+- Packet path: `reviews/task-122/003-spire-batched-materialization-prune`
+- Timestamp: 2026-06-27
+- Lane / fixture / storage format / rerank mode: SPIRE local unit validation; TurboQuant V2 leaf batched bounded candidate retention path.
+- Isolation: unit tests only; no index/table benchmark run.
+- Commands:
+  - `cargo test -p ecaz --lib --no-default-features --features pg18 collect_scan_placement_diagnostics_reports_candidate_truncation`
+  - `cargo test -p ecaz --lib --no-default-features --features pg18 collect_scan_placement_diagnostics`
+- Key result lines:
+  - `test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_reports_candidate_truncation ... ok`
+  - `test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2219 filtered out; finished in 0.05s`
+  - `test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_reports_candidate_truncation ... ok`
+  - `test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_reports_boundary_dedupe_and_winners ... ok`
+  - `test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_skips_degraded_unavailable_leaf ... ok`
+  - `test am::ec_spire::scan::tests::collect_scan_placement_diagnostics_counts_routed_store_rows ... ok`
+  - `test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 2216 filtered out; finished in 0.04s`
+

reviews/task-122/003-spire-batched-materialization-prune/request.md

@@ -0,0 +1,50 @@
+# Task 122 Packet 003: SPIRE Batched Materialization Prune
+
+## Summary
+
+This continues Task 122 Phase 2 by extending the bounded pre-materialization
+prune from Packet 002 into the SPIRE V2 leaf fully batched TQ path.
+
+Packet 002 added the threshold check to the per-column V2 path. The
+multi-segment batched path still routed each scored row through
+`append_quantized_v2_scored_column_candidate`, which materialized the row before
+the bounded heap could reject it. This change applies the same
+`pre_materialization_min_ip_to_keep()` gate in that shared scored-column helper
+before row decoding.
+
+## Scope
+
+Touched files:
+
+- `src/am/ec_spire/scan/candidates.rs`
+- `src/am/ec_spire/scan/tests/diagnostics.rs`
+
+The change remains limited to bounded, non-deduped scans where there are no
+delete-delta vec_ids requiring row materialization before delete filtering.
+
+## Diagnostic Accounting
+
+Pre-materialization skips are still reported as truncated candidates. The
+diagnostic test now distinguishes rows actually materialized into visible
+candidate rows from rows skipped before materialization:
+
+- `candidate_row_count` is the materialized visible candidate count.
+- `truncated_candidate_row_count` includes the pre-materialization skipped row.
+
+## Validation
+
+Passed:
+
+- `cargo test -p ecaz --lib --no-default-features --features pg18 collect_scan_placement_diagnostics_reports_candidate_truncation`
+- `cargo test -p ecaz --lib --no-default-features --features pg18 collect_scan_placement_diagnostics`
+
+See `artifacts/cargo-test-spire-truncation.log`,
+`artifacts/cargo-test-spire-placement-diagnostics.log`, and
+`artifacts/manifest.md`.
+
+## Measurement
+
+No benchmark claim is made in this packet. The next required step remains an
+`ecaz bench suite` A/B run for SPIRE bounded TQ scans at 10k/50k/100k before
+claiming latency or materialization wins.
+

reviews/task-122/004-spire-prune-ab-suite/artifacts/cargo-pgrx-install.log

@@ -0,0 +1,57 @@
+       Using PgConfig("pg18") and `pg_config` from /opt/homebrew/opt/postgresql@18/bin/pg_config
+    Building extension with features pg18
+     Running command "/opt/homebrew/bin/cargo" "build" "--lib" "--features" "pg18" "--no-default-features" "--message-format=json-render-diagnostics"
+warning: ecaz@0.1.1: In file included from csrc/pg18_pgstat_shim.c:10:
+warning: ecaz@0.1.1: In file included from /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:19:
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:537:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:   537 | dlist_next_node(dlist_head *head, dlist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:547:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:   547 | dlist_prev_node(dlist_head *head, dlist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:1054:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:  1054 | slist_next_node(slist_head *head, slist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: In file included from csrc/pg18_pgstat_shim.c:10:
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:875:58: warning: unused parameter 'size' [-Wunused-parameter]
+warning: ecaz@0.1.1:   875 | pgstat_cmp_hash_key(const void *a, const void *b, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                          ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:875:70: warning: unused parameter 'arg' [-Wunused-parameter]
+warning: ecaz@0.1.1:   875 | pgstat_cmp_hash_key(const void *a, const void *b, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                                      ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:882:56: warning: unused parameter 'arg' [-Wunused-parameter]
+warning: ecaz@0.1.1:   882 | pgstat_hash_hash_key(const void *d, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                        ^
+warning: ecaz@0.1.1: 6 warnings generated.
+    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.21s
+  Installing extension
+     Copying control file to /opt/homebrew/share/postgresql@18/extension/ecaz.control
+     Copying shared library to /opt/homebrew/lib/postgresql@18/ecaz.dylib
+  Discovered 237 SQL entities: 0 schemas (0 unique), 236 functions, 0 types, 0 enums, 1 sqls, 0 ords, 0 hashes, 0 aggregates, 0 triggers
+  Rebuilding pgrx_embed, in debug mode, for SQL generation with features pg18
+warning: ecaz@0.1.1: In file included from csrc/pg18_pgstat_shim.c:10:
+warning: ecaz@0.1.1: In file included from /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:19:
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:537:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:   537 | dlist_next_node(dlist_head *head, dlist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:547:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:   547 | dlist_prev_node(dlist_head *head, dlist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/lib/ilist.h:1054:29: warning: unused parameter 'head' [-Wunused-parameter]
+warning: ecaz@0.1.1:  1054 | slist_next_node(slist_head *head, slist_node *node)
+warning: ecaz@0.1.1:       |                             ^
+warning: ecaz@0.1.1: In file included from csrc/pg18_pgstat_shim.c:10:
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:875:58: warning: unused parameter 'size' [-Wunused-parameter]
+warning: ecaz@0.1.1:   875 | pgstat_cmp_hash_key(const void *a, const void *b, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                          ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:875:70: warning: unused parameter 'arg' [-Wunused-parameter]
+warning: ecaz@0.1.1:   875 | pgstat_cmp_hash_key(const void *a, const void *b, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                                      ^
+warning: ecaz@0.1.1: /opt/homebrew/include/postgresql@18/server/utils/pgstat_internal.h:882:56: warning: unused parameter 'arg' [-Wunused-parameter]
+warning: ecaz@0.1.1:   882 | pgstat_hash_hash_key(const void *d, size_t size, void *arg)
+warning: ecaz@0.1.1:       |                                                        ^
+warning: ecaz@0.1.1: 6 warnings generated.
+   Compiling ecaz v0.1.1 (/Users/peter/tqvector)
+    Finished `dev` profile [unoptimized + debuginfo] target(s) in 10.42s
+     Writing SQL entities to /opt/homebrew/share/postgresql@18/extension/ecaz--0.1.1.sql
+    Finished installing ecaz

reviews/task-122/004-spire-prune-ab-suite/artifacts/guc-check.log

@@ -0,0 +1,10 @@
+ ecaz_build_profile 
+--------------------
+ debug
+(1 row)
+
+ pre_materialization_prune 
+---------------------------
+ on
+(1 row)
+