Repository navigation
ci: auto-mirror ruvector-rulake → ruvnet/RuLake - #377
Merged
Merged
Conversation
…rate
Implements the M1 scope of docs/research/ruLake/ as an intermediary that
fans out vector queries across heterogeneous backends (Parquet, BigQuery,
Snowflake, Delta, Iceberg, local) behind a single RVF wire protocol, with
a RaBitQ-compressed cache in front.
## What ships
- **Research docs** under docs/research/ruLake/ (9 files, ~2.5k lines),
reframed from the earlier "plug RVF into BigQuery" shape to the
intermediary/federation shape. BigQuery-native compute becomes a Tier-2
push-down optimization inside the BigQueryBackend adapter, not a new
product shape.
- **ADR-155 v2** as "Proposed" — captures the seven alternatives
considered (plug-in-per-lake, standalone vector DB, Iceberg extension,
Trino connector, JVM intermediary, notebook-only, push-through-only),
consequences, and eight open questions.
- **crates/ruvector-rulake/** — new workspace member:
- `BackendAdapter` trait with minimum surface (id / list_collections /
pull_vectors / generation / supports_pushdown).
- `LocalBackend` in-memory reference implementation (thread-safe).
- `VectorCache` wrapping ruvector_rabitq::RabitqPlusIndex, with per-
collection generation tracking and `Consistency::{Fresh, Eventual}`
policies.
- `RuLake` entry point: register backends, search single or federated,
cache-stats introspection.
- 7 smoke tests (`tests/federation_smoke.rs`): byte-exact match vs
direct RaBitQ, cache-coherence after backend mutation, cross-backend
fan-out with correct score ordering, cache-hit-faster-than-miss,
three error-path tests.
- `rulake-demo` bin: unified benchmark producing the same-run table in
BENCHMARK.md.
## Measured numbers (LocalBackend, D=128, rerank×20, 300 queries)
| n | direct RaBitQ+ QPS | ruLake Fresh QPS | ruLake Eventual QPS | tax |
|--------:|-------------------:|-----------------:|--------------------:|------:|
| 5,000 | 17,311 | 17,874 | 17,858 | 0.97× |
| 50,000 | 5,162 | 5,123 | 5,050 | 1.01× |
| 100,000 | 3,122 | 3,117 | 3,114 | 1.00× |
**Intermediary tax is effectively zero on a local backend.** Federated
across 2 shards: 2,470 QPS @ n=100k (0.79× of single-shard); 4 shards:
1,781 QPS (0.57×) — sequential fan-out, parallel merge is the v2
optimisation per ADR-155 §Consequences.
## Build + test status (this crate only)
```
cargo build -p ruvector-rulake --release ✓
cargo test -p ruvector-rulake --release ✓ 7 passed
cargo clippy -p ruvector-rulake --release --all-targets -- -D warnings ✓ clean
cargo fmt -p ruvector-rulake -- --check ✓ clean
cargo run -p ruvector-rulake --release --bin rulake-demo ✓ reproduces BENCHMARK.md
```
## Scope this commit does NOT cover (M2-M5, see 07-implementation-plan.md)
- ParquetBackend, BigQueryBackend, SnowflakeBackend, IcebergBackend,
DeltaBackend (real-backend adapters).
- Push-down paths into backends with native vector ops.
- Governance / RBAC / PII / lineage / audit (M4).
- SIFT1M recall measurement on the real-backend path.
- Parallel fan-out via rayon.
- LRU cache eviction.
Co-Authored-By: claude-flow <ruv@ruv.net>
Applies the reviewer's architectural feedback (docs/research/ruLake/ chat thread): ruLake is a cache-first vector execution fabric, not a federation engine. Federation is the cache's refill mechanism. ## Perf fix — cache prime now runs lock-free `VectorCache::prime()` previously built a fresh `RabitqPlusIndex` (~400 ms at n=100k) while holding the cache mutex, serialising all other queries. Now builds entirely before touching `inner`; the lock is only taken to swap the finished entry in. No benchmark regression — intermediary tax still 1.00× on LocalBackend at n=100k. ## New: bundle sidecar (`table.rulake.json`) `ruvector_rulake::bundle` — the portable unit that defines ruLake's reproducibility + governance scope. Flagged by the reviewer as more important than the UDF because it's what travels between teams, clouds, and backups. Carries: `data_ref`, `dim`, `rotation_seed`, `rerank_factor`, `generation`, `rvf_witness` (SHAKE-256 over the preceding fields), `pii_policy`, `lineage_id`. `Generation` is a serde-untagged union of `Num(u64)` (Parquet mtime, Iceberg version, Snowflake offset) and `Opaque(String)` (UUIDs, hashes, base64 blobs) — fixes the "u64 doesn't fit an Iceberg snapshot id" open question from the M1 review. Witness fn is domain-separated, length-prefixed, and verifiable via `bundle.verify_witness()`. 6 new tests: determinism, field-change-detection, length-prefix-anti-collision, serde roundtrip, tamper-detection, format-version-downgrade-rejected. ## New: recall-vs-brute-force gate `rulake_recall_at_10_above_90pct_vs_brute_force` — the missing correctness test. Builds brute-force L2 truth over 5k clustered Gaussian vectors, asserts ruLake's top-10 hits ≥ 90% at rerank×20. Uses the same n + cluster-count + methodology as `ruvector-rabitq::BENCHMARK.md` so a regression shows up as a divergence from the known-good estimator baseline. ## ADR-155 v2 — cache-first decision explicit - Decision opens with "cache-first vector execution fabric; federation is the refill mechanism", lifts the reviewer's 5-axis decision matrix (cache-first wins 4/5 axes). - New Decision §6 declares the bundle sidecar as the portable unit (not the UDF) and documents how the witness acts as the cache-key anchor, closing the "cache invalidation drift" failure mode. ## Test + lint status ``` cargo test -p ruvector-rulake --release ✓ 14/0 cargo clippy -p ruvector-rulake --release --all-targets -- -D warnings ✓ clean cargo fmt -p ruvector-rulake -- --check ✓ clean cargo run -p ruvector-rulake --release --bin rulake-demo -- --fast ✓ no regression ``` Co-Authored-By: claude-flow <ruv@ruv.net>
…KE-256 Implements the reviewer's "use RVF witness chain hash as cache-key anchor" design. Cache entries are now keyed by the RuLakeBundle witness, not (backend_id, collection). Two backends advertising the same logical dataset (same data_ref + seed + rerank + generation) produce the same witness and share one compressed index. ## The change ### BackendAdapter::current_bundle() (new trait method) Returns the backend's authoritative bundle for a collection. Default impl synthesizes from `id() + generation()`; real backends override to report a shared data_ref when they're replicas of the same source of truth. LocalBackend overrides to avoid the default's pull-to-read-dim round-trip. ### VectorCache: two-layer storage - `entries: HashMap<WitnessKey, CacheEntry>` — content-addressed - `pointers: HashMap<CacheKey, WitnessKey>` — (backend, collection) → witness - `last_checked: HashMap<CacheKey, Instant>` — for Eventual-mode TTL `CacheEntry` now carries a `refcount` so an entry is GC'd only when its last pointer drops. New stat: `shared_hits` — incremented when a pointer move finds the target witness already cached. ### RuLake::ensure_fresh flow 1. Eventual within TTL → skip check (fast). 2. Witness matches pointer → hit, no-op. 3. Witness mismatch, target witness already in pool (another pointer has it) → just swap the pointer, zero prime work. This is the cross-backend share. 4. Witness not in pool → pull + prime as before. ### Prime is now race-tolerant A concurrent thread racing to prime the same witness doesn't rebuild — whichever thread gets the lock second observes the entry and drops its own build. Two builds for the same witness are byte-identical by determinism, so no data is lost. ## Test added `two_backends_share_cache_when_witness_matches` — uses a `SharedLocalBackend` shim that overrides `current_bundle()` to advertise a shared data_ref. Two distinct `LocalBackend`s behind shims report identical witnesses; the second search finds `primes=1, shared_hits=1` and only ONE compressed entry in the pool despite two pointers. Both pointers' `refcount_of(witness) == 2`. ## Lint + test status ``` cargo test -p ruvector-rulake --release ✓ 15/0 cargo clippy -p ruvector-rulake --release --all-targets -- -D warnings ✓ clean cargo fmt -p ruvector-rulake -- --check ✓ clean ``` ## Closes open question from earlier ADR review "Cache invalidation drift" — the witness is now the cache-key anchor. Backend generation bumps become witness changes; witness changes are content-addressable so old entries can drop but shared ones survive. "Where does freshness truth live?" — answered: in the bundle. Co-Authored-By: claude-flow <ruv@ruv.net>
MVP shipped an unbounded cache. v1 must-have: a hard cap on the number of distinct compressed entries, evicting the least-recently-used *unpinned* (refcount=0) entry when the cap is exceeded. Design note: entries pinned by a live `(backend, collection)` pointer are never evicted — dropping them would orphan a caller. If every entry is pinned, the cap is temporarily exceeded rather than return an error. Correctness over strict bounds. API: - `VectorCache::with_max_entries(n)` — builder-mode cap. - `RuLake::with_max_cache_entries(n)` — user-facing constructor flag. - `RuLake::invalidate_cache(key)` — drop a pointer explicitly so its entry becomes evictable. - `CacheEntry.last_used` bumped on every search_cached; LRU picks the oldest unpinned entry as victim. Eviction runs opportunistically at the end of each prime when a cap is set. Zero overhead when `max_entries == None` (default path). Test: `lru_eviction_caps_entry_count_when_pointers_dropped` pins three entries, invalidates one, asserts the cap=2 holds after the next prime runs the sweep. 16/16 tests pass. Clippy clean under -D warnings. Co-Authored-By: claude-flow <ruv@ruv.net>
Direct dependency of the BQ UDF + cache sidecar daemon: the daemon needs to read `table.rulake.json` off GCS (or a local mount) and verify its witness before swapping in a new compressed entry. - Atomic write via temp+rename so concurrent readers never see a truncated sidecar (matches the pattern a warehouse-push path needs). - Read verifies witness on-disk → malformed or tampered bundles surface as InvalidParameter with a "witness" message. - Canonical filename is exposed as SIDECAR_FILENAME so callers don't hardcode the string. Tests: - fs_roundtrip: write + read preserves witness + optional fields. - fs_read_rejects_tampered_sidecar: edit dim on disk → read errors. - fs_write_is_atomic_under_crash_simulation: leftover .tmp.* files don't corrupt reads of the canonical sidecar. 19/19 passing (9 bundle + 10 federation). Clippy -D warnings green. Co-Authored-By: claude-flow <ruv@ruv.net>
search_federated now par_iters over targets so that cache-miss primes (the expensive case — pulling from the backend + building a RabitqPlus index) run concurrently per shard. Measured speedups in BENCHMARK.md: n=100k: 1-shard prime 425ms → 2-shard 215ms (1.97×) → 4-shard 110ms (3.86×) n= 50k: 1-shard prime 213ms → 2-shard 110ms (1.95×) → 4-shard 56ms (3.83×) Warm-cache QPS on a single-threaded benchmark drops slightly because rayon's par_iter startup is measurable at sub-ms per-query. The win is in tail-latency under miss and in real remote-backend deployments where per-shard latency dominates — the bench understates this. Short-circuits on error (first shard to return Err wins), matching the sequential loop's semantics. Rayon pinned via workspace.dependencies (rayon = "1.10"). Co-Authored-By: claude-flow <ruv@ruv.net>
First concrete adapter that reads real persistent data. Uses a simple 'ruvec1' binary format (8-byte magic + u64 count + u32 dim + records) and takes the mtime as the generation token. This proves the full bundle → witness → cache → search loop works against the filesystem without pulling arrow/parquet deps — a real ParquetBackend reuses the exact same shape, only the decoder and generation source change. - current_bundle() reads only the 24-byte header to pick up dim — real-backend hot-path ergonomics; a full pull per coherence check would be catastrophic on a warehouse adapter. - Atomic write via temp+rename so concurrent reads never observe a torn record stream (matches the bundle sidecar write pattern). - data_ref is 'file://<path>', anchoring the witness on the local filesystem location — two FsBackends pointing at the same file share the cache entry (content-addressed, per ADR-155). Tests: - fs_write_then_pull_roundtrip: write vectors, read them back bitwise. - fs_bundle_has_file_uri_and_header_dim: verify witness + data_ref. - fs_pull_rejects_bad_magic: magic-byte guard on pull. - fs_backend_end_to_end_search_and_recache_on_mtime_bump (federation smoke): full RuLake → FsBackend → mtime bump → re-prime cycle. 23/23 passing (9 bundle + 3 fs_backend + 11 federation). Clippy green. Co-Authored-By: claude-flow <ruv@ruv.net>
8 threads × 50 queries against a shared RuLake, alternating single-shard and federated calls. Validates: - no deadlocks (bounded time to completion) - no panics from the cache Mutex or backend RwLock under contention - every returned hit is finite and the per-call result is sorted - prime count stays at ≤ 2 (one per shard) — hits serve the rest Closes the M3 "concurrent multi-client throughput" smoke item from BENCHMARK.md. The Send + Sync bound on RuLake is now exercised, not just declared. 12/12 federation + 9 bundle + 3 fs_backend tests passing (24 total). Clippy -D warnings green. Co-Authored-By: claude-flow <ruv@ruv.net>
M1 done and benchmarked. Update status from 'Proposed' → 'Accepted (M1)', collapse the implementation-plan M1 bullet to reflect everything that actually shipped on the branch, and move the open-question resolutions into a dedicated "Resolved in M1" block. New M1 evidence in the ADR: - Intermediary tax 1.00× at n=100k on LocalBackend - Byte-exact parity with direct RaBitQ at same (seed, rerank_factor) - Rayon fan-out 1.97× (2-shard) / 3.86× (4-shard) prime-time speedup - Recall@10 > 90% gate passes - Witness-addressed cache sharing verified - Send+Sync under 8-thread contention Remaining open questions rewritten for M2 focus: - Remote-backend tax measurement (Parquet-on-GCS prime) - Cache sidecar daemon protocol for bundle handoff - Push-down negotiation policy - Cost accounting for pushed-down BQ work Co-Authored-By: claude-flow <ruv@ruv.net>
Pairs with iter 4's read_from_dir: given a registered (backend, collection) key, emit the current table.rulake.json to a directory. This is what a cache sidecar daemon calls when the warehouse triggers a bundle refresh — the daemon publishes the new bundle, any serving ruLake watching that directory swaps in the new witness on next search. Does NOT prime the cache — publish is a metadata emission, not a data load. That keeps publish cheap and lets operators stage bundle updates without moving any compressed data. Test publish_bundle_roundtrips_through_disk: publish → read_from_dir on a third party → witness matches what a cache prime would see. 13 federation + 9 bundle + 3 fs_backend = 25 passing. Clippy green. Co-Authored-By: claude-flow <ruv@ruv.net>
Completes the sidecar loop (publish → disk → refresh). Given a key and a directory, read the on-disk table.rulake.json and: - UpToDate: witness matches cache pointer, nothing to do - Invalidated: witnesses differ, cache pointer for key is dropped - BundleMissing: no sidecar present (caller decides) A corrupt/tampered sidecar surfaces as InvalidParameter via RuLakeBundle::read_from_dir's witness verification — a poisoned publish cannot silently invalidate the cache. This is the minimal primitive a cache sidecar daemon needs. The daemon itself is a ~10-line loop in user code: for each watched (key, dir), call refresh_from_bundle_dir periodically or in response to inotify events; handle the three outcomes. Closes the "cache sidecar daemon protocol" open question from ADR-155. The protocol is: filesystem-based, witness-authenticated, atomic-write on publish, three-state on refresh. 14 federation + 9 bundle + 3 fs_backend = 26 tests passing. Co-Authored-By: claude-flow <ruv@ruv.net>
Iter 10 shipped the symmetric publish_bundle / refresh_from_bundle_dir primitives with witness-authenticated handoff. The protocol is: publisher → atomic-write table.rulake.json reader → read, verify witness, compare, invalidate if different Three-state refresh result (UpToDate / Invalidated / BundleMissing) covers all the daemon's logging / alerting needs. Tampered sidecars fail loudly instead of silently corrupting the cache. Move the question from "still open" to "resolved in M1" and drop the now-stale M2 placeholder. Co-Authored-By: claude-flow <ruv@ruv.net>
Added measure_concurrent_fed: N client threads hammering federated
search against K shards on one box. Expected result was "rayon helps
under concurrent load where single-thread bench masks it." Actual
result is different and worth recording honestly.
At n=100k, 8 clients × 300 queries:
1 shard: 810ms wall, 2,963 qps
2 shards: 960ms wall, 2,500 qps (0.84×)
4 shards: 1,350ms wall, 1,778 qps (0.60×)
More shards = LOWER concurrent throughput for this "same data split K
ways on one box" workload. Root cause: the RaBitQ rerank_factor × k =
200 rerank runs per shard, so K-shard federation does ~K× the rerank
work. Parallel fan-out cuts scan cost but not rerank cost.
Consequences documented in BENCHMARK.md:
- Don't shard for throughput on same-box same-data; shard for
reachability or memory bounds.
- Per-shard rerank factor reduction is an obvious M2 optimization:
fan out at rerank=50 per shard when K≥2 keeps global recall above
90% while approximately K× reducing the rerank cost. Measurement-
driven, not speculative.
- Real federation gain (disjoint data across network backends) is
genuine; this bench just doesn't measure it.
Rayon fan-out is NOT reverted — still correct for the miss-path prime
(1.97× / 3.86× speedup retained) and for remote-backend I/O overlap.
Co-Authored-By: claude-flow <ruv@ruv.net>
Iter 12's concurrent benchmark surfaced that K-shard federation pays
~K× rerank work because RaBitQ's rerank runs per-shard on candidates
that can't be globally merged before rerank without an API change.
Fix spec'd precisely so it's easy to land later:
1. ruvector-rabitq: add search_with_rerank(query, k, rerank_factor)
— same body as search() but takes rerank_factor as a parameter.
2. rulake: plumb through VectorCache and RuLake::search_federated
with an optional per_shard_rerank. Default policy: divide by K,
floor 5.
3. Re-bench the concurrent workload to verify; recall@10 should
stay > 85%.
Deferred to M2 because rabitq was just merged and changing its public
API mid-branch is out of scope. Filed as the explicit trigger for
the first rabitq follow-up.
Co-Authored-By: claude-flow <ruv@ruv.net>
Cache-first reframe (ADR-155) makes hit_rate the primary KPI. Before this, operators were flying blind — the raw hits/misses counters told them nothing about the 95% gate the acceptance test targets. - CacheStats gets total_prime_ms + last_prime_ms (prime timed inside prime() with Instant::now() around the lock-free build). - CacheStats::hit_rate() → Option<f64> (None when no searches yet). - CacheStats::avg_prime_ms() → Option<f64> (None when no primes). Test stats_expose_hit_rate_and_prime_duration: 1 prime + 99 warm queries → hit_rate ≥ 0.95, last_prime_ms in the expected range. This is step 1 of the strategic reframe the latest review surfaced: cache-coherent execution layer, federation as refill. The KPI it exposes is what the 95% acceptance gate will measure. 15 federation + 9 bundle + 3 fs_backend = 27 passing. Clippy green. Co-Authored-By: claude-flow <ruv@ruv.net>
Ships the cross-crate fix that iter 12's concurrent bench identified:
K-shard federation no longer pays K× the rerank cost.
Changes:
- rabitq: RabitqPlusIndex::search_with_rerank(query, k, rerank_factor)
— non-mutating per-call override, same body as search(). The stored
field stays the default used by plain search().
- rulake: VectorCache::search_cached_with_rerank(key, q, k, rf_opt)
forwards through. search_cached() remains the default path.
- rulake: RuLake::search_federated uses an adaptive default of
max(MIN_PER_SHARD_RERANK=5, global / K). search_federated_with_rerank
lets callers override explicitly (None = adaptive, Some(global) =
byte-exact parity with single-shard).
Bench (n=100k, 8 clients × 300 queries, same box):
shards before QPS after QPS per-shard rerank
1 2,963 2,854 20
2 2,500 2,959 (1.04×) 10
4 1,778 2,791 (0.98×) 5
4-shard federation went from 0.60× the single-shard baseline to
0.98×. At 2 shards, the mutex serialization overhead even nets us
slightly above 1-shard. Federation is genuinely free now.
Recall gate: adaptive_per_shard_rerank_preserves_recall asserts
recall@10 ≥ 0.85 at K=2 and K=4 on clustered D=128 n=5k.
This closes the M2 cross-crate task filed in ADR-155 (iter 13). The
strategic review's "immediate optimization, high impact" is shipped.
27 → 28 tests passing. Clippy -D warnings clean in both crates.
Co-Authored-By: claude-flow <ruv@ruv.net>
Acts on the 2026-04-22 strategic review. Three changes:
1. Sharpen the one-line decision:
'ruLake is a vector execution cache with deterministic compression
and federated refill.' Federation is the refill mechanism; the
cache is the product surface. Previous framing was correct but
fuzzy on which half was the headline.
2. New M1.5 acceptance test:
'95% of queries return exact top-k without touching the backend.'
Measurable from CacheStats::hit_rate() alone. Replaces the prior
'federation works across 4 shards' gate, which the concurrent
bench showed was a distraction from the real product claim.
3. Strategic questions section — two product choices recorded with
recommendations instead of resolutions:
a) Invisible infrastructure vs user-facing query layer?
→ Recommend invisible first (BQ UDF path).
b) Strict Fresh vs 10× Eventual?
→ Recommend both as a product knob, not a flag.
4. Close per-shard-rerank question (shipped in iter 15) and
cache-first KPI surface question (shipped in iter 14) as
resolved in M1.
Co-Authored-By: claude-flow <ruv@ruv.net>
Two changes from the 2026-04-22 strategic review reframing ruLake as the memory substrate for agent brain systems: 1. Consistency::Frozen variant — caller asserts bundle immutability; never automatic backend recheck. Maps to "Frozen for audit" from the reviewer's three-mode product knob. Automatic coherence is suppressed; explicit refresh_from_bundle_dir still works (lets operators invalidate frozen caches without needing Fresh mode). can_skip_check short-circuits when the pointer is already installed — first prime still runs, subsequent queries never round-trip to the backend. Test frozen_consistency_never_rechecks_after_prime: prime → bump backend → 10 warm searches still hit on the old witness, primes stay at 1. Explicit refresh on a re-published bundle correctly reports Invalidated, proving operator control remains. 2. ADR-156 — positioning addendum, not replacement of ADR-155. ruLake stays as substrate (memory hierarchy); brain system stays above (memory type, recall policy, mutation policy). Decomposes the reviewer's "recall / verify / forget / compact / rehydrate" acceptance test into six guarantees, five of which are shipped. Rejects: - absorbing the brain into ruLake (violates substrate separation) - a new rulake-memory crate (premature; M1 primitives suffice) - forking into two products (identical properties; no win) 17 federation + 9 bundle + 3 fs_backend = 29 tests passing. Clippy green. Co-Authored-By: claude-flow <ruv@ruv.net>
…atch
Locks the CPU-first, GPU-optional architecture from the 2026-04-22
strategic review. Scaffolding-only ADR — no kernel implementations
ship with this decision.
Key positions:
1. VectorKernel trait lives in ruvector-rabitq (kernels are RaBitQ
primitives); dispatch lives in ruvector-rulake (has the live
signals — batch size, hit rate, rerank pressure).
2. GPU implementations (CUDA/ROCm/Metal) ship as separate crates
(ruvector-rabitq-cuda, -rocm, -metal) on their own cadence.
Laptop and WASM builds never pay the dep cost.
3. WASM SIMD is feature-gated in ruvector-rabitq itself (same source,
different target).
4. Determinism as a hard gate: scan-phase must be bit-reproducible
across kernels; rerank-phase may be float-nondeterministic but
caps().deterministic=false kernels are refused on Fresh/Frozen
paths. Witness chain stays anchored on data, not kernel identity.
5. Acceptance gate for promotion past experimental:
p95 ≥ 2× lower OR cost per 1M queries ≥ 30% lower,
at identical recall@10 on a reference workload
(clustered D=768 n=1M rerank×20).
Considers and rejects: single-crate GPU kernels (build/CI bloat),
dispatch inside rabitq (wrong info), new ruvector-kernel crate
(premature), feature-flag-only static dispatch (no runtime detection),
wgpu-first (shader model not mature for popcount+reduction).
Co-Authored-By: claude-flow <ruv@ruv.net>
…queries
Batched single-collection search that amortizes per-query overhead
across the batch:
- ensure_fresh() runs once, not N times (big for Fresh consistency
where it's a backend RTT each time).
- VectorCache::search_cached_batch takes the cache mutex once and
runs all N scans under that single acquisition.
- pos_to_id clone is amortized across the whole batch.
Layered trait shape (ADR-157 preparation):
RuLake::search_batch
└─ VectorCache::search_cached_batch
└─ N × RabitqPlusIndex::search (today — CPU per-query in loop)
A future VectorKernel trait plugs in under the innermost step. The
signature up to search_cached_batch is already kernel-agnostic —
GPU / SIMD kernels cross over CPU only at batch sizes ≥ their
min_batch, so a per-query API would never let dispatch pick them.
Tests:
- search_batch_matches_per_query_results: byte-exact parity with
search_one called individually. No recall loss.
- search_batch_acquires_cache_lock_once: a batch of 32 registers
as 1 coherence-skip hit (Eventual), not 32 — proves the
amortization is real, not nominal.
19 federation + 9 bundle + 3 fs_backend = 31 tests passing. Clippy
-D warnings clean.
Co-Authored-By: claude-flow <ruv@ruv.net>
Lets operators see which backend is hot (high hit_rate) vs cold (high miss+prime cost). The global CacheStats are unchanged; the per-backend counters are populated lazily on first activity against a given backend id. Attribution touches four events: hits, misses, primes, and invalidations. Shared-hits (witness-match cache shares) attribute to the *receiving* backend — that's the one that saved the prime work. Mark_hit/mark_miss take &CacheKey so attribution is explicit, not threaded through shared state. API: - VectorCache::stats_by_backend() -> HashMap<BackendId, PerBackendStats> - RuLake::cache_stats_by_backend() (thin delegate) - PerBackendStats::hit_rate() mirrors CacheStats::hit_rate() Test cache_stats_by_backend_attributes_hits_to_the_right_backend: two backends, hammer one, cold one stays at 0 hits. Hot hit_rate ≥ 0.95 after warmup; both primed once. Motivation (ADR-157): kernel dispatch decisions need per-backend signals — a cold backend with high miss rate should not trigger GPU dispatch the same way a hot backend serving at high hit rate does. Also generally useful for capacity planning. 20 federation + 9 bundle + 3 fs_backend = 32 tests passing. Clippy green. Co-Authored-By: claude-flow <ruv@ruv.net>
Ships the runnable acceptance test ADR-156 spec'd. Drives a single
LocalBackend through the full substrate contract in one test:
1. Recall: search_one → results
2. Verify: publish_bundle → read_from_dir → verify_witness
→ cache pointer matches on-disk witness
3. Forget: invalidate_cache → pointer is None
4. Rehydrate: next search_one → primes+1, pointer reinstalled
5. Location- results before forget ≡ results after rehydrate
transparency (byte-exact ids + scores at the same seed); the
caller never touched data_ref or knew which tier
served the call
6. Compact: explicitly out of scope per ADR-156 — belongs to
RVM/Cognitum, not the substrate
If this test stays green on every commit, the agent-facing memory
substrate claim is mechanical, not aspirational.
Also closes ADR-156 open question #4 (substrate test needed) as
resolved.
21 federation + 9 bundle + 3 fs_backend = 33 tests passing. Clippy
-D warnings clean.
Co-Authored-By: claude-flow <ruv@ruv.net>
Adds a batch-vs-loop block to rulake-demo. Measures on an already-primed
LocalBackend under Eventual consistency (the hot path):
batch=8 qps=2874 1.01×
batch=32 qps=2961 1.04×
batch=128 qps=2943 1.03×
batch=300 qps=2986 1.05×
per-query loop 2855 baseline
Modest on this workload because the warm cache path is already
uncontended (single-threaded + Eventual TTL makes ensure_fresh a
HashMap lookup, not a backend RTT). BENCHMARK.md is updated to
record the honest number and name the three latent wins the bench
does not measure:
1. Fresh consistency — batch of N amortizes N backend RTTs to 1.
2. Concurrent contention — fewer mutex acquires under multi-client.
3. Kernel dispatch (ADR-157) — batch is the plug-point GPU / SIMD
kernels need to cross over CPU.
The mechanical guarantee is unchanged and already tested
(search_batch_acquires_cache_lock_once): batch=32 registers as 1
coherence check, not 32. Speedup is workload-dependent; the shape
is correct.
Co-Authored-By: claude-flow <ruv@ruv.net>
…ts + sidecar example
Four in-scope M1 items from the remaining backlog, landed together
because they cross-cut cleanly.
Iter 23 (rabitq): VectorKernel trait + CpuKernel default
- Trait: id(), caps() → KernelCaps, scan(ScanRequest) → ScanResponse.
Scan-phase determinism is the hard contract; rerank-phase nondet
is declared via caps().deterministic = false and the caller's
dispatch policy filters those out of Fresh/Frozen paths (ADR-157).
- CpuKernel wraps RabitqPlusIndex::search_with_rerank, always
available, unbounded dim, deterministic.
- Tests: CPU kernel matches direct search byte-exactly + respects
per-call rerank override + caps advertised correctly.
Iter 24 (rulake): memory_class on RuLakeBundle (ADR-156)
- Opaque caller-defined tag — agent systems write "episodic" /
"semantic" / etc; ruLake stores but never interprets.
- Not part of the witness: two bundles with identical data but
different memory_class share the cache.
- Serde default+skip_if_none keeps old bundles forward-compatible.
- Test: roundtrip + witness-unchanged + legacy bundles without the
field still parse.
Iter 25 (rulake): examples/sidecar_daemon.rs
- Runnable demo of publish_bundle / refresh_from_bundle_dir pair.
- Publisher mutates backend + re-publishes; daemon poll loop
detects witness change, invalidates; next query re-primes.
- Includes a bug fix in refresh_from_bundle_dir: when the cache
pointer is None (already invalidated), report UpToDate instead
of Invalidated so daemons don't re-fire on every poll between
"we invalidated" and "somebody queried."
Iter 26 (rulake): CacheStats::stats_by_collection
- Per-(backend, collection) counters, one level finer than
stats_by_backend. Operators can identify which specific
collection is hot and pin it in LRU or increase its shard count.
21 federation + 11 bundle + 3 fs_backend + 3 kernel = 38 tests
passing across both crates. Clippy -D warnings clean. Example runs
end-to-end.
Co-Authored-By: claude-flow <ruv@ruv.net>
… (8-12× QPS)
Two items in one commit because they both touch broad surface:
=== Iter 27: deep security review + fixes ===
CRITICAL — path traversal in FsBackend
FsBackend::register/write previously joined user-supplied filenames
with the root dir without validation. A filename of "../etc/passwd"
escaped the root. Also polluted the data_ref (witness input),
letting an attacker forge witness values via canonicalization.
Fix: validate_filename() rejects empty, ".", "..", control bytes,
path separators (both / and \), colons (Windows drive/UNC), and
names > 255 bytes (POSIX NAME_MAX). register() is now fallible.
Test fs_register_rejects_path_traversal covers 12 malicious forms
(../escape, ../../etc/passwd, /absolute, sub/foo, back\slash,
"." "..", null bytes, control bytes, C:drive, empty string).
LOW — unbounded JSON metadata fields
from_json had no cap on input size or field lengths. A 1 GB
memory_class string would force the reader to allocate that much.
Fix: MAX_JSON_BYTES=64 KiB total, MAX_FIELD_BYTES=4 KiB per
(data_ref, pii_policy, lineage_id, memory_class), rvf_witness
capped at 128 bytes (SHAKE-256(32) hex is exactly 64).
Tests: from_json_rejects_oversize_input + from_json_rejects_
oversize_metadata_field.
No unsafe in ruLake or the new kernel module. Bundle witness scheme
is already domain-separated + length-prefixed SHAKE-256. Mutex
poisoning on unwrap() is a deliberate fail-fast; documented.
=== Iter 28: Arc-drop-lock optimization ===
The cache Mutex was held for the entire scan duration. Under
concurrent load, every reader serialized behind the previous one.
Fix: CacheEntry::index is now Arc<RabitqPlusIndex>, pos_to_id is
Arc<Vec<u64>>. search_cached + search_cached_batch clone the Arcs
under the mutex (microseconds) and drop the lock before running
the scan. The index is immutable once built; no data race.
Measured on n=100k, 8 clients × 300 queries:
shards QPS before QPS after lift
1 2,854 23,681 8.3×
2 2,959 28,971 9.8×
4 2,791 33,094 11.9×
Federated sequential also wins big — previously each shard's scan
serialized on the same cache mutex; now rayon actually parallelizes:
n=100k, 4-shard sequential: 1,673 → 8,289 qps (5.0×)
Single-threaded single-shard numbers are unchanged (mutex is
uncontended in that case) — this is purely a concurrency win, and
it's the single biggest optimization on the M1 branch.
All 40 tests passing (17 lib + 21 federation + 3 fs_backend).
Clippy -D warnings clean in both rabitq and rulake.
Co-Authored-By: claude-flow <ruv@ruv.net>
Root-level README for the ruvector-rulake crate in the style of the workspace README. Sections: - Introduction: one-line positioning + what ruLake is/isn't - Features: 8 subsections covering the actual measured properties (cache-first, witness-authenticated, federated, three-mode consistency, sidecar protocol, KPIs, kernel plane, security) - Quick start: minimal usable example - Usage patterns: 5 concrete recipes (RAG, federated, Frozen audit, sidecar daemon, memory substrate tagging) - Benchmarks: 4 measurement tables from the commit history - Comparison: vs Pinecone / Weaviate / Milvus / LanceDB / BQ Vector Search (intermediary tax, federation, witness, cross-process cache sharing, GPU optionality, unsafe count) - Technical details: architecture diagram, bundle protocol, coherence protocol state machine, adaptive rerank formula, Arc concurrency model, security model - User guide: consistency-mode choice, cache sizing, observability metrics, writing a custom backend, running the examples - Status: M1 shipped, M2 roadmap - Links to ADR-155 / 156 / 157 Co-Authored-By: claude-flow <ruv@ruv.net>
RabitqPlusIndex::from_vectors_parallel rotates + bit-packs every vector in parallel using rayon, then commits the SoA serially. Produces a bit-identical index to the serial add loop — rotation matrix is seeded once at construction and encode is deterministic, so parallel ordering cannot affect output bytes. VectorCache::prime picks between serial add() and the new parallel constructor based on batch size (PARALLEL_PRIME_THRESHOLD = 1024). Below 1k vectors the rayon task-queue overhead outweighs the D×D rotation savings; above it the parallel path dominates. Measured (clustered D=128, rerank×20): n=5k prime 22.3 ms → 4.5 ms (4.9×) n=50k prime 213 ms → 19.6 ms (10.9×) n=100k prime 420 ms → 37.6 ms (11.2×) This is the biggest cold-start-latency win available in M1. Real backend deployments where prime cost is the critical-path latency on a cache miss now see p99 drop by an order of magnitude. rayon dep is no longer feature-gated in rabitq (it's already a runtime dep via the workspace-pinned 1.10 that ruLake uses). 40 tests passing. Clippy -D warnings clean. Co-Authored-By: claude-flow <ruv@ruv.net>
…tioning
Reworked the top-level README per review feedback. Changes:
- New intro: concrete one-line description + a plain-English
"what is ruLake" that leads with the user's existing data ("you
already have vectors somewhere"), not internal architecture.
- Added "Why ruLake exists" section contrasting the three existing
options (managed vector DB / lakehouse / local library) and where
ruLake fits.
- Features: simpler bullet-point descriptions, every claim backed
by a measured number.
- New "Benefits" section grouping wins by reader role (app dev,
platform, performance eng, security eng).
- Comparison section now leads with the "not a vector database"
statement verbatim and positions ruLake explicitly as part of
the RuVector ecosystem (RVF / rabitq / ruLake table).
- Usage recipes expanded with the 95% hit-rate target + acceptance
test mention.
- Benchmarks section now includes the parallel-prime numbers
(11× speedup at n=100k).
- "How it works" cleaned up: data-flow diagram, bundle structure
example, adaptive rerank explanation, Arc concurrency, parallel
prime.
- User guide: decision tables for consistency mode, metric
interpretation, custom backend writing, running examples.
- Status: M1 shipped list (60 tests, zero unsafe) + M2 roadmap.
Co-Authored-By: claude-flow <ruv@ruv.net>
…eneration tag byte
All three from the 2026-04-23 round-2 security audit. Format version
bumped to 2 because Generation::hash_bytes() now prepends a variant
tag; v1 witnesses don't match v2 witnesses by design.
HIGH — unbounded pull_vectors DoS
Hostile backend returning n=u64::MAX / dim=u32::MAX would OOM the
host at prime. Added MAX_PULLED_VECTORS=100M, MAX_PULLED_DIM=8192,
MAX_PULLED_BYTES=16 GiB caps with checked_mul overflow guard. Cache
prime() validates the batch before any allocation. FsBackend::pull
also bounds-checks the on-disk ruvec1 header fields before
Vec::with_capacity so a corrupt file can't crash the reader.
MEDIUM — u64/u32 → usize truncation on 32-bit targets
FsBackend::pull now uses usize::try_from(count_u64) and errors on
overflow instead of silently truncating. wasm32 / armv7 deployments
no longer have the content-smuggling-past-witness attack surface.
LOW — Generation::Num(n) vs Generation::Opaque(bytes) collision
Before: Num(7) and Opaque("\x07\0\0\0\0\0\0\0") produced identical
hash_bytes() output, so two different generations could share a
witness.
After: hash_bytes() prepends 0x00 for Num, 0x01 for Opaque. Tests
cover the regression (generation_num_and_opaque_cannot_collide).
Format version bumped 1→2 since witness bytes change.
Also added 4 backend-module tests for the pulled_batch validator
(accepts reasonable size, rejects dim=0, rejects dim>cap, rejects
ids/vectors length mismatch).
42 tests passing (21 lib + 21 federation). Clippy -D warnings clean.
Co-Authored-By: claude-flow <ruv@ruv.net>
Two memory/perf fixes from the 2026-04-23 audit round. Flatten (finding #3 of memory audit, top-priority): RabitqPlusIndex::originals was Vec<Vec<f32>> — one heap allocation per row, 24 B Vec header × n, pointer-chasing on rerank. Replaced with originals_flat: Vec<f32> of length n*dim. Row i is originals_flat[i*dim..(i+1)*dim], accessed via a new fn original(&self, pos) -> &[f32]. Memory win at n=1M, D=128: before: 512 MB data + 24 MB Vec headers + 1M heap allocations after: 512 MB data + 24 B Vec header + 1 allocation That's 24 MB + allocator fragmentation eliminated. Drop the double-clone (finding #5): RabitqPlusIndex::add previously did self.inner.add(id, vector.clone()) + self.originals.push(vector) — the clone was redundant since RabitqIndex::add takes owned Vec<f32>. Reordered: extend the flat buffer first (cheap slice copy), then hand the owned vector to the inner index. One less alloc per add on the serial prime path. Also tightened memory_bytes() accounting: 24 B header + n*dim*4 of payload (instead of 24 B × n + n*dim*4). Measured prime-time + QPS at n=100k (rayon parallel prime already landed; this layers on top): n=100k single-thread QPS: 2,975 → 3,132 (+5%) n=100k concurrent 4-shard: 33,094 → 33,663 (+2%) The memory win is the real prize — the perf uplift is small because rerank is a tiny fraction of scan cost at rerank_factor=20. 23 rabitq tests + 42 rulake tests passing. Clippy clean. Co-Authored-By: claude-flow <ruv@ruv.net>
Memory audit finding #4: encode_query_packed previously did q.to_vec() // alloc #1 (unit buffer) self.rotation.apply(&unit) // alloc #2 (rotated buffer) vec![0u64; n_words] // alloc #3 (returned packed words) per query. 3 heap allocations per search, firing at ~10k QPS, caused measurable allocator contention under concurrent load. Fix: thread_local scratch holds (unit_buf, rotated_buf) across queries on the same thread. RandomRotation gains an apply_into(&[f32], &mut [f32]) variant that writes into the scratch rather than allocating. Only the returned Vec<u64> is freshly allocated (the caller needs ownership). Net: 3 → 1 allocation per query on the hot path. New RandomRotation::apply_into is the building block for future in-place paths; apply() is now a thin wrapper around it. Measured QPS lift at n=100k (stacked with earlier iter-2/3 security + flatten): single-thread QPS: 2,975 → 3,137 (+5%) concurrent 1-shard: 23,681 → 24,255 (+2%) The uplift is smaller than the profiler's 30–50% estimate because at n=100k the scan dominates query encoding. On smaller collections (n=5k) where encoding is a larger fraction the relative win is similar. Allocator contention dominates only at much higher QPS. 23 rabitq tests passing. Clippy -D warnings clean. Co-Authored-By: claude-flow <ruv@ruv.net>
…re rule) Closes the data-skew recall gap the 2026-04-23 cache/federation SOTA agent flagged. Weaviate/Elasticsearch default to k-per-shard which under-recalls when the true top-K is concentrated in one shard. Policy: k' = k + ceil(sqrt(k * ln(S))), clamped to [k, 4k] Examples: k=10, S=2 → k'=13 k=10, S=4 → k'=14 k=10, S=16 → k'=16 k=10, S=64 → k'=17 At k=10 the over-request is ~30-70% of k — cheap insurance against skew. Formula is the folklore rule cited in SPIRE (arxiv 2512.17264), HARMONY (SIGMOD'25), and the OpenSearch recall guide. Extra cost per shard is O(k' × rerank) — negligible vs the scan cost at rerank=20. Single-shard (S=1) returns k unchanged. Callers can still override via search_federated_with_rerank to get exact parity. 21 federation tests passing. Clippy -D warnings clean. Co-Authored-By: claude-flow <ruv@ruv.net>
…t n=100k
Implements the profiler's top-priority optimization: a SIMD-friendly
scan kernel that decouples the XNOR+popcount agree-count pass from
the cos-LUT + score + TopK heap reduction.
Design (crates/ruvector-rabitq/src/scan.rs):
- scan_scalar: portable u64::count_ones, byte-identical to the
original inline loop.
- scan_avx2: #[target_feature(enable="avx2,popcnt")], 4-candidate
outer unroll via core::arch::x86_64::_popcnt64. Processes 4
rows per loop iteration, amortizing branch + stride overhead.
- scan: runtime dispatcher, cached in std::sync::OnceLock<fn(...)>
so the CPUID check runs once per process.
symmetric_scan_topk in index.rs now:
1. Calls scan::scan(...) once to fill a scratch Vec<u32> of
agree-counts (the whole-table popcount pass).
2. Walks the agree array with the cos-LUT + score + TopK heap —
a serial reduction that was never SIMD-amenable.
Determinism preserved: scan_avx2 and scan_scalar produce byte-
identical agree-count arrays. Two new tests verify this at D=128
(n=1000) and D=64/100/192/200 with tail cases n=1023/7.
Measured (single-thread, cargo run --release rulake-demo):
n= 5 000 direct RaBitQ+: 17,915 → 18,998 QPS (+6%)
n=50 000 direct RaBitQ+: 5,230 → 5,959 QPS (+14%)
n=100k direct RaBitQ+: 3,058 → 3,681 QPS (+20%)
Win grows with n as the per-query allocation overhead becomes a
smaller fraction of scan time. Smaller than the 2-4× upper-bound
profiler estimate because rerank=20 keeps ~30-40% of query time in
the exact-L2 rerank step (unchanged by this patch).
25 rabitq tests passing (23 prior + 2 new scan determinism tests).
Clippy -D warnings clean. No new deps. All unsafe confined to the
two SIMD functions in scan.rs.
Co-Authored-By: claude-flow <ruv@ruv.net>
… path Memory-audit finding #1: the hot router path cloned (String, String) keys ~3K times per federated query (one per mark_hit / mark_miss / per_backend_mut call). At 10 k QPS × 8 shards that's 7.6 MB/s of short-lived allocator traffic + hashmap rehashing on every step. Fix: intern at the RuLake boundary into Arc<str>. pub type CacheKey = (BackendId, CollectionId); // unchanged (public) pub(crate) type InternedKey = (Arc<str>, Arc<str>); // internal pub(crate) fn intern_key(b: &str, c: &str) -> InternedKey; ensure_fresh interns once at entry; every downstream mark_hit / mark_miss / per_backend_mut call takes refcount-cheap Arc<str> clones instead of cloning owned Strings. The public CacheKey alias stays (String, String) for API stability — callers passing owned tuples at rare diagnostic sites (cache_witness_of, invalidate_cache) keep working untouched. Bench delta (stacked with AVX2 popcount commit 5a4b0d7): n=100k single-thread Eventual: 2,963 → 3,626 QPS (+22%) n=100k concurrent 1-shard: 23,681 → 27,814 QPS (+17%) n=100k concurrent 4-shard: 33,094 → 36,715 QPS (+11%) vs original pre-optimization M1 baseline: **13.2× concurrent QPS**. 21 federation tests + 21 rulake lib + 25 rabitq = 67 tests passing. Clippy -D warnings clean. Co-Authored-By: claude-flow <ruv@ruv.net>
Refreshed the headline tables to reflect the AVX2 popcount + CacheKey Arc-intern commits. Wave-2 delta on top of the Arc-refactor baseline: n=100k direct RaBitQ+: 2,975 → 3,661 QPS (+23%) n=100k concurrent 1-shard: 23,681 → 27,814 QPS (+17%) n=100k concurrent 2-shard: 28,971 → 32,194 QPS (+11%) n=100k concurrent 4-shard: 33,094 → 36,715 QPS (+11%) vs original pre-optimization M1 baseline: concurrent 4-shard: 2,791 → 36,715 QPS = 13.2× lift. Intermediary tax on cache-hit path stays at 1.01-1.03× direct RaBitQ. Co-Authored-By: claude-flow <ruv@ruv.net>
Two parallel swarm agents delivered disjoint features for M2:
=== Agent A: seed-based index persistence ===
NEW: crates/ruvector-rabitq/src/persist.rs (+393 LoC)
save_index / load_index serialize a RabitqPlusIndex via its *build
inputs* (dim, seed, rerank_factor, ids, vectors) rather than the
opaque internal SoA state. Rationale: (dim, seed, data) →
bit-identical index by construction (RaBitQ is deterministic), and
the public API doesn't expose packed / rotation / cos_lut — so
seed-based reconstruction is the only path without touching index.rs.
On-disk format (32-byte header + payload):
magic "rbpx0001" | version:u32 | dim:u32 | seed:u64
| rerank_factor:u32 | n:u32 | (id:u32, v:f32[dim])*n
DoS caps: dim ≤ 8192, n ≤ 100M, rerank_factor ≤ 1024. Format is
portable — no matrix, no packed codes stored (rebuilt on load).
Tests: serialize_roundtrip_preserves_search_results (10 queries,
byte-exact ids + score bits), reject_bad_magic, reject_version_too_new,
reject_oversize_fields (4 sub-cases).
=== Agent B: randomized Hadamard (HD-HD-HD) rotation ===
MODIFIED: crates/ruvector-rabitq/src/rotation.rs (+219 LoC)
Adds RandomRotation::hadamard(dim, seed) as an opt-in O(D log D)
rotation. Storage is 3 × padded_dim × 4 bytes of ±1 signs instead
of D×D × 4 bytes of Haar matrix (1.5 KiB vs 64 KiB at D=128).
Based on TurboQuant 2025 (arXiv:2504.19874 §3.2): D₃·FWHT·D₂·FWHT·D₁
is close-to-Haar-uniform in the Johnson–Lindenstrauss sense, which
is all RaBitQ's error bound requires. For non-power-of-2 dim:
zero-pad to next_power_of_two, apply, truncate.
Backward-compatible: RandomRotation::random() still returns the
Haar matrix. New RandomRotationKind { HaarDense, HadamardSigned }
enum for introspection. RabitqIndex unchanged — integration into
the scan path is future work (ADR-158 pending).
Tests: hadamard_apply_preserves_norm_power_of_two (D=128, 256),
hadamard_apply_preserves_norm_non_power_of_two (D=1000 → pad 1024,
norm ∈ [0.95, 1.05] on 100 unit vectors), hadamard_is_deterministic,
hadamard_is_fast.
=== Totals ===
25 → 33 rabitq lib tests (+4 persist, +4 hadamard). All 21 rulake
federation + 21 rulake lib tests unchanged and passing. Clippy -D
warnings clean across both crates.
Both agents worked on strictly disjoint file scopes (persist.rs +
lib.rs one-liner vs rotation.rs only) — no merge conflicts.
Co-Authored-By: claude-flow <ruv@ruv.net>
Wires the previously-shipped RandomRotation::hadamard into RabitqIndex
as opt-in constructors. Completes the M2 feature from wave-3.
=== Agent A: integration (crates/ruvector-rabitq/src/index.rs) ===
New opt-in constructors, all backward-compatible:
- RabitqIndex::new_with_rotation(dim, seed, kind: RandomRotationKind)
- RabitqPlusIndex::new_with_rotation(dim, seed, rerank, kind)
- RabitqPlusIndex::from_vectors_parallel_with_rotation(dim, seed, rerank, kind, items)
- Existing RabitqIndex::new / RabitqPlusIndex::new delegate with
HaarDense kind — zero callsite breakage.
Measured at D=128, seed=131, rerank×20, clustered n=500, 50 queries:
Haar recall@10 vs brute-force L2²: 1.000
Hadamard recall@10 vs brute-force L2²: 1.000 (identical)
Haar rotation memory: 66,052 B
Hadamard rotation memory: 2,052 B (32.2× reduction)
Recall is indistinguishable from Haar at this scale/rerank. Rotation
storage shrinks by the expected D²/D log D factor (~3·D vs D² bytes).
=== Agent B: ADR-158 ===
docs/adr/ADR-158-optional-rotation-and-qvcache-positioning.md (new,
345 lines). Documents:
- Why rotation choice matters (cache-line coldness, D² cost)
- Decision: HaarDense default, HadamardSigned opt-in
- Math rationale (TurboQuant arXiv:2504.19874 §3.2)
- Why not default (recall sweep, non-pow2 padding, witness)
- Alternatives (Householder, Kac, butterflies)
- Consequences — including the WitnessV2 gap: the bundle witness
doesn't currently encode rotation kind, so flipping the default
is a witness-format breaking change.
- QVCache (arXiv:2602.02057, ETH/EPFL Feb 2026) positioning:
complementary not competitive. Both are query-level caches over
heterogeneous backends; ruLake has witness-authenticated cross-
process sharing + federation, QVCache has adaptive-threshold
region-local recall. Clean complementarity.
- 5 open questions incl. when to flip default + WitnessV2 plan.
33 → 36 rabitq lib tests (+3 Hadamard integration). Rulake 42
unchanged. Clippy -D warnings clean across both crates.
Co-Authored-By: claude-flow <ruv@ruv.net>
Adds direct comparison in rulake-demo. RandomRotationKind re-exported at the crate root so callers don't need to reach into the rotation module. Measured (clustered Gaussian, D=128, rerank×20): n= 5 000 Haar build: 22.4 ms Hadamard: 7.2 ms (3.09×) n=50 000 Haar build: 211.6 ms Hadamard: 72.7 ms (2.91×) n=100 000 Haar build: 421.1 ms Hadamard: 142.9 ms (2.95×) Matches the O(D²) → O(D log D) theoretical speedup: at D=128, ~16 K flops for the dense matrix multiply vs ~900 flops for three FWHT passes + three sign-vector multiplies. The 3× ceiling reflects that other allocations + SoA writes take non-negligible fraction of build time. Per-query QPS is flat (±3% noise) because the query-side rotation is only one of many per-query steps — the scan + rerank dominate, especially at n ≥ 50k. Hadamard's win is entirely on the prime / cold-start path, which was already the critical-path latency for cache-miss queries. Hadamard + existing parallel prime stack: n=100k total prime (incl. compression + SoA writes) still ~40 ms (parallel prime already dominates), but single-threaded rabitq- demo shows the pure-rotation win at 3×. Co-Authored-By: claude-flow <ruv@ruv.net>
…om_dir
Wires the previously-shipped rabitq::persist module into ruLake's
lake.rs as first-class cache-save/restore APIs. The architectural
blocker I've deferred across 3 waves is now closed.
=== Agent A: rabitq::RabitqPlusIndex::export_items() ===
crates/ruvector-rabitq/src/index.rs +1 method, +1 test.
Exposes `export_items() -> Vec<(usize, Vec<f32>)>` — each row as
(pos, original_vec) extracted from originals_flat with one clone per
row. Feeds directly into persist::save_index or
from_vectors_parallel_with_rotation. No new deps, no public API
breakage.
Regression test (`export_items_roundtrip_via_from_vectors_parallel`)
builds via serial add(), exports, rebuilds via the parallel path,
asserts byte-identical search results on 5 queries. Tests: 36 → 37.
=== Agent B: RuLake save_cache_to_dir + warm_from_dir ===
crates/ruvector-rulake/src/{cache.rs, lake.rs, tests/federation_smoke.rs}.
New API:
pub fn save_cache_to_dir(&self, key, dir) -> Result<PathBuf>
— writes dir/index.rbpx (atomic temp+rename+fsync) alongside
the table.rulake.json bundle sidecar. Uses export_items +
persist::save_index.
pub fn warm_from_dir(&self, key, dir) -> Result<usize>
— reads bundle, witness-verifies, loads index.rbpx via
persist::load_index, cross-checks dim+rerank_factor, installs
into cache via the new install_prebuilt path. Returns n vectors.
Does NOT require the backend to be registered — warm restart
without backend RTT is the point.
New on CacheStats: warm_installs counter (separate from primes so
warm-restart cost isn't confused with cold-prime cost).
New on VectorCache: install_prebuilt + install_prebuilt_interned —
insert a pre-built Arc<RabitqPlusIndex> at a known witness without
any prime-timer bookkeeping. Respects the LRU cap. Shared-entry
path reuses an existing witness entry if another pointer already
holds it (witness-addressed cache sharing remains the headline).
New test: `warm_from_dir_skips_backend_and_returns_bit_exact_results`
Prime a 50-vec D=8 collection, save, spin up a FRESH RuLake with
NO backend registered + Consistency::Frozen, warm_from_dir, run the
same query, assert byte-identical ids + f32 score bits,
warm_installs=1, primes=0. Closes the "restart without re-prime"
gap end-to-end.
Documented limitation: pos_to_id reconstructed as (0..n) identity
because RabitqPlusIndex doesn't expose outer ids() accessor, and
the rabitq agent's scope prohibited adding it. Every current prime
path uses positional ids so this is byte-equivalent to the real
ids; external non-dense u64 ids would collapse (a known M2+ issue
filed inline).
Tests: 37 rabitq + 21 rulake lib + 22 rulake federation = 80 total.
Clippy -D warnings clean across both crates.
Co-Authored-By: claude-flow <ruv@ruv.net>
…lose wave-5 gaps
=== Agent A: rabitq — non-dense ID preservation through persist ===
crates/ruvector-rabitq/src/{index,persist}.rs
Wave-5's warm_from_dir collapsed external u64 ids to (0..n) identity
because RabitqPlusIndex lacked an outer ids accessor. Surprise finding:
the persist LOAD path was already id-preserving — the pipeline reads
`id:u32` from disk and hands (id, v) into from_vectors_parallel, which
writes `id` into inner.ids. The only missing piece was the outer-layer
accessor so ruLake could read them back.
Added:
- RabitqPlusIndex::external_ids(&self) -> &[u32] (thin forward)
- RabitqPlusIndex::ids_u64(&self) -> Vec<u64> (widening clone)
Regression test `persist_preserves_non_dense_ids` builds an index with
non-dense external ids (13*i + 7 for i in 0..50), save/load, asserts
byte-identical ids after round-trip. 37 → 38 rabitq tests.
=== rulake: drop the (0..n) workaround ===
crates/ruvector-rulake/src/lake.rs
warm_from_dir now calls `idx.ids_u64()` instead of synthesizing
(0..n). Non-dense external ids round-trip faithfully. The
~15-line inline comment documenting the old limitation is gone;
replaced with a 4-line pointer to the wave-6 close.
=== Agent B: warm_restart runnable example ===
crates/ruvector-rulake/examples/warm_restart.rs (new)
Runnable demo of the full save → ship → warm-restart cycle:
- Phase 1: prime from backend, save to disk
- Phase 2: spin up a FRESH RuLake with NO backend, warm_from_dir,
query, assert warm_installs=1 / primes=0
- Phase 3: cold-prime from backend for comparison
- Final: report cold/warm speedup
Measured at n=5000 D=128 (agent's single-run numbers):
Phase 1 prime: 5.03 ms
save_cache_to_dir: 3.44 ms (2.46 MiB rbpx)
Phase 2 warm: 5.00 ms (warm_installs=1, primes=0)
Phase 3 cold: 3.60 ms
Speedup cold/warm: 0.70×
Honest finding: at n=5k D=128, cold-prime is actually faster than
warm-load because our parallel prime is <5ms and parsing 2.5 MB of
rbpx is slower. The warm-restart win shows up at larger n where
compression dominates; documented in the example's closing block.
Steady-state QPS matches within 1.2% (same compressed index in both).
38 rabitq + 21 rulake lib + 22 rulake federation = 81 tests. Clippy
-D warnings clean across both crates.
Co-Authored-By: claude-flow <ruv@ruv.net>
…t n=100k Extends the scan dispatch ladder to scalar → AVX2 → AVX-512 VPOPCNTDQ. The new kernel runs under #[target_feature(enable = "avx2,avx512f, avx512bw,avx512vpopcntdq")] and processes 8 u64s per zmm load via _mm512_popcnt_epi64. select_impl() now prefers avx512f+avx512vpopcntdq, falls back to avx2+popcnt, then scalar. All paths cached in the existing OnceLock. Measured on host with all three levels available (n=100k, D=128, rerank×20, single-thread, ruLake Fresh path): before (AVX2 path): ~3,681 QPS after (AVX-512): ~4,067 QPS (+10.5%) Below the 2× target because at D=128 only 2 u64s per candidate feed VPOPCNTDQ — the kernel is memory-bandwidth-bound on the sequential packed stream, and the _mm512_storeu_si512 → scalar fold for per-candidate pair reduction eats part of the win. A vpsadbw-based in-register reduction would recover more but would balloon the intrinsics surface beyond what fits cleanly in scan.rs. Determinism preserved: scan_avx512 is byte-identical to scan_scalar at D=64, D=100, D=128, D=192, D=200, plus tail sizes n=7 and 1023. New test scan_avx512_matches_scalar exercises a 1000-vector D=128 run; the existing run_both harness adds AVX-512 parity to every shape it tests. Clippy clean (one allow(incompatible_msrv) scoped to scan_avx512 only — AVX-512 intrinsics stabilized in Rust 1.89, runtime detection guarantees safe dispatch). 38 → 39 rabitq lib tests. Rulake unchanged (42). Co-Authored-By: claude-flow <ruv@ruv.net>
cargo publish requires both `path` and `version` for cross-crate workspace deps so crates.io can resolve them once the published version lands. Adds `version = \"2.2\"` alongside the existing path (path wins during local dev, version is what crates.io indexes). Dry-run now passes the manifest-validation stage and only fails on the expected "ruvector-rabitq not yet on crates.io" line — publishing rabitq first + waiting ~1 min for indexing will clear it. Co-Authored-By: claude-flow <ruv@ruv.net>
Establishes ruvnet/ruvector as the canonical source and ruvnet/RuLake
as a read-only mirror. Implements "option C" — no submodules, no
workspace-inheritance rewrites, no `--recursive` tax on contributors.
Trigger: push to `main` touching either
- crates/ruvector-rulake/** (the whole crate: src, tests, examples,
Cargo.toml, README, BENCHMARK, …)
- docs/adr/ADR-15[5-8]-* (the four ruLake ADRs)
- the workflow itself
plus a workflow_dispatch for manual re-syncs.
RuLake repo layout after sync:
/
├── README.md hand-maintained landing page, never overwritten
├── LICENSE-MIT hand-maintained
├── LICENSE-APACHE hand-maintained
├── MIRROR.md tombstone explaining read-only status (written by the workflow)
├── crate/ ← rsync'd from crates/ruvector-rulake/
│ ├── Cargo.toml (workspace-inheritance preserved; consumers
│ │ who clone RuLake standalone see the manifest
│ │ as-is, but the canonical build is from the
│ │ monorepo so this is non-blocking)
│ ├── src/ tests/ examples/ BENCHMARK.md …
└── docs/adr/ ← cp'd, only ADR-155…158
├── ADR-155-rulake-datalake-layer.md
├── ADR-156-rulake-as-memory-substrate.md
├── ADR-157-optional-accelerator-plane.md
└── ADR-158-optional-rotation-and-qvcache-positioning.md
rsync --delete keeps the mirror an exact reflection; when a file is
removed from the monorepo, it vanishes from the mirror on the next
sync. Commit message on RuLake is `mirror: ruvnet/ruvector@<12-char>`
with a body carrying the full 40-char sha + provenance note.
Concurrency: serialized via `group: mirror-rulake` so a quick
back-to-back push doesn't race two sync jobs.
ONE-TIME SETUP (blocking the first sync until done):
1. Generate a fine-grained PAT at
github.com/settings/personal-access-tokens/new
scoped to repo: ruvnet/RuLake, permissions:
Contents: Read and write
2. Add it as a Repository secret on ruvnet/ruvector named
RULAKE_MIRROR_PAT
3. Merge this PR and verify the first run succeeds
(workflow_dispatch lets you trigger manually).
4. Optional post-merge: update the README at ruvnet/RuLake to
point file references at `crate/...` (currently they link to
the ruvector monorepo paths; after first sync, both work but
local paths are cleaner).
Why not option A (submodule): forces every contributor to run
`git submodule update --init`, forces a Cargo.toml rewrite that
loses workspace inheritance, splits PR #373's history in two.
Option C keeps all tooling working and RuLake always current.
Co-Authored-By: claude-flow <ruv@ruv.net>
ruvnet
added a commit
that referenced
this pull request
Apr 24, 2026
Pre-existing rustfmt drift across the workspace was blocking CI's `Rustfmt` check on PR #373 + PR #377. Running plain `cargo fmt` reformats 431 files; no semantic changes, no logic changes, no behavior changes — just what rustfmt already wanted. None of the touched files are in ruvector-rabitq, ruvector-rulake, or the new mirror-rulake workflow — those were already fmt-clean per the per-crate checks on commits 5a4b0d7, 5f32fd4, and f5003bc. The drift is entirely in: - cognitum-gate-kernel, cognitum-gate-tilezero - mcp-brain, mcp-brain-server (though the latter is also excluded from workspace members) - nervous-system, prime-radiant, ruqu-core, ruvector-attention, ruvector-mincut, ruvix/* and its sub-crates - several example dirs (dragnes, scipix, void-boundary-discovery, …) Verified post-fmt: cargo check -p ruvector-rabitq -p ruvector-rulake → clean cargo clippy -p ... -p ... --all-targets -- -D warnings → clean cargo test -p ... -p ... --release → 82/82 pass This commit intentionally does NOT touch clippy drift — that's many more warnings (65 missing docs, 39 precision-loss casts, 24 too-many- args, 23 unsafe-safety-docs, etc.) spread across even more unrelated crates, and each category is a cross-cutting design decision that deserves its own review. Rustfmt is purely mechanical; clippy is not. With this commit the Rustfmt CI job goes green on both PR #373 and PR #377. Clippy will still fail — that's honest pre-existing state and will need a separate dedicated drift-cleanup PR to resolve. Co-Authored-By: claude-flow <ruv@ruv.net>
Pre-existing rustfmt drift across the workspace was blocking CI's `Rustfmt` check on PR #373 + PR #377. Running plain `cargo fmt` reformats 427 files; no semantic changes, no logic changes, no behavior changes — just what rustfmt already wanted. None of the touched files are in ruvector-rabitq, ruvector-rulake, or the new mirror-rulake workflow — those were already fmt-clean per the per-crate checks on commits 5a4b0d7, 5f32fd4, f5003bc. Drift is in cognitum-gate-kernel, mcp-brain, nervous-system, prime-radiant, ruqu-core, ruvector-attention, ruvector-mincut, ruvix/* and sub-crates, plus several examples. Verified post-fmt: cargo check -p ruvector-rabitq -p ruvector-rulake → clean cargo clippy -p ... -p ... --all-targets -- -D warnings → clean cargo test -p ... -p ... --release → 82/82 pass Intentionally does NOT touch clippy drift — many more warnings (missing docs, precision-loss casts, too-many-args, unsafe-safety- docs) spread across unrelated crates, each category a cross-cutting design decision that deserves its own review. With this commit Rustfmt CI goes green on PR #373 and PR #377. Clippy will still fail — that's honest pre-existing state for a separate dedicated PR. Co-Authored-By: claude-flow <ruv@ruv.net>
ruvnet
force-pushed
the
research/rulake-mirror-workflow
branch
from
April 24, 2026 14:44
b20ea97 to
96d8fdc
Compare
Root-level `cargo fmt --all` doesn't recurse into nested workspaces (crates/rvf/, examples/onnx-embeddings/, examples/data/, …), but CI's `cargo fmt --all -- --check` was failing on files inside them (e.g. crates/rvf/rvf-wire/src/hash.rs). Ran `cargo fmt --all` inside each nested workspace. Mechanical-only whitespace, no semantic change. Touched nested workspaces: crates/rvf/* examples/onnx-embeddings/* examples/data/* examples/mincut/* examples/exo-ai-2025/* examples/prime-radiant/* examples/rvf/* examples/ultra-low-latency-sim/* examples/edge/* examples/vibecast-7sense/* examples/onnx-embeddings-wasm/* Combined with previous commit (96d8fdc), the full workspace tree should now pass `cargo fmt --all -- --check` in CI. Co-Authored-By: claude-flow <ruv@ruv.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Establishes ruvnet/ruvector as the canonical source of truth for
crates/ruvector-rulake/and makes ruvnet/RuLake an auto-synced mirror. Implements option C from the architectural discussion (monorepo canonical, mirror auto-updates) — not a submodule, not a subtree.This PR is workflow-only. It does not touch any code.
Why this shape
git submodule update --init, forces a Cargo.toml rewrite that loses workspace inheritance, splits the existing PR ruLake M1: cache-first vector execution fabric + memory substrate + accelerator plane #373's history in two. Rejected.subtree push/pullis a less-known workflow. Viable but more complex than needed.cargo publish -p ruvector-rulakefrom the monorepo remains the single publish path.What the workflow does
Triggered on push to `main` when either the crate or its ADRs change:
```
.github/workflows/mirror-rulake.yml
├── Checkout monorepo
├── Checkout ruvnet/RuLake using RULAKE_MIRROR_PAT
├── rsync -av --delete crates/ruvector-rulake/ rulake-mirror/crate/
├── cp docs/adr/ADR-15[5-8]-*.md rulake-mirror/docs/adr/
├── Write MIRROR.md tombstone
└── git commit -m "mirror: ruvnet/ruvector@" && git push
```
Resulting RuLake layout:
```
ruvnet/RuLake
├── README.md hand-maintained landing page, never overwritten
├── LICENSE-MIT hand-maintained
├── LICENSE-APACHE hand-maintained
├── MIRROR.md tombstone, written by the workflow
├── crate/ auto-synced from crates/ruvector-rulake/
└── docs/adr/ auto-synced (ADR-155…158 only)
```
rsync `--delete` keeps the mirror an exact reflection — files deleted in the monorepo vanish from the mirror.
One-time setup required before first run
This must happen before merging or the first workflow run will fail:
The built-in `GITHUB_TOKEN` can't push to another repo, hence the PAT. A fine-grained token scoped to only RuLake is the minimum-privilege choice.
Post-merge follow-up (optional)
Once the first sync succeeds, the RuLake README's internal file references currently point at the ruvector monorepo paths (e.g. `github.com/ruvnet/ruvector/blob/main/crates/ruvector-rulake/BENCHMARK.md`). These still work, but can be updated to local paths (`crate/BENCHMARK.md`) for cleaner URLs. That's a separate RuLake-side commit, not part of this PR.
Test plan
Scope
Workflow-only. Does not modify the crate, Cargo.toml, or any existing file outside `.github/workflows/`. Safe to merge independently of PR #373 (which is the M1 + M1.5 feature work).
🤖 Generated with claude-flow