Repository navigation
RVF SDK silently drops metadata; native metadata is not durable or returned #704
Description
Versions
@ruvector/rvf@0.2.3@ruvector/rvf-node@0.2.0- darwin-arm64 native binary
0.1.7 - Node 26.4.0, macOS arm64
Public SDK reproduction
const db = await RvfDatabase.create(path, {
dimensions: 2,
metric: 'cosine',
}, 'node');
await db.ingestBatch([
{ id: 'red-item', vector: new Float32Array([1, 0]), metadata: { color: 'red' } },
{ id: 'blue-item', vector: new Float32Array([0, 1]), metadata: { color: 'blue' } },
]);Observed:
- Ingest reports both vectors accepted.
- Unfiltered results contain no metadata.
- A field-0 equality filter for
redreturns no results.
Native N-API reproduction
Using the low-level binding, pass one positional metadata record per vector:
db.ingestBatch(
new Float32Array([1, 0, 0, 1]),
[101, 102],
[
{ fieldId: 0, valueType: 'string', value: 'red' },
{ fieldId: 0, valueType: 'string', value: 'blue' },
],
);A field-0 red filter works while that database instance remains open. However:
- Results still contain only ID and distance.
- After close/reopen, the same filter returns no results.
Root cause
npm/packages/rvf/src/backend.tsacceptsRvfIngestEntry.metadatabut calls nativeingestBatch(flat, ids)without metadata.- Native
SearchResultcontains only ID/distance. - Runtime inserts metadata into an in-memory
MetadataStore, but no metadata segment is written and reloaded. - The SDK filter declaration omits the native parser's required
valueType.
Impact
Applications can successfully ingest records while silently losing metadata. Filtering may appear to work through the native API and then stop working after restart, which can produce incomplete or incorrect search results without an error.
Requested fix
As an immediate safety measure, reject non-empty SDK metadata until it is supported rather than silently accepting it.
A complete fix needs:
- A per-vector metadata shape across the SDK and N-API.
- Metadata segment persistence and reload.
- Aligned filter types, including
valueType. - Either metadata-bearing search results or
getMetadata(id). - Close/reopen tests, including vectors with unequal metadata-field counts.
Activity
fix: SONA zero-vector bug, RVF wasm persistence, metadata safety, doc…
chore(rvf): bump @ruvector/rvf to 0.3.0
ruvnet commentedon Jul 28, 2026
Release verification update (2026-07-28): the interim safety fix from #708 is present in published @ruvector/rvf@0.3.4. A clean registry install rejects non-empty metadata with MetadataNotSupported before ingest, and store status remains at zero vectors, so the public SDK no longer silently accepts or loses metadata. Keeping this issue open for the remaining native durable-metadata/search-result design described above.
thesoulpole commentedon Jul 29, 2026
thx(!) for the RVF/SONA batch (#708) and agentdb #15. saw your release-verification pass ystrdy.
One thing still blocks me: durable metadata (#704). use case is merchant catalog distribution — one RVF file a merchant hands over that carries its own product attributes (brand, size, colour, price). today those live in a sidecar manifest next to the file, so it's two artifacts that have to stay in sync, which defeats the point. MetadataNotSupported is the right interim call, but we need the real thing.
read through main at 597be6a to size the gap. smaller than I expected:
so the shape looks like write_cow_state_segments / restore_cow_state applied to MetadataStore over the Meta segment, plus a field-name→fieldId dictionary in the same payload.
happy to write it. four things I'd want your call on first, since they're format commitments rather than implementation:
say the word on 1-4 and I'll do the PR. if it's already scoped your side, ignore me and just say roughly when.
ruvnet commentedon Sep 12, 2026
Dream cycle follow-up (2026-09-12): PR #980 is relevant to the string-ID collapse portion of this issue, but it is not yet a release-qualified remediation. It changes only two manifest floors, leaves the committed lockfile untouched, has no republished-artifact or independent-reader receipt, and required Workspace CI run 34616661436 failed (3,546/3,547 tests; database-lock failure) with a sibling shard cancelled. The acceptance gate remains: exact lock/install graph, three distinct string IDs across close/reopen, explicit metadata rejection or durable metadata, native/WASM parity, independent-reader verification, and all required workspace jobs green.
ruvnet commentedon Sep 15, 2026
Dream cycle release holdout (2026-09-15): the string-ID collapse remains reproducible in the published
npx ruvector@0.3.0RVF CLI.Using the bundled ONNX
all-MiniLM-L6-v2embedder (384 dimensions) and a cosine RVF store, I ingested five distinct records shaped exactly as the CLI documents ({id, vector}) with string IDs. The CLI reportedIngested 5 vectors (0 rejected), butrvf statusreportedtotalVectors: 1, and a top-5 query returned onlyid=0.Numeric-ID holdout, using the identical five vectors and a fresh store, reported
totalVectors: 5; top-5 returned IDs 0–4, with the exact query vector at distance 0.000000. This isolates the failure to the string-ID path rather than embedding generation, vector count, dimension, metric, or query construction.The named-ID store was not used for evidence retrieval and no self-learning claim is made from it. The prior release gate still applies: exact published graph, distinct string identities across close/reopen, native/WASM/RVF parity, independent-reader verification, and all required Workspace jobs green.