Anvil 0.5.2: clustered index generations, cache and minimum usable engines

[zcourts]

Outcome

Ship Anvil 0.5.2 as the first clustered index and PersonalDB capability release. Keep the implementation KISS: build the minimum usable version of every engine in scope, test it early, record tolerable limitations rather than widening the design, and fix only true release blockers before tagging and publishing 0.5.2.

This is a capability release, not a compatibility restoration of 0.4.

Non-negotiable architecture

• One logical index has exactly one builder and up to three query-serving replicas.
• Weighted HRW over stable (tenant_id, bucket_id, index_id) selects rank 0 as builder and ranks 0..2 as query replicas.
• Any public node can accept a query; it proxies the whole query to one selected replica. There is no distributed scatter/join query or cross-node result merge.
• Raft supplies ACTIVE cluster membership and the existing serving fence only. Index definitions, assignments, bytes, generations, cursors and caches never enter Raft.
• All authoritative definitions, manifests and index files are ordinary Anvil objects under the reserved _anvil namespace. Small bytes follow the normal inline path and large bytes follow the normal erasure-coded path.
• There is no authoritative index column family, index registry, index-specific persistence plane or index-specific placement plane.
• Local disk and memory materialisations are disposable caches.
• Foreground object mutation does no synchronous index construction. It only retains the existing atomic head-plus-source-journal write.

How every node's changes reach one builder

A builder is not limited to the journal on its own node.

Every authoritative object coordinator writes the current head and one LocalChange to its own durable source journal in the same RocksDB batch. The ingress node is not necessarily the journal source: if a request enters node B and HRW forwards it to coordinator C, C owns the authoritative head and journal entry.

Each node that owns one or more index builders runs one IndexEventRouter:

1. Read the current ACTIVE membership and serving fence.
2. Maintain one contiguous cursor for every ACTIVE source ID/epoch.
3. Read each remote source through the existing private mTLS source-status and source-journal APIs.
4. Verify source identity, epoch, offset continuity and retention floor.
5. Match each invalidation against locally assigned index definitions.
6. Route it to bounded per-builder queues.
7. Retry from the last applied/published checkpoint after interruption.
8. Rebuild an index if a source changes epoch or its checkpoint is below the retention floor.

The router consumes each cluster source once and multiplexes it to all local builders. There is no per-index peer subscription and no all-node event broadcast.

A generation manifest contains a checkpoint vector with (source_id, source_epoch, next_offset) for every source. A generation is not advertised as covering a change unless its manifest checkpoint covers that change.

A global event order is not required. Events are invalidations, not historical truth. On every relevant event the builder rereads the authoritative current head and version. An older PUT delivered after a newer DELETE therefore cannot resurrect the path.

Atomic-program visibility

Index visibility must match ordinary API visibility.

• If a program is partially visible through the ordinary object APIs, the index may reflect that same visibility.
• If a program is atomically visible through ordinary APIs, an index generation must not publish only a subset of its affected paths.
• The implementation must derive this from the existing atomic-program commit/finalisation contract rather than inventing a weaker index-only rule.
• If the current source journal lacks the grouping/fence information needed to preserve the ordinary guarantee, that is a release-blocking gap to fix at the event contract.

Authoritative layout and publication

A representative layout is:

/_anvil/indexes/<index_id>/definition
/_anvil/indexes/<index_id>/current
/_anvil/indexes/<index_id>/generations/<generation>/manifest
/_anvil/indexes/<index_id>/generations/<generation>/files/<name>/segments/<ordinal>

Generation publication:

1. Builder captures its HRW membership fence and expected current generation.
2. Builder writes immutable sealed file segments with REPLICATED acknowledgement.
3. Builder writes the immutable generation manifest with REPLICATED acknowledgement.
4. Builder CAS-publishes current.
5. The current-pointer coordinator verifies the caller remains HRW rank 0 at the supplied fence.
6. Queries pin exactly one immutable generation.

The builder may use local disposable scratch files for sorting, graph construction and merging. Scratch files are not authoritative storage.

Shared cache

One IndexCacheManager per node serves all locally assigned query replicas.

Administrator configuration:

• absolute disk-byte budget;
• memory budget as bytes or percentage;
• materialisation concurrency;
• default/minimum/maximum logical segment size;
• generation count, age and authoritative-byte retention caps.

Cache identity is immutable content identity, allowing deduplication across generations and indexes. Concurrent misses coalesce. A fetched object is length/hash verified before atomic admission.

The memory tier uses memmap2 where appropriate. This dependency is approved: it is established, small and preferable to maintaining unsafe platform-specific mmap code.

A cache handle pins its backing entry. Zero external references make an entry eligible for eviction; they do not force immediate deletion. LRU/clock policy chooses unpinned entries under pressure. Pinned handles may temporarily exceed configured cache budgets and the sweeper restores the limits after they drop.

Indexes larger than the cache remain usable because only the requested logical segments and bounded prefetch are pinned.

Async IndexFile API

Do not accept caller-owned mutable buffers across async calls and do not require a mutable cursor-bearing file handle.

Use immutable positional reads:

pub struct IndexSlice {
    // Owned immutable view that pins its cache backing and dereferences to [u8].
}

impl IndexFile {
    pub async fn read_at(
        &self,
        offset: u64,
        max_length: usize,
    ) -> Result<IndexSlice, IndexError>;

    pub async fn prefetch(
        &self,
        offset: u64,
        max_length: usize,
    ) -> Result<(), IndexError>;

    pub async fn pin(
        &self,
        offset: u64,
        max_length: usize,
    ) -> Result<IndexSlice, IndexError>;
}

Semantics:

• IndexSlice owns/pins the immutable backing and exposes the exact returned bytes.
• Empty data means EOF.
• A read may return fewer than max_length, normally at a logical segment boundary, so it need not copy several mmap/cache entries into one allocation.
• The caller advances by the returned byte count.
• There is no &mut [u8] held across an await, mutable cursor or exposed node/path/cache/EC identity.
• Index snapshots are immutable multi-file generations.

Writers are append-only, multi-file and may request an earlier format-aware seal boundary. Automatic sealing clamps the requested target to administrator limits. Publication occurs only after all sealed files are durable.

Query results and freshness

Never fail a successful index query merely because its generation is behind.

Every query response returns results plus freshness evidence:

• index ID and definition version;
• generation ID;
• generation publication time;
• complete source checkpoint vector or an opaque integrity-protected equivalent;
• observed source tails when known;
• per-source and aggregate lag hints;
• whether a rebuild is in progress;
• authorization revision used for filtering.

The client decides whether the result is sufficiently fresh. There is no INDEX_LAGGING error and no server-side wait-for-freshness mode in 0.5.2.

Pagination tokens bind the principal, authorization revision, index generation, query shape and last sort position. Page size is bounded but total traversal has no arbitrary cap.

Authorization

Both ingress and query-execution nodes authenticate and authorize. The peer does not trust an ingress decision.

Eliminate work in this order:

1. authenticated tenant;
2. stable bucket;
3. fixed index path boundary;
4. Zanzibar permission to query the index;
5. bucket/boundary-wide object-read grant where available;
6. bounded exact-object Zanzibar evaluation for remaining candidates.

Unauthorized candidates are removed before results, facets, totals, scores or continuation state are exposed. Authoritative Zanzibar tuple/schema storage remains in the synchronous replicated metadata plane. Rebuildable authorization bitmaps may accelerate pruning but can never grant authority.

Retention and GC

Always preserve the current generation.

For obsolete generations, administrators configure three independent caps:

• maximum retained generation count N;
• maximum retained age M;
• maximum retained authoritative bytes O.

Hitting any cap triggers oldest-first cleanup until all configured bounds are satisfied. Cleanup deletes ordinary generation objects by exact version, after which normal reference counting and blob GC reclaim content.

Active local cache handles continue pinning already materialised bytes. Any unavoidable race for a non-materialised segment belonging to a concurrently deleted obsolete generation must be bounded/tested and documented; do not invent a distributed query-lease plane.

Cold discovery

Definitions are ordinary objects. A node that loses its disposable assignment cache reconstructs it by scanning definition paths, ranks each definition with HRW and activates the builders/query replicas assigned to it.

The startup cost is an accepted 0.5.2 limitation. Do not add a registry, catalogue column family or other optimization until measurement proves it necessary.

Index types in product scope

Implement the minimum usable product for every listed engine, move to integration testing quickly, and document non-blocking limitations.

Path/directory

Sorted current paths and exact versions, prefix seek, stable unbounded pagination, delta/tombstone runs and builder compaction. This is the future fast path for ListObjects.

Metadata filter

Typed value dictionary, equality/existence lookup, bitmap postings, path narrowing and authorization-safe facets. Associated metadata remains ordinary object data; no metadata side plane.

Typed JSON

JSON Pointer fields, scalar and array membership, null/missing distinction, eq, in, prefix, ranges, exists, multi-column order, deterministic path tie-break and exact indexed object version.

Full text

Analyzer configuration, term dictionary, independently addressable compressed postings/positions, document lengths, phrase lookup and a minimum usable BM25 query.

Vector

Fixed dimensions/metric, immutable vector and HNSW files, entrypoints/upper layers suitable for pinning, paged lower graph/vector reads, deletes and local top-k merge. Do not reconstruct every vector into a full in-memory graph per query.

Hybrid

One immutable generation containing shared document identity plus full-text and vector files. Execute both locally and provide one minimum usable deterministic fusion method.

Git source

Sorted repository/commit/tree-path/object records with exact lookup and prefix tree listing. It uses the generic files but remains available to the later Git gateway.

Tensor/model and Hugging Face manifests

Sorted tensor-name/metadata and target-file/metadata dictionaries with exact lookup and ordered listing. These prove later gateways need no new persistence system.

PersonalDB row metadata

Sorted (database group, table, primary key) authorization metadata driven by the ordered PersonalDB canonical log, not the object invalidation journal. Its generation reference advances with the PersonalDB canonical head, because it participates in commit authorization.

Bitmap/facet/range

These are physical/query capabilities rather than separate public kinds. Keep independently compressed blocks addressable through IndexFile. Evaluate a native Rust compressed-bitmap implementation; do not add FastBit merely by name.

PersonalDB capability in 0.5.2

Implement the spec-compliant minimum without restoring application-specific legacy usage:

• weighted-HRW primary per database group;
• any-node request proxying;
• authenticated group create/describe/open/join;
• leader epoch/lease handling;
• idempotent signed source proposals and voter acknowledgement;
• one predecessor-linked, hash-chained canonical log;
• witnessed commit and authoritative head;
• bounded streamed catch-up from (index, hash);
• snapshot registration, discovery and ranged streaming;
• watch/head notifications as wakeups;
• Zanzibar administration, proposal, read, catch-up, snapshot and watch checks;
• all log segments, payloads, snapshots, manifests and heads through ordinary inline/EC object storage.

PersonalDB itself is not an index and must not be implemented as one. Its row-metadata projection may use the generic index-file/cache substrate.

Delivery discipline

1. Implement the smallest coherent generic substrate.
2. Implement minimum usable engines instead of feature-complete engines.
3. Add narrow unit tests alongside each component.
4. Build only after the required code paths exist.
5. Validate with a three-node Docker Compose cluster and unique buckets.
6. Exercise cross-node writes, all-source builder catch-up, failover, cold cache, cache eviction, generation GC, Zanzibar filtering and atomic-program visibility.
7. Run representative typed, path, and PersonalDB qualification workloads.
8. Record tolerable gaps in docs/known-limitations.md.
9. Fix only true correctness, security, data-loss or release blockers.
10. Commit in small logical groups, push, tag 0.5.2, publish the GitHub release and multi-architecture GHCR image.