Single-node 0.5.3 loses readable large objects after reference delivery stalls

[zcourts]

Release

ghcr.io/worka-ai/anvil:0.5.3 on a pristine single-node Docker volume, LOCAL durability.

Reproduction

1. Start the published image using the same one-node settings as scripts/qualify-single-node.sh.
2. Provision one tenant and owner application.
3. Create a bucket.
4. Put a JSON object of roughly 32 KiB or larger with LOCAL durability.
5. Read it repeatedly while reference delivery runs.

The object is initially readable, then reads fail. Small inline objects remain readable. This reproduced on multiple pristine volumes.

Observed

Server repeatedly logs:

ordered reference delivery paused: metadata quorum cannot prove source event 14: source reference proof is missing

Subsequent reads fail with:

insufficient valid shard owners: need 2; healthy=0, missing=1, corrupt=0, unavailable=0

The default erasure profile is 2+1, while the published 0.5 release describes LOCAL as the satisfiable single-node durability. A locally acknowledged object must remain readable on that topology.

Impact

This is a data-availability/correctness blocker for single-node deployments using payloads that leave the inline path. Worka can temporarily bound its HMO release records below the affected size, but cannot safely use Anvil 0.5.3 for arbitrary object sizes until this is fixed.

Expected

A successful LOCAL write on the supported single-node topology remains durably readable. Reference/materialisation failure must not replace or retire the only readable local payload.

[zcourts]

Additional Worka HMO qualification evidence on a fresh official 0.5.3 single-node volume (2026-08-05): after reference delivery became permanently stuck at metadata quorum cannot prove source event 18: source reference proof is missing, the same node later began rejecting ordinary indexed reads with Unavailable: this node does not hold a current serving fence. This stopped template SDLC processing despite the container remaining up. The failure occurred after only sub-64-KiB application objects were admitted; Worka's compact HMO manifest was deliberately kept below the 65,536-byte payload-plane threshold. Full log retained locally as .tmp/worka/hmo-e2e73.log / Docker logs.