[v0.4.1] Remove per-bucket object journal serialization from unrelated MVCC mutations

[zcourts]

Problem

Every object mutation in a tenant bucket currently observes and rewrites two shared per-bucket stream heads and sequential event rows:

• object_watch:tenant:<id>:bucket:<id>
• object_metadata:tenant:<id>:bucket:<id>

plan_object_watch_appends and plan_metadata_events bind those shared head values and preallocated sequence rows into the user transaction snapshot. Concurrent mutations of unrelated object keys therefore certify as TransactionPointConflict. A steady but modest background writer can starve an unrelated multi-object control transaction through all client retries.

Production evidence

Worka HMO AMD64 E2E attempt 23 retried a three-object template trust mutation eight times. Persisted prepared bundles mapped every conflict hash to one of the two shared journal heads or their sequential record rows; none mapped to the trust, audit, or claim objects being mutated. The same failure reproduced on native ARM and AMD64 emulation.

Required architecture

• Mutations of unrelated object keys must not serialize on a per-bucket global head.
• Object mutation and durable watch/metadata evidence remain atomic.
• Watch ordering/cursors and tamper evidence remain correct across Raft apply and replay.
• Do not resolve this by dropping journal predicates, last-write-wins head updates, disabling watches, or requiring Worka to suppress normal concurrent writes.

Acceptance

• A regression opens concurrent transactions against different object keys in one bucket and both can commit without a journal-head point conflict.
• Same-key/precondition conflicts still abort correctly.
• Watch and metadata readers return complete, deterministic ordered history after concurrent commits.
• Hash/tamper validation and replay tests pass.
• Worka template trust mutation commits while normal device/build liveness traffic is active.

[zcourts]

Additional Worka attempt-24 evidence:

After decoupling the Worka build-worker heartbeat from its 2-second queue poll, trusted-org creation, binding, discovery, and authorization all progressed. The template processing run was then leased but workspace preparation repeatedly aborted under normal 15-second device and 30-second build-worker liveness traffic.

Observed multiple repeated PointConflict hashes while the processing record remained safely Leased. The processor retry loop prevents corruption but can starve indefinitely. AMD64 emulation widens transaction windows, but the conflicting keys are still unrelated per-bucket journal state, so native speed is not a correctness fix.

Anvil also emitted watch checkpoint cursor cannot move backwards from object materialization during the same clean-volume run. That may require a separate issue after correlating the exact checkpoint state, but it is recorded here because the affected subsystem overlaps journal sequencing.

[zcourts]

Native ARM64 confirmation from Worka release attempt 25 (2026-07-29): reproduced on the same fix-bearing source snapshot with no QEMU. Fresh Trixie ARM64 image/container is healthy; Worka API passed capabilities, identity, packs, workspaces, and auth initialization, then entered system_agents. During startup Anvil repeatedly emitted MVCC transaction predicate is false at its snapshot (tasks 7, 11, 15, 19, 47, 51, 53, 59, 63, 69...) and authorization userset publication aborted: PredicateConflict. Failures continue approximately every 10-20 seconds while some forward progress occurs. This rules out AMD64 emulation as the cause of the prior HMO starvation and leaves this issue as a Worka 0.1.0 release blocker.

[zcourts]

Implementation written, not yet built (2026-07-29):

• Foreground object mutations no longer call plan_metadata_events or plan_object_watch_appends and no longer observe/rewrite either per-bucket stream head.
• The certified bundle now carries canonical ObjectJournalCommitJob intent.
• At Raft apply, immutable metadata/watch facts are installed atomically in the same RocksDB batch using deterministic (commit_version, canonical journal-job ordinal, entry ordinal) cursors.
• Watch and metadata facts are self-authenticating; readers validate deterministic hashes and ordered cursors. Legacy v2 head/chained rows remain readable alongside v3 facts.
• Exact/latest watch receipts remain committed atomically with object projections.
• Focused tests cover cursor ordering and prove unrelated object jobs derive distinct keys without any TABLE_STREAM_HEAD_ROW mutation.
• Single-node shard-rebalance no-op traffic was also removed because it contributed unrelated topology/checkpoint churn during Worka startup.

This does not drop predicates, use last-write-wins heads, disable watches, or serialize the bucket. cargo fmt --all --check and git diff --check pass. Compile, concurrency tests, replay/tamper tests, and Worka QA are intentionally deferred to the one consolidated build.

[zcourts]

E2E33 isolated a residual public-CAS path after the internal per-object fence change.

Evidence:

• transaction tx-9fd58949d21a03e85fea0657d5a71620686a62cc6fc647f3b840fb860421c599 repeatedly certified against table 33025 / cf_object_heads / object-current for operations/operation-publisher-tenant-worka/operation.json;
• 9 PredicateConflict matches, 0 PointConflict, 0 object-id-counter conflicts, 0 typed-payload errors, and 0 transaction-expiry errors;
• the remaining caller was ObjectManager::object_mutation_precondition_snapshot, used by native/write MutationBatch preconditions.

Fix pushed at a6b8f71:

• semantic version/etag/existence evaluation remains against the current-head projection at the fixed transaction snapshot;
• certification is now against object-mutation-fence-current at the same base snapshot;
• staged same-key fences are deliberately excluded from predicate capture, avoiding self-conflict across sequential RPCs in one explicit transaction;
• upgrade case (existing projection, absent fence) certifies Absent so only one concurrent first post-upgrade mutation wins;
• focused regression asserts that public object preconditions never return the derived current-head key.

cargo fmt --all -- --check and a source-wide predicate call-path audit are clean. Per the consolidated-build policy, no rebuild/test compile was started yet. E2E34 must qualify the rebuilt artifact before this issue closes.

[zcourts]

E2E34 qualification checkpoint (2026-07-30): source a6b8f719f36190800dcdcceb0c06957270db84bf, ARM64 candidate image ID sha256:1e912999d6e8f44d63ab13beb113c3310c11d6f7d6007ef915c19b9dc831f8b8, fresh data/bootstrap volumes. Worka completed every persistence bootstrap stage and reached ready health with zero PredicateConflict/PointConflict/transaction-expiry/typed-payload-unavailable errors. Critically, the publisher identity transaction for tenant tenant-worka, owner user-worka-publisher completed at the exact gate where E2E33 repeatedly conflicted on the derived object-current head. Full HMO flow remains in progress; this confirms the public CAS fence fix at the original failure site.