bug: a failed branch merge wedges all writes on a served graph until the server is restarted

[ragnorc]

Summary

On a served graph, a branch merge that fails leaves a pending BranchMerge recovery
operation on main that blocks every write — not just merges — until an operator
restarts the server. Clients cannot recover: retries return the same operation id
indefinitely.

Hit twice in one day on a production graph (0.10.0), each time blocking unrelated agents
writing to the same graph. Same failure mode as #554, which was fixed for optimize /
Mutation barriers; this is the branch merge surface.

Error

recovery required for operation 01M22ZRK19HDA27Y0R3QSE56KE: pending BranchMerge
recovery operation on branch 'main' blocks the synchronous write/control recovery
barrier (datasets dataset for edge type 'ArtifactFromPerson', dataset for edge type
'IdentifiesPerson', dataset for edge type 'InConversation', dataset for node type
'Artifact', dataset for node type 'Conversation', dataset for node type 'ExternalID',
dataset for node type 'Person', dataset for node type 'SyncState');
reopen the graph read-write before retrying

What triggered it

Incident 1 — merge rejected by its own resource limit. 3,481 entities across 5 node
types and 3 edge types:

ordered_scan_input_batch_bytes: actual 39321601, limit 39321600

One byte over. No concurrency, no external writer, no crash — the server rejected its own
merge and wedged itself.

Incident 2 — the same wedge from a much smaller payload. After recovery I split the
payload and retried with 1,819 entities / 0.6 MB. It wedged again, new operation id.

That second data point matters: 0.6 MB is nowhere near a 37.5 MB ceiling, but the target
main holds 50,201 artifacts. It suggests ordered_scan_input_batch_bytes scales with
the merge target, not the branch delta — in which case splitting the payload cannot
help, and a sufficiently large graph becomes unmergeable. This looks related to #566
("read cost scales with commit/fragment count, not the branch delta") and #641.

Client cannot recover

Confirmed on production and again on a local server:

Path	Command	Result
control	branch create	recovery_required
write	load --mode merge	recovery_required
write, retried x3	load --mode merge	same operation id every time

The write probe was a no-op (an existing SyncState row re-merged with byte-identical
values), so this is not content- or conflict-dependent. Reads stay healthy throughout.

This is also why the retry-safe exit code proposed in #466 needs a topology split — on a
served graph a retry can never heal, because the barrier lives in the server (commented
there separately).

Restart clears it, and the merge never applied

main was byte-identical before and after both incidents (same entity counts; the target
data absent). So the barrier was guarding an operation that had not committed anything.

Scope: specific to branch merge

Reproduced locally (omnigraph-server 0.10.0, local cluster, 2 node types + 1 edge type).
Interrupting a client mid-operation and then probing writes:

Operation	Interrupted	Writes afterwards
load --mode merge	yes	fine
mutate	yes	fine
optimize	yes	fine
branch merge	yes	wedged (1 of 9 attempts)

load is documented as "one atomic commit"; branch merge is the multi-phase publish that
needs the recovery sidecar, and it is the only operation that wedged.

Caveat, stated plainly: the client-interruption wedge reproduced once in nine attempts
at various kill delays against a 6.7s merge, so interruption looks like a narrow race
(plausibly the Phase B -> manifest-commit window). The resource-limit path is the
deterministic one.

Secondary: the merge-side limit is undocumented

data.md documents the load bounds ("8,192 entities and 32 MiB of Arrow memory per touched
type"). My load respected them and succeeded; the merge then failed on
ordered_scan_input_batch_bytes, which is not documented anywhere I could find. Sizing a
payload to the published limit is therefore not sufficient to merge it.

Suggested direction

1. A failed merge should roll back its own barrier rather than requiring an operator to
   reopen the graph — especially when nothing committed.
2. Failing that, expose a supported way for a client (or the server itself) to clear a
   pending recovery operation without a process restart.
3. Bound or document the merge-side scan so payload sizing is possible.

Environment

• CLI omnigraph 0.10.0; server 0.10.0 (Railway, single node)
• Production graph: ~51k Artifact, ~4.5k Person, 8 datasets touched by the merge
• Local repro: omnigraph-server --cluster . --bind 127.0.0.1:8099 --unauthenticated

Happy to supply the local repro cluster, full logs, or to test a patch.

[azimafroozeh]

Could you share the local repro cluster, sanitized data, commands, and exact server build? The deterministic resource-limit failure would help most. I’d like to turn it into a GQT case that checks writes still work after the failed merge.

[azimafroozeh]

PR 699 is merged. The GQT reproduction now reaches the injected failure after recording merge recovery information and before creating the target table fork. Reads still work, but an unrelated write on the same Omnigraph returns RecoveryRequired, without reopening it. This reproduces with seeds 0 and 42 and fresh-process replay.

The case retains the successful-write expectation and marks only the exact recovery error as a known failure. This adds reproduction coverage, not an engine fix. The production resource-limit and client-cancellation triggers still need separate coverage.