⚡ Coalesce ready background completions into one wake-up turn (canary)

[danny-avila]

Summary

When several background results finish while their conversation is busy, each one gets its own wake-up turn. When the busy turn ends, they all become deliverable at once. One wins, and the others fail generation creation with JobPredecessorMismatchError, defer, and run one after another. Four finished results cost four agent turns and four model calls, and the last one arrives only after the other three turns.

This is a follow-up to #16349, #16339, #16345 and #16343. Those made waiting cheap and delivery prompt; they don't change how many turns a group of results costs. Implementation is maintainer-directed canary work: branch from origin/canary and open the PR with --base canary.

Evidence

A local stress run on dev (7238be3: #16349 + #16339), with 8 conversations each finishing 4 background results behind a 5-minute turn:

• 32/32 results delivered, each on its first attempt, a median of 2 s after the busy turn ended
• 32 separate wake-up turns, and 43 JobPredecessorMismatchError logs from wake-ups colliding at turn creation (12 on the earlier dev baseline, 29 with 🐢 fix: Back Off Waiting Completion Wake-ups and Keep Their Trace Out of Requests #16349 alone). Prompt delivery releases a conversation's results at the same moment, so collisions grow as delivery gets faster.
• On the demo deployment, real traffic produced 21 of these errors over 5 days. Most real results are collected by the agent's own same-turn poll, so the cost is concentrated in turns that park several results.

Why batching doesn't apply today

The message-backed path already claims a bounded sibling batch (completionResultBatchSize, default 8). The durable-receipt path, which is the normal path, claims exactly one result and reconciles that result's ownership against the parent message before dispatch. Extending bounded batching to durable receipts is the optimization. It's also where the delivery guarantees live, so it needs an explicit contract first.

Proposed contract

1. Ownership per result, not per batch. Each receipt claim is its own CAS; updateMany is atomic per row, not per batch. Concurrent roots may claim different subsets and start separate turns. Opportunistic batching reduces turns and collisions; it doesn't guarantee one turn per group.
2. Siblings stay responsible for their results. A sibling whose result a batch root claimed does not settle. It defers as a follower, and settles only once the root's continuation is confirmed applied. If the root dead-letters, is retired, or releases its claims, the follower claims and delivers its own result. Its row persists, so a crash at any point strands nothing.
3. Frozen membership. The root records its member list, the claims that actually succeeded, durably on its own row before dispatch. Retries rebuild the same input from that list and never add newly finished siblings. A lost write reply is idempotent: reading back by claimId returns the same set.
4. Explicit lifecycle for: partial claim success, lost write replies, definite pre-dispatch failure (release every member), dispatch that may already have started (keep ownership, reconcile), and root cancellation, dead-lettering and owner loss (followers take over).
5. Every member reconciled, with a precise scope. Message-backed ownership is checked for every selected result, including races with manual polls and late message persistence. The scope key is principal, tenant, conversation, parent branch and target agent, not just "same conversation".
6. Mixed-version gate. An older replica settles a claimed sibling the old way, which would reintroduce stranding. So batching must be gated: behind a new worker capability, or a flag that stays off until every replica runs it. Recovery is proven with a mixed-version test, not assumed.
7. Scope for a first version: ordinary background tools and code results. Subagent completions stay one per turn. No waiting window: only results already finished at claim time are folded in.

Per-branch election, meaning one turn per ready group, is a possible second phase if that becomes a hard requirement. It adds coordination and recovery complexity, so it needs its own design and estimate.

Acceptance

• Barrier-controlled multi-worker races and crash injection at each step: after claim, after the membership write, after dispatch, and during retry.
• The stress run reports unique results consumed, duplicates, stranded results, turns, and delivery latency. Target: 8 conversations × 4 results produce well under 32 turns, with zero duplicates, zero stranded results, and no latency regression against dev. The turn reduction is a benchmark target, not an established result.
• JobPredecessorMismatchError from wake-up collisions drops substantially in the same run.

[mehran-almma]

I traced this on dev (7b2362d). A few findings:

1. The batch of 8 effectively never forms. claimBackgroundToolResults narrows to a single task whenever it's receipt-backed (message.ts:1870), and every current registration is receipt-backed. The receipt claim also ignores limit and returns [result]. The root side is already batch-shaped, though: buildWakeupInput handles N results and the release goes by claimId. What's missing is sibling selection plus the follower lifecycle.

2. The scope key is narrower than the fence. Claims are scoped to the launching parentMessageId, but the fence that throws JobPredecessorMismatchError covers the whole conversation. Results launched from different turns can never batch, yet they still collide. "Parent branch" should mean the resolved dispatch parent (latestAssistantDescendant).

3. The repeated collisions are a herd, and there's a cheap fix. On settle, expediteCompletions releases every waiting delivery in the conversation at once, and readiness deferrals have no jitter. One wins, the rest 409, and the next settle re-expedites everything that's left, which works out to roughly k(k−1)/2 collisions per group. Expediting only the oldest waiting delivery per conversation, or adding jitter, would cut the error count without touching delivery semantics.

4. Gate per row, not per replica. A follower that sees another claimId returns settled immediately, so an old replica would strand it. Rows already carry requiredWorkerCapability, so a root could absorb only siblings enqueued with a new capability. Frozen membership also can't reuse batchMemberIds: every replica turns those rows into coalesced-event envelopes.

Canary has the same claim logic but no expedite or backoff, so the baseline should be measured there.