Exact-review queue Durable Object overloads on records/export; batch publications fail in bursts

[steipete]

Summary

The ExactReviewQueue Durable Object in clawsweeper-status intermittently becomes unavailable for a minute or more under load. Every in-flight exact-review-batch-publish.yml run then fails at once, because heartbeats and post-effect requests exhaust their bounded retries (3 attempts / 45 s, added in #1368) against HTTP 500 exact_review_queue_unavailable.

Evidence (2026-09-03 UTC)

• After the fix(publish): retry transient queue failures during batch publication #1368 deploy (https://github.com/openclaw/clawsweeper/actions/runs/33712345842), runs https://github.com/openclaw/clawsweeper/actions/runs/33712516195, https://github.com/openclaw/clawsweeper/actions/runs/33712795256 and https://github.com/openclaw/clawsweeper/actions/runs/33712836865 all failed between 04:00:47 and 04:00:48 with Batch queue ... failed (HTTP 500): exact_review_queue_unavailable after two retries each. Same shape earlier: four runs failed together at 02:37–02:38 with client TimeoutError (20 s), and 17 of 73 claimed batches failed between 22:00 and 01:29 in the same two classes.
• wrangler tail clawsweeper-status (110 s and 166 s windows around 02:13 and 02:42): exceptions Internal error in Durable Object storage caused object to be reset (4×) and Durable Object is overloaded. Requests queued for too long. (1×); one exact_review_queue_request_failed with failure_category: platform_retryable.
• CPU profile of the object's fetches in the 166 s window: about 37 s of CPU total, of which /internal/state/records/export accounted for about 24 s; next were lifecycle/router-receipt (~2.9 s), publication-batches/complete (~2.8 s), github-etag-cache/store (~2.1 s). Alarms took 3–19 s wall time each (waiting on GitHub reads).
• The object served ~28 fetches/s in that window. records/export is called by every batch run's state hydration (scripts/worker-records.ts, .github/actions/setup-state), so more successful concurrent batches after fix(queue): queued publications starve behind stale batch probes and superseded rows #1367 increase the load.

Suggested directions

1. Take records/export off the Durable Object hot path: serve hydration from the R2 snapshot (STATE_SNAPSHOTS) or a cached export keyed by the records revision, and only fall back to the object for deltas.
2. Make batch heartbeats tolerate an outage longer than 45 s: allow N consecutive heartbeat failures within the lease window (lease is 30 min) instead of failing the batch on the first exhausted retry; keep the fenced lease semantics.
3. Consider the queue-service split runbook (docs/queue-service-split-runbook.md) or splitting the records store into its own object so publication/lease traffic is isolated from export CPU.

Tail method and token location are documented in the maintainer notes; no secrets were captured in the logs above.

[clawsweeper]

Codex review: this still needs some work. Reviewed September 22, 2026, 7:22 PM ET / 23:22 UTC.

Summary
Keep open as the canonical overload investigation. Snapshot, heartbeat, cache-isolation, and lifecycle-retry mitigations are implemented, but sustained recovery under peak load remains unverified and records reads still share the queue object. The earlier lifecycle-retry decision is resolved.

Reproducibility: no. high-confidence current-main overload reproduction is established. Source confirms shared routing and the discussion supplies historical production failures, but neither proves the present peak-load failure threshold.

Root-cause cluster
Relationship: canonical
Canonical: #1372
Summary: This issue owns the remaining overload investigation; related merged fixes resolve narrower contributing mechanisms.

Members:

• partial_overlap: Exact-review queue Durable Object is overloaded: etag cache and state reads share the queue object, causing bursty 500/503s #1529 - Closed after ETag isolation, with remaining records and state-writer pressure explicitly consolidated here.
• partial_overlap: fix(queue): retry replay-safe batch lifecycle post-effects #1533 - Merged lifecycle retry repair explicitly leaves the broader queue-availability incident open.
• partial_overlap: fix(publish): tolerate queue outages within the batch lease #1378 - Merged heartbeat tolerance addresses outages within confirmed leases without removing shared queue load.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Maintainer decision needed

Question	Recommendation
Should further storage separation wait for a peak-load profile of the current mitigations, or should a records-read isolation design be commissioned now?	Measure before selecting isolation: Capture peak-load route cost, queue failures, and publication recovery on the current deployment before choosing a storage boundary.

Why: The remaining work changes storage ownership and rollout boundaries, while existing observations do not establish which residual traffic warrants migration.

Next step
Preserve the no-new-fix signal while the incident owners assess current peak-load evidence and select any necessary storage isolation.

Review details

Best possible solution:

Keep lease and publication coordination responsive under measured peak hydration load, preserving snapshot watermarks, replay safety, and Bay’s observer-only contract.

Do we have a high-confidence way to reproduce the issue?

No high-confidence current-main overload reproduction is established. Source confirms shared routing and the discussion supplies historical production failures, but neither proves the present peak-load failure threshold.

Is this the best way to solve the issue?

Unclear until current load is measured: the narrow retry repair has landed, and moving the entire queue namespace would not itself isolate record traffic.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• Available production observations do not establish sustained peak-load recovery after the latest mitigations.
• Record-read isolation requires a consistent snapshot-and-journal boundary; the proposed Worker namespace transfer alone retains the shared object.

Codex review notes: model internal, reasoning medium; reviewed against 258a5b1fa254.

Label changes

Label changes:

• add issue-rating: 🦪 silver shellfish: Current issue advisory state selects this label.
• remove issue-rating: 🦞 diamond lobster: Current issue advisory state no longer selects this label.
• remove clawsweeper:source-repro: Current issue advisory state no longer selects this label.

Label justifications:

• P1: The documented overload causes bursts of failed review intake and publication workflows; sustained recovery remains unresolved.
• impact:other: The affected product behavior is maintenance-automation availability and timely review publication.

Evidence reviewed

What I checked:

• Current main retains shared record routing: The checkout matches fetched main. Records list, export, latest-snapshot metadata, and snapshot chunks still forward through the queue request helper; exactReviewQueueStub selects one named queue object. The export handler reconstructs records inside that object. (dashboard/worker.ts:1209, 258a5b1fa254)
• Snapshots reduce work without isolating reads: Hydration loads a stored snapshot and then exports journal entries after its watermark. The batch workflow still supplies repository slugs without focused item selection. Scheduled snapshots use runner-built uploads every six hours. (scripts/worker-records.ts:348, 258a5b1fa254)
• Previous retry blocker is resolved: GitHub confirms fix(queue): retry replay-safe batch lifecycle post-effects #1533 merged on September 11. Current client code admits bounded retries for explicitly opted-in lifecycle requests with stable replay identities, and the batch CLI opts in. Tests cover identical signed bytes, attempt limits, legacy exclusions, and permanent rejection statuses. Publication enqueue remains single-attempt intentionally. (src/repair/exact-review-batch-queue-client.ts:318, e597eabd49d9)
• Production improvement has an explicit coverage limit: The discussion records substantial recovery after the initial mitigations. The later cache-isolation measurement reported zero 5xx responses and zero overload exceptions, but at roughly one quarter of baseline traffic; its author explicitly withheld a peak-load recovery claim. The closing comment on Exact-review queue Durable Object is overloaded: etag cache and state reads share the queue object, causing bursty 500/503s #1529 (comment) consolidates remaining records and state-writer pressure into this issue.
• Storage transfer remains an operational proposal: The runbook explicitly marks namespace transfer as proposed and unapproved, requiring namespace identity, drained leases, staging verification, and reverse-transfer rollback. Moving the existing namespace between Worker scripts does not itself separate record reads from lease coordination. (docs/queue-service-split-runbook.md:3, 258a5b1fa254)
• Recent maintenance and routing history: History identifies Peter Steinberger’s cache isolation, lifecycle retries, and recent passive-telemetry optimization, alongside Vincent Koc’s batch-capacity and status-read repairs. These establish relevant routing candidates, not causal attribution for overload. Some older blob comparisons were unavailable; current source and commit metadata were inspected successfully. (dashboard/exact-review-queue.ts, 01cd636eda9f)

Likely related people:

• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• vincentkoc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[steipete]

Update 2026-09-03 22:20 UTC: the overload is now a feedback loop, not just records/export.

Worker tail (147 s, deployed 69ec53b): 389 × Durable Object storage operation exceeded timeout which caused object to be reset, 52 × Internal error in Durable Object storage caused object to be reset, 104 fetches exceededWallTime, alarms running 38–58 s. Fetch CPU by route (ms): complete 27,838 · stats 26,967 · lifecycle-bay 11,312 · lifecycle/router-receipt 10,619 · terminal-finalization/attempt 5,418 · heartbeat 2,091 · records/export 1,562. That is ~90 s CPU in 147 s at only 5 requests/s, roughly one second of CPU per stats or complete call. At 02:42 UTC, with ~80 pending, the same routes cost 15–100 ms. Per-request cost grows with the backlog (whole-state reads/parses, N+1 head queries, full direct-publication row parses in stats), slow requests fail batch heartbeats/completes, retries grow the backlog, and it spirals: over the last three hours 12 of 13 claimed batches failed and 63% of dispatches were no-claim exits; publication pending is back to ~200.

The manual snapshot trigger (https://github.com/openclaw/clawsweeper/actions/runs/33812168504) also failed with exact_review_queue_unavailable, so refreshing the R2 base is blocked until the object is responsive again.

A perf PR targeting the per-request CPU (stats memoization, lean state reads, hoisted head queries, incremental Bay materialization) is in progress, alongside a PR for focused single-repo batch hydration plus a 6-hour snapshot schedule, and a PR letting batch heartbeats tolerate outages within the lease.