Skip to content

fix(cloud-agent-next): reconcile completed kilo turns on wrapper death - #5087

Merged
eshurakov merged 1 commit into
mainfrom
medieval-clarinet
Aug 6, 2026
Merged

fix(cloud-agent-next): reconcile completed kilo turns on wrapper death#5087
eshurakov merged 1 commit into
mainfrom
medieval-clarinet

Conversation

@eshurakov

Copy link
Copy Markdown
Contributor

Problem

When the wrapper OOM-dies after emitting wrapper_finalizing but before sending complete, the DO liveness watchdog (handleUnhealthyWrapper via wrapper_ping_timeout) and the disconnect-grace expiry terminalize all accepted messages as wrapper_failure — even though the turn actually completed. Incident: 8 sessions posted their work but were reported as failed.

Why the DO can settle this itself

Ingest is a single FIFO channel, and the wrapper only emits wrapper_finalizing after message.updated(time.completed) and session.idle have already been sent. Kilo events with an entity id are upserted last-wins into the DO's events table, so receiving wrapper_finalizing implies the DO already holds the terminal assistant state. (Verified via deadline arithmetic on the incident: 7 of 8 sessions provably reached wrapper_finalizing.)

Change

Both wrapper-death paths (ping/no-output deadline, disconnect-grace expiry) now reconcile each accepted message against the DO's stored kilocode events before failing:

  • assistant info.errorfailed/assistant_error with the existing classifyAssistantFailure classification (extracted into a shared helper also used by settleSealedBatch)
  • assistant info.time.completed present → completed with the assistant message id, completionSource: 'idle_reconciliation'
  • no positive terminal evidence → previous wrapper_failure / wrapper_disconnected behavior, unchanged

Unlike the sealed-batch path, bare assistant-message presence is not sufficient here — no wrapper complete vouches for turn finality, so the same completion marker the wrapper itself uses (time.completed or terminal error) is required. Emits a warn log with reconciled/fallback counts for observability.

Residual gap: if the final message.updated never reached ingest at all, the store has no evidence and the outcome is wrapper_failure as today; closing that needs a live Kilo terminal query or an HTTP terminal assertion (separate work).

Verification

  • 4 new unit tests: ping-timeout reconcile to completed (incl. callback fired), no-completion-marker fallback, assistant-error reconcile (not masked as wrapper failure), disconnect-grace reconcile
  • Full cloud-agent-next unit suite: 2451 passed; death-path integration tests in Workers runtime: 30 passed
  • tsgo --noEmit, oxlint, oxfmt clean
  • services/cloud-agent-next/AGENTS.md invariant bullet updated

@kilo-code-bot

kilo-code-bot Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (3 files)
  • services/cloud-agent-next/AGENTS.md
  • services/cloud-agent-next/src/session/wrapper-supervisor.ts
  • services/cloud-agent-next/src/session/wrapper-supervisor.test.ts
Previous Review Summary (commit 114b3f5)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 114b3f5)

Status: No Issues Found | Recommendation: Merge

Files Reviewed (3 files)
  • services/cloud-agent-next/AGENTS.md
  • services/cloud-agent-next/src/session/wrapper-supervisor.ts
  • services/cloud-agent-next/src/session/wrapper-supervisor.test.ts

Reviewed by kimi-k3 · Input: 67K · Output: 6.4K · Cached: 528.4K

Review guidance: REVIEW.md from base branch main

When the wrapper dies after wrapper_finalizing but before its terminal
complete, the liveness watchdog (ping/no-output deadline) and disconnect
grace expiry blanket-failed all accepted messages as wrapper_failure —
even when the DO's own event store already held the completed assistant
message (ingest is FIFO, so receiving wrapper_finalizing implies the
terminal kilocode events were persisted).

Both death paths now reconcile each accepted message against the stored
kilocode events first: a terminal assistant error settles as
assistant_error (reusing the sealed-batch classification), an assistant
reply with time.completed settles as completed via idle_reconciliation,
and anything without positive terminal evidence falls back to the
previous wrapper_failure behavior unchanged.
@eshurakov

Copy link
Copy Markdown
Contributor Author

Ran manual generic cloud agent verification

@eshurakov
eshurakov merged commit 2385fd1 into main Aug 6, 2026
17 checks passed
@eshurakov
eshurakov deleted the medieval-clarinet branch August 6, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants