Skip to content

fix(coding-agent): reconnect attached windows when the daemon restarts - #2458

Open
sethkarten wants to merge 5 commits into
mainfrom
rsi/tui-daemon-reconnect
Open

sethkarten wants to merge 5 commits into
mainfrom
rsi/tui-daemon-reconnect

Conversation

@sethkarten

@sethkarten sethkarten commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Motivation

Running /update in one window restarts the daemon, but every OTHER attached TUI window died permanently: it showed

The Prime Agent daemon shut down while this window was attached. The session transcript remains saved; restart Prime Agent and reopen it from Agents View.

and every further send failed with Cannot send daemon command "prompt" because the Prime Agent daemon is not connected. The window never reconnected.

Two causes, both verified against the code and a scripted unix-socket probe (daemon close + new daemon on the same socket path):

  1. DaemonSupervisor hardcoded daemon_closing reason "shutdown" for the shutdown command, even when an update-restart coordinator had just prepared the daemon. daemon-mode.ts derives the reason from the update phase; the supervisor did not. Attached windows therefore saw a plain shutdown instead of an update restart and skipped the existing update-recovery path.
  2. A shutdown-reason transport close was treated as a permanent session loss in DaemonAgentConnection. Any daemon restart that is not an update (manual stop + start, stale-daemon replacement) killed every attached window even though the daemon came back on the same socket path and the session transcript stayed on disk.

Implementation

  • daemon-supervisor.ts: the shutdown command now closes clients with reason "update" when the update restart is prepared (captured at dispatch so a later phase change cannot rewrite it); a plain stop still reports "shutdown".
  • daemon-agent-connection.ts: a shutdown-reason close now runs a bounded recovery loop (reconnectTimeoutMs, default 60s, 100ms poll) instead of an immediate terminal close. The loop reconnects to the same socket path, lists sessions, matches the SAME session by session file or session id (the active id changes across a restart), re-attaches, and emits session_resynced so the view re-renders. Session discovery waits no longer than the bound, so the terminal close lands within it. It never relaunches the daemon (an explicit stop stays stopped); on timeout it falls back to the existing saved-transcript hard-fail message. An update-restart recovery still outranks it (including mid-iteration, so the two recoveries cannot emit duplicate resyncs), and a generic reconnect in flight yields to it.
  • The shared session-restore loop is parameterized (restoreConnectionAfterDaemonRestart) and reused by both the update and shutdown flows.
  • agents-view-mode.ts: an update-reason close now reconnects without recoverDaemon, so the Agents View never spawns a competing daemon while the update coordinator owns the restart (a plain outage keeps the relaunch behavior).
  • connection_status now carries an optional daemonVersion (the restarted daemon's daemon_hello.appVersion) when recovery re-attached.
  • The TUI banner is version-honest: Daemon restarted (vX) - reconnected; when the restarted daemon is NEWER than this window's binary it warns Daemon restarted (vX), this window still runs vY - restart the window to pick up the update. (an older or unorderable daemon version is reported without the advice). No auto-relaunch: the repo has no safe relaunch mechanism for a passive reconnect path (/update owns its own relaunch).

Verification

  • New regression tests (CI-exact, 1:1 test-line budget enforced by the repo checker):
    • agent-connection-daemon.test.ts: a shutdown close with the daemon coming back recovers (reconnect, re-attach the same session under its new active id, resync, connected banner carries the daemon version, the shutdown-specific reconnecting reason is reported, no terminal close); a shutdown recovery that times out (fake timers) emits the existing saved-transcript close.
    • daemon-supervisor-admission.test.ts: the shutdown command closes with reason "update" while an update restart is prepared (with and without force), and "shutdown" otherwise.
    • agents-view-state.test.ts: an update-reason close polls without calling recoverDaemon; a plain outage still relaunches.
    • interactive-update-relaunch.test.ts: formatDaemonReconnectBanner maps missing/matching/newer/older daemon versions to message + tone.
  • End-to-end probe (pre-push): a scripted unix-socket daemon broadcast daemon_closing "shutdown", exited, and a new daemon bound the same socket path; the connection recovered with reconnecting -> session_resynced -> connected (daemon version) and exactly one new socket on the restarted daemon. On main, the same scenario emitted the terminal saved-transcript close (the reported bug).
  • Dual reviewer pass (correctness + cross-file contract, both requiring empirical probes): all blockers/majors/minors addressed (bound enforcement, takeover de-duplication, agents-view relaunch race, version-order honesty, synchronous reason capture); noted-and-kept items: the input-pause fence keeps its terminal close (pre-existing deliberate safety), parked-request re-sends keep the update path's inherited behavior.
  • Existing suites: agent-connection-daemon (87), daemon-agent-connection-reconnect-park, daemon-supervisor-admission/monitor, daemon-client, daemon-routed-client, daemon-peer-transport, agents-view-*, interactive-update-relaunch, rpc-*, acp-*, package-self-update-daemon all pass; npm run check (biome, test-policy, tsgo, installer, push-guard, browser-smoke) is green.
  • daemon-supervisor-process.test.ts fails 12 tests locally, verified identical on pristine origin/main (environment artifact: the suite inherits daemon-worker env vars that flip spawned supervisors into peer mode; it is excluded from test:ci).

Note

Medium Risk
Changes core daemon disconnect/reconnect and session-restore paths with subtle precedence rules between update, shutdown, and generic reconnect; regressions could leave windows stuck reconnecting or close sessions that should recover.

Overview
Attached TUI windows no longer die permanently when the Prime Agent daemon restarts. Shutdown recovery polls the same socket path (default 60s), re-attaches the prior session by session file or ID, emits session_resynced, and only falls back to the saved-transcript close if the daemon never returns. Recovery runs only when the daemon announced daemon_closing; a bare session stop without that notice still closes the window.

Update-restart fixes: DaemonSupervisor now emits an update close reason when stopping a prepared update-restart daemon (so windows other than the one running /update use update recovery). Update and shutdown recoveries share restoreConnectionAfterDaemonRestart, with update taking precedence so resyncs and reconnect races are not duplicated. Generic reconnect and Agents View defer to shutdown/update recovery; Agents View skips recoverDaemon on update closes so it does not spawn a competing daemon.

UI: connection_status connected events can include daemonVersion; the TUI shows a one-line reconnect banner and warns to restart the window when the recovered daemon is newer than the client binary.

Reviewed by Cursor Bugbot for commit f6dcbab. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add bounded shutdown recovery to DaemonAgentConnection for daemon restarts

  • Adds reconnectAfterShutdown and shared restoreConnectionAfterDaemonRestart to DaemonAgentConnection, which polls for the daemon without relaunching, matches the prior session by file or ID, reattaches, and emits session_resynced
  • DaemonSupervisor shutdown handler now passes update as the close reason during a prepared update-restart and shutdown otherwise; handleDaemonMessage triggers shutdown recovery only when a daemon_closing notice precedes the session close
  • Update recovery retains precedence over in-flight shutdown recovery, and generic reconnect yields when shutdown recovery has restored the session
  • connection_status events now carry the restarted daemon version; InteractiveMode shows a reconnect banner that warns the user to restart the window when the daemon is newer than the client binary
  • Risk: unannounced session closes (no daemon_closing notice) remain terminal; if restoreConnectionAfterDaemonRestart fails to find the prior session within the deadline, it emits the existing shutdown close instead of retrying indefinitely

Macroscope summarized f6dcbab.

@sethkarten
sethkarten enabled auto-merge (squash) September 18, 2026 18:09
@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown

Prime Agent performance — completed

PR f6dcbab3 compared with main e311d649.

Overall: 0 regressed · 0 improved · 41 no clear change.

Metric Main This PR Change
Cold startup 772.9 ms 764.6 ms ≈ -8.3 ms (-1.07%)
Warm startup 503.5 ms 486.7 ms ≈ -16.9 ms (-3.35%)
Installation 5.78 s 5.75 s ≈ -0.04 s (-0.63%)
Compressed release artifacts 72.86 MB 72.89 MB ≈ +0.04 MB (+0.05%)
Installed footprint 594.23 MB 593.95 MB ≈ -0.27 MB (-0.05%)
Idle memory, summed RSS 817.72 MB 816.66 MB ≈ -1.06 MB (-0.13%)

Python runtime

Metric Main This PR Change
Python kernel startup 32.9 ms 33.4 ms ≈ +0.5 ms (+1.59%)
Python cell round trip 0.082 ms 0.080 ms ≈ -0.002 ms (-2.55%)
Empty bash command 1.9 ms 1.9 ms ≈ -0.012 ms (-0.60%)
Bash git status 2.6 ms 2.6 ms ≈ +0.028 ms (+1.07%)
Bash 32 KiB output 2.0 ms 1.9 ms ≈ -0.1 ms (-6.37%)
35 cells / 9 shell calls 26.3 ms 25.5 ms ≈ -0.8 ms (-2.94%)
Python interrupt to done 0.502 ms 0.509 ms ≈ +0.007 ms (+1.35%)
Python state snapshot 10.0 ms 9.9 ms ≈ -0.055 ms (-0.55%)
Python state restore 134.9 ms 133.5 ms ≈ -1.5 ms (-1.08%)
Python idle RSS 21.09 MB 21.16 MB ≈ +0.08 MB (+0.36%)
Python RSS after pandas workload 75.53 MB 75.60 MB ≈ +0.07 MB (+0.09%)

Session transport

Metric Main This PR Change
Private frame decode, 32 MiB in 8 KiB chunks 14.1 ms 14.4 ms ≈ +0.3 ms (+1.85%)

UI interactions

Metric Main This PR Change
Resume large session (cold) 1,878.9 ms 2,067.7 ms ≈ +188.9 ms (+10.05%)
CPU, resume large session 2,150.0 ms 2,260.0 ms ≈ +110.0 ms (+5.12%)
Switch into large session 1,761.3 ms 1,783.9 ms ≈ +22.6 ms (+1.28%)
CPU, switch into large session 2,090.0 ms 2,130.0 ms ≈ +40.0 ms (+1.91%)
Open agents view from a session 478.9 ms 535.8 ms ≈ +56.9 ms (+11.88%)
CPU, open agents view 800.0 ms 890.0 ms ≈ +90.0 ms (+11.25%)
Full agents roster, many sessions 4.03 s 4.03 s ≈ +0.00041 s (+0.01%)
CPU, full agents roster 0.95 s 0.83 s ≈ -0.12 s (-12.63%)
Open another session from agents view 1,935.2 ms 1,901.0 ms ≈ -34.2 ms (-1.77%)
CPU, open from agents view 1,020.0 ms 1,120.0 ms ≈ +100.0 ms (+9.80%)
Reopen resident large session 291.2 ms 265.1 ms ≈ -26.1 ms (-8.96%)
CPU, reopen resident session 310.0 ms 300.0 ms ≈ -10.0 ms (-3.23%)
Open subagent session at depth 6 17,657.6 ms 17,837.3 ms ≈ +179.7 ms (+1.02%)
CPU, open subagent at depth 6 4,610.0 ms 4,720.0 ms ≈ +110.0 ms (+2.39%)
Open chain parent from agents view 3,080.2 ms 3,034.7 ms ≈ -45.5 ms (-1.48%)
CPU, open chain parent 1,450.0 ms 1,400.0 ms ≈ -50.0 ms (-3.45%)
Scheduled catalog, first request 509.1 ms 429.2 ms ≈ -80.0 ms (-15.71%)
CPU, scheduled catalog 860.0 ms 830.0 ms ≈ -30.0 ms (-3.49%)
Scheduled catalog, repeated request 0.5 ms 0.5 ms ≈ +0.037 ms (+7.36%)
CPU, repeated catalog 0.0 ms 0.0 ms ≈ +0.0 ms (N/A)
Cold worker with three catalog scans 394.5 ms 401.5 ms ≈ +7.1 ms (+1.79%)
CPU, cold worker and scans 390.0 ms 360.0 ms ≈ -30.0 ms (-7.69%)
UI memory after interactions 1,896.58 MB 1,752.22 MB ≈ -144.36 MB (-7.61%)

Sandbox cost: ~$0.0983 — no inference calls.
Run, logs, and downloadable raw results

Methodology and samples

Main resolved at 2026-09-20T05:01:35.876043+00:00. Harness e311d649.
Linux x64, 4 vCPU, 8 GB RAM, 20 GB disk; region us.
Image: node:24-bookworm@sha256:be23f54a88d34e8824c741b19b91064094f92c1c97b194144bfc8b50d67258e2.
Stock tools, skills, daemon, and Python bootstrap enabled; fresh homes and a fixed Git fixture.
Onboarding is dismissed; the editor starts without a selected model or submitted prompt.
Medians shown. Arrows require a 20% timing/memory change plus absolute floors and IQR.
These practical noise floors are not a statistical significance test.
Cold means stopped Prime processes; OS filesystem caches are not flushed.
No model requests or credentials. Installation excludes build/setup time.
Installer tarballs use loopback; npm/Python downloads use the network with fresh caches.
Artifact size counts release tarballs; footprint after first use includes registry packages.
MB is decimal. Summed RSS can double-count shared pages; PSS is recorded when available.
Provisioning, setup, and build durations are recorded separately in the raw results.
Kernel probes use the installed JSONL runtime, outside the TUI/TypeScript host.
Per trial: 50 Python cells, 5 calls per shell case, and one 35-cell mix (9 git status calls).
Cell/shell values are batch means; other runtime timings are single operations.
State fixture: a 10,000-row × 8-column integer DataFrame and a 10,000-integer list.
Restore runs in a fresh kernel, including pandas imports; kernel startup is excluded.
Kernel RSS covers the isolated Python process; loaded RSS follows the pandas workload.
Transport benches run node against the prepared source build, outside the installed home.
Frame decode times one 32 MiB private frame, snapshot-chunk header, pushed in
8 KiB chunks; the wire shape of multi-MB frames on the daemon-worker channels.
UI trials use a fresh fixture set: 194 top-level sessions including one ~40 MB transcript,
40 ledger fan-out children, and a 6-deep subagent chain (~46 spawn edges).
Large fixtures hold 1,999 complete triples (~5 MB JSONL); medium 119; subagents 399 each.
Interactions: cold --resume of a large session, warm /resume switch, left-arrow to agents view,
roster settle with many saved sessions, search-and-open of another large session,
reattaching to that resident session, opening the chain parent, and drilling to depth 6.
Readiness is the rendered transcript tail plus a confirmed editor echo.
CPU metrics sum utime+stime across the whole benchmark-user process tree per interaction.
UI memory sums RSS after the interactions; PTY byte counts are in the raw results.
A separate catalog fixture has 2,300 sessions, 2,298 edges, and 13 paused scheduled-job owners.
Catalog timings cover first/repeated reads and cold worker creation under three pending scans.
All expected jobs and owner metadata are checked; worker readiness excludes TUI rendering.
Costs estimate full sandbox lifetimes at configured rates, including setup and build.
Budget target: $1; not a billing cap. Performance changes are informational.
Failed or incomplete execution fails the workflow; saved artifacts remain available.
Each side stops a phase after 2 identical consecutive failures.
Skipped trials are not attempted samples. Warm startup requires a successful cold launch.

Metric Main successful/attempted PR successful/attempted Main spread PR spread
Cold startup 10/10 10/10 IQR 44.1 ms IQR 100.5 ms
Warm startup 10/10 10/10 IQR 43.6 ms IQR 38.5 ms
Installation 3/3 3/3 range 0.89 s range 0.46 s
Compressed release artifacts 1/1 1/1
Installed footprint 1/1 1/1
Idle memory, summed RSS 10/10 10/10 IQR 3.49 MB IQR 7.96 MB
Python kernel startup 10/10 10/10 IQR 2.9 ms IQR 0.7 ms
Python cell round trip 10/10 10/10 IQR 0.021 ms IQR 0.026 ms
Empty bash command 10/10 10/10 IQR 0.3 ms IQR 0.2 ms
Bash git status 10/10 10/10 IQR 0.3 ms IQR 0.2 ms
Bash 32 KiB output 10/10 10/10 IQR 0.2 ms IQR 0.2 ms
35 cells / 9 shell calls 10/10 10/10 IQR 2.9 ms IQR 3.5 ms
Python interrupt to done 10/10 10/10 IQR 0.070 ms IQR 0.023 ms
Python state snapshot 10/10 10/10 IQR 0.6 ms IQR 0.4 ms
Python state restore 10/10 10/10 IQR 8.8 ms IQR 5.8 ms
Python idle RSS 10/10 10/10 IQR 0.12 MB IQR 0.05 MB
Python RSS after pandas workload 10/10 10/10 IQR 0.43 MB IQR 0.11 MB
Private frame decode, 32 MiB in 8 KiB chunks 10/10 10/10 IQR 2.2 ms IQR 1.4 ms
Resume large session (cold) 3/3 3/3 range 285.2 ms range 710.2 ms
CPU, resume large session 3/3 3/3 range 70.0 ms range 280.0 ms
Switch into large session 3/3 3/3 range 47.8 ms range 147.8 ms
CPU, switch into large session 3/3 3/3 range 80.0 ms range 230.0 ms
Open agents view from a session 3/3 3/3 range 60.3 ms range 21.5 ms
CPU, open agents view 3/3 3/3 range 80.0 ms range 60.0 ms
Full agents roster, many sessions 3/3 3/3 range 0.40 s range 0.40 s
CPU, full agents roster 3/3 3/3 range 0.10 s range 0.08 s
Open another session from agents view 3/3 3/3 range 185.0 ms range 51.5 ms
CPU, open from agents view 3/3 3/3 range 60.0 ms range 230.0 ms
Reopen resident large session 3/3 3/3 range 31.2 ms range 16.8 ms
CPU, reopen resident session 3/3 3/3 range 50.0 ms range 40.0 ms
Open subagent session at depth 6 3/3 3/3 range 1,027.6 ms range 1,923.2 ms
CPU, open subagent at depth 6 3/3 3/3 range 120.0 ms range 280.0 ms
Open chain parent from agents view 3/3 3/3 range 59.0 ms range 16.7 ms
CPU, open chain parent 3/3 3/3 range 260.0 ms range 100.0 ms
Scheduled catalog, first request 3/3 3/3 range 98.2 ms range 40.8 ms
CPU, scheduled catalog 3/3 3/3 range 150.0 ms range 70.0 ms
Scheduled catalog, repeated request 3/3 3/3 range 0.3 ms range 0.2 ms
CPU, repeated catalog 3/3 3/3 range 10.0 ms range 0.0 ms
Cold worker with three catalog scans 3/3 3/3 range 80.3 ms range 32.0 ms
CPU, cold worker and scans 3/3 3/3 range 60.0 ms range 60.0 ms
UI memory after interactions 3/3 3/3 range 173.31 MB range 213.91 MB

Comment thread packages/coding-agent/src/modes/agent-connection/daemon-agent-connection.ts Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread packages/coding-agent/src/modes/agent-connection/daemon-agent-connection.ts Outdated
Every attached TUI window died permanently when the daemon restarted:
the supervisor broadcast daemon_closing "shutdown" even when an update
restart was prepared, and a shutdown-reason close was treated as a
permanent session loss, so windows showed the saved-transcript error and
then failed every further send.

- Supervisor: a shutdown command during a prepared update restart now
  closes clients with reason "update" so every attached window takes
  the existing update-recovery path (parity with daemon-mode.ts). The
  reason is captured at dispatch so a later phase change cannot rewrite
  it, and Agents View skips its daemon relaunch for that reason while
  the update coordinator owns the restart.
- DaemonAgentConnection: a shutdown-reason close now runs a bounded
  recovery loop (reconnectTimeoutMs, default 60s) that polls the same
  socket path, re-attaches the same session (matched by session file or
  session id, active id may differ after a restart), and resyncs the
  transcript; it never relaunches the daemon. Session discovery waits
  no longer than the bound, so the terminal close lands within it; on
  timeout it keeps the current saved-transcript hard-fail message.
- Reconnect banners are version-honest: the recovered window reports
  the daemon version, and when the daemon is newer than this window's
  binary it says so instead of pretending the window is updated.
@sethkarten
sethkarten force-pushed the rsi/tui-daemon-reconnect branch from 86127a9 to 5442ea4 Compare September 18, 2026 18:38
Comment thread packages/coding-agent/src/modes/interactive/interactive-mode.ts

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

…sync

- formatDaemonReconnectBanner now treats a numeric-equal daemon release as newer than the client own prerelease (per semver 1.2.3 > 1.2.3-beta.1), so prerelease windows see the restart-to-update warning.

- A generic shutdown recovery now yields permanently once an update recovery restored the session (its sessionRevision bump), not merely while updateRestartPending is set, so it can no longer emit a duplicate session_resynced/connected.
sethkarten and others added 2 commits September 18, 2026 16:32
…tdown

- Route a shutdown session_closed into the bounded shutdown-recovery reconnect only when the daemon announced itself closing first (daemon_closing): an orderly daemon shutdown now recovers attached windows like a reasoned socket close does, while a bare session stop (worker /stop, no notice) still terminal-closes immediately.

- The notice clears on every attach, so a later bare stop after a recovered shutdown is not misrouted into a reconnect hang.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 09c8308. Configure here.

…d close relay

- An orderly supervisor shutdown archive-stops its workers, so attached windows read the relayed session_closed with reason "killed" after the daemon_closing notice; route that close into the shutdown-recovery reconnect like a direct worker link's "shutdown" close. The notice, not the close reason, stays the discriminator: a bare session stop (worker /stop, no notice) still terminal-closes immediately.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant