Repository navigation
fix(infinity-daemon): reconcile persisted session statuses at startup - #161
Open
akainth015 wants to merge 1 commit into
Open
akainth015 wants to merge 1 commit into
akainth015 wants to merge 1 commit into
Conversation
Deploying infinity with
|
| Latest commit: |
a97b661
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://bcca4a0a.infinity-dc7.pages.dev |
| Branch Preview URL: | https://pr-session-status-reconcile.infinity-dc7.pages.dev |
akainth015
force-pushed
the
pr/session-status-reconcile
branch
from
October 20, 2026 00:05
cafeeee to
d46d069
Compare
akainth015
added a commit
that referenced
this pull request
Oct 20, 2026
…ing can't wake zombie sessions After a restart, #161 reports stale sessions Idle, but viewing one in the web UI flipped it permanently to Running: attaching a client wakes the thread's driver (`Live` clears the idle flag), and the driver parks forever because `awaiting_tool_result()` is true for its trailing unanswered call. Two causes: * **Orphaned calls**: the RAP server, input queue or timer that would deliver the result died with the old process. * **`sleep_until_event_or_input` held its driver**: its no-op call is never answered by design, but only `close_thread` was exempt from `awaiting_tool_result`, so any woken sleeping thread showed Running, restart or not. ## Fix * **core**: exported `tool_call_expects_result(name)` exempts `close_thread` and `sleep_until_event_or_input`; sleeping drivers now exit and are respawned by the router on the next event or input. * **daemon**: `PersistentConversationStore::settle_orphaned_tool_calls` runs at cold start, before the agent system starts, and appends the synthetic result an interrupting message would (with restart-specific text) to every trailing call that expects a result. A tail probe (`RawValue`, so only the last message is deserialized) keeps threads that don't end in such a call unloaded. ## Tests * `driver_idles_after_sleep_until_event_or_input_call` (core residency). * `settle_orphaned_tool_calls_answers_only_orphaned_calls` (per-tool selection, persistence, idempotence). * `restart_reconciles_stale_sessions` now also checks the sweep runs at startup. Co-authored-by: Infinity 🤖 <infinity@hydro.run>
After a crash or `kill -9`, sessions persisted mid-flight in `sessions.json` (`idle: false, shut_down: false`) loaded as `Running` forever: the web UI showed green "running" dots for sessions with no live driver (4 of 56 sessions on a real store). * `SessionStore::reconcile_cold_start()` marks those entries idle right after load, before any client can connect, and the result is persisted. Stopped and archived entries are untouched; persisted pending choices still surface as `WaitingForChoice`, which ranks above `idle`. * Test: `restart_reconciles_stale_sessions` (stale running -> Idle and written back to disk, stopped stays Stopped).
akainth015
added a commit
that referenced
this pull request
Oct 20, 2026
…ing can't wake zombie sessions After a restart, #161 reports stale sessions Idle, but viewing one in the web UI flipped it permanently to Running: attaching a client wakes the thread's driver (`Live` clears the idle flag), and the driver parks forever because `awaiting_tool_result()` is true for its trailing unanswered call. Two causes: * **Orphaned calls**: the RAP server, input queue or timer that would deliver the result died with the old process. * **`sleep_until_event_or_input` held its driver**: its no-op call is never answered by design, but only `close_thread` was exempt from `awaiting_tool_result`, so any woken sleeping thread showed Running, restart or not. ## Fix * **core**: exported `tool_call_expects_result(name)` exempts `close_thread` and `sleep_until_event_or_input`; sleeping drivers now exit and are respawned by the router on the next event or input. * **daemon**: `PersistentConversationStore::settle_orphaned_tool_calls` runs at cold start, before the agent system starts, and appends the synthetic result an interrupting message would (with restart-specific text) to every trailing call that expects a result. A tail probe (`RawValue`, so only the last message is deserialized) keeps threads that don't end in such a call unloaded. ## Tests * `driver_idles_after_sleep_until_event_or_input_call` (core residency). * `settle_orphaned_tool_calls_answers_only_orphaned_calls` (per-tool selection, persistence, idempotence). * `restart_reconciles_stale_sessions` now also checks the sweep runs at startup. Co-authored-by: Infinity 🤖 <infinity@hydro.run>
akainth015
force-pushed
the
pr/session-status-reconcile
branch
from
October 20, 2026 17:42
d46d069 to
a97b661
Compare
Member
|
I think this will be subsumed by better interrupt handling, so that we get rid of the "stopped" state entirely and an agent is always idle or running? Edit: although, we do want some way of saying "interrupt and do not be re-woken by anything other than user input, ignore subscription events", so we do need to think about that... |
akainth015
added a commit
that referenced
this pull request
Oct 22, 2026
…ing can't wake zombie sessions After a restart, viewing a stale session in the web UI flipped it permanently to Running (even once #161 reports it Idle): attaching a client wakes the thread's driver (`Live` clears the idle flag), and the driver parks forever because `awaiting_tool_result()` is true for its trailing unanswered call. Two causes: * **Orphaned calls**: the RAP server, input queue or timer that would deliver the result died with the old process. * **`sleep_until_event_or_input` held its driver**: its no-op call is never answered by design, but only `close_thread` was exempt from `awaiting_tool_result`, so any woken sleeping thread showed Running, restart or not. ## Fix Builds on #170 (`InfinityMessage::Interrupt`). * **core**: exported `tool_call_expects_result(name)` exempts `close_thread` and `sleep_until_event_or_input`; sleeping drivers now exit and are respawned by the router on the next event or input. * **daemon**: `PersistentConversationStore::settle_orphaned_tool_calls` runs at cold start, before the agent system starts, and settles every trailing call that expects a result with an `InfinityMessage::Interrupt` entry, as an interrupting message would (the model sees the usual "interrupted" result; the entry carries no cause, so the restart is recorded only in its `-orphaned-by-restart` message ID). History then ends on the interrupt: the thread is stopped, not waiting. A tail probe (`RawValue`, so only the last message is deserialized) keeps threads that don't end in such a call unloaded. ## Tests * `driver_idles_after_sleep_until_event_or_input_call` (core residency). * `settle_orphaned_tool_calls_answers_only_orphaned_calls` (per-tool selection, persistence, idempotence). * `startup_settles_orphaned_tool_calls` (the sweep runs at startup). Co-authored-by: Infinity 🤖 <infinity@hydro.run>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A session killed mid-completion persists
idle=false, shut_down=false, which loads as Running forever after a daemon restart: stale green dots in the web UI (4 of 56 sessions on a real store).SessionStore::reconcile_cold_start()runs right after load, before any client can connect. It marks those entries idle and the result is persisted. Stopped and archived entries are left alone. Sessions with persisted pending choices still showWaitingForChoice, which ranks above idle.Test:
restart_reconciles_stale_sessions(stale running → Idle, also on disk; stopped stays Stopped).