Skip to content

Commit bfc70ea

Browse files
committed
feat(infinity-slack-bot): resume streaming threads across restarts
Restarting the bot used to orphan in-flight responses: the daemon kept generating, nothing reconnected until the user's next message, and the output produced meanwhile never reached Slack (and dying mid-stream left Slack messages dangling in streaming state). ## Clean shutdown (SIGTERM / ctrl-c) * The inbound task closes the Socket Mode WebSocket with a proper close handshake (Slack sees the app offline immediately; envelopes already read were ACKed and forwarded, unread ones stay unACKed and are redelivered after the restart — a racing user message is delayed, never lost). * The outbound task drains: flushes buffered text, completes open task cards, finalizes every active stream with the disclaimer footer, and clears thread statuses. Queued-but-unprocessed actions are deliberately dropped — they were never counted as delivered, so the restart replays them. The whole drain is bounded by a 10s timeout. * The resume state is persisted to `~/.infinity/slack_resume.json` (atomic temp-file rename): per-stream delivery cursors, channel mappings, subagent thread routes, user-stop latches, and posted choice buttons. A periodic 30s snapshot bounds the damage of a crash (documented: a crash may re-post or drop a few seconds of output). ## Delivery cursors * `runtime.stream_progress` tracks, per (root thread, subagent), the cumulative text bytes and tool call/result cards *delivered to Slack*, counted by the I/O layer at delivery time (text when accepted into the flush buffer — flushed before shutdown completes — cards when posted). Suppressed (user-stopped) output is not counted and stays suppressed across restarts via the persisted latch. * `StreamStop` gains an `end_of_response` flag (set only by the ResponseDone-driven stop): a response-final stop drops the stream's cursor — cursor presence is exactly "this stream needs resuming". ## Startup resume * Each persisted stream gets its own daemon connection (`ConnectSession.target_thread`; replay — and an already-running subagent driver's live broadcast — are per thread, and a new root subscriber is not seeded into pre-existing child drivers). * The daemon's `Replay` is trimmed by the pure `resume::replay_delta`: skip the cursor-covered prefix (slicing inside replayed whole-message text chunks on char boundaries; clamped if history shrank via rewind or compaction), emit the remainder through the existing rendering arms, and append the synthetic `ResponseDone`s replay history never carries — counted to restore the exact end-state the live sequence would have produced (tool-executing: none; completion in flight: one to consume the round-continues flag; idle: one to close, plus one if the delta re-armed the flag). Replays for non-resumed streams are dropped (their content is already on screen). * Still-pending choices are re-posted only if their buttons weren't already on screen (restored `choice_messages`); pre-restart buttons keep working. Subagent output resumes into its existing dedicated thread (anchor map seeded from the persisted routes), and replies inside subagent threads keep routing to their subagent. ## Tests & docs * 10 unit tests for the resume store and `replay_delta` (trimming, mid-chunk slicing, multibyte boundaries, independent call/result cursors, synthetic-done counting, clamping, snapshot/restore round-trip). * 2 new sim tests: the ResponseDone stop is marked end-of-response, and the two-dones-close-a-replayed-tool-round contract `replay_delta` relies on. * "Restarts and Continuity" section in the Slack bot docs.
1 parent 70474b2 commit bfc70ea

11 files changed

Lines changed: 1732 additions & 97 deletions

File tree

‎crates/infinity-slack-bot/src/daemon_io.rs‎

Lines changed: 256 additions & 71 deletions
Large diffs are not rendered by default.

‎crates/infinity-slack-bot/src/main.rs‎

Lines changed: 168 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
//! synchronous output callbacks → mpsc → `slack_io`/`daemon_io` tasks.
1414
1515
use infinity_slack_dataflow::config::Config;
16+
use infinity_slack_dataflow::resume;
1617
use infinity_slack_dataflow::runtime;
1718
use infinity_slack_dataflow::session_store::SessionStore;
1819
use tokio_stream::wrappers::ReceiverStream;
@@ -98,20 +99,90 @@ async fn run() -> Result<(), BoxError> {
9899
let sessions = std::sync::Arc::new(std::sync::Mutex::new(store));
99100
runtime::init(config, sessions);
100101

102+
// Restore cross-restart state (channel mappings, subagent thread routes,
103+
// user-stop latches, posted choice buttons, and the delivery cursors of
104+
// streams that had an open response at shutdown) before the I/O tasks
105+
// spawn — the Slack outbound task seeds its subagent-anchor map from it.
106+
let resume_path = resume::default_path();
107+
let resume_streams = match resume::load(&resume_path) {
108+
Ok(state) => {
109+
let streams = state.streams.clone();
110+
if !state.is_empty() {
111+
tracing::info!(
112+
"restored resume state ({} stream(s) to catch up)",
113+
streams.len()
114+
);
115+
}
116+
resume::restore(runtime::get(), state);
117+
streams
118+
}
119+
Err(e) => {
120+
tracing::error!(
121+
"failed to load resume state from {}: {e}; starting without it \
122+
(in-flight responses from before the restart will not be caught up)",
123+
resume_path.display()
124+
);
125+
Vec::new()
126+
}
127+
};
128+
129+
// Shutdown coordination: flipping the watch to `true` makes the inbound
130+
// task close the Socket Mode WebSocket gracefully and the outbound task
131+
// drain (flush + finalize all streams) and exit.
132+
let (shutdown_tx, shutdown_rx) = tokio::sync::watch::channel(false);
133+
101134
// I/O tasks. Slack credentials are validated before the tasks spawn, so
102135
// a bad token fails startup here instead of killing a detached task.
103136
let slack_io::SlackIo {
104137
events: slack_events_rx,
105138
actions: slack_actions_tx,
106-
outbound: slack_outbound,
107-
inbound: slack_inbound,
108-
} = slack_io::spawn(config).await?;
139+
outbound: mut slack_outbound,
140+
inbound: mut slack_inbound,
141+
} = slack_io::spawn(config, shutdown_rx).await?;
109142
let daemon_io::DaemonIo {
110143
events: daemon_events_rx,
111144
commands: daemon_commands_tx,
112-
dispatch: daemon_dispatch,
145+
dispatch: mut daemon_dispatch,
113146
} = daemon_io::spawn();
114147

148+
// Reconnect every stream that had an open response at shutdown. Each
149+
// stream gets its own connection (the daemon replays — and an existing
150+
// subagent driver broadcasts — per thread); the replay is trimmed to the
151+
// undelivered remainder via the restored cursors (see `daemon_io`).
152+
for stream in resume_streams {
153+
let target_thread = match &stream.subagent {
154+
None => None,
155+
Some(sub) => match sub.parse::<infinity_protocol::ThreadRef>() {
156+
Ok(t) => Some(t),
157+
Err(e) => {
158+
tracing::error!(
159+
"skipping resume of subagent stream in {}: bad thread id {sub:?}: {e}",
160+
stream.thread_ts
161+
);
162+
continue;
163+
}
164+
},
165+
};
166+
if daemon_commands_tx
167+
.send(
168+
infinity_slack_dataflow::daemon::DaemonCommand::ConnectSession {
169+
thread_ts: stream.thread_ts,
170+
session_id: stream.session_id,
171+
target_thread,
172+
},
173+
)
174+
.is_err()
175+
{
176+
tracing::error!("daemon I/O task is gone, cannot resume streams");
177+
break;
178+
}
179+
}
180+
181+
// Periodically snapshot the resume state so a crash (no clean shutdown)
182+
// loses at most the last interval's worth of delivery accounting (which
183+
// then re-posts or drops a few chunks on resume).
184+
let snapshot_task = tokio::spawn(periodic_snapshots(resume_path.clone()));
185+
115186
// Output callbacks fire synchronously while the dataflow ticks; they hand
116187
// work off to the async I/O tasks through unbounded channels (see
117188
// `slack_io::spawn` / `daemon_io::spawn` for the backpressure rationale).
@@ -136,23 +207,107 @@ async fn run() -> Result<(), BoxError> {
136207

137208
tracing::info!("infinity-slack-bot running (Hydro embedded dataflow)");
138209

139-
// DFIR graphs are !Send; drive the dataflow on a LocalSet until ctrl-c.
210+
// DFIR graphs are !Send; drive the dataflow on a LocalSet until a
211+
// shutdown signal arrives.
140212
let local = tokio::task::LocalSet::new();
141-
local
213+
let result = local
142214
.run_until(async {
143215
tokio::select! {
144216
_ = flow.run() => unreachable!("bug: Dfir::run returns Never"),
145-
result = slack_outbound => Err(io_task_exited("Slack outbound", result)),
146-
result = slack_inbound => Err(io_task_exited("Slack inbound", result)),
147-
result = daemon_dispatch => Err(io_task_exited("daemon dispatch", result)),
148-
result = tokio::signal::ctrl_c() => {
149-
result.expect("failed to listen for ctrl-c");
150-
tracing::info!("shutting down");
217+
result = &mut slack_outbound => Err(io_task_exited("Slack outbound", result)),
218+
result = &mut slack_inbound => Err(io_task_exited("Slack inbound", result)),
219+
result = &mut daemon_dispatch => Err(io_task_exited("daemon dispatch", result)),
220+
result = shutdown_signal() => {
221+
result?;
151222
Ok(())
152223
}
153224
}
154225
})
155-
.await
226+
.await;
227+
// The dataflow (and the action/event channels it held) is dropped here.
228+
snapshot_task.abort();
229+
230+
if result.is_ok() {
231+
// Clean shutdown: signal the I/O tasks and let them drain (the
232+
// outbound task flushes buffers and finalizes every active stream;
233+
// the inbound task closes the WebSocket), bounded so shutdown can
234+
// never hang, then persist the resume state.
235+
tracing::info!("shutting down");
236+
if shutdown_tx.send(true).is_err() {
237+
tracing::warn!("Slack I/O tasks already gone at shutdown");
238+
}
239+
let drain = async {
240+
if let Err(e) = slack_outbound.await {
241+
tracing::warn!("Slack outbound task failed during drain: {e}");
242+
}
243+
if let Err(e) = slack_inbound.await {
244+
tracing::warn!("Slack inbound task failed during drain: {e}");
245+
}
246+
};
247+
if tokio::time::timeout(std::time::Duration::from_secs(10), drain)
248+
.await
249+
.is_err()
250+
{
251+
tracing::warn!("Slack I/O drain timed out after 10s; persisting state as-is");
252+
}
253+
let state = resume::snapshot(runtime::get());
254+
match resume::save(&resume_path, &state) {
255+
Ok(()) => tracing::info!(
256+
"persisted resume state ({} active stream(s)) to {}",
257+
state.streams.len(),
258+
resume_path.display()
259+
),
260+
Err(e) => tracing::error!(
261+
"failed to persist resume state to {}: {e}",
262+
resume_path.display()
263+
),
264+
}
265+
}
266+
result
267+
}
268+
269+
/// Resolves when the process receives SIGINT (ctrl-c) or SIGTERM.
270+
async fn shutdown_signal() -> Result<(), BoxError> {
271+
let mut sigterm = tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
272+
.map_err(|e| format!("failed to install SIGTERM handler: {e}"))?;
273+
tokio::select! {
274+
result = tokio::signal::ctrl_c() => {
275+
result.map_err(|e| format!("failed to listen for ctrl-c: {e}"))?;
276+
tracing::info!("received ctrl-c");
277+
}
278+
_ = sigterm.recv() => {
279+
tracing::info!("received SIGTERM");
280+
}
281+
}
282+
Ok(())
283+
}
284+
285+
/// Snapshot the resume state to disk every 30 seconds (when it changed), so
286+
/// a crash loses at most one interval of delivery accounting. The clean
287+
/// shutdown path writes the authoritative final snapshot.
288+
async fn periodic_snapshots(path: std::path::PathBuf) {
289+
const INTERVAL: std::time::Duration = std::time::Duration::from_secs(30);
290+
let mut last_written: Option<String> = None;
291+
let mut tick = tokio::time::interval(INTERVAL);
292+
tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
293+
loop {
294+
tick.tick().await;
295+
let state = resume::snapshot(runtime::get());
296+
let json = match serde_json::to_string(&state) {
297+
Ok(j) => j,
298+
Err(e) => {
299+
tracing::error!("failed to serialize resume snapshot: {e}");
300+
continue;
301+
}
302+
};
303+
if last_written.as_deref() == Some(&json) {
304+
continue;
305+
}
306+
match resume::save(&path, &state) {
307+
Ok(()) => last_written = Some(json),
308+
Err(e) => tracing::warn!("periodic resume snapshot failed: {e}"),
309+
}
310+
}
156311
}
157312

158313
/// An I/O task finished while the bot was running. Fatal regardless of how it

0 commit comments

Comments
 (0)