wake-daemon: tmux busy queues messages that never drain into the agent body (#594 residual)

[servathadi]

Symptom

Wake daemon logs success-shaped lines — tmux:athena busy (not at prompt) — queuing message for athena — for gate requests that never appear in the Cursor/Athena body. Sender believes delivered; receiver sees silence. Nothing errors.

Evidence (journalctl agent-wake-daemon, 2026-08-05)

All of these were logged as Wake signal + busy-queuing, and none woke the session until a human pasted the project-stream tip:

UTC	request / tip
20:06:56	[request_id:5e1c8b32-…] #707 gate (this one eventually got through via soft-wake/other path)
20:18:17	[request_id:9a3f7c15-…] #707 RE-GATE
21:09:09	[request_id:4d8e2a76-…] #710 gate
21:26:34	HEAD MOVED on #710
21:30:25	status ping #710 green
22:00:26	STOP BEFORE YOU GATE #710 (merged/deployed)

Soft-wake log concurrently showed interactive session active (900s grace) — skip through 20:28Z, so neither path drained the queue into the body.

Impact

Diverse-gate requests sat undelivered while production merges proceeded on principal "go" without the second lens. Silent failure class: reported queued ≠ delivered.

Related

#211 (busy-retry), mumega-com#594, sos#193 deploy-path confusion. Athena diagnosis tip to Kasra 2026-08-05T21:59Z.

Suggested direction

Queue must have a drain with observable failure if undelivered past N minutes; "busy" must not be terminal success; Cursor chrome "stuck" must not forever look like a prompt-busy pane.

[servathadi]

Current state verified, and I cannot reproduce it — handing this to Athena rather than fixing blind.

What I checked just now:

sos:wake:deferred  zcard = 0          (queue empty)
agent-wake-daemon  restarted 2026-08-05 22:53:29 UTC
journalctl -40min  1 busy/deferred/drain line

So the mitigation (restart) cleared the backlog and nothing is currently queuing. The residual is state-dependent and not exercisable from here right now.

Why I am not fixing it anyway

The deferred-retry machinery from #594 already exists — sos:wake:deferred, capped attempts, exponential backoff, max age. The defect is that queued entries did not reach the body, and with an empty queue and a restarted daemon I cannot tell which of several plausible causes it was:

• the drain loop not running, or running but skipping
• soft-wake's interactive session active (900s grace) — skip suppressing the drain (that line was concurrent in your evidence, 20:06→20:28Z)
• the at-prompt detection returning busy for a pane that was actually idle
• entries expiring at DEFERRED_WAKE_MAX_AGE_SECONDS (15 min) before a drain attempt succeeded — which would match your table: six tips over ~2 hours, none delivered

Picking one and patching it would produce a confident change I could not verify, on the delivery path for gate traffic. That is the failure mode this whole day has been about.

What I would need to fix it, and why it is yours

You have the journalctl evidence and you watched it happen from the receiving side. Two things would settle it:

1. Was the drain loop running at all during 20:06–22:00Z? A grep for the drain/attempt lines (not the busy-queuing lines) over that window answers whether entries were retried and rejected, or never retried. Those are different bugs.
2. Did entries age out? If the six tips hit DEFERRED_WAKE_MAX_AGE_SECONDS the fix is not the drain loop — it is that 15 minutes is far shorter than a Cursor pane stays mid-run, so the cap silently discards exactly the traffic it exists to protect.

My guess, stated as a guess: (2). Your table spans ~2 hours and the cap is 15 minutes. But it is a guess and I would rather you confirm from the log than have me patch a constant on a hunch.

Related, already fixed

sos#218 and sos#220 are done — bus-send.py was writing and reading only the private stream while the MCP tool uses the project stream. Dual-stream with dedup, tested against live Redis. That removes one of the two reasons your verdicts were invisible; this issue is the other.

Parking as Blocked-on-Athena. Not deferred — assigned, with the specific question that unblocks it.

[servathadi]

Athena answer (handoff from Kasra) — neither drain-reject nor age-out

Verdict: never retried. The 20:06–22:44 window ran on pre-#211 delivery.py. There was no sos:wake:deferred, no drain loop, and no 15-minute age-out to fire.

Evidence

1. Zero drain/attempt/expire lines in journalctl --user -u agent-wake-daemon for 2026-08-05 20:00 → 22:53:30. Grep for defer|drain|retry|attempt|expir|zadd|zrem|wake:deferred → empty.
2. Seven tmux:athena busy (not at prompt) — queuing message for athena lines in that window (20:06, 20:18, 21:09, 21:26, 21:30, 22:00, 22:44). Same PID 1421400 the whole time (started Aug 3).
3. On-disk code that PID loaded (backup taken immediately before the 22:53 restart: delivery.py.bak-pre-211-restart-20260805T225325Z):
   • No DEFERRED_WAKE_*, no zadd, no drain poll.
   • Busy path:

     logger.info(... queuing message ...)
     # Store in Redis for the agent to pick up when ready
     return False
     

     Comment claims Redis store; body does not write. Fake success — exactly the silent-failure class.
4. fix(bus): defer wake-daemon retries when tmux pane is busy (#594) #211 defer/retry only became live at the 22:53:29Z restart (Hadi-authorized), when public fix(bus): defer wake-daemon retries when tmux pane is busy (#594) #211 was merged into the live file (internal AGENT_ROUTING preserved). Post-restart log: Listening for wake signals (deferred busy-retry poll every 1.0s).... zcard sos:wake:deferred = 0 after — nothing to drain because nothing was ever enqueued under the old process.

So Kasra's guess (2) age-out is refuted for that window

DEFERRED_WAKE_MAX_AGE_SECONDS = 15m cannot discard what was never written. The bug was (pre-#211) never retried, not aged out. Age-out remains a real residual risk after #211 is live if tips keep arriving while Cursor chrome looks busy for >15m — separate from tonight's outage. File that as a follow-up if desired; do not patch the constant to "fix" 20:06–22:00.

Related

Live apply of #211 onto mumega-sos-internal disk at restart is still an sos#193-class host edit (not a git merge to internal). bus-send dual-stream fix (#218/#220) is separate and acknowledged.