ops: coordinator-worker pod restart drops in-flight LLM calls silently - investigate PDB and disruption budget

[sabbour]

Summary

During demo capture (beatall9, 2026-08-19), the staging agentweaver-worker pod restarted at 08:07 UTC, immediately before a new Define Outcome orchestration was created. A SocketException at 08:31 UTC interrupted the in-flight LLM streaming call. The combination caused a 49-minute stall before outcome-spec finally served 200 at 08:57 UTC.

Evidence (App Insights - agentweaver-insights/agentweaver-rg)

Time (UTC)	Event
08:07	Coordinator reconciler: re-armed 10 orphaned coordinator loop(s) - pod restarted
08:08	POST /api/projects/{id}/orchestrations - 201
08:31	System.Net.Http.HttpRequestException + SocketException - LLM stream dropped
08:31-08:57	~26 min retry/backoff cycle
08:57	GET /api/runs/{id}/outcome-spec - 200 OK (451ms) - plan finally served

Also: agentweaver-worker service account logs RBAC 403s for services.list every ~60s.

Questions / Scope

1. PDB: Is there a PodDisruptionBudget for agentweaver-worker? Without one, voluntary disruptions (node drain, rolling upgrade) evict the pod mid-LLM-call with no graceful handoff.
2. LLM streaming retry: SocketException caused a ~26-min backoff. Should there be a shorter circuit-breaker or faster retry with fresh connection?
3. Coordinator orphan recovery: Reconciler re-arms terminal orphans on startup, but runs mid-LLM-call may not be detected as orphaned and stall silently.
4. RBAC 403 - services.list: system:serviceaccount:agentweaver:agentweaver-worker cannot list services in agentweaver namespace. Fires every 60s - intentional degraded feature or missing Role binding?

Suggested mitigations

• Add PDB with minAvailable: 1 for agentweaver-worker
• Add SIGTERM handler to checkpoint/re-queue in-flight LLM dispatches before exit
• Reduce LLM retry backoff ceiling (currently >26 min) - prefer faster failure + re-queue
• Fix services.list RBAC gap