Summary
The coordinator reconciler enters a tight infinite loop after every pod restart, continuously re-arming 10 specific orphaned run IDs every ~10 seconds (6 cycles/minute). All 10 runs are already terminal/stopped but the reconciler never stops trying them. This starves new coordinator-draft work from completing.
Evidence (App Insights - agentweaver-insights/agentweaver-rg, 2026-08-19)
Reconciler thrash rate
10:16:10Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s) 10:16:20Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s) 10:16:30Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s) 10:16:40Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s) 10:16:50Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s) 10:17:00Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s)
6 full cycles/minute, each cycling the same 10 run IDs. All immediately resolve as "already terminal/stopped; no subtasks will be dispatched" — yet the next cycle re-arms them again.
Orphaned run IDs (same set every cycle)
- 18eb9563-bdef-4890-9321-5b9e39d47ed8
- 69317bce-7b16-4ea1-9fdb-7080fb42cd02
- 944c3a24-1ad1-40ec-8f0e-b503d11a54f0
- 33ab1922-a4dd-468d-ac99-5fca1ce18ba5
- 96654eb5-ff8e-4d71-adbf-5bd5611dc73b
- bf56f06a-ff76-4fa2-9795-1058864b7cd3
- 7dfa8744-7d41-4305-ac49-5fb4e934fc3f
- 2e264551-a24e-46f3-a3c7-b07c28f900a8
- a85038f7-6e46-4dbd-a626-13ebbfc17dbd
- 76f5f19f-1914-4e34-9ec2-4f0f81d49da2
Impact on new coordinator work
New orchestration aaa6b185-27e2-4ac7-88d9-4e6d46ae7178 submitted at 10:15:22Z. coordinator-draft SetupAsync entered at 10:16:24Z. Zero further traces in 1.5 hours. /work-plan returned 499 at 10:16:53Z (29s after setup). Plan never completed.
Root cause
The reconciler's orphan-scan query does not exclude runs already confirmed terminal. When a new pod starts, all orphaned runs re-enter the scan queue. The dispatch step confirms terminal status but does NOT mark the run as reconciled/excluded. The scan runs again 10s later and re-discovers the same runs. No backoff, no deduplication, no exit condition.
Impact
- CPU/thread contention on the worker pod
- New coordinator-draft work enters SetupAsync but silently fails to produce output (LLM stream likely starved)
- Pod restarts every ~18 minutes (possibly OOMKilled or liveness probe failing due to load)
- Cumulative: 3 pod restarts observed since 08:07Z, each adding new orphan re-arm cycles
Fix suggestions
- Reconciler scan query: add filter WHERE coordinator_status NOT IN ('terminal', 'stopped', 'completed', 'failed')
- Or: per-pod in-memory set of already-reconciled terminal run IDs (skip on next cycle)
- Or: after dispatch finds run is terminal, update a DB flag so it's excluded from future scans
Immediate ops workaround
Manually mark the 10 run IDs above as reconciled/excluded OR delete their coordinator_loop records so the scan no longer finds them.
Summary
The coordinator reconciler enters a tight infinite loop after every pod restart, continuously re-arming 10 specific orphaned run IDs every ~10 seconds (6 cycles/minute). All 10 runs are already terminal/stopped but the reconciler never stops trying them. This starves new coordinator-draft work from completing.
Evidence (App Insights - agentweaver-insights/agentweaver-rg, 2026-08-19)
Reconciler thrash rate
10:16:10Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s) 10:16:20Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s) 10:16:30Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s) 10:16:40Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s) 10:16:50Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s) 10:17:00Z Coordinator reconciler: re-armed 10 orphaned coordinator loop(s)6 full cycles/minute, each cycling the same 10 run IDs. All immediately resolve as "already terminal/stopped; no subtasks will be dispatched" — yet the next cycle re-arms them again.
Orphaned run IDs (same set every cycle)
Impact on new coordinator work
New orchestration aaa6b185-27e2-4ac7-88d9-4e6d46ae7178 submitted at 10:15:22Z. coordinator-draft SetupAsync entered at 10:16:24Z. Zero further traces in 1.5 hours. /work-plan returned 499 at 10:16:53Z (29s after setup). Plan never completed.
Root cause
The reconciler's orphan-scan query does not exclude runs already confirmed terminal. When a new pod starts, all orphaned runs re-enter the scan queue. The dispatch step confirms terminal status but does NOT mark the run as reconciled/excluded. The scan runs again 10s later and re-discovers the same runs. No backoff, no deduplication, no exit condition.
Impact
Fix suggestions
Immediate ops workaround
Manually mark the 10 run IDs above as reconciled/excluded OR delete their coordinator_loop records so the scan no longer finds them.