fix(fleet): recycle five phantom agent seats + stop the poller swallowing 401s - #221
Open
servathadi wants to merge 1 commit into
Open
fix(fleet): recycle five phantom agent seats + stop the poller swallowing 401s#221servathadi wants to merge 1 commit into
servathadi wants to merge 1 commit into
Conversation
…wing 401s
Measured, not inferred: 167 x HTTP 401 in six hours from sos-squad —
sol 36, mizan 36, gemma 36, dandan 36, worker 23 — one poll each per agent every
~10 minutes, forever. Hadi: "we never worked with sol or mizan or gemma".
Three separate defects, one symptom.
1. DEAD SEATS. task_poller iterates AGENT_ROUTING and polls the squad service for
every key. Five seats nobody uses were still in the dict, so each produced a
401 per cycle. Removed rather than set to "none": the poller fetches tasks
BEFORE it consults the route value, so "none" would have kept the polling and
only hidden the wake. Same treatment the webdev seats already got in this file.
2. THE POLLER SWALLOWED IT. `if resp.status_code == 200:` with no else — so "the
squad refused my credential" and "this agent has no queued work" were the SAME
observation from the poller's side. The only evidence anywhere was the squad
service's access log. Any agent with real queued work would have been starved
for as long as the credential stayed wrong and NOTHING would have said so.
401/403 now logs at error and says explicitly that it is our credential, not
an idle agent. Root cause is visible in the code: _squad_headers() returns {}
when SQUAD_TOKEN is unset — no Authorization header at all.
3. RECYCLING COULD HAVE STRANDED WORK. analytics/act.py still dispatched blog
work to sol (x2) and business work to mizan. Removing a seat from routing
while something keeps assigning to it creates a task that is created,
assigned, and never delivered — strictly worse than the noise, because the
noise at least left a trace. That is mupot#744's shape. Guarded generally at
the dispatch helper rather than per-callsite, so the NEXT recycle cannot
strand work either. analytics-act.service is static/inactive, so this was
latent rather than live.
8 tests. Mutations: re-adding sol to AGENT_ROUTING turns it red; disabling the
dispatch guard turns it red. Live seats (kasra, athena, mumega, mumcp, river,
gemini) are pinned so a future recycle cannot quietly take out a working one.
NOT fixed here, and it is the real root: the squad system token is missing or
wrong in the poller's environment. That is a credential, not code — flagged to
Hadi. Defect 2 means the next occurrence will at least be loud.
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measured first
One poll per seat every ~10 minutes, forever. Hadi: "we never worked with sol or mizan or gemma."
Three defects, one symptom
1. Dead seats.
task_polleriteratesAGENT_ROUTINGand polls the squad service for every key. Five unused seats were still in the dict.Removed rather than set to
"none"— and that distinction is the whole fix. The poller fetches tasks before it ever consults the route value, so"none"would have kept the polling and only hidden the wake. Same treatment thewebdevseats already received in this file.2. The poller swallowed it.
So "the squad refused my credential" and "this agent has no queued work" were the same observation from the poller's side. The only evidence anywhere was the squad service's access log. Any agent with real queued work would have been starved for as long as the credential stayed wrong, and nothing would have said so.
401/403 now logs at
errorand states explicitly that it is our credential rather than an idle agent. The root cause is visible in the code —_squad_headers()returns{}whenSQUAD_TOKENis unset, i.e. noAuthorizationheader at all, which is exactly how this presented.3. Recycling could have stranded work.
analytics/act.pystill dispatched blog work tosol(×2) and business work tomizan. Removing a seat from routing while something keeps assigning to it produces a task that is created, assigned, and never delivered — strictly worse than the noise it replaced, because the noise at least left a trace. Same shape as mupot#744.Guarded at the dispatch helper rather than per-callsite, so the next recycle cannot strand work either.
analytics-act.serviceisstatic/inactive, so this was latent rather than live.Evidence
8 tests. Mutations both bite:
soltoAGENT_ROUTINGLive seats (
kasra,athena,mumega,mumcp,river,gemini) are pinned by test, so a future recycle cannot quietly remove a working one. Roster goes 24 → 19.Not fixed here
The squad system token is missing or wrong in the poller's environment. That is a credential, not code, and it belongs to Hadi. Defect 2 means the next occurrence will at least be loud instead of silent.