Skip to content

fix(fleet): recycle five phantom agent seats + stop the poller swallowing 401s - #221

Open
servathadi wants to merge 1 commit into
mainfrom
fix/recycle-phantom-agent-seats
Open

fix(fleet): recycle five phantom agent seats + stop the poller swallowing 401s#221
servathadi wants to merge 1 commit into
mainfrom
fix/recycle-phantom-agent-seats

Conversation

@servathadi

Copy link
Copy Markdown
Collaborator

Measured first

401 Unauthorized, 6h window, from sos-squad.service
  sol 36 · mizan 36 · gemma 36 · dandan 36 · worker 23   = 167
LLM tier failures in the same window: 0

One poll per seat every ~10 minutes, forever. Hadi: "we never worked with sol or mizan or gemma."

Three defects, one symptom

1. Dead seats. task_poller iterates AGENT_ROUTING and polls the squad service for every key. Five unused seats were still in the dict.

Removed rather than set to "none" — and that distinction is the whole fix. The poller fetches tasks before it ever consults the route value, so "none" would have kept the polling and only hidden the wake. Same treatment the webdev seats already received in this file.

2. The poller swallowed it.

if resp.status_code == 200:
    ...
# no else

So "the squad refused my credential" and "this agent has no queued work" were the same observation from the poller's side. The only evidence anywhere was the squad service's access log. Any agent with real queued work would have been starved for as long as the credential stayed wrong, and nothing would have said so.

401/403 now logs at error and states explicitly that it is our credential rather than an idle agent. The root cause is visible in the code — _squad_headers() returns {} when SQUAD_TOKEN is unset, i.e. no Authorization header at all, which is exactly how this presented.

3. Recycling could have stranded work. analytics/act.py still dispatched blog work to sol (×2) and business work to mizan. Removing a seat from routing while something keeps assigning to it produces a task that is created, assigned, and never delivered — strictly worse than the noise it replaced, because the noise at least left a trace. Same shape as mupot#744.

Guarded at the dispatch helper rather than per-callsite, so the next recycle cannot strand work either. analytics-act.service is static/inactive, so this was latent rather than live.

Evidence

8 tests. Mutations both bite:

mutation result
re-add sol to AGENT_ROUTING 2 red
disable the dispatch guard 2 red

Live seats (kasra, athena, mumega, mumcp, river, gemini) are pinned by test, so a future recycle cannot quietly remove a working one. Roster goes 24 → 19.

Not fixed here

The squad system token is missing or wrong in the poller's environment. That is a credential, not code, and it belongs to Hadi. Defect 2 means the next occurrence will at least be loud instead of silent.

…wing 401s

Measured, not inferred: 167 x HTTP 401 in six hours from sos-squad —
sol 36, mizan 36, gemma 36, dandan 36, worker 23 — one poll each per agent every
~10 minutes, forever. Hadi: "we never worked with sol or mizan or gemma".

Three separate defects, one symptom.

1. DEAD SEATS. task_poller iterates AGENT_ROUTING and polls the squad service for
   every key. Five seats nobody uses were still in the dict, so each produced a
   401 per cycle. Removed rather than set to "none": the poller fetches tasks
   BEFORE it consults the route value, so "none" would have kept the polling and
   only hidden the wake. Same treatment the webdev seats already got in this file.

2. THE POLLER SWALLOWED IT. `if resp.status_code == 200:` with no else — so "the
   squad refused my credential" and "this agent has no queued work" were the SAME
   observation from the poller's side. The only evidence anywhere was the squad
   service's access log. Any agent with real queued work would have been starved
   for as long as the credential stayed wrong and NOTHING would have said so.
   401/403 now logs at error and says explicitly that it is our credential, not
   an idle agent. Root cause is visible in the code: _squad_headers() returns {}
   when SQUAD_TOKEN is unset — no Authorization header at all.

3. RECYCLING COULD HAVE STRANDED WORK. analytics/act.py still dispatched blog
   work to sol (x2) and business work to mizan. Removing a seat from routing
   while something keeps assigning to it creates a task that is created,
   assigned, and never delivered — strictly worse than the noise, because the
   noise at least left a trace. That is mupot#744's shape. Guarded generally at
   the dispatch helper rather than per-callsite, so the NEXT recycle cannot
   strand work either. analytics-act.service is static/inactive, so this was
   latent rather than live.

8 tests. Mutations: re-adding sol to AGENT_ROUTING turns it red; disabling the
dispatch guard turns it red. Live seats (kasra, athena, mumega, mumcp, river,
gemini) are pinned so a future recycle cannot quietly take out a working one.

NOT fixed here, and it is the real root: the squad system token is missing or
wrong in the poller's environment. That is a credential, not code — flagged to
Hadi. Defect 2 means the next occurrence will at least be loud.
@cursor

cursor Bot commented Aug 6, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant