fix(ai): bound retries on persistent rate-limit to prevent infinite loop (H1)

[ocervell]

Finding: H1 (High) — persistent rate-limit causes an effectively infinite loop.

Root cause

In secator/tasks/ai.py, the main loop's exception handler did, for litellm.RateLimitError:

iteration -= 1
sleep(5)
continue

Decrementing iteration pins the loop counter, so a persistent 429 (exhausted quota, billing block, provider throttle) never advances toward max_iterations and spins forever — bounded only by the 3h K8s Job deadline — holding a worker slot the whole time. call_llm in secator/ai/utils.py already retries 429 (up to 3×, ~2/4/8s exponential backoff) before re-raising, so the outer sleep(5) was redundant double-waiting.

Fix

secator/tasks/ai.py, rate-limit handler region only:

• Stop pinning the counter — removed iteration -= 1 so a 429 lets the iteration advance.
• Added a bounded consecutive-429 guard (rate_limit_streak, mirroring the existing empty_streak pattern): hard-abort with a clear Error after 4 consecutive rate-limit failures, then _save_history() + return. The streak resets to 0 on any successful call_llm, so transient throttling is forgiven; only a persistent 429 trips the abort. This bounds termination at min(4, max_iterations) independent of max_iterations (which is user-settable and can be raised by modes / query extensions).
• Removed the redundant outer sleep(5); backoff is preserved inside call_llm. Dropped the now-orphaned from time import sleep import.
• AuthenticationError / APIConnectionError handling left untouched.

secator/ai/utils.py call_llm retry/backoff already provides the spacing between attempts (2/4/8s) and was left as-is — no change needed.

Validation

• python3 -m py_compile secator/tasks/ai.py secator/ai/utils.py → OK.
• New focused test tests/unit/test_ai_tokens.py::TestAiRateLimitTermination::test_persistent_rate_limit_aborts_bounded reuses the existing _loop_patches harness, patches call_llm to raise RateLimitError on every call with max_iterations=50, drives the real _run_loop, and asserts the loop calls call_llm exactly 4 times (the consecutive-429 cap, far below the 50-iteration budget) and ends with a rate-limit abort Error — i.e. it provably terminates instead of looping unbounded. Passes.
• No new regressions: pre-existing failures on the base branch (test_ai_loop.py guardrail/e2e tests and test_ai_tokens.py::test_loop_sums_token_usage) are unchanged with vs. without this patch (verified by stash/compare).

Extra issues surfaced

Noted in adjacent code; not fixed here (separate PRs / other lanes):

• Pre-existing test failures on ai-resiliency (not introduced by this PR, flagged for triage): in tests/unit/test_ai_loop.py, 9 guardrail/e2e tests fail with Action denied: shell command not approved (e.g. TestGuardrailsAutoMode::test_allowed_command_passes, TestMainLoopAutoE2E::test_multi_turn_auto_loop), and tests/unit/test_ai_tokens.py::TestAiTokenAccountingEndToEnd::test_loop_sums_token_usage asserts 600 but gets 100. These look like guardrail/loop drift, owned by other regions.
• Empty-response handling (empty_streak) hard-stops after 3 with return (no saved abort Error type distinction) — consistent with this fix's pattern, but worth confirming it's the intended UX.
• call_llm retryable set (utils.py) lumps BadRequestError and generic APIError into the same retry/backoff path as transient 429s — a permanently malformed request will still burn 3 backoff attempts (~6s) before surfacing. Candidate to split non-retryable client errors out.
• query_extensions / mode configs raise max_iterations at runtime — independent of this fix (the new guard is mode-independent), but a reminder that the iteration budget is not a tight upper bound on wall-clock by itself.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NNjPggRSVZ2xnLb7ZxWP5H

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7089c484-0641-437e-aa22-9009c2d400e8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/ratelimit-loop

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}