feat(loop): classify failures + retry transient ones with backoff#7
Merged
Conversation
A coder/infra failure used to mean Blocked (or, with a ladder, a model-tier climb).
But a rate limit or a transient git/network error isn't the feature's fault and a
stronger model won't fix it — it should be retried with backoff. New failures.py is
a small ordered regex table (a lean distillation of protoMaker's failure-classifier)
returning a retry Policy: category, retryable, base_delay_s, max_attempts.
loop._drive now, on a NoChangesError/WorktreeError:
1. transient (rate_limit / network / merge_conflict) → back off + retry the SAME
tier (a fresh worktree off latest base also clears a conflict), up to the
policy's attempt cap;
2. capability failure (no diff / dispatch error) + a ladder → climb a model tier
(with a fresh retry budget);
3. terminal (auth / unknown) or retries exhausted → Blocked, tagged with the
category.
This is the first of the P0 resilience ports from protoMaker (the others: a stuck-
drive watchdog, and crash recovery on boot).
Tests: +20 (92 total) — test_failures.py (every category, ordering, terminal
fallback) + loop retry-then-succeed / exhaust-then-block / terminal-immediate-block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
👀 Quinn is reviewing — verdict (PASS / WARN / FAIL) + findings to follow. |
There was a problem hiding this comment.
QA Audit — PR #7 | feat(loop): classify failures + retry transient ones with backoff
VERDICT: WARN (non-blocking — CI still queued; formal PASS/FAIL on re-dispatch)
CI Status
- test: ⏳ queued
Diff Review
- New
failures.py(63 lines): ordered regex table mapping error messages →Policydataclass. Pure, deterministic, trivially testable. Pattern ordering (rate_limit → transient → merge_conflict → auth → terminal) is correct. loop.py(~30 line delta):_drivenow classifiesNoChangesError/WorktreeErrorbefore acting. Transient infra errors (rate-limit, network, merge-conflict) retry same tier with backoff; capability failures still climb the ladder; exhausted/terminal blocks with category tag.retries < policy.max_attempts - 1gate is correct for the docstring contract ("total dispatch attempts"): rate_limit gets 5 total, transient 3, merge_conflict 2.tests/test_failures.py(61 lines, new): exhaustive parametrized coverage of every category, unknown fallback, empty input, first-match-wins ordering, and retry budget assertions.tests/test_loop.py(truncated in diff): stubbedasyncio.sleepfor retry-then-succeed, exhaust-then-block, terminal-immediate-block scenarios.
Observations
- LOW: clawpatch structural review unavailable — repo not in project registry. Manual diff review completed; no structural concerns identified.
- LOW:
transientrule includes bareunavailablepattern — could match non-infra messages like "model unavailable." Mitigated by first-match ordering (rate_limit patterns fire first for capacity messages). Consider tightening toservice unavailableor anchoring with\b. - LOW:
merge_conflictrule uses bareconflict— broad match. In the loop context, errors come from git/gh operations, so false positives are unlikely. Worth monitoring.
— Quinn, QA Engineer
|
Submitted COMMENT review on #7. |
This was referenced Jun 13, 2026
mabry1985
added a commit
that referenced
this pull request
Jun 13, 2026
…ve (#9) A drive doesn't survive a server restart, so every in_progress feature the previous run was building was stranded forever (the puller only claims `ready`). The loop now runs a one-time recovery before the puller spins (at the top of _run): for each in_progress feature, if its PR actually got opened (a crash between open_pr and open_review) adopt it → in_review; otherwise reset it to ready for a clean rebuild (a stale worktree is cleaned when the puller re-claims). in_review features are left alone — they have a PR and the webhook/merge-poll resolves them. Recovery is best-effort per feature (one failure doesn't strand the rest or stop the loop). worktree.pr_url_for_branch() is the probe (the open PR for feat/<id>, or ""). This completes the P0 resilience batch (failure classification #7, stuck-coder watchdog #8, crash recovery here): the loop now survives restarts, hung coders, and transient failures unattended. Tests: +3 (98 total) — pr_url_for_branch found/absent; _recover adopts-PR-else-requeue and is resilient to a per-feature error. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The first P0 resilience port from protoMaker. A coder/infra failure used to mean Blocked (or, with a ladder, a model-tier climb) — but a rate limit or a transient git/network error isn't the feature's fault, and a stronger model won't clear it. It should be retried with backoff.
New
failures.py— a small ordered regex table (a lean distillation of protoMaker'sfailure-classifier-service.ts) returning a retryPolicy:category,retryable,base_delay_s,max_attempts. Pure + deterministic.loop._drive, on aNoChangesError/WorktreeError, now:rate_limit/ network /merge_conflict) → back off and retry the same tier (a fresh worktree off the latest base also clears a conflict), up to the policy's attempt cap;auth/ unknown) or retries exhausted → Blocked, tagged with the category.A rate limit no longer burns a stronger model or permanently blocks a feature; a flaky push retries instead of blocking.
Tests
+20 (92 total), all green:
test_failures.py(every category, first-match-wins ordering, terminal fallback) + loop retry-then-succeed, exhaust-then-block, terminal-immediate-block (withasyncio.sleepstubbed — no real backoff waits).ruffclean.Next in the P0 batch
in_progressfeatures)🤖 Generated with Claude Code