Skip to content

feat(loop): classify failures + retry transient ones with backoff#7

Merged
mabry1985 merged 1 commit into
mainfrom
feat/failure-classification
Jun 13, 2026
Merged

feat(loop): classify failures + retry transient ones with backoff#7
mabry1985 merged 1 commit into
mainfrom
feat/failure-classification

Conversation

@mabry1985

Copy link
Copy Markdown
Member

What

The first P0 resilience port from protoMaker. A coder/infra failure used to mean Blocked (or, with a ladder, a model-tier climb) — but a rate limit or a transient git/network error isn't the feature's fault, and a stronger model won't clear it. It should be retried with backoff.

New failures.py — a small ordered regex table (a lean distillation of protoMaker's failure-classifier-service.ts) returning a retry Policy: category, retryable, base_delay_s, max_attempts. Pure + deterministic.

loop._drive, on a NoChangesError/WorktreeError, now:

  1. transient (rate_limit / network / merge_conflict) → back off and retry the same tier (a fresh worktree off the latest base also clears a conflict), up to the policy's attempt cap;
  2. capability failure (no diff / dispatch error) + a ladder → climb a model tier (fresh retry budget);
  3. terminal (auth / unknown) or retries exhausted → Blocked, tagged with the category.

A rate limit no longer burns a stronger model or permanently blocks a feature; a flaky push retries instead of blocking.

Tests

+20 (92 total), all green: test_failures.py (every category, first-match-wins ordering, terminal fallback) + loop retry-then-succeed, exhaust-then-block, terminal-immediate-block (with asyncio.sleep stubbed — no real backoff waits). ruff clean.

Next in the P0 batch

  • a stuck/stall watchdog on running drives
  • crash recovery on boot (resume/reset stranded in_progress features)

🤖 Generated with Claude Code

A coder/infra failure used to mean Blocked (or, with a ladder, a model-tier climb).
But a rate limit or a transient git/network error isn't the feature's fault and a
stronger model won't fix it — it should be retried with backoff. New failures.py is
a small ordered regex table (a lean distillation of protoMaker's failure-classifier)
returning a retry Policy: category, retryable, base_delay_s, max_attempts.

loop._drive now, on a NoChangesError/WorktreeError:
  1. transient (rate_limit / network / merge_conflict) → back off + retry the SAME
     tier (a fresh worktree off latest base also clears a conflict), up to the
     policy's attempt cap;
  2. capability failure (no diff / dispatch error) + a ladder → climb a model tier
     (with a fresh retry budget);
  3. terminal (auth / unknown) or retries exhausted → Blocked, tagged with the
     category.

This is the first of the P0 resilience ports from protoMaker (the others: a stuck-
drive watchdog, and crash recovery on boot).

Tests: +20 (92 total) — test_failures.py (every category, ordering, terminal
fallback) + loop retry-then-succeed / exhaust-then-block / terminal-immediate-block.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@protoquinn

protoquinn Bot commented Jun 13, 2026

Copy link
Copy Markdown

👀 Quinn is reviewing — verdict (PASS / WARN / FAIL) + findings to follow.

@mabry1985
mabry1985 merged commit 2150fc3 into main Jun 13, 2026
1 check passed
@mabry1985
mabry1985 deleted the feat/failure-classification branch June 13, 2026 05:15

@protoquinn protoquinn Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QA Audit — PR #7 | feat(loop): classify failures + retry transient ones with backoff

VERDICT: WARN (non-blocking — CI still queued; formal PASS/FAIL on re-dispatch)


CI Status

  • test: ⏳ queued

Diff Review

  • New failures.py (63 lines): ordered regex table mapping error messages → Policy dataclass. Pure, deterministic, trivially testable. Pattern ordering (rate_limit → transient → merge_conflict → auth → terminal) is correct.
  • loop.py (~30 line delta): _drive now classifies NoChangesError/WorktreeError before acting. Transient infra errors (rate-limit, network, merge-conflict) retry same tier with backoff; capability failures still climb the ladder; exhausted/terminal blocks with category tag.
  • retries < policy.max_attempts - 1 gate is correct for the docstring contract ("total dispatch attempts"): rate_limit gets 5 total, transient 3, merge_conflict 2.
  • tests/test_failures.py (61 lines, new): exhaustive parametrized coverage of every category, unknown fallback, empty input, first-match-wins ordering, and retry budget assertions.
  • tests/test_loop.py (truncated in diff): stubbed asyncio.sleep for retry-then-succeed, exhaust-then-block, terminal-immediate-block scenarios.

Observations

  • LOW: clawpatch structural review unavailable — repo not in project registry. Manual diff review completed; no structural concerns identified.
  • LOW: transient rule includes bare unavailable pattern — could match non-infra messages like "model unavailable." Mitigated by first-match ordering (rate_limit patterns fire first for capacity messages). Consider tightening to service unavailable or anchoring with \b.
  • LOW: merge_conflict rule uses bare conflict — broad match. In the loop context, errors come from git/gh operations, so false positives are unlikely. Worth monitoring.

— Quinn, QA Engineer

@protoquinn

protoquinn Bot commented Jun 13, 2026

Copy link
Copy Markdown

Submitted COMMENT review on #7.

mabry1985 added a commit that referenced this pull request Jun 13, 2026
…ve (#9)

A drive doesn't survive a server restart, so every in_progress feature the previous
run was building was stranded forever (the puller only claims `ready`). The loop now
runs a one-time recovery before the puller spins (at the top of _run): for each
in_progress feature, if its PR actually got opened (a crash between open_pr and
open_review) adopt it → in_review; otherwise reset it to ready for a clean rebuild
(a stale worktree is cleaned when the puller re-claims). in_review features are left
alone — they have a PR and the webhook/merge-poll resolves them. Recovery is
best-effort per feature (one failure doesn't strand the rest or stop the loop).

worktree.pr_url_for_branch() is the probe (the open PR for feat/<id>, or "").

This completes the P0 resilience batch (failure classification #7, stuck-coder
watchdog #8, crash recovery here): the loop now survives restarts, hung coders, and
transient failures unattended.

Tests: +3 (98 total) — pr_url_for_branch found/absent; _recover adopts-PR-else-requeue
and is resilient to a per-feature error.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant