Skip to content

watch: no per-issue failure tracking - a permanently failing issue is re-dispatched every round forever #1713

Open

Description

@swigerb

Summary

squad watch has no notion of a work item that keeps failing. An issue that an agent cannot complete is re-selected and re-dispatched on every round, indefinitely, with no escalation and no backoff.

Evidence

The only failure counter in the watch path is the circuit breaker, and it is scoped to GitHub API quota, not to agent outcomes.

packages/squad-cli/src/cli/commands/watch/index.ts:431-449:

interface CircuitBreakerState {
  status: 'closed' | 'open' | 'half-open';
  openedAt: string | null;
  cooldownMinutes: number;
  consecutiveFailures: number;
  consecutiveSuccesses: number;
  lastRateLimitRemaining: number | null;
  lastRateLimitTotal: number | null;
}

Every site that increments consecutiveFailures is a rate-limit path:

  • index.ts:886 - inside the ghRateLimitCheck() traffic-light branch
  • index.ts:1009 - inside if (adapter.type === 'github' && isRateLimitError(err))

The non-rate-limit error branch at index.ts:1014-1016 only logs:

} else {
  console.error(`${RED}✗${RESET} Round error: ${err.message}`);
}

And on a clean round, index.ts:983-986 resets both counters to zero.

Meanwhile issue selection in execute.ts:182-191 re-queries all open squad-labelled issues each round and applies only a label filter. There is no per-issue attempt count, no cooldown, no state carried between rounds about which items have already burned agent turns.

Impact

  • A single malformed or impossible issue silently consumes an agent turn every interval, forever. Over an overnight run at a 5 minute interval that is ~100 wasted turns on one bad item.
  • Because it is re-offered every round, it also crowds out genuinely actionable work in the same dispatch.
  • There is no signal to a human that a specific item is stuck - it looks identical to normal activity in the board report.

Proposal

Track attempts per work item, persisted alongside the existing circuit-breaker state in .squad/:

  1. Record { itemId, attempts, lastAttemptAt, lastError } after each dispatch.
  2. After N consecutive failed attempts on the same item (default 3, configurable), stop offering it to findExecutableIssues().
  3. On the final strike, escalate visibly - comment on the issue, apply a squad:stuck label, and surface it in reportBoard().
  4. Reset the counter when the item is edited or the label is removed, so a human fixing the issue puts it back in play.

This depends on having a real success signal, so it pairs naturally with adding a verification gate to the watch path.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    wave1:P5Wave 1 PR group P5wave:1-nextWave 1 — immediate next sprint

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions