copilot_harness: "Failed to get response from the AI model … Last error: 429 Too Many Requests" misclassified as partial_execution → retries with --continue until timeout

[mur6]

Summary

When the Copilot CLI exhausts its own internal retries against a rate-limited or unavailable model endpoint, it exits 1 with a message of the form:

Failed to get response from the AI model; retried 5 times (total retry wait time: 380.35 seconds) (Request-ID …) Last error: 429 Too Many Requests

copilot_harness.cjs classifies this as partial_execution and retries it with --continue. But the session made zero progress (tokenCount=0, Changes +0 -0), so --continue has nothing to resume — every retry re-issues the same request into the same 429. The harness burns its full retry budget and the job dies on the soft-timeout guard.

The net effect is that a ~6.5-minute failure is amplified into a 20-minute one, and --continue masks a non-retryable condition as a retryable one.

Root cause

shouldRetryFailedExecution() already has a non-retryable path for quota/rate-limit failures, but the pattern it depends on requires a literal CAPIError: prefix:

// actions/setup/js/detect_agent_errors.cjs
const CAPI_QUOTA_EXCEEDED_PATTERN = /CAPIError:\s*(?:429\s+)?(?:429\s+quota exceeded|Too Many Requests)/i;

The Failed to get response from the AI model; … Last error: 429 Too Many Requests form carries no CAPIError: prefix, so isCAPIQuotaExceededError() returns false and the failure falls through to:

// actions/setup/js/copilot_harness.cjs — classifyCopilotFailure()
if (detection.hasOutput && (detection.tokenCount ?? 0) > LONG_RUN_TOKEN_THRESHOLD) return "long_run_exit";
return detection.hasOutput ? "partial_execution" : "no_output";

partial_execution here means only "the process printed something and exited non-zero" — it does not check whether any work actually happened. tokenCount=0 and Changes +0 -0 prove nothing was accomplished, yet the attempt still qualifies for a --continue retry.

#39479 closed the CAPIError:-prefixed form; this variant was not covered. The pattern is unchanged on main as of today.

Reproduction

Real failing run (gh-aw v0.84.3, Copilot CLI 1.0.80, engine: copilot, model: auto → claude-sonnet-5, timeout-minutes: 20), agent job log:

[copilot-harness] starting: command=/usr/local/bin/copilot maxRetries=3 initialDelayMs=5000 backoffMultiplier=2 maxDelayMs=60000
[copilot-harness] copilot model alias resolution: 'auto' -> 'claude-sonnet-5'

Failed to get response from the AI model; retried 5 times (total retry wait time: 380.35 seconds) (Request-ID AC21:F5CEC:33A719:40DD88:6A83AA27) Last error: 429 Too Many Requests
Changes    +0 -0
Duration   6m 25s
[copilot-harness] attempt 1 failed: exitCode=1 failureClass=partial_execution isCAPIError400=false isCAPIQuotaExceededError=false isInvocationCapExceeded=false … hasOutput=true tokenCount=0 attemptDurationMs=386484 retriesRemaining=3
[copilot-harness] attempt 1: partial execution — will retry with --continue (attempt 2/4)
[copilot-harness] retry 1/3: sleeping 5000ms before next attempt (--continue)

… Last error: 429 Too Many Requests
Duration   13m 7s
[copilot-harness] attempt 2 failed: exitCode=1 failureClass=partial_execution isCAPIQuotaExceededError=false … tokenCount=0 attemptDurationMs=397086 retriesRemaining=2
[copilot-harness] attempt 2: partial execution — will retry with --continue (attempt 3/4)

… Last error: 429 Too Many Requests
Duration   19m 45s
[copilot-harness] attempt 3 failed: exitCode=1 failureClass=partial_execution isCAPIQuotaExceededError=false … tokenCount=0 attemptDurationMs=388021 retriesRemaining=1
[copilot-harness] attempt 3: partial execution — will retry with --continue (attempt 4/4)
[copilot-harness] soft-timeout guard reached before attempt 4: timeoutMinutes=20 bufferMs=90000
[copilot-harness] done: exitCode=1 totalDuration=19m 47s

All three attempts resumed the same session id (copilot --resume=f2163d1a-…) and all three reported Changes +0 -0.

Notably this is not a network or sandbox problem — the firewall summary for the same run shows every request reaching the endpoint:

▼ 38 requests | 38 allowed | 0 blocked | 1 unique domain
| api.githubcopilot.com | 38 | 0 |

Not limited to 429

The same message shape appears with other terminal upstream statuses. #53550 (Failure Investigator Report — 2026-08-18, Cluster C) records:

4 consecutive Copilot-harness attempts (~33-38s each) all failed with Failed to get response from the AI model; retried 5 times ... Last error: 503 Service Unavailable

Same amplification, different status code — so a fix keyed only to 429 would leave the general case open.

Suggested fix

Two independent changes, either of which would have prevented the 20-minute burn:

1. Recognize the CLI's exhausted-retry message as a terminal upstream failure. Match Failed to get response from the AI model; retried N times … Last error: <status> and treat 429 (and arguably 5xx) as non-retryable in shouldRetryFailedExecution(), independent of the CAPIError: prefix. The Copilot CLI has already performed its own 5 retries with ~380s of backoff at that point; the harness re-running it adds no new information.

2. Do not classify a zero-progress attempt as partial_execution. When tokenCount === 0 and the CLI reported no changes, there is by definition no partial state for --continue to resume. A separate class (e.g. no_progress) that is retried at most once — or restarted fresh rather than with --continue — would be more faithful than the current "printed something ⇒ partially executed" heuristic.

Impact

Workflows triggered on a one-shot event (e.g. pull_request: [opened, ready_for_review] with no synchronize) have re-run as their only recovery path. When the upstream condition persists, every re-run reproduces the same 20-minute timeout, so the failure is both expensive and self-perpetuating.