Skip to content

Treat Copilot CLI's own retry-exhaustion message as non-retryable in harness classification #53569

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 3 commits into from
Aug 18, 2026
Merged

Treat Copilot CLI's own retry-exhaustion message as non-retryable in harness classification #53569

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bca0fc5
Initial plan
Copilot Aug 18, 2026
995acfd
Recognize Copilot CLI's own retry-exhaustion message as non-retryable
Copilot Aug 18, 2026
754fb30
Anchor exhausted-retry pattern to a single line to avoid cross-messag…
Copilot Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .github/workflows/daily-team-evolution-insights.lock.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 7 additions & 2 deletions .github/workflows/mcp-inspector.lock.yml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 27 additions & 0 deletions actions/setup/js/copilot_harness.test.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,33 @@ describe("copilot_harness.cjs", () => {
expect(isCAPIQuotaExceededError("Authentication failed")).toBe(false);
expect(isCAPIQuotaExceededError("")).toBe(false);
});

it("matches the Copilot CLI's own retry-exhaustion message without a CAPIError: prefix (429)", () => {
const output =
"Failed to get response from the AI model; retried 5 times (total retry wait time: 380.35 seconds) " +
"(Request-ID AC21:F5CEC:33A719:40DD88:6A83AA27) Last error: 429 Too Many Requests\nChanges +0 -0";
expect(isCAPIQuotaExceededError(output)).toBe(true);
});

it("matches the Copilot CLI's own retry-exhaustion message for 5xx statuses (503)", () => {
const output = "Failed to get response from the AI model; retried 5 times (total retry wait time: 300 seconds) Last error: 503 Service Unavailable";
expect(isCAPIQuotaExceededError(output)).toBe(true);
});

it("does not retry a zero-progress attempt that exhausted the CLI's own 429 retries", () => {
const output =
"Failed to get response from the AI model; retried 5 times (total retry wait time: 380.35 seconds) " +
"(Request-ID AC21:F5CEC:33A719:40DD88:6A83AA27) Last error: 429 Too Many Requests";
expect(
shouldRetryFailedExecution({
exitCode: 1,
hasOutput: true,
output,
attempt: 0,
maxRetries: 3,
})
).toBe(false);
});
});

it("matches CAPIError: 400 with various spacing", () => {
Expand Down
19 changes: 13 additions & 6 deletions actions/setup/js/detect_agent_errors.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,12 @@
* response (for example "Response status code does not indicate success: 400 (Bad Request)").
* - capi_quota_exceeded_error: The Copilot CAPI quota has been exhausted
* or rate-limited (e.g., "CAPIError: 429 429 quota exceeded",
* "CAPIError: Too Many Requests"). All matched forms are treated as
* non-retryable because the Copilot SDK has already retried internally
* before surfacing the error.
* "CAPIError: Too Many Requests", or the Copilot CLI's own
* retry-exhaustion message "Failed to get response from the AI model;
* retried N times ... Last error: 429/5xx" which carries no "CAPIError:"
* prefix). All matched forms are treated as non-retryable because the
* Copilot CLI/SDK has already retried internally before surfacing the
* error.
* - invocation_cap_exceeded: The per-run pooled LLM invocation cap is
* fully exhausted (e.g., "CAPIError: 429 Maximum LLM invocations exceeded (N/N)"
* or `"type":"max_runs_exceeded"`). This is more specific than generic
Expand Down Expand Up @@ -165,9 +168,13 @@ const MISSING_MODEL_PRICING_PATTERN = /Model\s+"([^"]+)"\s+has no AI credits pri
// "CAPIError: 429 429 quota exceeded" (original observed form)
// "CAPIError: 429 Too Many Requests" (HTTP 429 form)
// "CAPIError: Too Many Requests" (no status code in message)
// All forms are treated as non-retryable; the Copilot SDK has already retried
// internally before surfacing this error (evidenced by "retried 5 times" context).
const CAPI_QUOTA_EXCEEDED_PATTERN = /CAPIError:\s*(?:429\s+)?(?:429\s+quota exceeded|Too Many Requests)/i;
// "Failed to get response from the AI model; retried 5 times ... Last error: 429 Too Many Requests"
// (Copilot CLI's own retry-exhaustion message, no "CAPIError:" prefix — seen with both
// 429 and 5xx terminal statuses, e.g. "Last error: 503 Service Unavailable")
// All forms are treated as non-retryable; the Copilot CLI/SDK has already retried
// internally before surfacing this error (evidenced by "retried N times" context).
const CAPI_QUOTA_EXCEEDED_PATTERN =
/CAPIError:\s*(?:429\s+)?(?:429\s+quota exceeded|Too Many Requests)|Failed to get response from the AI model;\s*retried\s+\d+\s+times[^\n]{0,300}?Last error:\s*(?:429|5\d{2})\b/i;
Comment on lines +176 to +177

/**
* Build a case-insensitive merged RegExp from literal/regex patterns.
Expand Down
16 changes: 16 additions & 0 deletions actions/setup/js/detect_agent_errors.test.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,22 @@ describe("detect_agent_errors.cjs", () => {
// it should NOT match the CAPI quota pattern.
expect(isCAPIQuotaExceededError("CAPIError: 429 Maximum LLM invocations exceeded (25/25)")).toBe(false);
});

it("matches the Copilot CLI's own retry-exhaustion message with no CAPIError: prefix (429)", () => {
const message =
"Failed to get response from the AI model; retried 5 times (total retry wait time: 380.35 seconds) " +
"(Request-ID AC21:F5CEC:33A719:40DD88:6A83AA27) Last error: 429 Too Many Requests";
expect(isCAPIQuotaExceededError(message)).toBe(true);
});

it("matches the Copilot CLI's own retry-exhaustion message for 5xx statuses (503)", () => {
const message = "Failed to get response from the AI model; retried 5 times (total retry wait time: 300 seconds) Last error: 503 Service Unavailable";
expect(isCAPIQuotaExceededError(message)).toBe(true);
});

it("does not match a 'Failed to get response' message without retry-exhaustion context", () => {
expect(isCAPIQuotaExceededError("Failed to get response from the AI model due to a network error")).toBe(false);

@github-actions github-actions Bot Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] Missing negative test for non-429, non-5xx status codes — the boundary (?:429|5\d{2}) is not regression-tested, so a future pattern edit could silently start treating client errors like 400 as non-retryable.

💡 Suggested test 
it("does not match a retry-exhaustion message with a non-quota 4xx status (e.g. 400)", () => {
  expect(
    isCAPIQuotaExceededError(
      "Failed to get response from the AI model; retried 5 times Last error: 400 Bad Request"
    )
  ).toBe(false);
});

This pins the intended semantics: only 429 and 5xx terminal statuses are treated as non-retryable quota errors. A 400 (malformed request) is a different failure class and should flow through normal retry logic.

@copilot please address this.

});
});

describe("INVOCATION_CAP_EXCEEDED_PATTERN / isInvocationCapExceededError", () => {
Expand Down
Loading