Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 42 additions & 3 deletions actions/setup/js/detect_agent_errors.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,14 @@
* Detected from the agent stdio log (text pattern) and the AWF firewall audit
* JSONL log (`unknown_model_ai_credits` event type). Both sources are checked
* and their results merged.
* - shell_expansion_guard_rejected: The sandbox's shell command-injection guard
* rejected a shell command for containing (or appearing to contain) bash
* expansion patterns (command substitution, indirect expansion, parameter
* transformation, etc.), e.g. "...could enable arbitrary code execution.
* Please rewrite the command without these expansion patterns." This can
* misfire on benign multi-line `printf`/`safeoutputs` CLI invocations; agents
* should switch to the `jq -Rs` file-piping pattern instead of retrying the
* same command verbatim.
* This replaces the individual bash scripts (detect_inference_access_error.sh,
* detect_mcp_policy_error.sh) with a single JavaScript step.
*
Expand Down Expand Up @@ -173,6 +181,16 @@ const INVOCATION_CAP_EXCEEDED_PATTERN = buildCombinedPattern(MAX_RUNS_EXCEEDED_P
// parseMaxCacheMissesExceededFromEventLog().
const MAX_CACHE_MISSES_EXCEEDED_PATTERN = /(?:\bmax_cache_misses_exceeded\b|\bmaximum\s+consecutive\s+cache\s+misses\s+exceeded\b)/i;

// Pattern: the sandbox's shell command-injection guard rejected a shell command believed to
// contain dangerous bash expansion patterns (command substitution, indirect expansion, parameter
// transformation, backtick substitution, etc.). Observed message form:
// "...indirect expansion, or nested command substitution) that could enable arbitrary code
// execution. Please rewrite the command without these expansion patterns."
// This guard can misfire on benign multi-line printf/safeoutputs CLI invocations. Retrying the
// identical command is pointless — it will be rejected again — so this is surfaced as a distinct,
// actionable diagnostic instead of a generic shell failure.
const SHELL_EXPANSION_GUARD_REJECTED_PATTERN = /could enable arbitrary code execution\b[\s\S]{0,200}?\brewrite the command without these expansion patterns\b/i;

/**
* Determines if the collected output contains the observed Copilot/CAPI quota exhaustion error.
* @param {string} output - Collected stdout+stderr from the process
Expand Down Expand Up @@ -205,6 +223,18 @@ function isMaxCacheMissesExceededError(output) {
return MAX_CACHE_MISSES_EXCEEDED_PATTERN.test(output);
}

/**
* Determines if the collected output shows the sandbox's shell command-injection guard
* rejected a command for containing (or appearing to contain) dangerous bash expansion
* patterns. Retrying the same command verbatim will not succeed; the agent should switch
* to the `jq -Rs` file-piping pattern for multi-line safeoutputs CLI bodies instead.
* @param {string} output - Collected stdout+stderr from the process
* @returns {boolean}
*/
function isShellExpansionGuardRejectedError(output) {
return SHELL_EXPANSION_GUARD_REJECTED_PATTERN.test(output);
}

/**
* Normalize model names to a single safe line for GitHub Actions outputs and issue titles.
* @param {string} value
Expand All @@ -227,7 +257,7 @@ function extractMissingModelPricingModelName(logContent) {
/**
* Detect known error patterns in a log string and return detection results.
* @param {string} logContent - Contents of the agent stdio log
* @returns {{ inferenceAccessError: boolean, mcpPolicyError: boolean, agenticEngineTimeout: boolean, modelNotSupportedError: boolean, http400ResponseError: boolean, capiQuotaExceededError: boolean, invocationCapExceeded: boolean, maxCacheMissesExceeded: boolean, missingModelPricingError: boolean, missingModelPricingModelName: string }}
* @returns {{ inferenceAccessError: boolean, mcpPolicyError: boolean, agenticEngineTimeout: boolean, modelNotSupportedError: boolean, http400ResponseError: boolean, capiQuotaExceededError: boolean, invocationCapExceeded: boolean, maxCacheMissesExceeded: boolean, missingModelPricingError: boolean, missingModelPricingModelName: string, shellExpansionGuardRejected: boolean }}
*/
function detectErrors(logContent) {
const missingModelPricingModelName = extractMissingModelPricingModelName(logContent);
Expand All @@ -242,12 +272,13 @@ function detectErrors(logContent) {
maxCacheMissesExceeded: isMaxCacheMissesExceededError(logContent),
missingModelPricingError: missingModelPricingModelName !== "",
missingModelPricingModelName,
shellExpansionGuardRejected: isShellExpansionGuardRejectedError(logContent),
};
}

/**
* Build GitHub Actions output lines from detection results.
* @param {{ inferenceAccessError: boolean, mcpPolicyError: boolean, agenticEngineTimeout: boolean, modelNotSupportedError: boolean, http400ResponseError: boolean, capiQuotaExceededError: boolean, invocationCapExceeded: boolean, maxCacheMissesExceeded: boolean, missingModelPricingError: boolean, missingModelPricingModelName: string }} results
* @param {{ inferenceAccessError: boolean, mcpPolicyError: boolean, agenticEngineTimeout: boolean, modelNotSupportedError: boolean, http400ResponseError: boolean, capiQuotaExceededError: boolean, invocationCapExceeded: boolean, maxCacheMissesExceeded: boolean, missingModelPricingError: boolean, missingModelPricingModelName: string, shellExpansionGuardRejected: boolean }} results
* @returns {string[]}
*/
function buildOutputLines(results) {
Expand All @@ -263,12 +294,13 @@ function buildOutputLines(results) {
`max_cache_misses_exceeded=${results.maxCacheMissesExceeded}`,
`missing_model_pricing_error=${results.missingModelPricingError}`,
`missing_model_pricing_model_name=${results.missingModelPricingModelName}`,
`shell_expansion_guard_rejected=${results.shellExpansionGuardRejected}`,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ac847b8448: the flag is now promoted to agent job outputs, passed to the conclusion job as GH_AW_SHELL_EXPANSION_GUARD_REJECTED, and classified/rendered by handle_agent_failure.cjs ahead of generic timeout handling.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new output is never promoted beyond the detect step, so downstream jobs still only see a generic timeout and cannot act on at all.

💡 The plumbing is incomplete

This file now emits , but the repo already has a separate propagation layer that maps selected detect-step outputs into job/workflow outputs and downstream env vars (for example and ). Nothing in this PR updates that layer, so the new flag is effectively trapped inside the step that produced it.

As a result, the conclusion job and workflow callers still cannot distinguish this failure mode from the existing generic timeout classification, which undercuts the main remediation described in the PR body.

Please thread the new output through the same promotion path as the other engine-error flags before merge.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ac847b8448: shell_expansion_guard_rejected now follows the same generated output/env propagation path as the other engine-error flags, with lock files regenerated.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] shell_expansion_guard_rejected is emitted to $GITHUB_OUTPUT but handle_agent_failure.cjs never reads GH_AW_SHELL_EXPANSION_GUARD_REJECTED from the environment, so the diagnostic does not influence the failure report or categorization surfaced to authors.

All peer outputs (GH_AW_MAX_CACHE_MISSES_EXCEEDED, GH_AW_MISSING_MODEL_PRICING_ERROR, etc.) are read in handle_agent_failure.cjs around line 3246 and fed into failure-context templates. Without a corresponding read + context block, this new detection is visible only in the step's stderr log — not in the issue/comment the author sees.

💡 Suggested follow-up

Add to handle_agent_failure.cjs:

const shellExpansionGuardRejected = process.env.GH_AW_SHELL_EXPANSION_GUARD_REJECTED === "true" && agentConclusion === "failure";

...and a corresponding shell_expansion_guard_rejected.md template and context block, following the same pattern as max_cache_misses_exceeded.md.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ac847b8448: handle_agent_failure.cjs now reads GH_AW_SHELL_EXPANSION_GUARD_REJECTED, adds a dedicated failure category/title/context template, and suppresses the generic timeout path when this diagnosis is present.

];
}

/**
* Write GitHub Actions outputs to $GITHUB_OUTPUT.
* @param {{ inferenceAccessError: boolean, mcpPolicyError: boolean, agenticEngineTimeout: boolean, modelNotSupportedError: boolean, http400ResponseError: boolean, capiQuotaExceededError: boolean, invocationCapExceeded: boolean, maxCacheMissesExceeded: boolean, missingModelPricingError: boolean, missingModelPricingModelName: string }} results
* @param {{ inferenceAccessError: boolean, mcpPolicyError: boolean, agenticEngineTimeout: boolean, modelNotSupportedError: boolean, http400ResponseError: boolean, capiQuotaExceededError: boolean, invocationCapExceeded: boolean, maxCacheMissesExceeded: boolean, missingModelPricingError: boolean, missingModelPricingModelName: string, shellExpansionGuardRejected: boolean }} results
*/
function writeOutputs(results) {
const outputFile = process.env.GITHUB_OUTPUT;
Expand Down Expand Up @@ -351,6 +383,11 @@ function main() {
if (results.missingModelPricingError && !auditMissingPricing) {
process.stderr.write(`[detect-agent-errors] Detected missing model pricing: model "${results.missingModelPricingModelName}" has no AI credits pricing configured\n`);
}
if (results.shellExpansionGuardRejected) {
process.stderr.write(
"[detect-agent-errors] Detected sandbox shell expansion guard rejection: a shell command was rejected for dangerous bash expansion patterns; use the jq -Rs file-piping pattern for multi-line safeoutputs CLI bodies instead of retrying\n"
);
}

writeOutputs(results);
}
Expand Down Expand Up @@ -378,5 +415,7 @@ module.exports = {
INVOCATION_CAP_EXCEEDED_PATTERN,
MAX_CACHE_MISSES_EXCEEDED_PATTERN,
MISSING_MODEL_PRICING_PATTERN,
SHELL_EXPANSION_GUARD_REJECTED_PATTERN,
isShellExpansionGuardRejectedError,
buildOutputLines,
};
84 changes: 84 additions & 0 deletions actions/setup/js/detect_agent_errors.test.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ const {
INVOCATION_CAP_EXCEEDED_PATTERN,
MAX_CACHE_MISSES_EXCEEDED_PATTERN,
MISSING_MODEL_PRICING_PATTERN,
SHELL_EXPANSION_GUARD_REJECTED_PATTERN,
isShellExpansionGuardRejectedError,
extractMissingModelPricingModelName,
buildOutputLines,
} = require("./detect_agent_errors.cjs");
Expand Down Expand Up @@ -326,6 +328,7 @@ describe("detect_agent_errors.cjs", () => {
expect(result.maxCacheMissesExceeded).toBe(false);
expect(result.missingModelPricingError).toBe(false);
expect(result.missingModelPricingModelName).toBe("");
expect(result.shellExpansionGuardRejected).toBe(false);
});

it("detects inference access error only", () => {
Expand Down Expand Up @@ -416,6 +419,24 @@ describe("detect_agent_errors.cjs", () => {
expect(result.invocationCapExceeded).toBe(true);
});

it("detects shell expansion guard rejection only (issue github/gh-aw#52254 payload shape)", () => {
const log = [
"[copilot-harness] attempt 1: shell(safeoutputs create_discussion --title 'MCP toolset unavailable' --body \"...\\n...\")",
"Command rejected: shell command contains dangerous patterns (command substitution, indirect expansion, or nested command substitution) that could enable arbitrary code execution. Please rewrite the command without these expansion patterns.",
"[copilot-harness] attempt 2: retrying identical command",
"Command rejected: shell command contains dangerous patterns (command substitution, indirect expansion, or nested command substitution) that could enable arbitrary code execution. Please rewrite the command without these expansion patterns.",
"##[error]The action 'Execute GitHub Copilot CLI' has timed out after 5 minutes.",
].join("\n");
const result = detectErrors(log);
expect(result.inferenceAccessError).toBe(false);
expect(result.mcpPolicyError).toBe(false);
expect(result.modelNotSupportedError).toBe(false);
expect(result.http400ResponseError).toBe(false);
expect(result.capiQuotaExceededError).toBe(false);
expect(result.invocationCapExceeded).toBe(false);
expect(result.shellExpansionGuardRejected).toBe(true);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The test is named "only" but does not assert agenticEngineTimeout: false. The log fixture includes ##[error]The action 'Execute GitHub Copilot CLI' has timed out after 5 minutes. — if the AGENTIC_ENGINE_TIMEOUT_PATTERN were ever broadened to match that format, both flags would be true and the test would not catch the regression.

Adding expect(result.agenticEngineTimeout).toBe(false) here makes the test a precise specification for the "only" claim and guards against that regression.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ac847b8448: the "only" regression test now asserts agenticEngineTimeout remains false for the shell-expansion guard payload.

});

it("detects both capi quota and invocation-cap flags when both signatures are present", () => {
const result = detectErrors("CAPIError: Too Many Requests\nCAPIError: 429 Maximum LLM invocations exceeded (25/25)");
expect(result.capiQuotaExceededError).toBe(true);
Expand Down Expand Up @@ -621,6 +642,51 @@ commentary" has no AI credits pricing`;
});
});

describe("SHELL_EXPANSION_GUARD_REJECTED_PATTERN / isShellExpansionGuardRejectedError", () => {
// Exact payload shape from the issue report (github/gh-aw#52254): the sandbox's shell
// command-injection guard rejected a benign multi-line `printf` call to `safeoutputs
// create_discussion` for containing bash expansion patterns.
const ISSUE_REJECTION_MESSAGE =
"Command rejected: shell command contains dangerous patterns (command substitution, " +
"indirect expansion, or nested command substitution) that could enable arbitrary code " +
"execution. Please rewrite the command without these expansion patterns.";

it("matches the exact rejection message from the issue report", () => {
expect(SHELL_EXPANSION_GUARD_REJECTED_PATTERN.test(ISSUE_REJECTION_MESSAGE)).toBe(true);
expect(isShellExpansionGuardRejectedError(ISSUE_REJECTION_MESSAGE)).toBe(true);
});

it("matches when embedded in larger multi-line log output", () => {
const log = [
"[copilot-harness] attempt 1: invoking shell(safeoutputs create_discussion --title ... --body ...)",
ISSUE_REJECTION_MESSAGE,
"[copilot-harness] attempt 2: retrying identical command",
ISSUE_REJECTION_MESSAGE,
"##[error]The action 'Execute GitHub Copilot CLI' has timed out after 5 minutes.",
].join("\n");
expect(isShellExpansionGuardRejectedError(log)).toBe(true);
});

it("is case-insensitive", () => {
expect(SHELL_EXPANSION_GUARD_REJECTED_PATTERN.test(ISSUE_REJECTION_MESSAGE.toUpperCase())).toBe(true);
});

it("matches when the two anchor phrases are split across a line break", () => {
const wrapped = "Command rejected: ...that could enable arbitrary code execution.\nPlease rewrite the command without these expansion patterns.";
expect(isShellExpansionGuardRejectedError(wrapped)).toBe(true);
});

it("does not match unrelated shell errors", () => {
expect(isShellExpansionGuardRejectedError("bash: safeoutputs: command not found")).toBe(false);
expect(isShellExpansionGuardRejectedError("permission denied by workflow tool permissions")).toBe(false);
expect(isShellExpansionGuardRejectedError("")).toBe(false);
});

it("does not match arbitrary code execution mentions without the rewrite guidance", () => {
expect(isShellExpansionGuardRejectedError("This could enable arbitrary code execution if left unchecked.")).toBe(false);
});
});

describe("WATCHDOG_SIGTERM_PATTERN", () => {
it("matches a process closed line with SIGTERM and watchdogFired=true", () => {
const log = "[copilot-harness] attempt 1: process closed exitCode=1 signal=SIGTERM duration=12m 38s hasOutput=true watchdogFired=true";
Expand Down Expand Up @@ -799,5 +865,23 @@ commentary" has no AI credits pricing`;

expect(lines).toContain("max_cache_misses_exceeded=false");
});

it("emits shell_expansion_guard_rejected=true when detected", () => {
const lines = buildOutputLines({
inferenceAccessError: false,
mcpPolicyError: false,
agenticEngineTimeout: false,
modelNotSupportedError: false,
http400ResponseError: false,
capiQuotaExceededError: false,
invocationCapExceeded: false,
maxCacheMissesExceeded: false,
missingModelPricingError: false,
missingModelPricingModelName: "",
shellExpansionGuardRejected: true,
});

expect(lines).toContain("shell_expansion_guard_rejected=true");
});
});
});
11 changes: 11 additions & 0 deletions actions/setup/md/mcp_cli_tools_prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,17 @@ printf '{"item_number":42,"body":"### Title\n\nBody."}' | safeoutputs add_commen
# or write to a file: safeoutputs create_pull_request . < /tmp/payload.json
```

**Multi-line or long `body` content:** do NOT build the JSON payload with `printf`/`echo` embedding raw newlines or many escaped characters directly in the command line — the sandbox's shell command-injection guard may reject long or complex quoted arguments (reporting "expansion patterns"/"command substitution" even though none are present) and retrying the identical command will fail again. Instead, write the content to a temp file with a heredoc, then use `jq -Rs` to inject it as the `body` field:
```bash
cat <<'EOF' > /tmp/gh-aw/body.md
### Title

Multi-line body content goes here.
EOF
jq -Rs '{title: "My title", body: .}' /tmp/gh-aw/body.md | safeoutputs create_discussion .
```
If a shell command is rejected for containing expansion patterns, do not retry the same command — switch to the heredoc + `jq -Rs` pattern above.

To inject an entire local file as the `body` field without re-embedding its content in the model context, use `jq -Rs`:
```bash
jq -Rs --arg discussion_number "$DISCUSSION_NUMBER" \
Expand Down
Loading