threat-detect: "compact fallback prompt" returns 400 Bad Request when the full prompt exceeds 100KB

[jaroslawgajewski]

Summary

When the threat-detection prompt built for a PR review exceeds 100KB, threat-detect (v0.5.1) switches to a "compact fallback prompt" instead of the full one. That fallback path fails immediately with a 400 Bad Request from the Copilot backend, with zero retries configured, so threat detection reports agent_failure on every run where the diff is large enough to trigger the fallback — even though the underlying PR review itself completes successfully.

Environment

• gh-aw compiler: v0.88.4
• threat-detect: v0.5.1
• Engine: copilot, model gpt-5.4-mini
• A PR-review workflow with native (compiler-default) threat detection, continue-on-error: true (default)

Observed behavior

Two consecutive runs on the same PR (large diff, big codebase) both failed identically:

[copilot-harness] resolved --prompt-file: path=/tmp/threat-detect-prompt-XXXXXXXXXX.txt size=118029B
[copilot-harness] prompt file exceeds 100KB; using compact fallback prompt
[copilot-harness] attempt 1: spawning: /usr/local/bin/copilot --add-dir /tmp --log-level all --disable-builtin-mcps --no-ask-user --allow-all-tools --add-dir <repo-path> -p <redacted>
[copilot-harness] attempt 1: process started (pid=184)

400 400 400 Bad Request

Changes    +0 -0
Duration   0s
Resume     copilot --resume=<session-id>
[copilot-harness] attempt 1: process exit event exitCode=1
[copilot-harness] attempt 1: process closed exitCode=1 duration=1s stdout=0B stderr=123B hasOutput=true
[copilot-harness] attempt 1 failed: exitCode=1 failureClass=partial_execution ... tokenCount=0 attemptDurationMs=1657 retriesRemaining=0
[copilot-harness] all 0 retries exhausted — giving up (exitCode=1)
Error running detection: engine execution failed: node exited with code 1: ...
THREAT_DETECTION_STATUS: reason=engine_error exit=2
##[error]ERR_SYSTEM: ❌ Detection result file not found at: /tmp/gh-aw/threat-detection/detection_result.json

The failure is immediate (attemptDurationMs≈1657, duration=1s) and 100% reproducible on this PR — both runs (7 minutes apart, one after a fresh commit) hit the identical 400 Bad Request at the identical stage, right after the "using compact fallback prompt" line.

Impact

• safe_outputs, conclusion, and the main PR review comment all complete successfully (continue-on-error: true on the detection job absorbs this).
• The gh-aw-managed "Detection Runs" tracking issue accumulates a failure / agent_failure comment on every affected run, with no way to distinguish "genuine security concern" from "tooling crashed" other than reading the run log.
• Any repo whose typical PR diffs regularly exceed 100KB (large codebases, generated/lock files, vendored code) will see this on effectively every PR review.

Suspected root cause

The "compact fallback prompt" that gets substituted once the full prompt exceeds 100KB appears to produce a request the Copilot backend rejects outright (400, zero tokens, sub-2-second failure) — as opposed to the full prompt, which presumably works (no 100KB-related failures are seen on smaller PRs). This suggests the fallback-prompt construction path itself is malformed (e.g. missing a required field, invalid encoding, or truncating mid-structure) rather than the request being legitimately too large for the model.

Suggested fix direction

• Validate/test the compact-fallback-prompt code path explicitly (it looks under-exercised, since it seems to always fail rather than degrade gracefully).
• Alternatively, raise the compaction threshold, or compact more conservatively (e.g. truncate the diff content rather than whatever is currently dropped) so the resulting request is still well-formed.
• At minimum, retry the compact-fallback path at least once before giving up (currently retriesRemaining=0 on the very first attempt), since this looks like a request-construction bug rather than a transient error.

Reproduction notes

Happy to share the full job log / the exact >100KB prompt that triggers this if that's useful, but the pattern above should be reproducible on any sufficiently large PR diff with threat-detection enabled.