Review agent should explicitly calibrate robustness findings for experiment/throwaway code

[fullsend-ai-retro]

What happened

On PR #40, qodo-code-review flagged unchecked API responses in run_matrix.py as a bug/reliability issue, noting the code calls .json() without checking status codes. The author dismissed this, explaining the harness already captures status codes without crashing, error outcomes (403/405) are expected signals documented in the experiment, and this was a one-shot script against throwaway infrastructure. The fullsend-ai-review agent correctly did NOT flag this issue — its correctness analysis noted the script handles API responses appropriately. Agents repo used at commit ef7e8c3c64b1a8af87aec2d253d26ea81a6310de. Review workflow run: 29277635106.

What could go better

The fullsend-ai-review agent happened to avoid this false positive, but the reasoning appears incidental rather than deliberate. The correctness analysis did not explicitly note that robustness standards differ for concluded experiment scripts vs production code. For experiment repos (identifiable via AGENTS.md context describing the repo as holding experiments), the review agent should deliberately suppress or downgrade findings about defensive coding, error handling robustness, and API response validation when: (a) the code already records the relevant signals (status codes, error bodies), (b) the experiment status is Concluded, and (c) the code is not intended for reuse. This avoids wasting author time dismissing findings. Note: issue #150 covers suppressing low-value compliance findings on trivial config PRs but does not address experiment-specific robustness calibration. Confidence is medium — the review agent may already handle this in practice, but the reasoning should be explicit to be reliable.

Proposed change

Add experiment-context awareness to the review agent's correctness sub-agent or the review policy:

1. When the target repo's AGENTS.md describes the repo as holding experiments or prototypes, the review agent should note this context explicitly in its analysis.
2. For PRs with experiment status 'Concluded' or targeting throwaway infrastructure, suppress findings about error handling robustness and defensive coding when the code already captures relevant error signals (status codes, response bodies) — these are adequate for experiment purposes.
3. If such findings are still surfaced, they should be marked as informational/low rather than actionable, with a note that the code is experiment/throwaway in nature.

This could be implemented as a conditional in the review policy (e.g., policies/review.yaml) or as guidance in the correctness sub-agent skill.

Validation criteria

1. On a review of PR fix(triage): enforce question action and label consistency #40's code, the correctness sub-agent should explicitly note the experiment context and either suppress or downgrade the unchecked-API-responses finding.
2. The same finding should still fire at normal severity for production code in non-experiment repos.
3. The review output should contain explicit reasoning about experiment context when analyzing code in experiment repos.

---

Generated by retro agent from fullsend-ai/experiments#40