-
Notifications
You must be signed in to change notification settings - Fork 4
feat: add generic critique agent #87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| # Critique Agent | ||
|
|
||
| Adversarial reviewer and quality gate for refinement plans. Evaluates a proposed decomposition from the refine agent and decides: approve (ready for issue creation), revise (send back to refine), or needs_input (escalate to a human for clarification). | ||
|
|
||
| > **Note**: The explore and refine agents are added in separate PRs. Cross-references to their docs pages will resolve once all three PRs merge. | ||
|
|
||
| ## How the agent works | ||
|
|
||
| The critique agent runs after the refine agent has produced a decomposition plan. It reads the original work item, exploration context, the proposed plan, and any prior critique history (for revision rounds). It evaluates the plan across seven dimensions — coverage, granularity, dependency coherence, implementability, scope accuracy, assumption grounding, and description clarity — and produces a structured verdict. | ||
|
|
||
| The agent runs in a read-only sandbox. It cannot modify issues or create children directly — those mutations are performed by the post-script on the runner. The sandbox policy permits read-only egress to GitHub and Jira APIs plus inference endpoints, but the agent's only output is a structured JSON critique result consumed by the post-script, which posts a summary comment, attaches feedback, and applies labels to signal the next pipeline step. | ||
|
|
||
| ## How it helps | ||
|
|
||
| - Prevents over-decomposition (15 issues when 6 would suffice) and under-decomposition (one giant epic). | ||
| - Catches scope creep — children that exceed what the parent asked for. | ||
| - Validates that acceptance criteria are testable, not vague ("should be fast"). | ||
| - Detects assumption laundering — plans that look confident but are built on unverified guesses. | ||
| - Creates child issues automatically on approval when `AUTO_CREATE=true`. | ||
|
|
||
| ## Commands | ||
|
|
||
| | Command | Where | Effect | | ||
| |---------|-------|--------| | ||
| | `/fs-create` | Issue comment | Creates child issues from an approved plan | | ||
| | `/fs-refine` | Issue comment | Restarts the refinement pipeline | | ||
|
|
||
| ## Pipeline integration | ||
|
|
||
| The critique agent is the third stage of a three-stage refinement pipeline: | ||
|
|
||
| 1. **Explore** — gathers technical context | ||
| 2. **Refine** — decomposes the work item | ||
| 3. **Critique** — reviews and approves/revises the decomposition (this agent) | ||
|
|
||
| ### Verdict outcomes | ||
|
|
||
| | Verdict | Post-script action | | ||
| |---------|-------------------| | ||
| | `approved` + `AUTO_CREATE=true` | Creates child issues immediately via `create-children.sh` | | ||
| | `approved` + `AUTO_CREATE=false` | Posts approval, adds `refine-approved` label for human gate | | ||
| | `revise` + under round limit | Posts feedback, adds `ready-to-refine` label for revision | | ||
| | `revise` + at round limit | Escalates to human, adds `refine-needs-human` and `refine-escalated` labels | | ||
| | `needs_input` | Posts question, adds `refine-needs-input` label | | ||
|
|
||
| ## Control labels | ||
|
|
||
| These labels are managed by the critique pipeline: | ||
|
|
||
| | Label | Meaning | | ||
| |-------|---------| | ||
| | `refine-approved` | Plan approved; ready for human review or auto-creation | | ||
| | `ready-to-refine` | Revisions requested; refine agent should re-run | | ||
| | `refine-revision-round-N` | Tracks which revision round the pipeline is on | | ||
| | `refine-needs-input` | Human must answer a question before proceeding | | ||
| | `refine-needs-human` | Max review rounds reached; human decision needed | | ||
| | `refine-escalated` | Distinguishes max-rounds escalation from genuine approval | | ||
|
|
||
| ## Child issue creation | ||
|
|
||
| When the critique agent approves a plan and `AUTO_CREATE=true`, the post-script sources `create-children.sh` to create child issues. This script: | ||
|
|
||
| - Creates issues in topological order (parents before children) | ||
| - Supports both GitHub (sub-issues API) and Jira (parent-child hierarchy with fallback to "Relates" links) | ||
| - Deduplicates against existing children to prevent double-creation on re-runs | ||
| - Resolves Jira issue types against the project's available types | ||
| - Handles cross-project creation when a routing skill provides `target_project` | ||
|
|
||
| ## Source | ||
|
|
||
| [`harness/critique.yaml`](../harness/critique.yaml) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| export ISSUE_CONTEXT=/sandbox/workspace/issue-context.json | ||
| export EXPLORE_CONTEXT=/sandbox/workspace/exploration_context.json | ||
| export REFINE_RESULT=/sandbox/workspace/refine-result.json | ||
| export CRITIQUE_HISTORY=/sandbox/workspace/critique-history.json | ||
| export PROJECT_ROUTING=/sandbox/workspace/routing-skill.md | ||
| export PLATFORM_CONTEXT=/sandbox/workspace/platform-context.md |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| --- | ||
| agent: agents/critique.md | ||
| doc: docs/critique.md | ||
| model: opus | ||
| image: ghcr.io/fullsend-ai/fullsend-sandbox:latest | ||
| policy: policies/critique.yaml | ||
|
|
||
|
qodo-code-review[bot] marked this conversation as resolved.
|
||
| role: critique | ||
| slug: fullsend-ai-critique | ||
|
|
||
| host_files: | ||
| - src: common/env/gcp-vertex.env | ||
| dest: /sandbox/workspace/.env.d/gcp-vertex.env | ||
| expand: true | ||
| - src: env/critique.env | ||
| dest: /sandbox/workspace/.env.d/critique.env | ||
| expand: true | ||
| - src: ${GOOGLE_APPLICATION_CREDENTIALS} | ||
| dest: /tmp/.gcp-credentials.json | ||
| - src: ${GCP_OIDC_TOKEN_FILE} | ||
| dest: /sandbox/workspace/.gcp-oidc-token | ||
| optional: true | ||
| - src: /tmp/workspace/issue-context.json | ||
| dest: /sandbox/workspace/issue-context.json | ||
| optional: true | ||
| - src: /tmp/workspace/exploration_context.json | ||
| dest: /sandbox/workspace/exploration_context.json | ||
| optional: true | ||
| - src: /tmp/workspace/refine-result.json | ||
| dest: /sandbox/workspace/refine-result.json | ||
| optional: true | ||
| - src: /tmp/workspace/critique-history.json | ||
| dest: /sandbox/workspace/critique-history.json | ||
| optional: true | ||
| - src: /tmp/workspace/routing-skill.md | ||
| dest: /sandbox/workspace/routing-skill.md | ||
| optional: true | ||
| - src: /tmp/workspace/platform-context.md | ||
| dest: /sandbox/workspace/platform-context.md | ||
| optional: true | ||
|
|
||
| pre_script: scripts/pre-critique.sh | ||
| post_script: scripts/post-critique.sh | ||
|
|
||
| validation_loop: | ||
| script: scripts/validate-output-schema.sh | ||
| schema: schemas/critique-result.schema.json | ||
| max_iterations: 2 | ||
|
|
||
| timeout_minutes: 15 | ||
|
|
||
| env: | ||
| runner: | ||
| ISSUE_KEY: "${ISSUE_KEY}" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [medium] contract-violation FULLSEND_OUTPUT_FILE set to critique-result.json in env.runner but agent prompt writes agent-result.json and post-script searches for agent-result.json. Currently benign but creates a latent inconsistency. Suggested fix: Remove the FULLSEND_OUTPUT_FILE override or align the agent prompt to write critique-result.json.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed earlier (6d0c112): removed |
||
| ISSUE_SOURCE: "${ISSUE_SOURCE}" | ||
| REPO_FULL_NAME: "${REPO_FULL_NAME}" | ||
| REFINE_RUN_ID: "${REFINE_RUN_ID}" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [medium] excessive-privilege GH_TOKEN passed to sandbox via env.sandbox but agent reads pre-staged context files and does not need live GitHub API access. Violates least privilege. Suggested fix: Remove GH_TOKEN from env.sandbox.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed in 6d0c112: removed
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed earlier (6d0c112): removed |
||
| GITHUB_ISSUE_NUMBER: "${GITHUB_ISSUE_NUMBER}" | ||
| REVIEW_ROUND: "${REVIEW_ROUND}" | ||
| MAX_REVIEW_ROUNDS: "${MAX_REVIEW_ROUNDS}" | ||
| AUTO_CREATE: "${AUTO_CREATE}" | ||
| sandbox: | ||
| REVIEW_ROUND: "${REVIEW_ROUND}" | ||
| MAX_REVIEW_ROUNDS: "${MAX_REVIEW_ROUNDS}" | ||
|
qodo-code-review[bot] marked this conversation as resolved.
|
||
|
|
||
| forge: | ||
| github: | ||
| pre_script: scripts/pre-critique.sh | ||
| post_script: scripts/post-critique.sh | ||
| env: | ||
| runner: | ||
| GH_TOKEN: "${GH_TOKEN}" | ||
| PUSH_TOKEN: "${PUSH_TOKEN}" | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| --- | ||
| version: 1 | ||
|
|
||
| # Sandbox policy for the critique agent. | ||
| # | ||
| # Read-only agent: needs GitHub API for issue context and Jira for | ||
| # attachment retrieval. No write access — the post-script handles | ||
| # mutations on the runner. | ||
|
|
||
| filesystem_policy: | ||
| include_workdir: true | ||
| read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log] | ||
| read_write: [/sandbox, /tmp, /dev/null] | ||
| landlock: | ||
| compatibility: best_effort | ||
| process: | ||
| run_as_user: sandbox | ||
| run_as_group: sandbox | ||
|
|
||
| network_policies: | ||
| vertex_ai: | ||
| name: vertex-ai | ||
| endpoints: | ||
| - host: "api.anthropic.com" | ||
| port: 443 | ||
| protocol: rest | ||
| enforcement: enforce | ||
| access: read-write | ||
| - host: "*.googleapis.com" | ||
| port: 443 | ||
| protocol: rest | ||
| enforcement: enforce | ||
| access: read-write | ||
| binaries: | ||
| - path: "**/claude" | ||
| - path: "**/node" | ||
|
|
||
| github_api: | ||
| name: github-api | ||
| endpoints: | ||
| - host: "api.github.com" | ||
| port: 443 | ||
| protocol: rest | ||
| enforcement: enforce | ||
| access: read-only | ||
| - host: "github.com" | ||
| port: 443 | ||
| protocol: rest | ||
| enforcement: enforce | ||
| access: read-only | ||
| - host: "raw.githubusercontent.com" | ||
| port: 443 | ||
| protocol: rest | ||
| enforcement: enforce | ||
| access: read-only | ||
| binaries: | ||
| - path: "**/gh" | ||
| - path: "**/git" | ||
| - path: "**/node" | ||
|
|
||
| jira_api: | ||
| name: jira-api | ||
| endpoints: | ||
| - host: "*.atlassian.net" | ||
| port: 443 | ||
| protocol: rest | ||
| enforcement: enforce | ||
| access: read-only | ||
|
qodo-code-review[bot] marked this conversation as resolved.
|
||
| binaries: | ||
| - path: "**/curl" | ||
| - path: "**/node" | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this to the harness, then
env.sandbox.