Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ script-test:
$(call run-timed,bash scripts/post-fix-test.sh)
$(call run-timed,bash scripts/post-retro-test.sh)
$(call run-timed,bash scripts/post-scribe-test.sh)
$(call run-timed,bash scripts/post-critique-test.sh)
$(call run-timed,bash scripts/validate-output-schema-test.sh)

test: script-test
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ First-class agents for the [fullsend](https://github.com/fullsend-ai/fullsend) p
| **Review** | Dispatches parallel sub-agents across six review dimensions | PR events, `/fs-review` |
| **Fix** | Implements targeted fixes from review feedback | Review comments, `/fs-fix` |
| **Prioritize** | Scores issues using the RICE framework | Schedule, `/fs-prioritize` |
| **Critique** | Adversarial reviewer and quality gate for refinement plans | `ready-to-critique` label |
| **Retro** | Analyzes completed workflows and proposes improvements | PR close, `/fs-retro` |

See [`docs/`](docs/) for detailed documentation on each agent.
Expand Down
423 changes: 423 additions & 0 deletions agents/critique.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ agents:
- source: harness/review.yaml
- source: harness/retro.yaml
- source: harness/prioritize.yaml
- source: harness/critique.yaml
71 changes: 71 additions & 0 deletions docs/critique.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Critique Agent

Adversarial reviewer and quality gate for refinement plans. Evaluates a proposed decomposition from the refine agent and decides: approve (ready for issue creation), revise (send back to refine), or needs_input (escalate to a human for clarification).

> **Note**: The explore and refine agents are added in separate PRs. Cross-references to their docs pages will resolve once all three PRs merge.

## How the agent works

The critique agent runs after the refine agent has produced a decomposition plan. It reads the original work item, exploration context, the proposed plan, and any prior critique history (for revision rounds). It evaluates the plan across seven dimensions — coverage, granularity, dependency coherence, implementability, scope accuracy, assumption grounding, and description clarity — and produces a structured verdict.

The agent runs in a read-only sandbox. It cannot modify issues or create children directly — those mutations are performed by the post-script on the runner. The sandbox policy permits read-only egress to GitHub and Jira APIs plus inference endpoints, but the agent's only output is a structured JSON critique result consumed by the post-script, which posts a summary comment, attaches feedback, and applies labels to signal the next pipeline step.

## How it helps

- Prevents over-decomposition (15 issues when 6 would suffice) and under-decomposition (one giant epic).
- Catches scope creep — children that exceed what the parent asked for.
- Validates that acceptance criteria are testable, not vague ("should be fast").
- Detects assumption laundering — plans that look confident but are built on unverified guesses.
- Creates child issues automatically on approval when `AUTO_CREATE=true`.

## Commands

| Command | Where | Effect |
|---------|-------|--------|
| `/fs-create` | Issue comment | Creates child issues from an approved plan |
| `/fs-refine` | Issue comment | Restarts the refinement pipeline |

## Pipeline integration

The critique agent is the third stage of a three-stage refinement pipeline:

1. **Explore** — gathers technical context
2. **Refine** — decomposes the work item
3. **Critique** — reviews and approves/revises the decomposition (this agent)

### Verdict outcomes

| Verdict | Post-script action |
|---------|-------------------|
| `approved` + `AUTO_CREATE=true` | Creates child issues immediately via `create-children.sh` |
| `approved` + `AUTO_CREATE=false` | Posts approval, adds `refine-approved` label for human gate |
| `revise` + under round limit | Posts feedback, adds `ready-to-refine` label for revision |
| `revise` + at round limit | Escalates to human, adds `refine-needs-human` and `refine-escalated` labels |
| `needs_input` | Posts question, adds `refine-needs-input` label |

## Control labels

These labels are managed by the critique pipeline:

| Label | Meaning |
|-------|---------|
| `refine-approved` | Plan approved; ready for human review or auto-creation |
| `ready-to-refine` | Revisions requested; refine agent should re-run |
| `refine-revision-round-N` | Tracks which revision round the pipeline is on |
| `refine-needs-input` | Human must answer a question before proceeding |
| `refine-needs-human` | Max review rounds reached; human decision needed |
| `refine-escalated` | Distinguishes max-rounds escalation from genuine approval |

## Child issue creation

When the critique agent approves a plan and `AUTO_CREATE=true`, the post-script sources `create-children.sh` to create child issues. This script:

- Creates issues in topological order (parents before children)
- Supports both GitHub (sub-issues API) and Jira (parent-child hierarchy with fallback to "Relates" links)
- Deduplicates against existing children to prevent double-creation on re-runs
- Resolves Jira issue types against the project's available types
- Handles cross-project creation when a routing skill provides `target_project`

## Source

[`harness/critique.yaml`](../harness/critique.yaml)
6 changes: 6 additions & 0 deletions env/critique.env

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this to the harness, then env.sandbox.

Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
export ISSUE_CONTEXT=/sandbox/workspace/issue-context.json
export EXPLORE_CONTEXT=/sandbox/workspace/exploration_context.json
export REFINE_RESULT=/sandbox/workspace/refine-result.json
export CRITIQUE_HISTORY=/sandbox/workspace/critique-history.json
export PROJECT_ROUTING=/sandbox/workspace/routing-skill.md
export PLATFORM_CONTEXT=/sandbox/workspace/platform-context.md
73 changes: 73 additions & 0 deletions harness/critique.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
agent: agents/critique.md
doc: docs/critique.md
model: opus
image: ghcr.io/fullsend-ai/fullsend-sandbox:latest
policy: policies/critique.yaml

Comment thread
qodo-code-review[bot] marked this conversation as resolved.
role: critique
slug: fullsend-ai-critique

host_files:
- src: common/env/gcp-vertex.env
dest: /sandbox/workspace/.env.d/gcp-vertex.env
expand: true
- src: env/critique.env
dest: /sandbox/workspace/.env.d/critique.env
expand: true
- src: ${GOOGLE_APPLICATION_CREDENTIALS}
dest: /tmp/.gcp-credentials.json
- src: ${GCP_OIDC_TOKEN_FILE}
dest: /sandbox/workspace/.gcp-oidc-token
optional: true
- src: /tmp/workspace/issue-context.json
dest: /sandbox/workspace/issue-context.json
optional: true
- src: /tmp/workspace/exploration_context.json
dest: /sandbox/workspace/exploration_context.json
optional: true
- src: /tmp/workspace/refine-result.json
dest: /sandbox/workspace/refine-result.json
optional: true
- src: /tmp/workspace/critique-history.json
dest: /sandbox/workspace/critique-history.json
optional: true
- src: /tmp/workspace/routing-skill.md
dest: /sandbox/workspace/routing-skill.md
optional: true
- src: /tmp/workspace/platform-context.md
dest: /sandbox/workspace/platform-context.md
optional: true

pre_script: scripts/pre-critique.sh
post_script: scripts/post-critique.sh

validation_loop:
script: scripts/validate-output-schema.sh
schema: schemas/critique-result.schema.json
max_iterations: 2

timeout_minutes: 15

env:
runner:
ISSUE_KEY: "${ISSUE_KEY}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] contract-violation

FULLSEND_OUTPUT_FILE set to critique-result.json in env.runner but agent prompt writes agent-result.json and post-script searches for agent-result.json. Currently benign but creates a latent inconsistency.

Suggested fix: Remove the FULLSEND_OUTPUT_FILE override or align the agent prompt to write critique-result.json.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed earlier (6d0c112): removed FULLSEND_OUTPUT_FILE override — all paths use agent-result.json.

ISSUE_SOURCE: "${ISSUE_SOURCE}"
REPO_FULL_NAME: "${REPO_FULL_NAME}"
REFINE_RUN_ID: "${REFINE_RUN_ID}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] excessive-privilege

GH_TOKEN passed to sandbox via env.sandbox but agent reads pre-staged context files and does not need live GitHub API access. Violates least privilege.

Suggested fix: Remove GH_TOKEN from env.sandbox.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6d0c112: removed GH_TOKEN from env.sandbox.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed earlier (6d0c112): removed GH_TOKEN from env.sandbox.

GITHUB_ISSUE_NUMBER: "${GITHUB_ISSUE_NUMBER}"
REVIEW_ROUND: "${REVIEW_ROUND}"
MAX_REVIEW_ROUNDS: "${MAX_REVIEW_ROUNDS}"
AUTO_CREATE: "${AUTO_CREATE}"
sandbox:
REVIEW_ROUND: "${REVIEW_ROUND}"
MAX_REVIEW_ROUNDS: "${MAX_REVIEW_ROUNDS}"
Comment thread
qodo-code-review[bot] marked this conversation as resolved.

forge:
github:
pre_script: scripts/pre-critique.sh
post_script: scripts/post-critique.sh
env:
runner:
GH_TOKEN: "${GH_TOKEN}"
PUSH_TOKEN: "${PUSH_TOKEN}"
71 changes: 71 additions & 0 deletions policies/critique.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
version: 1

# Sandbox policy for the critique agent.
#
# Read-only agent: needs GitHub API for issue context and Jira for
# attachment retrieval. No write access — the post-script handles
# mutations on the runner.

filesystem_policy:
include_workdir: true
read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log]
read_write: [/sandbox, /tmp, /dev/null]
landlock:
compatibility: best_effort
process:
run_as_user: sandbox
run_as_group: sandbox

network_policies:
vertex_ai:
name: vertex-ai
endpoints:
- host: "api.anthropic.com"
port: 443
protocol: rest
enforcement: enforce
access: read-write
- host: "*.googleapis.com"
port: 443
protocol: rest
enforcement: enforce
access: read-write
binaries:
- path: "**/claude"
- path: "**/node"

github_api:
name: github-api
endpoints:
- host: "api.github.com"
port: 443
protocol: rest
enforcement: enforce
access: read-only
- host: "github.com"
port: 443
protocol: rest
enforcement: enforce
access: read-only
- host: "raw.githubusercontent.com"
port: 443
protocol: rest
enforcement: enforce
access: read-only
binaries:
- path: "**/gh"
- path: "**/git"
- path: "**/node"

jira_api:
name: jira-api
endpoints:
- host: "*.atlassian.net"
port: 443
protocol: rest
enforcement: enforce
access: read-only
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
binaries:
- path: "**/curl"
- path: "**/node"
Loading
Loading