You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
AgentSWE is a Platform benchmark toolkit for building, exporting, and evaluating software engineering agent tasks. It extends SWE-Forge with a Cursor-style synthetic task pipeline: start from a real repository, delete a testable feature, then ask an agent to restore the behavior using tests as the reward signal.
16
+
Agent-SWE turns real repositories into benchmark tasks for autonomous software engineering agents. It keeps the parts that make coding work hard in practice: existing project structure, real tests, install commands, patches, Docker evaluation, and a clear fail-to-pass scoring contract.
17
17
18
-
The project can mine real GitHub pull requests, generate synthetic feature-deletion tasks, export benchmark workspaces, and run Docker-based evaluation against gold or model-generated patches.
18
+
The synthetic task pipeline is inspired by Cursor's public writing on Composer, Composer 2, and Composer 2.5. Cursor described training coding agents on tasks grounded in real codebases, including a feature-deletion style setup: remove a testable behavior, ask the agent to restore it, and use tests as the reward signal. Agent-SWE adapts that idea into an open benchmark-generation workflow for Platform agents.
19
19
20
-
## What Agent SWE Does
20
+
This project is not affiliated with Cursor. It is an implementation inspired by the public methodology described in their posts and reports.
21
21
22
-
Agent SWE creates reproducible benchmark tasks for autonomous coding agents:
22
+
## Why Agent-SWE Exists
23
23
24
-
1. Select a real repository or pull request.
25
-
2. Discover install and test commands.
26
-
3. Generate or select fail-to-pass and pass-to-pass tests.
27
-
4. Export a task workspace with hidden solution artifacts.
28
-
5. Evaluate a gold patch or model patch in an isolated environment.
29
-
6. Produce task scores that can be consumed by Platform challenge validators.
24
+
Most coding benchmarks are either real but scarce, or synthetic but too detached from real development. Agent-SWE aims for the middle ground: tasks are synthetic enough to scale, but grounded enough that agents still need to inspect a real repository, understand context, edit code, and run tests.
30
25
31
-
## Key Features
26
+
A good Agent-SWE task should answer three questions:
32
27
33
-
- Real PR mining from GH Archive and GitHub metadata.
34
-
- Synthetic Cursor-style feature deletion tasks.
35
-
- Docker verification for fail-to-pass and pass-to-pass tests.
36
-
- Workspace export with `workspace.yaml`, `patch.diff`, tests, and optional `deletion_patch.diff`.
37
-
- JSONL and Parquet exports for benchmark datasets.
38
-
- Simple CLI commands for mining, synthetic generation, and evaluation.
28
+
1. Can the agent understand the existing codebase?
29
+
2. Can it restore the intended behavior without seeing the oracle patch?
30
+
3. Can the result pass both targeted reward tests and regression tests?
39
31
40
-
## Evaluation Flow
32
+
## Inspired by Cursor Composer
33
+
34
+
Cursor's Composer work is the main public inspiration for the synthetic path in Agent-SWE:
35
+
36
+
-[Composer: Building a fast frontier model with RL](https://cursor.com/blog/composer)
The important idea is simple: instead of only collecting issues and pull requests, generate new tasks from real repositories. In the feature-deletion variant, a known behavior is removed from the codebase, the inverse patch becomes the oracle solution, and tests define whether the agent recovered the behavior.
43
+
44
+
Agent-SWE currently implements this idea for Python functions and methods. It keeps the public signature, replaces the body with a synthetic failure, writes that mutation to `deletion_patch.diff`, and stores the inverse repair as `patch.diff`.
45
+
46
+
## What Agent-SWE Does
47
+
48
+
Agent-SWE supports two sources of benchmark tasks:
49
+
50
+
1.**Real pull requests** mined from GitHub and converted into SWE-style workspaces.
51
+
2.**Synthetic feature-deletion tasks** generated from real repositories, inspired by the public Composer 2.5 training method.
52
+
53
+
Both flows export a workspace that can be evaluated in Docker. The agent being tested should never see the oracle patch or hidden benchmark files.
41
54
42
55
```mermaid
43
56
flowchart LR
44
-
Repo[Real repo or PR] --> Task[Task generation]
45
-
Task --> Tests[Test discovery]
46
-
Tests --> Export[Workspace export]
47
-
Export --> Eval[Docker evaluation]
48
-
Eval --> Score[Task score]
49
-
Score --> Platform[Platform weights]
57
+
Repo[Real repo] --> Build[Build task]
58
+
Build --> Export[Export workspace]
59
+
Export --> Run[Docker eval]
60
+
Run --> Score[Task score]
61
+
Score --> Plat[Platform]
50
62
```
51
63
52
-
For synthetic tasks, Agent SWE applies `deletion_patch.diff` first, verifies that reward tests fail, applies the candidate or oracle patch, and then verifies that reward and regression tests pass.
When `--output-folder`is used, tasks are exported as directories:
154
+
A task workspace is the portable benchmark unit:
137
155
138
156
```text
139
157
tasks/
@@ -147,44 +165,24 @@ tasks/
147
165
└── evaluate.sh
148
166
```
149
167
150
-
`patch.diff` is the oracle solution and must be hidden from agents. `deletion_patch.diff` exists only for synthetic feature-deletion tasks and is applied before evaluation.
151
-
152
-
Example `workspace.yaml`:
153
-
154
-
```yaml
155
-
task_id: owner-repo-1234
156
-
repo:
157
-
url: https://github.com/owner/repo.git
158
-
base_commit: abc123def456
159
-
merge_commit: abc123def456
160
-
language: python
161
-
prompt: Restore the deleted behavior for `target_function`.
162
-
install:
163
-
commands:
164
-
- pip install -e .
165
-
tests:
166
-
fail_to_pass:
167
-
- pytest tests/test_target.py -v
168
-
pass_to_pass:
169
-
- pytest tests/ -v
170
-
synthetic:
171
-
source_type: synthetic_feature_deletion
172
-
deletion_patch_file: deletion_patch.diff
173
-
strategy: feature_deletion
174
-
```
168
+
The files have different audiences:
169
+
170
+
-`workspace.yaml` describes the task, repo, install commands, tests, and synthetic metadata.
171
+
-`patch.diff` is the oracle solution and must be hidden from the evaluated agent.
172
+
-`deletion_patch.diff` is the synthetic mutation applied before evaluation.
173
+
-`tests/` contains generated or extracted benchmark tests.
174
+
-`evaluate.sh` is a simple local scoring script.
175
175
176
-
## Synthetic Task Method
176
+
For details, read [docs/architecture/workspace-format.md](docs/architecture/workspace-format.md).
177
177
178
-
The synthetic pipeline follows the public Cursor Composer 2.5-style feature-deletion method:
178
+
## Documentation
179
179
180
-
1. Start from a real repository with tests.
181
-
2. Replace a target Python function or method body with a synthetic failure.
182
-
3. Store the mutation as `deletion_patch.diff`.
183
-
4. Store the inverse patch as `patch.diff`.
184
-
5. Use supplied tests as the reward contract.
185
-
6. Export the task for isolated evaluation.
180
+
The architecture docs explain how the pieces fit together:
186
181
187
-
This makes the benchmark grounded in real code while preserving an objective fail-to-pass signal.
AgentSWE is designed to feed Platform challenge validators with deterministic repository-repair tasks. Validators can use exported workspaces or JSONL predictions to run isolated evaluations and convert task completion rates into raw challenge scores.
216
+
Agent-SWE is designed to feed Platform challenge validators with deterministic repository-repair tasks. Validators can sample tasks, run agent patches in isolated workspaces, and turn task completion rates into raw challenge scores for Platform.
Agent-SWE is a benchmark-generation and evaluation toolkit for software engineering agents. It builds task workspaces from real repositories, then evaluates whether an agent can produce a patch that makes the right tests pass.
4
+
5
+
The system has two task sources:
6
+
7
+
1.**Real PR tasks**: mined from GitHub pull requests and exported in a SWE-style format.
8
+
2.**Synthetic feature-deletion tasks**: generated from real repositories by deleting a known behavior and asking the agent to restore it.
9
+
10
+
The synthetic path is inspired by Cursor's public descriptions of how Composer 2 and Composer 2.5 use synthetic, codebase-grounded tasks for agent training.
11
+
12
+
## High-Level Flow
13
+
14
+
```mermaid
15
+
flowchart LR
16
+
Repo[Real repo] --> Gen[Task generation]
17
+
Gen --> Del[Deletion patch]
18
+
Del --> Export[Workspace]
19
+
Export --> Eval[Evaluation]
20
+
Eval --> Score[Score]
21
+
Score --> Plat[Platform]
22
+
```
23
+
24
+
Legend:
25
+
26
+
-**Real repo**: a real GitHub repository or local checkout.
27
+
-**Task generation**: mining, command discovery, or synthetic task creation.
28
+
-**Deletion patch**: only used for synthetic feature-deletion tasks.
3.`synthetic.feature_deletion.build_python_function_deletion` parses the source file and replaces the target body with a synthetic failure.
51
+
4. The mutation is saved as `deletion_patch.diff`.
52
+
5. The inverse mutation is saved as `patch.diff`, the oracle solution.
53
+
6.`export.workspace.export_task_to_workspace` writes the workspace directory.
54
+
7. The evaluation scripts apply the deletion patch before testing model patches.
55
+
56
+
## Evaluation Flow
57
+
58
+
For a synthetic task, the evaluator checks three things:
59
+
60
+
1. After `deletion_patch.diff`, `fail_to_pass` tests must fail.
61
+
2. After applying the candidate patch, `fail_to_pass` tests must pass.
62
+
3.`pass_to_pass` tests must also pass, so the candidate did not break unrelated behavior.
63
+
64
+
For a mined PR task, the same fail-to-pass idea applies, but there is no synthetic deletion patch. The repository starts at the base commit and the candidate patch is expected to repair the original issue.
65
+
66
+
## Why This Shape Works
67
+
68
+
This architecture gives Agent-SWE a practical balance:
69
+
70
+
- real codebases keep tasks realistic;
71
+
- synthetic deletion makes task generation scalable;
72
+
- Docker keeps evaluation isolated;
73
+
- fail-to-pass tests create a clear scoring contract;
74
+
- workspace exports make tasks portable for validators and offline experiments.
75
+
76
+
## Cursor References
77
+
78
+
The synthetic approach is inspired by public Cursor material:
79
+
80
+
-[Composer: Building a fast frontier model with RL](https://cursor.com/blog/composer)
0 commit comments