Skip to content

[examples] feat: blackbox mini-swe-agent training recipe - #86

Open
zhaizhiqiangA wants to merge 9 commits into
verl-project:mainfrom
zhaizhiqiangA:blackbox-recipe-sync-upstream
Open

[examples] feat: blackbox mini-swe-agent training recipe#86
zhaizhiqiangA wants to merge 9 commits into
verl-project:mainfrom
zhaizhiqiangA:blackbox-recipe-sync-upstream

Conversation

@zhaizhiqiangA

Copy link
Copy Markdown
Collaborator

What does this PR do?

This PR adds a blackbox RL training recipe for mini-swe-agent under examples/blackbox_recipes/. The agent runs entirely inside an remote sandbox via a sidecar tool-image mount: the host-side runner creates the sandbox, pipes the task config to the in-sandbox agent over stdin, parses the result from stdout, and evaluates the reward in the same sandbox. The agent reaches the LLM through the gateway via an upstream tunnel, so training is fully "blackbox" — the trainer only sees prompts/responses through the gateway. Training uses the V1 unified trainer (Megatron backend, GRPO, separate_async).
Related work:

[train] feat: add blackbox agent gateway (#25) — the gateway this recipe runs against

Checklist Before Starting

  • Search for similar PRs or issues and paste at least one relevant link here:
    gh pr list --repo verl-project/uni-agent --state open --search "mini-swe-agent"
    No pull requests match your search in verl-project/uni-agent
  • Format the PR title as [examples] feat: blackbox mini-swe-agent training recipe

Test

A full RL training recipe is not practical to cover in CI, so validation was manual:

  • Inference smoke test — ran a single sample end-to-end against an remote sandbox.
  • Short training run — bash examples/blackbox_recipes/scripts/run_train.sh with the V1 separate_async trainer on a single 8-GPU node (4 trainer + 4 rollout).

API and Usage Example

This PR only adds files under examples/ plus minor internal import-path updates; there are no public API changes.

Design & Code Changes

New recipe — examples/blackbox_recipes/mini_swe_agent/

mini_swe_agent_runner.py — host-side runner. Creates a YRSandbox with the sidecar mounted at /opt/mini-swe-agent, base64-encodes the task config (task text + tunnel-rewritten gateway URL + step limit) and pipes it to run_agent.py via stdin, parses the JSON result from stdout (robust to litellm noise), then evaluates the reward in the same sandbox via SandboxEnvForReward and POSTs reward_info. Sandbox is always cleaned up in finally.
run_agent.py — in-sandbox entrypoint. Builds a LocalEnvironment + LitellmModel (pointed at the gateway tunnel) + DefaultAgent from mini-swe-agent's SWE-bench defaults, runs the task, emits a result JSON.
Dockerfile.mini-swe-agent-tool — self-contained, glibc-portable sidecar image (FROM scratch) so the sandbox base image needs no Python/Node.
dataset.py (SWEBenchDataset) injects verl-standard reward fields; reward.py reuses the uni_agent reward-spec registry to score resolved/unresolved in-env.
config/swe_agent_blackbox_megatron_v1.yaml + scripts/run_train.sh — V1 unified trainer, separate_async by default (4 GPU trainer + 4 GPU rollout on one node), vLLM async rollout, GRPO, Megatron offload.

Checklist Before Submitting

  • Read the Contribute Guide
  • Run pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
  • Add or update docs/examples for user-facing changes
  • Add tests or explain why tests are not practical
  • Confirm the PR title matches the required format
  • Confirm the placeholder text in this template has been replaced with real content

zhaizhiqiangA and others added 7 commits June 26, 2026 03:32
add debug

use global_steps fill min_global_steps & max_global_steps

add infer

Add agent framework concurrency diagnostics

Stabilize blackbox rollout logging

npu config
fix reward bug && npu config
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@zhaizhiqiangA
zhaizhiqiangA force-pushed the blackbox-recipe-sync-upstream branch from 5e314b1 to d51e978 Compare July 23, 2026 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants