English | 中文
Reinforcement learning for a hybrid action space — low-level GUI control (pyautogui) and high-level MCP tool calls — on the OSWorld computer-use environment, evaluated on the OSWorld-MCP benchmark (309 tasks), using Qwen3-VL-8B-Thinking as the policy.
This is the code companion to our paper Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI–MCP Computer-Use Agents (arXiv:2608.03327). It contains the full inference, evaluation, and RL-training pipeline, the experiment configs behind every table, and the task splits. Model weights, the VM image, and the MCP tool set are external — see Setup.
Scope note. Two levels of results. (1) Action level (behavior is steerable, competence is not): a dense tool bonus raises spreadsheet tool-adoption 0.03 → 0.33 and this carries into greedy decoding, but held-out accuracy does not follow. The bottleneck is tool-call semantics, not adoption. (2) Context level (a concrete efficiency win): a successful tool call often makes the next screenshot redundant. Dropping it and halving image history cuts input tokens by about a third at a small accuracy cost; retraining under the same observation rule removes that cost. The compressed agent reaches 37.8% vs 33.0% for the uncompressed operating point, at 53% of the input cost, and closes the rich–lean gap on a pre-registered degraded subset to zero (in-distribution).
Takeaway: at the action level, tools are available but rarely called, and calls often fail semantically. At the context level, the tool result is already in context as text, so the following screenshot is redundant, yet it is kept by default. Both trace back to the same cause: training provides no signal for either. Full per-checkpoint numbers, configs, and provenance for every figure above are in
results/context_rl/PROVENANCE.mdandresults/dense_bonus/; see alsoREPRODUCE.md.
Same task, same model — the only difference is whether MCP tools are exposed. Pure GUI burns all 50 steps scrolling a font list; the hybrid agent selects all, makes one tool call, and finishes in 3 steps:
A complete successful hybrid rollout ("add page number for every page at the bottom left", LibreOffice Writer) — GUI steps to reach the footer, then one MCP call finishes the job:
- Check a paper number →
REPRODUCE.mdmaps every claim to a command and its result artifact. - Run something → Quickstart below; every RL experiment is one YAML under
configs/experiments/. - Confused by the split files → the TL;DR table in
data/README.md; at runtime only three sets matter (train 74 / held-out 48 / eval 309). - Building on this repo, with or without an AI coding agent →
AGENTS.mdis the repo map for coding agents: architecture, invariants, footguns, and the cheap-feedback-loop ladder. Recommended workflow: treatREPRODUCE.mdas the spec, express changes as a new YAML underconfigs/experiments/(never edit the base config in place), and validate with compileall → smoke test → probe before any full run.
osworld_rl.py # top-level RL orchestrator (serve→rollout→reward→preproc→train)
train/ # accelerate + DeepSpeed training worker (osworld_train.py, preproc)
reward/ # reward / advantage computation (outcome, dense MCP bonus, pos-adv-only)
agents/ # BM25 tool retriever
prompts/ # policy / reward prompt templates
tools/ # tools_registry.json (MCP tool corpus the retriever ranks)
mcp_tools/ # vendored MCP server + tools (injected into the VM; the paper tool set)
configs/
osworld_rl.yaml # base config (paths via ${oc.env:...})
experiments/ # one YAML per paper run (RL run series + probes/profiles)
env.example.sh # machine-specific env vars (copy to env.sh, source it)
data/splits/ # task-id splits + metadata
scripts/ # run_mcp_eval.sh, run_puregui_eval.sh, prepare_splits_v2.py, ...
OSWorld-main/ # vendored, modified OSWorld (see OSWorld-main/PATCHES.md)
docs/ # ENVIRONMENT.md + research notes
results/outcome_only/, results/context_rl/ # paper numbers, per-checkpoint cells, provenance, recompute
Our setup (the source of every number in the paper): 8× 80 GB-class GPUs, Docker with /dev/kvm, a fast local disk for scratch, Python 3.10. The scripts assume this layout (8 vLLM instances + 96 parallel VMs); Qwen3-VL-8B fits on a single GPU, so fewer cards also work — lower the instance count and NUM_ENVS accordingly, at reduced parallelism.
# 1. environment
conda env create -f setup/environment_rlanything.yml # or: pip install -r requirements.txt
pip install -r OSWorld-main/requirements.txt
# 2. machine-specific paths / secrets (nothing is hardcoded)
cp configs/env.example.sh env.sh && $EDITOR env.sh && source env.sh
cp OSWorld-main/.env.example OSWorld-main/.env # local vLLM: key can stay "dummy"
# 3. external assets (not in this repo) — see docs/ENVIRONMENT.md
# - model: Qwen/Qwen3-VL-8B-Thinking -> $MODEL_DIR/
# - VM: happysixd/osworld-docker + patched Ubuntu-MCP.qcow2 -> /dev/shm/
# - MCP: ships in-repo (mcp_tools/); set MCP_SRC_ROOT=0 for stock OSWorld-MCPSee docs/ENVIRONMENT.md for the Docker+QEMU provider, the MCP-in-VM architecture, the patched VM image, and hardware notes.
Two RL policy checkpoints are released on the HuggingFace Hub. Each is a complete, directly-loadable Qwen3-VL-8B model directory (config + model.safetensors + tokenizer/processor).
| Checkpoint | Run | What it is | HF repo |
|---|---|---|---|
| outcome-only | outcome_only_rl |
pure outcome-only RLVR — tool adoption 0.02→0.33 but held-out flat (results/outcome_only) | redai-infra/hybrid-routing-outcome-only |
| context-RL | context_rl |
train=eval context-compression RL — epoch-40 paper operating point, 37.8% @ 51% input cost, img2 + skip_on_mcp_success (results/context_rl) |
redai-infra/hybrid-routing-context-rl |
# download
huggingface-cli download redai-infra/hybrid-routing-context-rl --local-dir ckpts/context_rl
# evaluate a checkpoint (point MODEL at the downloaded dir)
MODEL=ckpts/context_rl bash scripts/run_mcp_eval.shSmoke test (a few tasks, end-to-end):
TEST_META=evaluation_examples/smoke_test.json bash scripts/run_mcp_eval.shReproduce the two inference baselines on the 309-task set:
# B1 — pure GUI (pyautogui, no MCP) -> overall 30.5% (5-run mean)
bash scripts/run_puregui_eval.sh
python OSWorld-main/show_result.py --result_dir baselines/b1_puregui_thinking
# B2 — GUI + MCP (tool retrieval + MCP calls) -> overall 34.5% (5-run mean; best single run 37.9%)
bash scripts/run_mcp_eval.sh
python OSWorld-main/show_result.py --result_dir baselines/b2_mcp_thinkingAdd MODEL_TYPE=instruct to either for the Instruct model (B1/B2-Instruct).
Inference context-policy sweep (baseline / skip_on_mcp_success / skip_on_no_change, img4 / img2) runs via env vars on the same script, e.g. CONTEXT_POLICY=skip_on_mcp_success MAX_IMAGE_HISTORY_LENGTH=2 bash scripts/run_mcp_eval.sh — see REPRODUCE.md §A.
Run an RL experiment (Context-Compression RL):
OSWORLD_LOCAL_TEMP="$OSWORLD_LOCAL_TEMP" TQDM_DISABLE=1 \
nohup python osworld_rl.py config=configs/experiments/context_rl.yaml \
> logs/context_rl.log 2>&1 &Each experiment YAML has a self-contained header describing its hypothesis, read-out criteria, and kill-switch red lines.
| Model | GUI only | GUI + MCP (win.4) | Δ(MCP) |
|---|---|---|---|
| Thinking | 30.5% | 34.5% | +4.0pp |
| Instruct | 25.4% | 19.5% | −5.9pp |
The same MCP injection helps the Thinking model (+4.0pp) and hurts the Instruct model (−5.9pp), both beyond 2 SE — the paper characterizes when MCP injection helps. All headline numbers are 5-run means (baselines/COMPARISON.md, baselines/REPEATS.md); the best single Thinking run reaches 37.9%. Full tables, the adoption–competence result, and the context-compression DiD are in REPRODUCE.md.
Action level — the dense tool bonus moves adoption, not competence (paper Figure 2; training record in results/dense_bonus/). Spreadsheet tool adoption rises 0.03→0.33 and carries into greedy decoding (0.02→0.29), while held-out accuracy stays at the base level with zero sustained fail→pass flips:
Context level — matched training recovers the compression cost (paper Figure 4; curves in results/context_rl/train-reward.csv and results/outcome_only/train-reward.csv, cells in results/context_rl/cells.json). (a) Training reward of the compressed run vs. the rich-observation control; (b) the D13 rich–lean gap collapsing to zero at step 30:
RL before/after — the same VS Code file-save task, run by the base checkpoint (uncompressed img4) and by the step-40 context-RL checkpoint (compressed img2). The base model loops for all 50 steps and never saves; the RL'd model walks the save dialog and finishes in 12 steps:
One running example — LibreOffice Calc matrix transposition ("transpose B2:F5, paste at B8") — shows all three findings. Full walk-through in docs/CASES.md.
Base fails with pure GUI — 9 pyautogui steps fumbling Paste Special; only a stray 30 lands in B8:
Context-RL succeeds via one MCP call — the full transposed table appears at B8:
{ "action_type": "mcp", "tool_name": "libreoffice_calc.transpose_range",
"params": { "source_range": "B2:F5", "target_cell": "B8" } }- Capability flip — base 0/3 → context-RL 3/3 (one of 15 such flips).
- Tool adoption — base uses
guionly; the RL'd model chooses the MCP tool. - Context compression — on a task both settings solve, img2 carries 2 screenshots/step vs img4's 4 → 61% of the input (−39%) at the same result; the post-tool screenshot is skipped (
skip_on_mcp_success).
If you find this code or our paper useful in your research, please consider citing:
@article{fan2026screenshots,
title = {Screenshots or Tools? Eliciting Tool Use and Managing Multimodal
Context in Hybrid GUI-MCP Computer-Use Agents},
author = {Fan, Siqi and Li, Minghao and Ma, Xiaoqian and Tan, Wenhui and
Huang, Xiusheng and Wu, Juntong and Zhang, Liujie and Shang, Shuo
and Chen, Weihang},
journal = {arXiv preprint arXiv:2608.03327},
year = {2026},
url = {https://arxiv.org/abs/2608.03327}
}Apache-2.0 (see LICENSE and NOTICE). Built on verl (scaffolding), OSWorld, and Qwen3-VL.
This project would not exist without the following open efforts — our sincere thanks to all of them:
- OSWorld — the execution-based desktop benchmark and VM environment everything here runs on.
- OSWorld-MCP — the verified MCP tool benchmark; our vendored tool set derives from it.
- ToolCUA — the closest concurrent work on GUI–tool path orchestration; its observations on MCP injection shaped our study design.
- RLAnything — open-source RL for LLMs and agentic scenarios.
- Relax — high-performance distributed RL framework for multimodal LLM post-training.