fix(scripts): stop pinning vm=False on sandbox creation - #2456
Conversation
Prime Sandboxes are VM-only: an explicit vm=False hard-fails on the new backend and the field is gone from prime-sandboxes 0.3.0. Drop it from the benchmark controller and the short-SWE build controller, and add smoke tests that the create requests carry no vm value. Update sandbox docs to the VM-only surface (no --vm/--start-command/expose).
Prime Agent performance — completedPR Overall: 0 regressed · 0 improved · 41 no clear change.
Python runtime
Session transport
UI interactions
Sandbox cost: ~$0.1463 — no inference calls. Methodology and samplesMain resolved at 2026-09-18T18:50:43.459769+00:00. Harness
|
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
Co-authored-by: macroscopeapp[bot] <170038800+macroscopeapp[bot]@users.noreply.github.com>
2 participants
Summary
vm=Falsehard-fails (410) on the new backend and the field is deleted inprime-sandboxes0.3.0. This dropsvm=Falsefrom the two create sites —scripts/benchmarks/controller.py(agent-bench roles) andscripts/evals/short_swe/build_controller.py(behavioral base/head builders) — so both controllers work against the VM-only backend.docker_imagestays: the VM backend auto-builds from the image on first launch, so no image handling changes.vmvalue (model_dump(exclude_none=True)has novmkey). Works on the pinnedprime-sandboxes==0.2.40(vmdefaults toNone) and on 0.3.0 (field gone). No SDK pin bump is required: omittingvmworks with both.--vm/--containermentions,--start-commandreplaced by the--token-list form, removedclient.expose()from the SDK snippet, network isolation via allow/deny lists, and "Docker environment"/"container" wording updated without deprecation framing.Sweep results
vm=False/vm=Truesites, noregistry_credentials_id, no--container, noprime registry,images transfer-bulk,ssh/expose/unexpose/list-portsCLI usage in scripts.--no-network-accessflag reference replaced (network is now allow/deny lists viaprime sandbox network/ SDK fields).Validation
uv run --locked ruff check .andruff format --check .inscripts/benchmarks: pass.uv run --locked python -m unittest discover -s tests(127 tests): pass.uv run --locked --project scripts/benchmarks pytest -q scripts/evals/short_swe/tests(40 tests): pass.uv run --locked --project scripts/benchmarks ruff check/ruff format --checkonscripts/evals/short_swe: pass.npm run checknot run: no TS/JS files changed (biome/tsgo only coverpackages/*/src|test).Note
Stop pinning
vm=Falseon sandbox creation requests in benchmark and eval scriptsvmfield fromCreateSandboxRequestin controller.py (Controller.start) and build_controller.py so the server default applies instead of forcing Docker mode.vmwhile retaining a Docker image value.vmis notfalse, benchmark and short-SWE build sandboxes will now start as VMs instead of containers, changing runtime behavior for those workflows.Changes since #2456 opened
LifecycleTests.test_sandbox_creation_does_not_inject_credentials_or_pin_vmtest to verify that sandbox creation requests do not include avmfield [1646b8b]Macroscope summarized 3298f60.