Description
DockerSandboxAgentRuntime.run_tasks() limits the number of sandbox tasks executing concurrently according to the configured parallelism, but creates a child asyncio task for every input task before waiting for them to complete.
As a result, parallelism bounds active sandbox work but does not bound the number of admitted pending tasks.
I reproduced this behavior with 512 input tasks and parallelism=16.
Observed behavior:
Input tasks: 512
Configured parallelism: 16
Peak active sandbox tasks: 16
Pending child tasks before release: 512
External service contacted: No
The number of pending asyncio tasks therefore scales with the total input size rather than the configured parallelism. Large evaluation batches can consequently create substantial numbers of pending tasks even when active sandbox execution is intentionally limited.
Affected Version
Confirmed on:
nemo-platform-plugin==0.3.0
DockerSandboxAgentRuntime.run_tasks()
nemo_platform/beta/evaluator/agent_eval/runtimes/docker_sandbox.py:113-128
PoC runtime:
Python 3.11.0rc1
Other versions have not been tested.
Reproduction
The attached PoC loads the released DockerSandboxAgentRuntime.run_tasks() implementation from the 0.3.0 wheel and supplies 512 tasks with parallelism set to 16.
The per-task operation is replaced with a controlled local implementation that holds active work pending long enough to measure both execution concurrency and the number of child tasks already created.
Observed result:
input tasks: 512
configured parallelism: 16
peak active sandbox tasks: 16
pending child tasks before release: 512
input-proportional pending task creation reproduced: yes
No external service is contacted during reproduction.
Attachment
run.py
Description
DockerSandboxAgentRuntime.run_tasks() limits the number of sandbox tasks executing concurrently according to the configured parallelism, but creates a child asyncio task for every input task before waiting for them to complete.
As a result, parallelism bounds active sandbox work but does not bound the number of admitted pending tasks.
I reproduced this behavior with 512 input tasks and parallelism=16.
Observed behavior:
Input tasks: 512
Configured parallelism: 16
Peak active sandbox tasks: 16
Pending child tasks before release: 512
External service contacted: No
The number of pending asyncio tasks therefore scales with the total input size rather than the configured parallelism. Large evaluation batches can consequently create substantial numbers of pending tasks even when active sandbox execution is intentionally limited.
Affected Version
Confirmed on:
nemo-platform-plugin==0.3.0
DockerSandboxAgentRuntime.run_tasks()
nemo_platform/beta/evaluator/agent_eval/runtimes/docker_sandbox.py:113-128
PoC runtime:
Python 3.11.0rc1
Other versions have not been tested.
Reproduction
The attached PoC loads the released DockerSandboxAgentRuntime.run_tasks() implementation from the 0.3.0 wheel and supplies 512 tasks with parallelism set to 16.
The per-task operation is replaced with a controlled local implementation that holds active work pending long enough to measure both execution concurrency and the number of child tasks already created.
Observed result:
input tasks: 512
configured parallelism: 16
peak active sandbox tasks: 16
pending child tasks before release: 512
input-proportional pending task creation reproduced: yes
No external service is contacted during reproduction.
Attachment
run.py