Description
CallableAgentTaskRunner.run_tasks() limits concurrent agent execution using its configured parallelism value.
However, the complete set of per-trial operations is still scheduled at once. The concurrency control limits how many agent calls execute simultaneously, but it does not bound the number of outstanding child tasks created for the input set.
I reproduced this behavior with 512 controlled tasks and parallelism set to 8.
Observed behavior:
Input tasks: 512
Configured parallelism: 8
Peak active agent calls: 8
Pending child tasks before release: 512
External service contacted: No
The execution limit therefore works as intended, but task admission still grows directly with the total number of trials.
For large evaluation runs, this creates avoidable task-management and memory overhead compared with keeping only a bounded amount of work outstanding.
Affected Version
Confirmed on:
nemo-platform-plugin==0.3.0
Python 3.13.14
nemo_platform/beta/evaluator/agent_eval/runtimes/callable_runtime.py:56-73
CallableAgentTaskRunner.run_tasks()
Other versions have not been tested.
Reproduction
The attached PoC executes the released CallableAgentTaskRunner.run_tasks() implementation with 512 controlled tasks and parallelism set to 8.
The agent callable is replaced with a local asynchronous operation that records active calls and waits on a shared gate. This allows the harness to observe both the active-agent concurrency limit and the number of child tasks created before any agent call completes.
Observed result:
input tasks: 512
configured parallelism: 8
peak active agent calls: 8
pending child tasks before release: 512
task-admission behavior reproduced: yes
Only eight agent calls execute concurrently, but all 512 per-task child operations are already pending.
No agent service or other external endpoint is contacted during reproduction.
Expected behavior
If the configured parallelism is intended to provide bounded execution for large evaluation runs, the number of outstanding per-trial tasks should also remain bounded rather than growing directly with the total number of input tasks.
A bounded worker pool, queue, or batched scheduling strategy could preserve the existing parallelism limit without eagerly creating one task for every trial.
Attachment
run.py
Description
CallableAgentTaskRunner.run_tasks() limits concurrent agent execution using its configured parallelism value.
However, the complete set of per-trial operations is still scheduled at once. The concurrency control limits how many agent calls execute simultaneously, but it does not bound the number of outstanding child tasks created for the input set.
I reproduced this behavior with 512 controlled tasks and parallelism set to 8.
Observed behavior:
Input tasks: 512
Configured parallelism: 8
Peak active agent calls: 8
Pending child tasks before release: 512
External service contacted: No
The execution limit therefore works as intended, but task admission still grows directly with the total number of trials.
For large evaluation runs, this creates avoidable task-management and memory overhead compared with keeping only a bounded amount of work outstanding.
Affected Version
Confirmed on:
nemo-platform-plugin==0.3.0
Python 3.13.14
nemo_platform/beta/evaluator/agent_eval/runtimes/callable_runtime.py:56-73
CallableAgentTaskRunner.run_tasks()
Other versions have not been tested.
Reproduction
The attached PoC executes the released CallableAgentTaskRunner.run_tasks() implementation with 512 controlled tasks and parallelism set to 8.
The agent callable is replaced with a local asynchronous operation that records active calls and waits on a shared gate. This allows the harness to observe both the active-agent concurrency limit and the number of child tasks created before any agent call completes.
Observed result:
input tasks: 512
configured parallelism: 8
peak active agent calls: 8
pending child tasks before release: 512
task-admission behavior reproduced: yes
Only eight agent calls execute concurrently, but all 512 per-task child operations are already pending.
No agent service or other external endpoint is contacted during reproduction.
Expected behavior
If the configured parallelism is intended to provide bounded execution for large evaluation runs, the number of outstanding per-trial tasks should also remain bounded rather than growing directly with the total number of input tasks.
A bounded worker pool, queue, or batched scheduling strategy could preserve the existing parallelism limit without eagerly creating one task for every trial.
Attachment
run.py