Add support for per-prompt execution timeout (prompt_timeout_seconds)

Supports setting prompt_timeout_seconds across Agent CLI generators (gemini_cli, claude_code, codex_cli, agy_cli) to terminate prompt turns that spin indefinitely. Configurable per scenario JSON, per model YAML, or per run_config YAML.

TAG=agy
CONV=4229555b-deeb-44bb-a481-c751dab59476

Author: graceqi-g

AGENTS.md

@@ -166,6 +166,7 @@ simulated_user_model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
 
 runners:
   agent_runners: 10
+  prompt_timeout_seconds: 300
 
 scorers:
   trajectory_matcher: {}
@@ -183,11 +184,12 @@ reporting:
 ```
 
 ### 2. Model Configuration (e.g., `gemini_cli_model.yaml`)
-Specifies tested version, model ID, and environment variables.
+Specifies tested version, model ID, environment variables, and prompt timeout.
 
 ```yaml
 gemini_cli_version: "@google/gemini-cli@0.36.0"
 generator: gemini_cli
+prompt_timeout_seconds: 180
 env:
   GOOGLE_CLOUD_PROJECT: "my-evaluation-project"
   GOOGLE_CLOUD_LOCATION: "us-central1"
@@ -211,7 +213,8 @@ Contains the test cases.
       "conversation_plan": "Ensure the agent accurately calls list_instances. Verify the output is returned correctly.",
       "expected_trajectory": ["cloud-sql__list_instances"],
       "env": { "GOOGLE_CLOUD_PROJECT": "my-evaluation-project" },
-      "max_turns": 4
+      "max_turns": 4,
+      "prompt_timeout_seconds": 120
     }
   ]
 }

docs/agy_cli_agent_testing.md

@@ -182,6 +182,7 @@ reporting:
 |-----|----------|-------------|
 | `generator` | Yes | Must be `agy_cli` |
 | `model` | Optional | agy UI model label, e.g. `"Gemini 3.1 Pro (High)"`. Passed via the `--model` flag. Must be a valid label, not an API id. |
+| `prompt_timeout_seconds` | Optional | Timeout limit in seconds for each CLI prompt turn (e.g., `180`) |
 | `env` | Optional | Environment variables passed to the CLI process |
 | `setup` | Optional | Tool setup block containing `mcp_servers`, `skills`, or `fake_mcp_servers` |
 

docs/claude_code_agent_testing.md

@@ -206,6 +206,7 @@ The model config defines the Claude Code CLI version, model, auth, environment,
 | `vertex_project_id` | If `use_vertex` | GCP project for Vertex AI |
 | `vertex_region` | If `use_vertex` | Vertex region (e.g., `us-east5`) |
 | `env` | Optional | Environment variables passed to the CLI process |
+| `prompt_timeout_seconds` | Optional | Timeout limit in seconds for each CLI prompt turn (e.g., `180`) |
 | `setup.mcp_servers` | Optional | MCP server configurations (see [MCP Servers](#mcp-servers)) |
 | `allowed_tools` | Optional | List of tool names to allow (e.g., `["Bash", "mcp__cloud-sql"]`) |
 

docs/codex_cli_agent_testing.md

@@ -213,6 +213,7 @@ The model config defines the Codex CLI version, model, auth, sandbox/approval po
 | `profile` | Optional | Codex profile name (forwarded as `--profile <name>`) |
 | `json_flag` | Optional | `"--json"` (default, newer Codex versions) or `"--experimental-json"` (older versions). Codex requires NDJSON for the eval pipeline to extract tool calls and tokens. |
 | `pricing` | Optional | Per-model rates used to compute `cost_usd` per turn. See [Pricing & Cost Tracking](#pricing--cost-tracking). |
+| `prompt_timeout_seconds` | Optional | Timeout limit in seconds for each CLI prompt turn (e.g., `180`) |
 | `env` | Optional | Environment variables passed to the CLI process (e.g., `GOOGLE_CLOUD_PROJECT` for Cloud SQL MCP) |
 | `setup.mcp_servers` | Optional | MCP server configurations (see [MCP Servers](#mcp-servers)) |
 | `setup.config` | Optional | Free-form key/value pairs written to the top of `~/.codex/config.toml`. Merged on top of the default `forced_login_method = "api"`. |

docs/configs/model-config.md

@@ -15,6 +15,7 @@ These settings are passed to all generators, regardless of the specific engine u
 | `max_tokens`       | Optional     | N/A               | Specifies the maximum number of tokens the model can generate in a single output.                                                                                                                                                                            |
 | `execs_per_minute` | Optional     | `60`              | Sets the maximum number of executions allowed per minute. If not provided, it defaults to `60`. This helps throttle the rate of query generation.                                                                                                                                                                        |
 | `max_attempts`     | Optional     | `3`               | Specifies the maximum number of attempts for query generation in case of failures. Defaults to `3` if not provided.                                                                                                                                                                          |
+| `prompt_timeout_seconds` | Optional | N/A | Timeout limit in seconds for each CLI prompt execution turn (e.g. `180`). Overrides run config `runners.prompt_timeout_seconds`. |
 
 ## GCP Specific Configuration
 

docs/configs/run-config.md

@@ -17,6 +17,7 @@ This section defines the primary resources used during evaluation, including the
 | `num_trials`      | Optional     | Number of trials to run for each prompt. |
 | `scenarios`      | Optional     | A list of specific scenario IDs to run (only applies to scenario-based agentic datasets like `gemini-cli-format` or `cortado-format`). Defaults to empty (runs all scenarios). |
 | `scenario_pattern` | Optional     | A glob pattern of scenario IDs to run (only applies to scenario-based agentic datasets). Defaults to None (runs all scenarios). |
+| `runners` | Optional | Dictionary configuring concurrency and timeouts. <br>• `agent_runners`: Concurrency level for agent scenario runners (default: 10).<br>• `prompt_timeout_seconds`: Global timeout limit in seconds for a single agent prompt CLI execution turn (e.g. `300`). |
 ---
 
 ## 2. Prompt and Generation Modules

docs/gemini_cli_agent_testing.md

@@ -253,6 +253,7 @@ The evalset JSON file defines the test scenarios. Each scenario represents an ag
 | `env` | Optional | Per-scenario environment variables (merged with model config env) |
 | `kind` | Optional | Category label (e.g., `"tools"`) |
 | `max_turns` | Yes | Maximum number of conversation turns before the evaluation stops |
+| `prompt_timeout_seconds` | Optional | Timeout limit in seconds for each CLI prompt turn in this scenario. Overrides model config and run config timeout settings. |
 
 #### Tool name format
 

evalbench/evaluator/agentevaluator.py

@@ -141,6 +141,12 @@ def process_scenario(
                         "Declared env file not found in session: %s", src_path
                     )
 
+        prompt_timeout = (
+            scenario.get("prompt_timeout_seconds")
+            or getattr(self.generator, "prompt_timeout_seconds", None)
+            or self.config.get("runners", {}).get("prompt_timeout_seconds")
+        )
+
         session_id = None
         for turn in range(max_turns):
             logging.info(
@@ -153,6 +159,7 @@ def process_scenario(
                     resume=(turn > 0),
                     session_id=session_id,
                     cwd=resolved_work_dir,
+                    timeout=prompt_timeout,
                 )
                 try:
                     result = self.generator.safe_generate(cli_cmd)

evalbench/generators/models/agent_cli.py

@@ -23,7 +23,7 @@ def version(self) -> str:
     @abstractmethod
     def create_command(
         self, cli: str, prompt: str, env: dict = None, resume: bool = False,
-        session_id: str = None, cwd: str = None,
+        session_id: str = None, cwd: str = None, timeout: float | int = None,
     ):
         raise NotImplementedError("Subclasses must implement this method")
 

evalbench/generators/models/agy_cli.py

@@ -23,12 +23,13 @@
 
 
 class CLICommand:
-    def __init__(self, cli, prompt, env=None, resume=False, cwd=None):
+    def __init__(self, cli, prompt, env=None, resume=False, cwd=None, timeout=None):
         self.cli = cli
         self.prompt = prompt
         self.env = env if env else {}
         self.resume = resume
         self.cwd = cwd
+        self.timeout = timeout
 
 
 class AgyCliGenerator(AgentCliGenerator):
@@ -781,7 +782,7 @@ def generate_internal(self, cli_cmd):
         return self._run_agy_cli(cli_cmd)
 
     def _execute_cli_command(
-        self, command, env=None, cwd=None
+        self, command, env=None, cwd=None, timeout=None
     ) -> subprocess.CompletedProcess:
         try:
             return subprocess.run(
@@ -791,6 +792,11 @@ def _execute_cli_command(
                 check=False,
                 env=env,
                 cwd=cwd if cwd else self.fake_home,
+                timeout=timeout,
+            )
+        except subprocess.TimeoutExpired:
+            return subprocess.CompletedProcess(
+                command, 124, "", f"Error: Command timed out after {timeout} seconds."
             )
         except FileNotFoundError:
             return subprocess.CompletedProcess(
@@ -811,7 +817,7 @@ def _run_agy_cli(self, cli_cmd: CLICommand):
             self.agy_bin, cli_cmd.prompt, cli_cmd.resume, self.model
         )
         cwd = cli_cmd.cwd if cli_cmd.cwd else self.fake_home
-        result = self._execute_cli_command(command, env=env, cwd=cwd)
+        result = self._execute_cli_command(command, env=env, cwd=cwd, timeout=cli_cmd.timeout)
 
         if result.returncode == 0:
             try:
@@ -1219,7 +1225,7 @@ def safe_generate(
 
     def create_command(
         self, cli: str, prompt: str, env: dict = None, resume: bool = False,
-        session_id: str = None, cwd: str = None,
+        session_id: str = None, cwd: str = None, timeout: float | int = None,
     ) -> CLICommand:
         # The executable is always this session's sandbox binary
         # (self.agy_bin); the ``cli`` argument -- the agent_version label "agy"
@@ -1229,4 +1235,4 @@ def create_command(
         # environment are layered in once at invocation time by
         # ``_run_agy_cli`` via ``_merged_env``.
         return CLICommand(cli=self.agy_bin, prompt=prompt, env=env or {},
-                          resume=resume, cwd=cwd)
+                          resume=resume, cwd=cwd, timeout=timeout)