Add support for per-prompt execution timeout (prompt_timeout_seconds)

Supports setting prompt_timeout_seconds across Agent CLI generators (gemini_cli, claude_code, codex_cli, agy_cli) to terminate prompt turns that spin indefinitely. Configurable per scenario JSON, per model YAML, or per run_config YAML.

- Extracts process_timeout context manager and _kill_process_group helper to encapsulate process group termination (start_new_session=True + os.killpg) across all CLI generators.
- Fixes Codex CLI generator to support timeout in CLICommand, create_command, and _run_codex_cli.
- Cleans up unused imports across CLI generators.
- Preserves partial stdout and stream output captured before timeout to retain execution trajectories.

TAG=agy
CONV=4229555b-deeb-44bb-a481-c751dab59476

Author: graceqi-g

AGENTS.md

@@ -166,6 +166,7 @@ simulated_user_model_config: datasets/model_configs/gemini_2.5_pro_model.yaml
 
 runners:
   agent_runners: 10
+  prompt_timeout_seconds: 300
 
 scorers:
   trajectory_matcher: {}
@@ -183,11 +184,12 @@ reporting:
 ```
 
 ### 2. Model Configuration (e.g., `gemini_cli_model.yaml`)
-Specifies tested version, model ID, and environment variables.
+Specifies tested version, model ID, environment variables, and prompt timeout.
 
 ```yaml
 gemini_cli_version: "@google/gemini-cli@0.36.0"
 generator: gemini_cli
+prompt_timeout_seconds: 180
 env:
   GOOGLE_CLOUD_PROJECT: "my-evaluation-project"
   GOOGLE_CLOUD_LOCATION: "us-central1"
@@ -211,7 +213,8 @@ Contains the test cases.
       "conversation_plan": "Ensure the agent accurately calls list_instances. Verify the output is returned correctly.",
       "expected_trajectory": ["cloud-sql__list_instances"],
       "env": { "GOOGLE_CLOUD_PROJECT": "my-evaluation-project" },
-      "max_turns": 4
+      "max_turns": 4,
+      "prompt_timeout_seconds": 120
     }
   ]
 }

docs/agy_cli_agent_testing.md

@@ -167,6 +167,7 @@ Specifies the generator, model label, execution timeouts, and environment:
 | `generator` | Yes | Must be `agy_cli` |
 | `model` | Optional | Model label (e.g. `"Gemini 3.1 Pro (Low)"` or `"Gemini 3.5 Flash (Medium)"`). Omit to use agy's default. |
 | `timeout` | Optional | CLI turn timeout string (e.g. `"20m"`, passed to `--print-timeout`). Defaults to 5m. |
+| `prompt_timeout_seconds` | Optional | Timeout limit in seconds for each CLI prompt turn (e.g., `180`). |
 | `env` | Optional | Environment block. Set `GOOGLE_CLOUD_PROJECT` (see below); `GOOGLE_CLOUD_LOCATION` defaults to `global`. |
 | `setup` | Optional | Tool setup block for `mcp_servers`, `skills`, or `fake_mcp_servers`. |
 

docs/claude_code_agent_testing.md

@@ -206,6 +206,7 @@ The model config defines the Claude Code CLI version, model, auth, environment,
 | `vertex_project_id` | If `use_vertex` | GCP project for Vertex AI |
 | `vertex_region` | If `use_vertex` | Vertex region (e.g., `us-east5`) |
 | `env` | Optional | Environment variables passed to the CLI process |
+| `prompt_timeout_seconds` | Optional | Timeout limit in seconds for each CLI prompt turn (e.g., `180`) |
 | `setup.mcp_servers` | Optional | MCP server configurations (see [MCP Servers](#mcp-servers)) |
 | `allowed_tools` | Optional | List of tool names to allow (e.g., `["Bash", "mcp__cloud-sql"]`) |
 

docs/codex_cli_agent_testing.md

@@ -213,6 +213,7 @@ The model config defines the Codex CLI version, model, auth, sandbox/approval po
 | `profile` | Optional | Codex profile name (forwarded as `--profile <name>`) |
 | `json_flag` | Optional | `"--json"` (default, newer Codex versions) or `"--experimental-json"` (older versions). Codex requires NDJSON for the eval pipeline to extract tool calls and tokens. |
 | `pricing` | Optional | Per-model rates used to compute `cost_usd` per turn. See [Pricing & Cost Tracking](#pricing--cost-tracking). |
+| `prompt_timeout_seconds` | Optional | Timeout limit in seconds for each CLI prompt turn (e.g., `180`) |
 | `env` | Optional | Environment variables passed to the CLI process (e.g., `GOOGLE_CLOUD_PROJECT` for Cloud SQL MCP) |
 | `setup.mcp_servers` | Optional | MCP server configurations (see [MCP Servers](#mcp-servers)) |
 | `setup.config` | Optional | Free-form key/value pairs written to the top of `~/.codex/config.toml`. Merged on top of the default `forced_login_method = "api"`. |

docs/configs/model-config.md

@@ -15,6 +15,7 @@ These settings are passed to all generators, regardless of the specific engine u
 | `max_tokens`       | Optional     | N/A               | Specifies the maximum number of tokens the model can generate in a single output.                                                                                                                                                                            |
 | `execs_per_minute` | Optional     | `60`              | Sets the maximum number of executions allowed per minute. If not provided, it defaults to `60`. This helps throttle the rate of query generation.                                                                                                                                                                        |
 | `max_attempts`     | Optional     | `3`               | Specifies the maximum number of attempts for query generation in case of failures. Defaults to `3` if not provided.                                                                                                                                                                          |
+| `prompt_timeout_seconds` | Optional | N/A | Timeout limit in seconds for each CLI prompt execution turn (e.g. `180`). Overrides run config `runners.prompt_timeout_seconds`. |
 
 ## GCP Specific Configuration
 

docs/configs/run-config.md

@@ -19,6 +19,7 @@ This section defines the primary resources used during evaluation, including the
 | `num_trials`      | Optional     | Number of trials to run for each prompt. |
 | `scenarios`      | Optional     | A list of specific scenario IDs to run (only applies to scenario-based agentic datasets like `gemini-cli-format` or `cortado-format`). Defaults to empty (runs all scenarios). |
 | `scenario_pattern` | Optional     | A glob pattern of scenario IDs to run (only applies to scenario-based agentic datasets). Defaults to None (runs all scenarios). |
+| `runners` | Optional | Dictionary configuring concurrency (`agent_runners`, default: 10) and prompt timeouts (`prompt_timeout_seconds`, e.g. `300` seconds per CLI execution turn). |
 ---
 
 ## 2. Prompt and Generation Modules

docs/gemini_cli_agent_testing.md

@@ -255,6 +255,7 @@ The evalset JSON file defines the test scenarios. Each scenario represents an ag
 | `max_turns` | Yes | Maximum number of conversation turns before the evaluation stops |
 | `env` | Optional | Per-scenario environment variables (merged with model config env) |
 | `kind` | Optional | Category label (e.g., `"tools"`) |
+| `prompt_timeout_seconds` | Optional | Timeout limit in seconds for each CLI prompt turn in this scenario. Overrides model config and run config timeout settings. |
 
 #### Tool name format
 

evalbench/evaluator/agentevaluator.py

@@ -141,6 +141,12 @@ def process_scenario(
                         "Declared env file not found in session: %s", src_path
                     )
 
+        prompt_timeout = (
+            scenario.get("prompt_timeout_seconds")
+            or getattr(self.generator, "prompt_timeout_seconds", None)
+            or self.config.get("runners", {}).get("prompt_timeout_seconds")
+        )
+
         session_id = None
         for turn in range(max_turns):
             logging.info(
@@ -153,6 +159,7 @@ def process_scenario(
                     resume=(turn > 0),
                     session_id=session_id,
                     cwd=resolved_work_dir,
+                    timeout=prompt_timeout,
                 )
                 try:
                     result = self.generator.safe_generate(cli_cmd)

evalbench/generators/models/agent_cli.py

@@ -1,13 +1,17 @@
 from abc import abstractmethod
+from contextlib import contextmanager
+import logging
+import os
+import shutil
+import signal
+import subprocess
+import threading
 
 from mcp import types as mcp_types
 
 from . import mcp_client
 from .generator import QueryGenerator
 from .tool_naming import canonical_tool_name
-import logging
-import os
-import shutil
 
 
 class AgentCliGenerator(QueryGenerator):
@@ -91,7 +95,7 @@ def version(self) -> str:
     @abstractmethod
     def create_command(
         self, cli: str, prompt: str, env: dict = None, resume: bool = False,
-        session_id: str = None, cwd: str = None,
+        session_id: str = None, cwd: str = None, timeout: float | int = None,
     ):
         raise NotImplementedError("Subclasses must implement this method")
 
@@ -110,3 +114,70 @@ def extract_tools(self, stdout: str) -> list:
     @abstractmethod
     def extract_skills(self, stdout: str) -> list:
         raise NotImplementedError("Subclasses must implement this method")
+
+
+def parse_timeout_seconds(timeout: float | int | str | None) -> float | None:
+    if timeout is None:
+        return None
+    if isinstance(timeout, (int, float)):
+        return float(timeout)
+    if isinstance(timeout, str):
+        s = timeout.strip()
+        unit = 1.0
+        if s.lower().endswith("s"):
+            s = s[:-1]
+        elif s.lower().endswith("m"):
+            s = s[:-1]
+            unit = 60.0
+        elif s.lower().endswith("h"):
+            s = s[:-1]
+            unit = 3600.0
+
+        try:
+            return float(s) * unit
+        except ValueError as e:
+            logging.warning("Failed to parse timeout string %r: %s", timeout, e)
+    return None
+
+
+def _kill_process_group(proc: subprocess.Popen):
+    """Terminates proc's process group with SIGKILL, falling back to proc.kill()."""
+    try:
+        os.killpg(os.getpgid(proc.pid), signal.SIGKILL)
+        return
+    except (ProcessLookupError, OSError) as e:
+        logging.warning("os.killpg failed for pid %s: %s; trying proc.kill()", proc.pid, e)
+    except Exception as e:
+        logging.warning("Unexpected error in os.killpg for pid %s: %s; trying proc.kill()", proc.pid, e)
+
+    try:
+        proc.kill()
+    except (ProcessLookupError, OSError) as e:
+        logging.warning("proc.kill() failed for pid %s: %s", proc.pid, e)
+    except Exception as e:
+        logging.warning("Unexpected error in proc.kill() for pid %s: %s", proc.pid, e)
+
+
+@contextmanager
+def process_timeout(proc: subprocess.Popen, timeout: float | int | str | None):
+    """Context manager that sets a timer to kill ``proc``'s process group on timeout.
+
+    Yields a callable ``is_timed_out() -> bool``.
+    """
+    timed_out = False
+    timer = None
+    timeout_sec = parse_timeout_seconds(timeout)
+    if timeout_sec:
+        def _on_timeout():
+            nonlocal timed_out
+            timed_out = True
+            _kill_process_group(proc)
+
+        timer = threading.Timer(timeout_sec, _on_timeout)
+        timer.start()
+
+    try:
+        yield lambda: timed_out
+    finally:
+        if timer:
+            timer.cancel()

evalbench/generators/models/agy_cli.py

@@ -1,4 +1,4 @@
-from .agent_cli import AgentCliGenerator
+from .agent_cli import AgentCliGenerator, process_timeout
 from .tool_naming import canonicalize_agy_tool_name, parse_agy_mcp_tool_call
 import subprocess
 import os
@@ -35,12 +35,13 @@ def _shred_credential(path: str) -> None:
 
 
 class CLICommand:
-    def __init__(self, cli, prompt, env=None, resume=False, cwd=None):
+    def __init__(self, cli, prompt, env=None, resume=False, cwd=None, timeout=None):
         self.cli = cli
         self.prompt = prompt
         self.env = env if env else {}
         self.resume = resume
         self.cwd = cwd
+        self.timeout = timeout
 
 
 class AgyCliGenerator(AgentCliGenerator):
@@ -872,17 +873,27 @@ def generate_internal(self, cli_cmd):
         return self._run_agy_cli(cli_cmd)
 
     def _execute_cli_command(
-        self, command, env=None, cwd=None
+        self, command, env=None, cwd=None, timeout=None
     ) -> subprocess.CompletedProcess:
         try:
-            return subprocess.run(
+            proc = subprocess.Popen(
                 command,
-                stdin=subprocess.DEVNULL, capture_output=True,
+                stdin=subprocess.DEVNULL,
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE,
                 text=True,
-                check=False,
                 env=env,
                 cwd=cwd if cwd else self.fake_home,
+                start_new_session=True,
             )
+            with process_timeout(proc, timeout) as is_timed_out:
+                stdout, stderr = proc.communicate()
+
+            if is_timed_out():
+                stderr = (stderr + "\n" if stderr else "") + f"Error: Command timed out after {timeout} seconds."
+                return subprocess.CompletedProcess(command, 124, stdout or "", stderr)
+
+            return subprocess.CompletedProcess(command, proc.returncode, stdout or "", stderr)
         except FileNotFoundError:
             return subprocess.CompletedProcess(
                 command, 127, "", f"Error: Command not found: {command[0]}"
@@ -901,10 +912,10 @@ def _run_agy_cli(self, cli_cmd: CLICommand):
         command = self._base_agy_command(
             self.agy_bin, cli_cmd.prompt, cli_cmd.resume, self.model,
             output_format="stream-json", log_file=self.cli_log_path,
-            timeout=self.timeout,
+            timeout=cli_cmd.timeout or self.timeout,
         )
         cwd = cli_cmd.cwd if cli_cmd.cwd else self.fake_home
-        result = self._execute_cli_command(command, env=env, cwd=cwd)
+        result = self._execute_cli_command(command, env=env, cwd=cwd, timeout=cli_cmd.timeout or self.timeout)
 
         # Parse whenever agy emitted a stream, even on a non-zero exit: a
         # timed-out/errored run still ends in a ``result`` event carrying real
@@ -1212,7 +1223,7 @@ def safe_generate(
 
     def create_command(
         self, cli: str, prompt: str, env: dict = None, resume: bool = False,
-        session_id: str = None, cwd: str = None,
+        session_id: str = None, cwd: str = None, timeout: float | int = None,
     ) -> CLICommand:
         # The executable is always this session's sandbox binary
         # (self.agy_bin); the ``cli`` argument -- the agent_version label "agy"
@@ -1222,4 +1233,4 @@ def create_command(
         # environment are layered in once at invocation time by
         # ``_run_agy_cli`` via ``_merged_env``.
         return CLICommand(cli=self.agy_bin, prompt=prompt, env=env or {},
-                          resume=resume, cwd=cwd)
+                          resume=resume, cwd=cwd, timeout=timeout)