Allow parallel function calling for gateway-owned Responses tools

[maralbahari]

Problem statement / motivation

The Responses API accepts parallel_tool_calls, but currently rejects
parallel_tool_calls: true whenever a gateway-owned tool such as
web_search_preview or mcp is declared. When the field is omitted, requests
containing such tools are normalized to parallel_tool_calls: false.

This restriction no longer matches the executor's capabilities:

• Multiple model-generated function calls are parsed and preserved.
• Gateway ownership is resolved independently for every call.
• Gateway-owned calls are already executed concurrently with a bounded
  concurrency of five and a per-call timeout.
• Multiple client-owned calls and mixed gateway/client rounds are already
  supported.
• /models advertises supports_parallel_tool_calls: true.

The relevant current implementation is:

• Request restriction: RequestPayload::to_upstream_request
• Concurrent gateway dispatch: execute_output_calls
• Existing concurrent web-search test: execute_runs_multiple_web_search_calls_concurrently
• Advertised model capability: supports_parallel_tool_calls

As a result, the public API rejects an execution mode that the tool loop can
already process.

The parallel_tool_calls field should control whether the model may produce
multiple tool invocations in one inference round. Gateway execution concurrency
should remain a separate, bounded server-side policy.

Proposed solution

1. Permit explicit opt-in for gateway-owned tools

Change request normalization so the presence of a gateway-owned tool does not
cause parallel_tool_calls: true to be rejected:

Request	Effective initial behavior
parallel_tool_calls: true	Forward true after tool normalization.
parallel_tool_calls: false	Forward false.
Omitted, with a gateway-owned tool	Keep the current false default during the initial rollout.
Omitted, with client-owned tools only	Preserve the current upstream/default behavior.

Keeping gateway parallelism opt-in initially avoids changing workloads that
currently omit the field. Aligning the omitted default with the OpenAI/vLLM
default can be considered separately.

2. Preserve ownership-based dispatch

For a model round containing multiple invocations:

1. Classify every invocation independently.
2. Execute every gateway-owned invocation.
3. Preserve every client-owned invocation for client execution.
4. For a mixed round, finish the gateway-owned subset and return the
   client-owned subset without beginning another inference round.
5. Associate every result with its original call_id.

This retains the existing ownership model: enabling parallel function calling
must not change which component owns an invocation.

3. Keep execution concurrency bounded and server-controlled

parallel_tool_calls is an admission policy for model output, not an execution
concurrency setting. The gateway should continue to enforce an independent
limit.

The current Responses implementation uses buffered(5). This polls up to five
calls concurrently and preserves output order, but it can introduce
head-of-line blocking when there are more than five calls. A complete sliding
window can be implemented by:

1. Attaching the original output ordinal to every call future.
2. Executing with buffer_unordered(limit).
3. Sorting completed results back into model-output order before appending them
   to context, persistence, or the public response.

This provides bounded fan-out without making response history nondeterministic.

Every failure or timeout should remain isolated to its invocation and become a
tool result for the next inference round. One failed call must not cancel
successful siblings.

4. Preserve deterministic state and streaming behavior

Although gateway execution is concurrent:

• Function call outputs sent into the next inference round remain in the
  model's output order.
• Persistence retains every call/result pair without dangling call_id
  references.
• Every public gateway tool item receives a distinct output_index.
• Streaming sequence_number values remain monotonic.
• Internal normalized function-call events do not leak for gateway-owned
  built-ins.
• The terminal streamed response matches the non-streaming response.

For the initial implementation, completion events may remain in deterministic
output order. Emitting them in wall-clock completion order can be considered
separately.

5. Document the compatibility boundary

OpenAI documents that functions may be called in parallel when built-in tools
are available, but OpenAI-hosted built-ins cannot themselves participate in a
parallel function-call batch:

https://developers.openai.com/api/docs/guides/function-calling#parallel-function-calling

Agentic API normalizes gateway-owned web-search and MCP tools into model-visible
function tools and then executes them locally. Supporting multiple
gateway-owned invocations in one round should therefore be documented as an
intentional Agentic API extension rather than a claim about OpenAI-hosted tool
behavior.

parallel_tool_calls: true permits multiple invocations; it does not guarantee
that a particular model or tool-call parser will generate them. vLLM support
and quality vary by model and parser:

https://github.com/vllm-project/vllm/blob/main/docs/features/tool_calling.md

Alternatives considered

No response

Additional context

Acceptance criteria

• A Responses request containing web_search_preview or mcp and
  parallel_tool_calls: true is accepted.
• The normalized upstream vLLM request contains
  parallel_tool_calls: true.
• Two gateway-owned invocations from one model round overlap in execution.
• Both calls and both results are included in the next inference request in
  stable model-output order.
• Mixed gateway-owned and client-owned calls retain the current ownership
  behavior.
• Streaming produces a complete lifecycle and unique output_index for
  every public tool item.
• One failed or timed-out invocation does not cancel its siblings.
• No more than the gateway concurrency limit execute simultaneously.
• Existing serial behavior remains unchanged for gateway-tool requests that
  omit the field.
• Tests cover streaming and non-streaming fan-out for web search or MCP.

The existing concurrent web-search test should explicitly set
parallel_tool_calls: true. It currently exercises multiple calls even though
the normalized upstream request is implicitly serial.

Suggested test matrix

Tools	parallel_tool_calls	Expected behavior
Client-owned functions	true	Preserve and return all calls.
Client-owned functions	false	Forward the serial policy without regression.
Web search	true	Accept and concurrently execute emitted calls.
MCP	true	Accept and concurrently execute emitted calls.
Gateway-owned tool	omitted	Preserve the current serial default.
Mixed gateway/client tools	true	Execute the gateway subset and return the client subset.
Multiple gateway calls with one failure	true	Preserve successful siblings and feed back one failed result.
Multiple streamed gateway calls	true	Emit valid per-item lifecycles with monotonic sequence numbers.