benchmark: add response_provider benchmark for Agentic API vs vLLM (vllm-project#185)

## Summary

Adds a new `response_provider` benchmark
(`crates/agentic-server/benches/client/response_provider/`)
that compares Agentic API (WebSocket, HTTP/SSE, HTTP/JSON) against
direct vLLM across three workloads:
transport (repeated tool-loop rounds), tool-call (BFCL function
selection), and history-rehydration
(multi-turn conversation state). Goal is to measure what actually
differentiates the gateway from
direct vLLM, not just raw inference latency, which is identical either
way since both hit the same
model.

- **Wire-byte instrumentation**: `TurnResult`/`ProviderSummary` track
`request_bytes` /
`response_bytes` for every transport. This is the direct measurement of
the state-scaling finding
  below (flat request size for the gateway vs growing for direct vLLM).
- **`--depths 5,10,25` (history-rehydration)**: each depth runs as its
own independent batch of
sessions instead of bucketing one long run after the fact, so sample
counts stay even across depths
instead of thinning out at the tail. Produces
`depth_summary.md`/`.json`.
- **Reporting**: a standing correctness-check disclaimer on accuracy
metrics (a gap is a bug signal,
not a competition result), and a per-BFCL-case pass/fail matrix instead
of a single aggregate
  tool-call accuracy number.
- Defaults: 5 concurrent sessions, 50 requests/session.

(A separate, non-repo orchestration script at the workspace level runs
this benchmark across all
providers/workloads with fresh vLLM/gateway restarts between each; not
part of this PR's diff.)

<details>
<summary><h3>📊 Benchmark findings (click to expand)</h3></summary>

Model: `Qwen/Qwen3.5-35B-A3B-FP8`. Full methodology and caveats in

[`benches/client/response_provider/README.md`](crates/agentic-server/benches/client/response_provider/README.md).
More benchmark detail and discussion: [design/results
doc](https://docs.google.com/document/d/1zJUfxdloxu9_oBYfbfTtgbz5sSZxwAnqvlnt72T67oU/edit?tab=t.2qh4vijnc2n1#heading=h.u692ockfn047).

**Headline: state-scaling.** At 25 turns of conversation, Agentic API
sends a constant ~450-480 byte
request regardless of history length. Direct vLLM sends ~84.7 KB for the
same turn (99.4% more),
because the client has to reconstruct and resend the whole transcript
itself. Over the full 25-turn
batch, cumulative client-sent bytes were 44.8 KB (Agentic API) vs 12.1
MB (direct vLLM), a 99.6%
reduction. (Full byte-vs-depth table in the README; not charted here.)

**Latency crossover with depth.** Direct vLLM is actually *faster* at
shallow depth (5 turns: 9.6s vs
12.5-13.3s for Agentic API), since the extra gateway hop costs something
before there's much history to
save. By depth 25 that inverts: direct vLLM is *slowest* (35.7s) while
Agentic API WS is fastest
(23.5s). The resend penalty compounds with depth; the gateway-hop cost
doesn't.

```mermaid
---
config:
  theme: base
  themeVariables:
    xyChart:
      plotColorPalette: "#2a78d6, #eb6834, #1baf7a"
  xyChart:
    width: 700
    height: 420
---
xychart-beta
    title "History-rehydration: p50 end-to-end latency (ms) vs turn depth"
    x-axis "Turn depth" [5, 10, 25]
    y-axis "p50 end-to-end latency (ms)" 0 --> 40000
    line [13315, 13289, 23478]
    line [12507, 12605, 30768]
    line [9564, 14079, 35663]
```

Lines in plot order (blue, orange, green): Agentic API (WS), Agentic API
(HTTP/SSE), direct vLLM.

**Transport latency and tool-call accuracy are parity results, and
that's the correct outcome.** Both
arms hit the same underlying model, so raw inference latency and
tool-selection accuracy should be
identical, not a competition:

```mermaid
---
config:
  theme: base
  themeVariables:
    xyChart:
      plotColorPalette: "#2a78d6, #eb6834, #1baf7a"
  xyChart:
    width: 700
    height: 420
---
xychart-beta
    title "Transport workload: p50 end-to-end latency by provider (ms)"
    x-axis ["Agentic API (WS): 8500", "Agentic API (HTTP): 8513", "Direct vLLM: 8515.5"]
    y-axis "p50 end-to-end latency (ms)" 0 --> 10000
    bar [8500, 0, 0]
    bar [0, 8513, 0]
    bar [0, 0, 8515.5]
```

```mermaid
---
config:
  theme: base
  themeVariables:
    xyChart:
      plotColorPalette: "#2a78d6, #eb6834, #1baf7a"
  xyChart:
    width: 700
    height: 420
---
xychart-beta
    title "BFCL tool-call accuracy by provider (%)"
    x-axis ["Agentic API (WS): 81.6%", "Agentic API (HTTP): 82.0%", "Direct vLLM: 80.8%"]
    y-axis "Tool-call accuracy (%)" 0 --> 100
    bar [81.6, 0, 0]
    bar [0, 82.0, 0]
    bar [0, 0, 80.8]
```

All three land within 16ms / a 1.2-point band of each other on both
metrics: the gateway adds no
measurable inference overhead and doesn't change what the model selects.
Even so, a single transport
turn already shows the byte gap that compounds into the state-scaling
story above: direct vLLM resends
the growing tool-call history on every round of the *same* 4-round turn,
so its per-turn request
payload (12,179 B p50) is already ~3.3x larger than either Agentic API
transport (3,698 / 3,573 B),
despite the identical latency (see README for the full bytes table).

</details>

## Test Plan

- `cargo fmt -- --check`, `cargo check --package agentic-server --bench
response_provider`,
`cargo clippy --package agentic-server --bench response_provider -- -D
warnings` all pass.
- `--print-prompts` smoke tests for `--depths` (valid list, `--depths`
on a non-history-rehydration
  workload correctly rejected, `--depths 0,...` correctly rejected).
- Full `./scripts/run_all_benchmarks.sh` executed end to end against all
3 providers x 3 workloads
(9 provider/workload pairs, all completed successfully); results above
are from that run.

Signed-off-by: maral <maralbahari.98@gmail.com>

Author: maralbahari

crates/agentic-server/Cargo.toml

@@ -32,7 +32,7 @@ url.workspace = true
 bytes.workspace = true
 criterion.workspace = true
 futures.workspace = true
-reqwest = { workspace = true, features = ["json"] }
+reqwest = { workspace = true, features = ["json", "stream"] }
 rand = "0.8"
 rsa = "0.9"
 serde_json.workspace = true
@@ -46,6 +46,11 @@ uuid = { version = "1", features = ["v7"] }
 name = "benches"
 harness = false
 
+[[bench]]
+name = "response_provider"
+path = "benches/client/response_provider/main.rs"
+harness = false
+
 [lints]
 workspace = true
 

crates/agentic-server/benches/client/response_provider/README.md

@@ -0,0 +1,107 @@
+use std::path::PathBuf;
+
+use clap::{Parser, ValueEnum};
+
+use crate::types::Workload;
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq, ValueEnum)]
+pub enum ProviderSelection {
+    /// Gateway WebSocket, HTTP/SSE, and JSON plus direct vLLM HTTP/SSE and JSON.
+    All,
+    /// Gateway WebSocket and direct vLLM HTTP/SSE (backwards-compatible pair).
+    Both,
+    AgenticApi,
+    AgenticApiHttp,
+    AgenticApiJson,
+    Vllm,
+    VllmJson,
+}
+
+#[derive(Debug, Parser)]
+#[command(
+    name = "agentic-responses-benchmark",
+    about = "Run concurrent Responses workloads against Agentic API and direct vLLM"
+)]
+pub struct Cli {
+    /// Model slug exposed by each selected provider.
+    #[arg(long, env = "RESPONSE_PROVIDER_BENCH_MODEL")]
+    pub model: String,
+
+    /// Capability measured by this run.
+    #[arg(long, value_enum, default_value_t = Workload::HistoryRehydration)]
+    pub workload: Workload,
+
+    /// Number of concurrent sessions per selected provider.
+    #[arg(long, default_value_t = 5)]
+    pub sessions: usize,
+
+    /// Sequential turns in each session. Defaults depend on `--workload`.
+    #[arg(long, short = 'n')]
+    pub requests_per_session: Option<usize>,
+
+    /// Fixed session depths (history-rehydration only), each run as its own independent batch of
+    /// `--sessions` sessions instead of bucketing one long run after the fact. Overrides
+    /// `--requests-per-session` when set. Example: --depths 1,5,10,25,50,100
+    #[arg(long, value_delimiter = ',')]
+    pub depths: Vec<usize>,
+
+    /// Responses created inside each transport workload turn.
+    #[arg(long, default_value_t = 4)]
+    pub transport_rounds: usize,
+
+    /// BFCL question JSONL for the tool-call workload.
+    #[arg(long, requires = "dataset_answers")]
+    pub dataset_questions: Option<PathBuf>,
+
+    /// BFCL possible-answer JSONL paired by case ID.
+    #[arg(long, requires = "dataset_questions")]
+    pub dataset_answers: Option<PathBuf>,
+
+    /// Zero-based BFCL case offset before selecting deterministic cases.
+    #[arg(long, default_value_t = 0)]
+    pub dataset_offset: usize,
+
+    /// Providers to benchmark. `all` includes both gateway transports.
+    #[arg(long, value_enum, default_value_t = ProviderSelection::Both)]
+    pub provider: ProviderSelection,
+
+    /// Agentic API OpenAI-compatible base URL.
+    #[arg(long, default_value = "http://localhost:9000/v1")]
+    pub agentic_url: String,
+
+    /// Direct vLLM OpenAI-compatible base URL.
+    #[arg(long, default_value = "http://localhost:5050/v1")]
+    pub vllm_url: String,
+
+    /// Timeout for one logical turn, including every model/tool round.
+    #[arg(long, default_value_t = 300)]
+    pub timeout_seconds: u64,
+
+    /// Seed for deterministic generated prompts and continuation secrets.
+    #[arg(long, default_value_t = 2_026_081_7)]
+    pub seed: u64,
+
+    /// Result directory. Defaults to target/response-provider-benchmark/<unix-milliseconds>.
+    #[arg(long)]
+    pub output_dir: Option<PathBuf>,
+
+    /// Stream timestamped, provider-tagged Responses events to stdout.
+    #[arg(long)]
+    pub live_jsonl: bool,
+
+    /// Validate inputs, print generated prompt specifications as JSONL, and exit.
+    #[arg(long)]
+    pub print_prompts: bool,
+
+    /// Automatically appended by `cargo bench` for custom harnesses.
+    #[arg(long, hide = true)]
+    pub bench: bool,
+}
+
+impl Cli {
+    #[must_use]
+    pub fn requests_per_session(&self) -> usize {
+        self.requests_per_session
+            .unwrap_or_else(|| self.workload.default_requests())
+    }
+}