benchmark: add response_provider benchmark for Agentic API vs vLLM

[maralbahari]

Summary

Adds a new response_provider benchmark (crates/agentic-server/benches/client/response_provider/)
that compares Agentic API (WebSocket, HTTP/SSE, HTTP/JSON) against direct vLLM across three workloads:
transport (repeated tool-loop rounds), tool-call (BFCL function selection), and history-rehydration
(multi-turn conversation state). Goal is to measure what actually differentiates the gateway from
direct vLLM, not just raw inference latency, which is identical either way since both hit the same
model.

• Wire-byte instrumentation: TurnResult/ProviderSummary track request_bytes /
  response_bytes for every transport. This is the direct measurement of the state-scaling finding
  below (flat request size for the gateway vs growing for direct vLLM).
• --depths 5,10,25 (history-rehydration): each depth runs as its own independent batch of
  sessions instead of bucketing one long run after the fact, so sample counts stay even across depths
  instead of thinning out at the tail. Produces depth_summary.md/.json.
• Reporting: a standing correctness-check disclaimer on accuracy metrics (a gap is a bug signal,
  not a competition result), and a per-BFCL-case pass/fail matrix instead of a single aggregate
  tool-call accuracy number.
• Defaults: 5 concurrent sessions, 50 requests/session.

(A separate, non-repo orchestration script at the workspace level runs this benchmark across all
providers/workloads with fresh vLLM/gateway restarts between each; not part of this PR's diff.)

📊 Benchmark findings (click to expand)

Model: Qwen/Qwen3.5-35B-A3B-FP8. Full methodology and caveats in
benches/client/response_provider/README.md.
More benchmark detail and discussion: design/results doc.

Headline: state-scaling. At 25 turns of conversation, Agentic API sends a constant ~450-480 byte
request regardless of history length. Direct vLLM sends ~84.7 KB for the same turn (99.4% more),
because the client has to reconstruct and resend the whole transcript itself. Over the full 25-turn
batch, cumulative client-sent bytes were 44.8 KB (Agentic API) vs 12.1 MB (direct vLLM), a 99.6%
reduction. (Full byte-vs-depth table in the README; not charted here.)

Latency crossover with depth. Direct vLLM is actually faster at shallow depth (5 turns: 9.6s vs
12.5-13.3s for Agentic API), since the extra gateway hop costs something before there's much history to
save. By depth 25 that inverts: direct vLLM is slowest (35.7s) while Agentic API WS is fastest
(23.5s). The resend penalty compounds with depth; the gateway-hop cost doesn't.

---
config:
  theme: base
  themeVariables:
    xyChart:
      plotColorPalette: "#2a78d6, #eb6834, #1baf7a"
  xyChart:
    width: 700
    height: 420
---
xychart-beta
    title "History-rehydration: p50 end-to-end latency (ms) vs turn depth"
    x-axis "Turn depth" [5, 10, 25]
    y-axis "p50 end-to-end latency (ms)" 0 --> 40000
    line [13315, 13289, 23478]
    line [12507, 12605, 30768]
    line [9564, 14079, 35663]

Loading

Lines in plot order (blue, orange, green): Agentic API (WS), Agentic API (HTTP/SSE), direct vLLM.

Transport latency and tool-call accuracy are parity results, and that's the correct outcome. Both
arms hit the same underlying model, so raw inference latency and tool-selection accuracy should be
identical, not a competition:

---
config:
  theme: base
  themeVariables:
    xyChart:
      plotColorPalette: "#2a78d6, #eb6834, #1baf7a"
  xyChart:
    width: 700
    height: 420
---
xychart-beta
    title "Transport workload: p50 end-to-end latency by provider (ms)"
    x-axis ["Agentic API (WS): 8500", "Agentic API (HTTP): 8513", "Direct vLLM: 8515.5"]
    y-axis "p50 end-to-end latency (ms)" 0 --> 10000
    bar [8500, 0, 0]
    bar [0, 8513, 0]
    bar [0, 0, 8515.5]

Loading

---
config:
  theme: base
  themeVariables:
    xyChart:
      plotColorPalette: "#2a78d6, #eb6834, #1baf7a"
  xyChart:
    width: 700
    height: 420
---
xychart-beta
    title "BFCL tool-call accuracy by provider (%)"
    x-axis ["Agentic API (WS): 81.6%", "Agentic API (HTTP): 82.0%", "Direct vLLM: 80.8%"]
    y-axis "Tool-call accuracy (%)" 0 --> 100
    bar [81.6, 0, 0]
    bar [0, 82.0, 0]
    bar [0, 0, 80.8]

Loading

All three land within 16ms / a 1.2-point band of each other on both metrics: the gateway adds no
measurable inference overhead and doesn't change what the model selects. Even so, a single transport
turn already shows the byte gap that compounds into the state-scaling story above: direct vLLM resends
the growing tool-call history on every round of the same 4-round turn, so its per-turn request
payload (12,179 B p50) is already ~3.3x larger than either Agentic API transport (3,698 / 3,573 B),
despite the identical latency (see README for the full bytes table).

Test Plan

• cargo fmt -- --check, cargo check --package agentic-server --bench response_provider,
  cargo clippy --package agentic-server --bench response_provider -- -D warnings all pass.
• --print-prompts smoke tests for --depths (valid list, --depths on a non-history-rehydration
  workload correctly rejected, --depths 0,... correctly rejected).
• Full ./scripts/run_all_benchmarks.sh executed end to end against all 3 providers x 3 workloads
  (9 provider/workload pairs, all completed successfully); results above are from that run.

[franciscojavierarceo]

this is great! would be also cool to have more cut outs for latency buckets as well.