fix(stargate-bench): measure validated stream output

[barrygreengus]

Why

The benchmark counted empty or truncated HTTP 200 responses as successful inference and credited requested token limits as generated output. Its native mock omitted streamed usage, leaving no reliable output count for the scorer.

Examples

• Empty or truncated success: HTTP 200 with an empty body, or an SSE body ending before data: [DONE], previously counted as success. These responses now fail validation.
• Requested versus generated: a request allows 100 output tokens, but a five-token context with a two-token prompt leaves room for three generated tokens. The mock-to-driver test records requested output_tokens: 100 and observed_output_tokens: 3. Scoring uses three.
• Backend shares: two backends receive 100-token requests but generate two and eight tokens. Their output shares are 20% and 80%, instead of the 50% split implied by requested limits.
• Missing usage: a completed external stream can be successful without providing a count. Aggregate output throughput and shares remain unavailable in that case. Older result files remain readable, and a reported zero remains distinct from an absent count.
• First output: keepalive comments and empty role chunks do not establish the first-output timestamp. Content, reasoning, and refusal text do. A refusal-only completion with three generated tokens must have a first-output timestamp; ignoring refusal text leaves that measurement missing.

What changed

• Use the public sse-core decoder for SSE framing and payload limits. The benchmark driver handles completion markers, application errors, output timing, and token usage.
• Keep requested workload tokens separate from observed completion-token counts, and finish the request when the completion marker arrives.
• Honor stream_options.include_usage in mock-dynamo, reporting actual bounded output before [DONE] and preserving the existing response shape when usage is not requested.
• Report throughput and output shares as unavailable when successful external or legacy responses have no usable count.
• Expose benchmark tests through Bazel and test the actual mock and driver executables together.

Customer Release Notes

Stargate benchmarks report validated output usage and reject incomplete success responses. Native mock streams provide actual token counts when requested; missing external usage is shown as unavailable.

Plan Summary

Not applicable.

Usage

Request results include observed_output_tokens. Existing output_tokens fields retain the requested workload limit. The driver requests streamed usage automatically; mock-dynamo also accepts stream_options.include_usage on direct chat-completion requests.

Testing

The YAGNI follow-up passed all 160 benchmark tests and 36 mock-dynamo tests, plus all-target Clippy for both packages with warnings denied. It replaces internal missing-usage counters with boolean flags; existing behavioral tests cover scoring, legacy results, and streamed usage. No runtime schema or dependency changed in this follow-up.

Earlier combined-stack validation passed 1,611 Cargo workspace tests, with 3 existing performance tests ignored, workspace Clippy, and 6 scoped Bazel targets. The native mock-to-driver integration recorded three observed tokens for a request allowing 100. A refusal-only stream recorded first-output timing and three observed tokens; a separate real-driver reproduction verified that later uncounted refusal text invalidates an earlier cumulative count. The full workspace and Bazel suites were not rerun for this internal state simplification. No GPU or live-cluster performance campaign was run.

Notes

The decoder limits each buffered SSE data, event-name, or ID field to 1 MiB. Discarded comments do not consume that buffer budget. Network requests remain owned by the existing HTTP client; decoding does not retry them.

Final layer of the Stargate maintenance stack. Earlier layers fix protocol handling, proxy and relay lifetime, and registration recovery. The streaming deadline finding was already fixed by #1452.

Issues

Closes #1817

References

sse-core decoder

Chat streamed usage

Related Pull Requests

Depends on #1823. Stack, bottom to top: #1818, #1820, #1823, #1826.

Dependencies

Added sse-core 0.2.3 with default features disabled and only std enabled. Its enabled dependencies were already present; no other package versions changed. The crate is dual-licensed MIT OR Apache-2.0. Apache-2.0 was selected and verified against the repository allowlist and the published source revision. Added benchmark attribution, regenerated the root NOTICE index, and refreshed Cargo and Bazel lockfiles, and updated the generated dependency inventory.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🛡️ CodeQL Analysis

🚨 Found 5 issue(s)

Severity Breakdown:

• 🔴 Errors: 0
• 🟡 Warnings: 0
• 🔵 Notes: 0

📋 Top Issues

🔗 View full details in Security tab

_{🕐 Last updated: 2026-09-11 22:27:38 UTC | Commit: 54b3e6e}