Bug: cumulative response budget rejects small outputs based on SSE chunking (0.6.0/main)

[SProst]

Description

Executor-backed responses can fail after only 7,655 bytes of answer text because the request's fixed 1 MiB cumulative response budget counts serialized SSE envelopes, every delta, and repeated done/terminal snapshots. The same 8,000-byte answer succeeds when sent with coarser deltas or as non-streaming JSON. A fast, continuously draining client does not prevent the failure.

This is still reproducible at main commit 2509342dcbc12b89350cabdda17f7bb1a6b8a41b, including its new WebSocket store: false session path. It affects the shared executor, not only WebSocket transport. The fixed upstream SSE-line and JSON-body ceilings are separate constraints.

The resource protections added in #240 are useful, but lifetime wire-byte accounting makes ordinary response capacity depend on chunking and repeated representations. This report proposes separating configurable retained-response, upstream-wire, and client-event limits while keeping bounded queues and cancellation.

Steps to reproduce

The following deterministic upstream uses Python's standard library. It supplies an ordinary Responses lifecycle with deltas and matching full done/terminal snapshots. It requires no model, GPU, SDK, or external service.

1. Save the script below as public_repro.py, then run python public_repro.py server.

2. In a separate terminal, build/run the gateway from the selected checkout with its locked dependencies:

   cargo run --locked -p agentic-server --bin agentic-server -- --llm-api-base http://127.0.0.1:18001 --gateway-host 127.0.0.1 --gateway-port 18002 --db-url 'sqlite://budget-repro.db?mode=rwc'

3. Run python public_repro.py client. It requests the same 8,000-byte answer with one-byte and 1,024-byte chunks, then a 220,000-byte answer with 1,024-byte chunks.

Use an isolated config/home and database; the recorded harness clears unrelated gateway/provider environment overrides. On Windows, use the native-build caveat in Environment below. The fixture bytes in this reduced script were checked against the fixture bytes used by the full HTTP/SSE/WebSocket harness.

Self-contained synthetic upstream and HTTP SSE client

"""Run `python public_repro.py server`, start the gateway, then run `... client`."""
import http.client
import json
import sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

def encode(value):
    return json.dumps(value, separators=(',', ':')).encode()

def fixture(size, chunk):
    text = 'x' * size
    part = dict(type='output_text', text=text, annotations=[])
    item = dict(id='msg_fixture', type='message', role='assistant', status='completed', content=[part])
    response = dict(id='resp_fixture', object='response', created_at=1789151000,
                    model='fixture', status='completed', output=[item], error=None, incomplete_details=None)
    initial = dict(response, status='in_progress', output=[])
    events = []
    def add(kind, **fields):
        events.append(dict(type=kind, sequence_number=len(events), **fields))
    add('response.created', response=initial)
    add('response.in_progress', response=initial)
    add('response.output_item.added', output_index=0, item=dict(item, status='in_progress', content=[]))
    add('response.content_part.added', output_index=0, item_id=item['id'], content_index=0, part=dict(part, text=''))
    for at in range(0, size, chunk):
        add('response.output_text.delta', output_index=0, item_id=item['id'], content_index=0, delta=text[at:at+chunk])
    add('response.output_text.done', output_index=0, item_id=item['id'], content_index=0, text=text)
    add('response.content_part.done', output_index=0, item_id=item['id'], content_index=0, part=part)
    add('response.output_item.done', output_index=0, item=item)
    add('response.completed', response=response)
    stream = b''.join(b'data: ' + encode(event) + b'\n\n' for event in events) + b'data: [DONE]\n\n'
    return response, stream

class Upstream(BaseHTTPRequestHandler):
    protocol_version = 'HTTP/1.1'
    def log_message(self, *args):
        pass
    def reply(self, body, content_type):
        self.send_response(200)
        self.send_header('Content-Type', content_type)
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        try:
            self.wfile.write(body)
        except (BrokenPipeError, ConnectionResetError, ConnectionAbortedError):
            pass
    def do_GET(self):
        self.reply(encode({'object':'list','data':[{'id':'fixture','object':'model','owned_by':'fixture'}]}), 'application/json')
    def do_POST(self):
        request = json.loads(self.rfile.read(int(self.headers['Content-Length'])))
        _, size, chunk = request['model'].split('-')
        response, stream = fixture(int(size), int(chunk))
        self.reply(stream if request.get('stream') else encode(response),
                   'text/event-stream' if request.get('stream') else 'application/json')

def client():
    for size, chunk in [(8000, 1), (8000, 1024), (220000, 1024)]:
        conn = http.client.HTTPConnection('127.0.0.1', 18002, timeout=30)
        conn.request('POST', '/v1/responses', encode(dict(model=f'fixture-{size}-{chunk}',
                     input='size reproduction', stream=True, store=True)), {'Content-Type':'application/json'})
        response = conn.getresponse()
        raw = response.read()
        conn.close()
        events = [json.loads(line[5:]) for line in raw.splitlines()
                  if line.startswith(b'data:') and line[5:].strip() != b'[DONE]']
        print(json.dumps({'size':size, 'chunk':chunk, 'http':response.status,
              'received_text_bytes':sum(len(e.get('delta','')) for e in events if e.get('type')=='response.output_text.delta'),
              'terminal':[e['type'] for e in events if e['type'] in ('response.completed','response.failed','response.incomplete')],
              'errors':[e for e in events if e['type']=='error']}))

if __name__ == '__main__':
    if sys.argv[1:] == ['server']:
        ThreadingHTTPServer(('127.0.0.1', 18001), Upstream).serve_forever()
    elif sys.argv[1:] == ['client']:
        client()
    else:
        raise SystemExit('Use: python public_repro.py server|client')

Additional controls in the full harness: non-streaming JSON, HTTP pass-through with store: false, WebSocket response.create, exact JSON-body boundary and boundary + 1, and a healthy request on the same WebSocket after a failed request.

Expected behavior

• An otherwise identical retained response should not fail solely because the upstream sends more, smaller deltas or repeats the same text in completion snapshots.
• Keep independent, configurable limits on actual retained response data, each upstream JSON body/SSE line, and each client stream event. Keep shared limits across inference rounds, gateway tool results, and MCP discovery, plus bounded queues/backpressure.
• Raising one allowance must not silently disable the others. Exceeding a resource limit should produce an explicit resource-specific error, terminate the affected request, and avoid publishing a successful durable response or transient checkpoint.
• Preserve fix: honor WebSocket response storage with bounded sessions #257's transient/durable continuation rules independently of per-request response resource accounting.

Actual behavior

The same 14-case fixture set was run against these source revisions:

Build	Completed	Size failures
3bb55fdf55ea1a7e756cf79694b187909faf62b7, parent of #240	14	0
e87eeae1a8ca09156ececbf18d04487d80324701	8	6
2509342dcbc12b89350cabdda17f7bb1a6b8a41b, including #257	8	6
Local response-resource patch applied on 2509342	14	0

The six baseline failures are four HTTP cases (fine SSE, repeated-snapshot SSE, oversized SSE line, JSON body above the fixed cap) and two WebSocket cases (fine deltas and repeated snapshots).

1. 8,000 answer bytes / one-byte deltas: each relevant delta line is only 137 bytes; the largest fixture line is 8,364 bytes. After delivering 7,655 answer bytes, the budget has charged 1,048,445 bytes. The next line would increase this to 1,048,582, exceeding 1,048,576. HTTP SSE and WebSocket fail at that same predicted point. With 1,024-byte chunks, the identical answer uses 42,775 cumulative data-line bytes and completes. The full fine-grained fixture would use 1,128,619 bytes.
2. 220,000 answer bytes / 1,024-byte deltas: all answer deltas arrive, but the terminal response.completed snapshot increases cumulative accounting from 910,273 to 1,130,636 bytes and is rejected. Every individual line remains below 256 KiB. JSON with the same answer succeeds.
3. 270,000 answer bytes: a full-content done event exceeds the distinct 262,144-byte upstream SSE-line cap. HTTP pass-through succeeds.
4. JSON-body boundary: exactly 1,048,576 upstream bytes succeeds; one more byte returns HTTP 500 (upstream response exceeded 1048576 bytes). HTTP pass-through succeeds.

For the cumulative SSE failure, HTTP has already returned 200; the stream ends with an error event (status: 500, code: "server_error", message containing executor response budget exceeded 1048576 bytes) and [DONE]. It has no response.completed, response.failed, or response.incomplete terminal event. The failed responses are not stored. These are resource errors, not token-limit truncation.

Environment

• Native Windows execution; Rust 1.98.1 (48a229cea), Cargo 1.98.1 (797e8a9bc), each revision's Cargo.lock, --locked.
• The pre-feat: multiplex response streams over WebSocket #240 parent reports package 0.5.0; the two tested main revisions report 0.6.0. Source commits are necessary to distinguish them.
• Python 3.12.14; standard-library HTTP upstream/client; websockets 15.0.1 for WebSocket controls; isolated SQLite databases.
• No vLLM server, model, LiteLLM, or GPU was used in this reproduction. This isolates the gateway behavior; it is not a live-model performance or memory benchmark.
• Native-build caveat: each baseline needed the same unrelated borrow correction in config.rs: after Windows replaces path separators and obtains a String, pass a borrowed string to utf8_percent_encode. No budget, parser, or storage behavior was changed in the baseline builds. The local response patch contains a Windows-only as_str() binding for this correction.

Additional context

Source cause. At 2509342, response_budget.rs still initializes a 1 MiB atomic allowance that only decreases. fetch_stream_payload charges line.len() before semantic accumulation. It counts the normalized data: prefix and serialized event data, including repeated content, rather than current retained output or queued bytes. Framing newlines and [DONE] are not charged. inference.rs separately caps upstream lines and JSON bodies. Increasing only an outbound WebSocket event cap cannot fix the cumulative failure.

Effect of #257 / 2509342. The new commit honors store: false and adds bounded connection-local checkpoints. It does not modify the cumulative response budget or upstream wire caps. A separate seven-check baseline session run reproduced the unstored fine-delta failure while coarse output, transient continuation, and durable restart controls passed. Its checkpoint ceilings (16 MiB per checkpoint, 32 MiB aggregate serialized retained checkpoints) address a different resource and should remain separate. In particular, cached/pinned/prepared checkpoint accounting should not be replaced by the per-request fix below.

Implemented and tested local approach (proposed defaults, not current upstream settings):

Resource	Proposed default	Config field in [responses]	Environment override
Logical retained response data across the turn	8 MiB	max_retained_bytes	AGENTIC_MAX_RETAINED_RESPONSE_BYTES
One upstream JSON body	16 MiB	max_upstream_json_bytes	AGENTIC_MAX_UPSTREAM_JSON_BYTES
One upstream SSE line	16 MiB	max_upstream_sse_line_bytes	AGENTIC_MAX_UPSTREAM_SSE_LINE_BYTES
One serialized client stream event	1 MiB	max_stream_event_bytes	AGENTIC_MAX_STREAM_EVENT_BYTES

• Positive, validated limits; environment overrides TOML independently. No lifetime streamed-byte limit. Raising defaults alone is insufficient: retained accounting replaces the previous item-state charge as deltas/completions update that state, rather than charging all wire snapshots.
• Keep the existing single normalization/ingestion path. Wire reads are bounded in inference.rs; retained output is measured in typed accumulator slots; stream delivery validates its own serialized event size; the engine owns the shared multi-round/tool/discovery allowance and persistence order.
• The prototype's retained-output metric uses UTF-8 content and 32 bytes per serialized value/container, with active text/argument buffers measured by length. Strings are not rescanned on every delta. Raw gateway tool output and serialized MCP discovery retain their existing byte measurements. This is logical resource accounting, not an RSS/allocator bound; transient parsing/serialization, execution copies, and allocator capacity require separate consideration. The configurable values are policy choices to review, not benchmark-derived safe memory ceilings.
• Preserve bounded delivery queues and cancellation. Retain the 1 MiB default client-event bound; reserve WebSocket routing overhead before core terminal validation so a response is not stored and then rejected only after stream_id is attached. generate: false validates both events before commit.
• Use error code response_resource_limit_exceeded with the resource/limit. Non-streaming executor failures return 502. Started streams emit response.failed when the diagnostic fits, otherwise a bounded error diagnostic. Reject terminal serialization before durable persistence or checkpoint publication; keep healthy-request recovery and fix: honor WebSocket response storage with bounded sessions #257's failure/eviction semantics.

Verification of the local patch on 2509342:

• 52 production-binary acceptance checks passed: 14 original regression/control cases; 24 independent-limit, boundary, TOML/environment precedence, failure, recovery, and database-count checks; two tiny-event-limit diagnostic checks; 12 transient/durable session and restart checks.
• Session checks include an unstored fine-delta response, same-connection continuation, a stored child of an unstored parent, full history restored after process restart, an unstored ID rejected after restart, and no cached/stored failed response. A failed same-lane continuation evicts its referenced parent; an independent request then succeeds.
• Eight new Rust unit tests passed: chunk/snapshot invariance, genuinely oversized active arguments, shared retained rounds, atomic boundary/overflow behavior, UTF-8/escape/structural accounting, retained JSON metadata, defaults/partial overrides, and invalid settings. Existing focused coverage also passed for MCP/tool aggregate limits, oversized-terminal non-persistence, lifecycle validation, backpressure/disconnect, and session budget/cancellation behavior.
• cargo fmt --all -- --check, git diff --check, and cargo clippy --locked --workspace --all-targets --all-features -- -D warnings passed.
• Full native Rust suite is not green: cargo test --locked --workspace --all-features --no-fail-fast reports the same 193 failing test cases on unpatched 2509342 and the patch, with matching names/counts: 192 fail opening Windows temporary SQLite paths; one oversized-request transport test fails with a connection-abort error. There are no additional failing test cases from the patch. The production harness uses working relative SQLite URLs and verifies persistence separately. Linux CI and live-model validation remain to be done.

The patch is a locally verified proposal, not an upstream change or a claim of complete platform validation. The regression report and the proposed default sizes can be evaluated independently.

[SProst]

Confirmed on Linux with a real vLLM engine serving Nemotron, including a successful comparison against the local response-resource patch described above. The cumulative-stream failure reproduces without the Windows build adaptation.

Tested on 2026-09-11:

• Linux x86_64 Docker container on Hugging Face Spaces; NVIDIA A100-SXM4-80GB, driver 580.178.04; kernel 6.12.100, glibc 2.39.
• vLLM 0.29.0, serving nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 at revision cc84af2fe71647d87f4486c064f320e1e7535243, alias nemotron-3.5-lightning.
• Model recipe retained: Humming MoE/linear backends, ModelOpt FP4, FlashInfer Mamba, nemotron_v3 reasoning parser, thinking enabled. No LiteLLM routing was used for these requests.
• Unmodified Linux baseline: 2509342dcbc12b89350cabdda17f7bb1a6b8a41b. Patched comparison: that same commit plus the local proposal. Both built with cargo build --locked --release -p agentic-server --bin agentic-server, Rust 1.98.0 / Cargo 1.98.0. The Windows-only borrow adaptation is inactive on Linux.

Live vLLM → gateway result

The following payload was sent directly to native vLLM, then separately through each gateway:

{
  "model": "nemotron-3.5-lightning",
  "input": "Print integers from 1 through 20000 in ascending order, one integer per line. Do not add any explanation. Continue counting without stopping.",
  "temperature": 0,
  "max_output_tokens": 8192,
  "stream": true,
  "store": true
}

Path	Observed result
Native vLLM	Reached 8,192 output tokens. Terminal response status incomplete, reason max_output_tokens.
Unpatched Linux gateway → vLLM	HTTP 200, then error / server_error: stream error: executor response budget exceeded 1048576 bytes. No normal response terminal or final usage.
Patched Linux gateway → vLLM	Delivered the stream through 8,192 output tokens, ending with response.incomplete, reason max_output_tokens, and final usage. No resource error.

Here incomplete is the expected token-budget termination, not the resource failure. Native vLLM names its terminal event response.completed while its embedded response has status: "incomplete"; the patched gateway emits the corresponding response.incomplete event.

Same captured model output, replayed through both Linux binaries

The live generations differed despite temperature 0, so I also replayed the exact native vLLM SSE capture, unchanged, through isolated baseline and patched Linux gateways. For the JSON control, the replay upstream returned that capture's terminal response object. No new model generation occurs in this control.

The capture contains:

• 1,594,460 raw SSE bytes, of which 1,301,571 bytes are charged by the baseline's normalized data: line accounting.
• Only 14,036 UTF-8 bytes of generated text: 6,771 reasoning bytes and 7,265 answer bytes. The largest data line is 17,119 bytes, far below the separate 256 KiB line limit.
• 1,672 reasoning deltas and 6,312 answer deltas, plus lifecycle/done snapshots.

Captured-output control	Unpatched 2509342	Local patch
HTTP SSE, store: true	Cumulative-budget error after 12,700 generated-text bytes	All 14,036 text bytes; expected response.incomplete
WebSocket, store: true	Same error at the same point	All text; expected response.incomplete
WebSocket, store: false	Same error at the same point	All text; expected response.incomplete
HTTP JSON, store: true	Succeeds with the complete captured response content	Succeeds with the same content

The failure point matches the source calculation exactly. Immediately before native event 6,657, cumulative charged bytes are 1,048,571. Its next 157-byte response.output_text.delta line would bring that to 1,048,728, above 1,048,576. All three unpatched streaming controls instead emit the budget error at sequence number 6,657.

For every patched replay mode, the final reasoning text and answer text match the captured native response byte for byte, as do response status, incomplete reason, and supported usage fields: 48 input tokens, 8,192 output tokens, 8,240 total, 1,870 reasoning, zero cached. This is not a claim of identical full JSON: the gateway rewrites streamed item IDs, resolves the reasoning item's status, and omits null optional fields and vLLM-specific empty/zero usage extensions. Those differences were inspected separately.

Limits and persistence still enforced

With the patched binary's AGENTIC_MAX_RETAINED_RESPONSE_BYTES=4096, the same capture is rejected with response_resource_limit_exceeded: HTTP JSON returns 502; HTTP SSE and both WebSocket store modes emit response.failed. The streaming failure occurs when the reasoning done snapshot becomes retained data; already-forwarded reasoning deltas are not a lifetime retained-byte charge. The isolated database remains empty after all four failures.

Across the ordinary replay controls, the baseline database contains exactly one response (the successful JSON control), and the patched database contains exactly three (store: true SSE, JSON, and WebSocket). The store: false WebSocket response adds no durable row.

Build and capture SHA-256 provenance

Baseline Linux binary:
a79f8deb4d41a79fa6372e58b187db401d67bc83bb83a334f1b284e6d389c1a7
Patched Linux binary:
b22af2a161f275d145a5571ca4bf8d16083a09d2569c7165f64bfb43f1ce8981
Applied local patch:
f9e3be040082d4a247a26387c9ffd85b3f678c7e0f952fb4b5a7ca6d7556df58
Native vLLM SSE capture:
00fd58e6257a09381c670fd427260e947c256c059d96e517f882a96076681128

The image build verified the patch checksum and git apply --check before application. Runtime hashes matched the build's baseline and patched binary hashes.

This adds live-model and Linux production-binary evidence to the original report: the regression is not specific to Windows, and the proposed accounting change fixes this observed failure on Linux too. It does not establish an RSS/allocator ceiling or replace a full Linux Rust test-suite/CI run.