Skip to content

Proxy uses a total-request timeout for streaming; should use read_timeout (needs reqwest 0.12) #190

Description

@cashcon57

The local API proxy forwards chat-completion requests to llama-server with a reqwest client built using a total-request timeout:

// src-tauri/src/core/server/proxy.rs (both the general and no_proxy local clients)
let client = Client::builder()
    .timeout(std::time::Duration::from_secs(proxy_timeout)) // default 600s
    .build()?;

reqwest::ClientBuilder::timeout() caps the entire request, including reading the streamed response body. For a streaming generation that means the timeout budget covers prompt prefill + the whole decode. On a slow-prefill model with a large context (e.g. the ternary 27B after a long agentic thread), a single healthy turn can approach or exceed the cap and get cut mid-stream, surfacing to the user as a generation error / unexpected EOF even though the server is fine and still working.

The correct semantic for a streaming proxy is an idle / read timeout (abort only if no bytes arrive for N seconds), not a total-duration cap. That way a long generation with steady token output never trips it, but a genuinely hung server still does.

reqwest::ClientBuilder::read_timeout() provides exactly this, but it was added in reqwest 0.12; this crate is on 0.11.27, so the fix requires a dependency upgrade.

Scope / plan:

  • Upgrade reqwest 0.11 → 0.12 (note blast radius: reqwest is used across src-tauri and pulled in by the MCP client via the transport-*-reqwest features, so the bump likely cascades to rmcp and related deps).
  • Switch the two proxy forwarding clients (proxy.rs) to .read_timeout(...) for the streaming path, keeping a total .timeout() only where a hard cap is actually wanted (non-streaming calls).
  • Verify a long local generation no longer gets cut, and that a killed llama-server still fails promptly.

Tracked separately from the partial-work-preservation fix (which addresses losing output when a generation does get interrupted, regardless of cause).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions