Skip to content

Use one total retry budget for Anthropic stream requests #177

Description

@Anteriousis

Problem

One logical Anthropic generation can attempt the HTTP request 18 times on repeated connection/read timeouts. The SDK is configured with max_retries=2 (three transport attempts), and VibeCAD wraps it in another six-attempt retry loop.

The outer progress event reports at most six attempts, so it understates the actual request count. A read timeout after submission can cause the same prompt to be submitted again; billing depends on whether the service accepted and processed each attempt. Connection failures before delivery do not imply token charges.

Evidence and reproduction

Audited main at 7289751459ceace471ac771f3a0ec39b972d1949. Area: AI provider orchestration. Findings come from source inspection and isolated Python execution of the relevant source functions with mocked CAD/model boundaries; no paid model requests or live CAD mutations were made.

Validated with Anthropic SDK 0.116.0, an in-memory httpx.MockTransport that raises httpx.ReadTimeout, and the extracted production _stream_response_with_retries / classifier functions. The stream callback used the real client.messages.stream; sleeps were disabled for the test. Result: 18 mock HTTP requests for one logical generation. No request reached the network.

Proposed fix

Make SDK and application retries share one explicit total attempt/deadline budget. Preserve recovery for stream failures after headers, which the outer loop handles, while avoiding nested retry multiplication for pre-stream SDK failures. The choice of which layer owns each retry class should be explicit.

Expose logical generation ID, transport attempt count, and whether any response content was observed in content-free telemetry. Keep CAD tool execution tied to a completed valid response so retries do not duplicate mutations.

Acceptance tests

Use the actual packaged SDK with a mock HTTP transport:

  • Repeated connection/read timeouts remain within the configured total request budget.
  • A transient failure followed by success still completes.
  • A stream interrupted after content begins follows the bounded stream-recovery policy.
  • Bad requests fail immediately; 429/5xx backoff behavior remains deliberate.
  • Cancellation/deadline exhaustion prevents further attempts.
  • Report transport request counts, not just outer-loop iterations.

Suggested priority: high for API efficiency and failure latency. This does not assume every failed attempt is billable.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions