Skip to content

Cache Anthropic model capabilities across turns and defer unused compaction lookups #178

Description

@Anteriousis

Problem

Every Anthropic provider child retrieves the selected model's output limit before it can start generation. That metadata is not cached across user turns. If a different compaction model is configured, its metadata is also fetched immediately, even when the turn never needs compaction.

This introduces one or two extra Models API calls on each user request, increases startup latency, and makes ordinary generation depend on an unused compaction-model lookup succeeding. These are API/latency costs, not generation-token charges.

Evidence and reproduction

Audited main at 7289751459ceace471ac771f3a0ec39b972d1949. Area: AI provider orchestration. Findings come from source inspection and isolated Python execution of the relevant source functions with mocked CAD/model boundaries; no paid model requests or live CAD mutations were made.

Executing the actual lookup helper ten times with the same fake client/model produced ten metadata retrievals. Source tracing confirms startup invokes it on each new child. With distinct primary/compaction models, the current startup path performs two lookups even for a successful one-response turn.

Proposed fix

Add a bounded capability cache in the parent/provider lifetime and pass validated capability metadata to the child. Scope entries to endpoint, model, and authentication context; use a TTL or explicit refresh so aliases and capabilities can change. Do not store credentials in cache telemetry.

Resolve a different compaction model lazily when compaction is actually needed. Preserve the model-reported output limits and SDK compatibility behavior introduced deliberately in #153; do not replace them with hard-coded small output caps.

Acceptance tests

  • Repeated turns using the same endpoint/model reuse fresh metadata across child processes.
  • Model, endpoint, authentication changes, and expiry trigger appropriate refresh.
  • No compaction-model lookup occurs on a turn that does not compact.
  • A compaction turn resolves the correct model's capabilities.
  • Invalid/failed metadata is not cached as a valid capability.
  • Existing maximum-output and older-SDK compatibility tests continue to pass.

Suggested priority: low; a small independent API-efficiency PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions