Skip to content

feat: provider adapter conformance test suite #118

Description

@arthurpanhku

Why (roadmap: Next)

Every model provider is an egress destination that user prompts — often proprietary source — get routed to. Today src/providers/ (anthropic.ts, openaiCompatible.ts) has per-adapter tests, but there's no shared contract every provider must satisfy. That makes "should we trust a new provider?" a subjective call each time (cf. #109). A conformance suite turns it into an objective, reviewable gate.

Sketch

  • A single parametrized suite in tests/providers/conformance.test.ts that runs the same assertions against each registered adapter.
  • Contract covers: streaming token deltas, tool/function-call round-trips, error/timeout normalization, and — most importantly for governance — that every request goes through the egress chokepoint (src/providers/egress.ts + trustedBaseUrls.ts), honors redaction, and produces the expected audit events.
  • A bypass-style assertion proving an adapter cannot reach a non-allow-listed base URL.
  • Fixture/mock transport so no real network calls are made.

Acceptance

  • Adding a new provider means implementing the adapter + passing the suite — no bespoke review of egress/redaction/audit behavior.
  • Existing anthropic and openaiCompatible adapters pass unchanged (or gaps they expose are fixed).

Constraints

Contract is narrowing-only: it asserts governance invariants, it can't be satisfied by weakening a control. Design the assertion list in the PR description first.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions