Why (roadmap: Next)
Every model provider is an egress destination that user prompts — often proprietary source — get routed to. Today src/providers/ (anthropic.ts, openaiCompatible.ts) has per-adapter tests, but there's no shared contract every provider must satisfy. That makes "should we trust a new provider?" a subjective call each time (cf. #109). A conformance suite turns it into an objective, reviewable gate.
Sketch
- A single parametrized suite in
tests/providers/conformance.test.ts that runs the same assertions against each registered adapter.
- Contract covers: streaming token deltas, tool/function-call round-trips, error/timeout normalization, and — most importantly for governance — that every request goes through the egress chokepoint (
src/providers/egress.ts + trustedBaseUrls.ts), honors redaction, and produces the expected audit events.
- A bypass-style assertion proving an adapter cannot reach a non-allow-listed base URL.
- Fixture/mock transport so no real network calls are made.
Acceptance
- Adding a new provider means implementing the adapter + passing the suite — no bespoke review of egress/redaction/audit behavior.
- Existing
anthropic and openaiCompatible adapters pass unchanged (or gaps they expose are fixed).
Constraints
Contract is narrowing-only: it asserts governance invariants, it can't be satisfied by weakening a control. Design the assertion list in the PR description first.
Why (roadmap: Next)
Every model provider is an egress destination that user prompts — often proprietary source — get routed to. Today
src/providers/(anthropic.ts,openaiCompatible.ts) has per-adapter tests, but there's no shared contract every provider must satisfy. That makes "should we trust a new provider?" a subjective call each time (cf. #109). A conformance suite turns it into an objective, reviewable gate.Sketch
tests/providers/conformance.test.tsthat runs the same assertions against each registered adapter.src/providers/egress.ts+trustedBaseUrls.ts), honors redaction, and produces the expected audit events.Acceptance
anthropicandopenaiCompatibleadapters pass unchanged (or gaps they expose are fixed).Constraints
Contract is narrowing-only: it asserts governance invariants, it can't be satisfied by weakening a control. Design the assertion list in the PR description first.