diff --git a/docs/integrations/passthrough.mdx b/docs/integrations/passthrough.mdx index 68dda7e36e8..8ae2a3ebb9a 100644 --- a/docs/integrations/passthrough.mdx +++ b/docs/integrations/passthrough.mdx @@ -25,15 +25,18 @@ When you use passthrough endpoints, the request still flows through Bifrost core - `/bedrock_passthrough` Default provider: `bedrock` (restricted to three operations, see [Bedrock Passthrough](#bedrock-passthrough)) +A custom provider built on `openai`, `anthropic`, `gemini` or `bedrock` is reached on its base provider's endpoint with the `x-model-provider: ` header, once it has allowed routes (see [Route allowlist](#route-allowlist)). + --- ## How It Works 1. Send your request to a passthrough endpoint (OpenAI, Anthropic, Azure, or GenAI passthrough). 2. The integration strips the passthrough prefix and forwards the remaining provider-native path/body. -3. **Bifrost picks the provider key.** Client-supplied provider credentials (`authorization`, `api-key`, `x-api-key`, `x-goog-api-key`) are stripped from the request, and Bifrost selects a key from its own key config for the resolved provider and model. -4. Bifrost handles provider execution through core inference and plugin pipelines. -5. Response status, headers, and body are returned as passthrough output (for both stream and non-stream requests). +3. Bifrost checks the method and path against the provider's [route allowlist](#route-allowlist) and refuses a route it does not allow with `403`, before any key is selected. +4. **Bifrost picks the provider key.** Client-supplied provider credentials (`authorization`, `api-key`, `x-api-key`, `x-goog-api-key`) are stripped from the request, and Bifrost selects a key from its own key config for the resolved provider and model. +5. Bifrost handles provider execution through core inference and plugin pipelines. +6. Response status, headers, and body are returned as passthrough output (for both stream and non-stream requests). **Authenticate to Bifrost with your Bifrost virtual key, not your provider API key.** Passthrough is not a credential proxy — provider keys in the incoming request are never forwarded upstream. Bifrost always injects the key it selects from its configured keys. @@ -91,6 +94,98 @@ Claude Code OAuth sign-in (`Authorization: Bearer sk-ant-oat…`) is handled sep --- +## Route allowlist + +Each provider has a list of passthrough routes Bifrost forwards to it. A request whose method and path match no rule is refused with `403` before a key is selected. + +| Mode | Built-in provider | Custom provider | Saved as | +|---|---|---|---| +| **All routes** | Default | Opt-in | Built-in: no policy. Custom: `[{"path": "/**"}]` | +| **Custom** | Opt-in | Opt-in | Your rules | +| **Off** | Opt-in | Default | Built-in: `[]`. Custom: no policy | + +The **Passthrough** tab is shown for OpenAI, Anthropic, Azure, Gemini and Vertex, and for custom providers built on `openai`, `anthropic`, `gemini` or `bedrock`. `/chatgpt_passthrough` is not affected. + +### Rules + +Each rule has a `path` and optional `methods`. A request is forwarded when any rule matches. + +- `path` is the path after the endpoint prefix, as the client calls it: `/v1/chat/completions` for `/openai_passthrough/v1/chat/completions`. +- `*` matches within one segment, so `/v1/files/*` matches `/v1/files/file-abc` but not `/v1/files/file-abc/content`. It can also match part of a segment: `*:generateContent`. +- A final `/**` matches the path and everything below it. `/**` alone matches every route. +- `methods` takes `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD` and `OPTIONS`. Omit it to allow any method. +- Paths must start with `/` and must not contain `?`, `#`, `%`, `\`, `[`, `]`, or empty, `.` or `..` segments. `**` is allowed only as the last segment. + +### Configuration + + + + +1. Open **Providers** and select the provider. +2. Open the **Passthrough** tab and pick a mode under **Allowed routes**. +3. For **Custom**, add a row per rule with its **Methods** and **Path**. +4. Click **Save Passthrough**. + +![Passthrough allowed routes](../media/ui-provider-passthrough-routes.png) + + + + +`PUT /api/providers/{provider}` replaces the provider's network and concurrency settings with what you send, so read the current config, set `passthrough`, and send it back. Omitting `passthrough` keeps the saved policy; `"passthrough": null` restores the default. + +```bash +curl -s 'http://localhost:8080/api/providers/openai' \ + | jq 'del(.name, .keys, .provider_status, .status, .description, .config_hash) | .passthrough = {"allowed_routes": [{"methods": ["POST"], "path": "/v1/chat/completions"}, {"methods": ["GET"], "path": "/v1/files/**"}]}' \ + | curl --location --request PUT 'http://localhost:8080/api/providers/openai' \ + --header 'Content-Type: application/json' \ + --data @- +``` + +An invalid rule returns `400` with `Invalid passthrough config: ...`. + + + + +```json +{ + "providers": { + "openai": { + "keys": [{ "name": "openai-key", "value": "env.OPENAI_API_KEY", "models": ["*"], "weight": 1.0 }], + "passthrough": { + "allowed_routes": [ + { "methods": ["POST"], "path": "/v1/chat/completions" }, + { "methods": ["GET"], "path": "/v1/files/**" } + ] + } + } + } +} +``` + +| Field | Type | Required | Description | +|---|---|---|---| +| `passthrough.allowed_routes` | array | Yes | Rules; an empty array forwards nothing | +| `allowed_routes[].path` | string | Yes | Path after the endpoint prefix | +| `allowed_routes[].methods` | array of strings | No | HTTP methods; omit for any | + + + + +### Custom providers + +A custom provider forwards no passthrough routes until you pick **All routes** or **Custom**. Call its base provider's endpoint with the `x-model-provider` header: + +```bash +curl -X POST http://localhost:8080/openai_passthrough/v1/chat/completions \ + -H "Content-Type: application/json" \ + -H "x-model-provider: my-openai-compatible" \ + -d '{"model": "my-model", "messages": [{"role": "user", "content": "hi"}]}' +``` + +If the provider sets [`allowed_requests`](../providers/custom-providers#allowed-request-types), it must also allow `passthrough`, and `passthrough_stream` for streaming routes; otherwise an allowed route is refused with `400` `passthrough is not supported by provider`. Saving routes in the UI turns both on. With the API or `config.json`, set them yourself. + +--- + ## Usage Examples ### OpenAI Passthrough diff --git a/docs/media/ui-provider-passthrough-routes.png b/docs/media/ui-provider-passthrough-routes.png new file mode 100644 index 00000000000..94f07533191 Binary files /dev/null and b/docs/media/ui-provider-passthrough-routes.png differ diff --git a/docs/openapi/openapi.json b/docs/openapi/openapi.json index b714c979d3a..5b62b67e866 100644 --- a/docs/openapi/openapi.json +++ b/docs/openapi/openapi.json @@ -139135,6 +139135,45 @@ "type": "number", "minimum": 0 }, + "input_cost_per_token_voice_live": { + "type": "number", + "minimum": 0, + "description": "Azure Voice Live input rate per text token, used on Voice Live turns in place of input_cost_per_token. Each _voice_live rate falls back to its normal rate when unset." + }, + "output_cost_per_token_voice_live": { + "type": "number", + "minimum": 0 + }, + "cache_read_input_token_cost_voice_live": { + "type": "number", + "minimum": 0 + }, + "input_cost_per_audio_token_voice_live": { + "type": "number", + "minimum": 0 + }, + "output_cost_per_audio_token_voice_live": { + "type": "number", + "minimum": 0 + }, + "input_cost_per_standard_speech_audio_token_voice_live": { + "type": "number", + "minimum": 0, + "description": "Azure Voice Live rate per input audio token handled by Azure speech to text." + }, + "output_cost_per_standard_speech_audio_token_voice_live": { + "type": "number", + "minimum": 0, + "description": "Azure Voice Live rate per output audio token from an Azure standard voice." + }, + "input_cost_per_custom_speech_audio_token_voice_live": { + "type": "number", + "minimum": 0 + }, + "output_cost_per_custom_speech_audio_token_voice_live": { + "type": "number", + "minimum": 0 + }, "output_cost_per_video_per_second_480p": { "type": "number", "minimum": 0, diff --git a/docs/providers/custom-providers.mdx b/docs/providers/custom-providers.mdx index eece2fc5aa8..d112c38416e 100644 --- a/docs/providers/custom-providers.mdx +++ b/docs/providers/custom-providers.mdx @@ -259,6 +259,10 @@ Custom providers can be built on these supported providers: - `replicate` - Replicate - `typesafe` - TypeSafe (decisions and model listing only; see [TypeSafe](/providers/supported-providers/typesafe#custom-typesafe-providers)) +### Passthrough + +A custom provider built on `openai`, `anthropic`, `gemini` or `bedrock` can be called through its base provider's passthrough endpoint with the `x-model-provider: ` header. Passthrough is off until you allow routes in the provider's **Passthrough** tab or its `passthrough.allowed_routes`. See [Route allowlist](/integrations/passthrough#route-allowlist). + ### Authentication (OAuth-minted keys) A custom provider on the `openai` base normally carries a static API key in each key's `value`. Upstreams that sit behind an identity provider and issue short-lived access tokens can instead be given an `oauth_key_config` on the key: leave `value` empty and Bifrost obtains the token from the token endpoint, sends it as the bearer, caches it per credential, collapses concurrent requests onto one mint, and refreshes it a minute before it expires. A rejected credential is backed off rather than retried on every request. The same machinery serves Databricks OAuth M2M; it applies to the standard `openai` provider too.