Self-host: allow a local / OpenAI-compatible chat model (vLLM, Ollama) instead of forcing OpenRouter

[Santofer]

Problem

The in-app chat agents (onboarding + SAM) are hardwired to OpenRouter. buildChatAgentModel (src/server/lib/openrouter.ts) calls createOpenRouter({ apiKey }) against OpenRouter's fixed endpoint, and the only knobs exposed to self-hosters are OPENROUTER_API_KEY and OPENROUTER_MODEL (src/env.d.ts). There's no way to point the chat at a self-hosted, OpenAI-compatible model.

For a self-hosted, privacy-conscious deployment this is the one place data still has to leave the box. We run OpenSEO on our own hardware specifically so client SEO data stays in-house — but the chat agent forces prompts (which include project domains, keywords, GSC-derived context) out to OpenRouter. We already run an OpenAI-compatible endpoint on the same host that could serve it:

$ curl http://<host>:8000/v1/models
{"object":"list","data":[{"id":"qwen3.5-9b","object":"model","owned_by":"vllm","root":"...Qwen3.5-9B...","max_model_len":16384,...}]}

vLLM (and Ollama, LM Studio, LiteLLM, etc.) all expose the OpenAI /v1/chat/completions shape, so the AI SDK could talk to them directly.

Proposal

Let self-hosters override the chat provider's base URL and key, and degrade the OpenRouter-only request options when a custom endpoint is set. Minimal shape:

• New env: CHAT_BASE_URL (or OPENROUTER_BASE_URL) + reuse OPENROUTER_API_KEY as the bearer (vLLM accepts any/no key).
• In buildChatAgentModel, when CHAT_BASE_URL is set:
  • pass baseURL to createOpenRouter (the provider already accepts it), or swap to @ai-sdk/openai's createOpenAI({ baseURL, apiKey });
  • drop the OpenRouter-specific options that a generic endpoint won't honor: usage: { include: true } (cost metering), provider: { order, zdr, allow_fallbacks } (routing), and the OpenRouter reasoning channel;
  • skip OpenRouter usage-cost metering for that path (openRouterCostUsd → 0 / "self-hosted").
• Docs note in SELF_HOSTING_DOCKER.md.

Caveats worth stating in the issue

The SAM agent is tool-calling (it drives the MCP tools). So a local model only fully replaces it if it supports OpenAI-style function calling. With vLLM that means launching the server with --enable-auto-tool-choice --tool-call-parser hermes (or the model-appropriate parser). Models without reliable tool-calling would get a degraded chat (no agentic tool use). Might be worth a startup check / doc caveat rather than silently failing.

Why upstream and not a fork

We deliberately track the official image (ghcr.io/every-app/open-seo:latest) to keep getting updates. A one-line base-URL override is small, self-contained, and useful to every self-hoster who wants a local/sovereign model or a cheaper gateway (LiteLLM, etc.). Happy to contribute the PR, including the tool-calling doc caveat, if a maintainer is open to the direction.

[bensenescu]

@Santofer Yup, I'm very open to this! Could you just provide clear instructions for how to set it up / test it with ollama once you put up the PR?

CHAT_BASE_URL sounds good too, I just picked OpenRouter since I thought the most people would be familiar with it