Skip to content

feat: add local Ollama provider support with prefix stripping #561

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
niko4244 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: add local Ollama provider support with prefix stripping #561

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 7 additions & 16 deletions server/src/providers/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,23 +94,14 @@ register(new OpenAICompatProvider({
}));

// Moonshot direct integration was dropped in V4 (paid-only); MiniMax direct
// was dropped in V4 (superseded by the OpenRouter route).

// Ollama Cloud — OpenAI-compatible. Free plan: 1 concurrent model, 5h session
// caps, GPU-time-based quota (not per-token). Many catalog models on the
// /v1/models list are subscription-only — Free returns 403 with an explicit
// "this model requires a subscription" message. Catalog rows are filtered to
// confirmed-Free entries.
//
// Frontier reasoning models (glm-4.7, kimi-k2-thinking, cogito-2.1:671b)
// regularly take 30-90s on Ollama Cloud Free, so the timeout is bumped from
// the default 15s. Ollama returns reasoning in `message.reasoning` (not
// `reasoning_content`) — handled by normalizeChoices.
// was dropped in V4 (superseded by the OpenRouter route).// Local Ollama — OpenAI-compatible. Runs at 127.0.0.1:11434. Models
// are the local models pulled via `ollama pull`. No auth required.
// Timeout matches the custom-provider extended timeout.
register(new OpenAICompatProvider({
platform: 'ollama',
name: 'Ollama Cloud',
baseUrl: 'https://ollama.com/v1',
timeoutMs: 120000,
platform: 'ollama',
name: 'Local Ollama',
baseUrl: 'http://127.0.0.1:11434/v1',
timeoutMs: 120000,
}));

// Kilo AI Gateway — OpenAI-compatible aggregator. Anonymous access works
Expand Down
8 changes: 8 additions & 0 deletions server/src/providers/openai-compat.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ export class OpenAICompatProvider extends BaseProvider {
modelId: string,
options?: CompletionOptions,
): Promise<ChatCompletionResponse> {
// Strip platform prefix for providers that use bare model names (local Ollama).
if (this.platform === 'ollama') {
modelId = modelId.replace(/^ollama\//, '');
}
const res = await this.fetchWithTimeout(`${this.baseUrl}/chat/completions`, {
method: 'POST',
headers: {
Expand Down Expand Up @@ -80,6 +84,10 @@ export class OpenAICompatProvider extends BaseProvider {
modelId: string,
options?: CompletionOptions,
): AsyncGenerator<ChatCompletionChunk> {
// Strip platform prefix for providers that use bare model names (local Ollama).
if (this.platform === 'ollama') {
modelId = modelId.replace(/^ollama\//, '');
}
const res = await this.fetchWithTimeout(`${this.baseUrl}/chat/completions`, {
method: 'POST',
headers: {
Expand Down