[Question]: Receiving a connection error while connecting to Ollama on my host machine

[joelclimbsthings]

What is your question?

I've been trying to get my local instance of Ollama to work with LibreChat for a while to no avail. I'll post my librechat.yml file in the "More Details" section below. I'm runng Librechat within Docker.

Ollama is running fine on my local machine--if I run the following, I get a valid response as expected:

curl http://localhost:11434/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "phi3",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Hello!"
            }
        ]
    }'

However I'm getting the following error within the ./logs directory.

{"cause":{"code":"ECONNREFUSED","errno":"ECONNREFUSED","message":"request to http://host.docker.internal:11434/v1/chat/completions failed, reason: connect ECONNREFUSED 172.17.0.1:11434","type":"system"},"level":"error","message":"[handleAbortError] AI response error; aborting request: Connection error.","stack":"Error: Connection error.\n    at OpenAI.makeRequest (/app/api/node_modules/openai/core.js:292:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async ChatCompletionStream._createChatCompletion (/app/api/node_modules/openai/lib/ChatCompletionStream.js:53:24)\n    at async ChatCompletionStream._runChatCompletion (/app/api/node_modules/openai/lib/AbstractChatCompletionRunner.js:314:16)"}

• [ ]

More Details

Here is my librechat.yml file:

For more information, see the Configuration Guide:

https://www.librechat.ai/docs/configuration/librechat_yaml

Configuration version (required)

version: 1.0.9

Cache settings: Set to true to enable caching

cache: true

endpoints:
custom:
- name: "Ollama"
apiKey: "ollama"
# use 'host.docker.internal' instead of localhost if running LibreChat in a docker container
baseURL: "http://host.docker.internal:11434/v1/chat/completions"
models:
default: [
"codestral",
"phi3",
"mixtral:8x7b-instruct-v0.1-q4_1",
]
# fetching list of models is supported but the name field must start
# with ollama (case-insensitive), as it does in this example.
fetch: true
titleConvo: true
titleModel: "current_model"
summarize: false
summaryModel: "current_model"
forcePrompt: false
modelDisplayLabel: "Ollama"

What is the main subject of your question?

Endpoints

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[fuegovic]

try to set the baseURL to : http://host.docker.internal:11434/v1/

[joelclimbsthings]

Thanks for the quick response, although (I should have mentioned) I had tried that as well. Same error:

{"cause":{"code":"ECONNREFUSED","errno":"ECONNREFUSED","message":"request to http://host.docker.internal:11434/v1/chat/completions failed, reason: connect ECONNREFUSED 172.17.0.1:11434","type":"system"},"level":"error","message":"[handleAbortError] AI response error; aborting request: Connection error.","stack":"Error: Connection error.\n    at OpenAI.makeRequest (/app/api/node_modules/openai/core.js:292:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async ChatCompletionStream._createChatCompletion (/app/api/node_modules/openai/lib/ChatCompletionStream.js:53:24)\n    at async ChatCompletionStream._runChatCompletion (/app/api/node_modules/openai/lib/AbstractChatCompletionRunner.js:314:16)"}

[danny-avila]

What's your operating system? is Ollama running without docker? you may need to use the IP of your machine

[ic-mw]

We have the same problem, running Ubuntu 23.10 on a server with the default docker version from apt/Ubuntu repositories.

Getting the following errors.
warn: [OpenAIClient.chatCompletion][stream] API error
error: [handleAbortError] AI response error; aborting request: Connection error.

Everything else is running fine, we only have a problem connecting to the Ollama server running on the host. And the Ollama instance is definitely working, a curl on the relevant endpoints goes through without a hitch.

[danny-avila]

How are you running Ollama? Networking for this is a bit tricky, the easiest solution is to share the same container network

[ic-mw]

Ollama is running as service on the host machine, and LibreChat is running via Docker.
Since your default setup looks like this:

So this should work, but it doesn't:
baseURL: "http://host.docker.internal:11434/v1"

I'm running out of ideas here...

[danny-avila]

So this should work, but it doesn't: baseURL: "http://host.docker.internal:11434/v1"

I'm running out of ideas here...

you're right it should, but this is mainly expected behavior for Docker Desktop:

This might be relevant: https://stackoverflow.com/questions/48546124/what-is-the-linux-equivalent-of-host-docker-internal

Try curling from the container shell.

AI convo on this topic:
New_Chat(3).md

[ic-mw]

Thanks for your reply and assistance so far. The "AI powered" suggestions are mostly valid (from a logical view), but the only configuration which actually works (at least when I try to curl it up from inside the librechat container) is the one where I allow Host networking for the container, which is not a robust option because of security concerns.
Unfortunately I don't have the time to do many more hours of trial & error, but if I happen to find a solution I'll share.

[joelclimbsthings]

I've still been unable to resolve this. I'm on linux, and oddly enough I've found that I can access a basic HTTP server running on my host with the host.docker.internal hostname, but still not ollama which is also work fine on the host. Perhaps the problem is with how the ollama service is made available on the host?

If I add the network_mode: host option to the docker-compose.yaml file, then I can access Ollama from the container, but the Librechat UI suddenly doesn't load. Would I need to tweak something else to make that work?

[joelclimbsthings]

It turns out the issue was that Ollama by default only responds to the 127.0.0.1 interface. Since my host service is run with systemd, I was able to add this to the ollama.service file to resolve the issue:

Environment="OLLAMA_HOST=0.0.0.0:11434"

[danny-avila]

Thank you for looking into this!

[ic-mw]

Yes, excellent find, thanks a lot.

[scoooducks]

Where is the ollama.service file?

[joelclimbsthings]

I'm on an Ubuntu system, and it's located at /etc/systemd/system