Local LM Studio/Ollama Custom Providers Support

[mintisan]

Do we currently support Local LM Studio/Ollama Custom Providers, considering that the recently released local models are also very powerful, especially in terms of tool calling ability?

If not supported, is there any plan to support it and when do you intend to add this feature?

If supported, how should it be set up? I can help with testing.
[I have a Mac Studio M3 512GB memory version and can test almost all open-source local models].

Thanks!

[minounou]

(same question: I tried the following on MacStudio but not connected yet), please let me know how to connect to local codeLLM Thanks!
...
cat < ~/.local/share/crush/crush.json
{
"provider": "openai",
"api_key": "EMPTY",
"base_url": "http://localhost:8000/v1",
"model": "qwen3-coder-local",
"default": true
}
EOF
...

[kujtimiihoxha]

Hey there, here is an example for both

{
  "$schema": "https://charm.land/crush.json",
  "providers": {
    "ollama": {
      "name": "Ollama",
      "base_url": "http://localhost:11434/v1/",
      "type": "openai",
      "models": [
        {
          "name": "Qwen 3 30B",
          "id": "qwen3:30b",
          "context_window": 256000,
          "default_max_tokens": 20000
        }
      ]
    },
    "lmstudio": {
      "name": "LM Studio",
      "base_url": "http://localhost:1234/v1/",
      "type": "openai",
      "models": [
        {
          "name": "Qwen 3 30B",
          "id": "qwen/qwen3-30b-a3b-2507",
          "context_window": 256000,
          "default_max_tokens": 20000
        }
      ]
    }
  }
}

This will make the providers and models available in the model selector

[snapo]

I did try that, unfortunately there seems a issue with tool calling....

Here is my config (/home/user/.config/crush/crush.json):

{
  "$schema": "https://charm.land/crush.json",
  "providers": {
    "ollama": {
      "name": "Ollama",
      "base_url": "http://mysuperaiserver.address:11434/v1/",
      "type": "openai",
      "models": [
        {
          "name": "Qwen 3 30B Thinking",
          "id": "huggingface.co/unsloth/Qwen3-30B-A3B-Thinking-2507-GGUF:Q6_K_XL",
          "context_window": 65536,
          "default_max_tokens": 20000
        },
        {
          "name": "Qwen 3 30B Coder",
          "id": "huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q6_K_XL",
          "context_window": 65536,
          "default_max_tokens": 20000
        }
      ]
    }
  }
}

After setting this i selected the coding model (the new qwen3 30B)

respnose i got was the model does not support tool calling.
when from the unsloth documentation. the chat template should in theory support it as far as i see:

or is this a issue from ollama itself?

Here is a example that works for tools:

# Your tool implementation
def square_the_number(num: float) -> dict:
    return num ** 2

# Define Tools
tools=[
    {
        "type":"function",
        "function":{
            "name": "square_the_number",
            "description": "output the square of the number.",
            "parameters": {
                "type": "object",
                "required": ["input_num"],
                "properties": {
                    'input_num': {
                        'type': 'number', 
                        'description': 'input_num is a number that will be squared'
                        }
                },
            }
        }
    }
]

import OpenAI
# Define LLM
client = OpenAI(
    # Use a custom endpoint compatible with OpenAI API
    base_url='http://localhost:8000/v1',  # api_base
    api_key="EMPTY"
)
 
messages = [{'role': 'user', 'content': 'square the number 1024'}]

completion = client.chat.completions.create(
    messages=messages,
    model="Qwen3-Coder-30B-A3B-Instruct",
    max_tokens=65536,
    tools=tools,
)

print(completion.choice[0])

[lefamoffat]

Found this issue looking for the same answer. @snapo try changing the domain, i.e.

"id": "hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q6_K_XL",

instead of

"id": "huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q6_K_XL",

and it should work. I got it working using the UD-Q4_K_XL version.

Here's my full config ($HOME/.config/crush/crush.json) :

{
  "$schema": "https://charm.land/crush.json",
  "providers": {
    "ollama": {
      "type": "openai",
      "base_url": "http://localhost:11434/v1/",
      "name": "Ollama",
      "models": [
        {
          "id": "hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL",
          "name": "Qwen3-Coder-30B (Unsloth)",
          "context_window": 256000,
          "default_max_tokens": 20000
        }
      ]
    }
  }
}

[snapo]

@lefamoffat OMG , i had to re-download it but it realy was because of the hf/huggingface domain... i still dont get why / how... and even then i would never have found out about that.... how did you find that?

now its at least doing something... even initialization dosent look correct (might be because of my _ folder :-)

[lefamoffat]

Nice 🙌 I followed their ollama configuration tutorial, so I was only familiar with the hf.co domain and that was the only difference that I could see between our configs. Best to check what ollama IDs are available with ollama list

[DeveloperPaul123]

I tried the above config with LM Studio but crush is not showing any response even though LM studio shows that it's generating a response. This is the config I used:

{
  "$schema": "https://charm.land/crush.json",
  "providers": {
    "lmstudio": {
      "name": "LM Studio",
      "base_url": "http://localhost:1234/v1/",
      "type": "openai",
      "models": [
        {
          "supports_attachments": true,
          "has_reasoning_efforts": false,
          "can_reason": false,
          "name": "Qwen 3 Coder 30B",
          "id": "qwen/qwen3-coder-30B",
          "context_window": 256000,
          "default_max_tokens": 10000
        }
      ]
    }
  },
  "models": {
    "large": {
      "model": "qwen/qwen3-coder-30B",
      "provider": "lmstudio"
    }
  }
}

[AbeEstrada]

Downloaded the newest ollama run qwen3-coder

My config ($HOME/.config/crush/crush.json)

{
	"$schema": "https://charm.land/crush.json",
	"providers": {
		"ollama": {
			"type": "openai",
			"base_url": "http://localhost:11434/v1/",
			"name": "Ollama",
			"models": [
				{
					"id": "qwen3-coder",
					"name": "Qwen3-Coder (Ollama)",
					"context_window": 256000,
					"default_max_tokens": 20000,
					"cost_per_1m_in": 0,
					"cost_per_1m_out": 0,
					"cost_per_1m_in_cached": 0,
					"cost_per_1m_out_cached": 0,
					"supports_attachments": true,
					"has_reasoning_efforts": false,
					"can_reason": false
				}
			]
		}
	}
}

I get this error:

POST "http://localhost:11434/v1/chat/completions": 400 Bad Request 
{
  "message": "registry.ollama.ai/library/qwen3-coder:latest does not support tools",
  "type": "api_error",
  "param": null,
  "code": null
}

I tried changing the id to:

• registry.ollama.ai/library/qwen3-coder
• registry.ollama.ai/library/qwen3-coder:latest

[DaveChini]

any idea why tool calls are failing?

[AbeEstrada]

qwen3-coder

ollama/ollama#11621

They need to update the template to include the tool calling

[gitmeatarru]

Who can help setup crush file in windows for using LM Studio?
i`m not see any crush.json in c:\Users\user.config\crush\

Next, i`m create this file in this DIR
And crush cant see this models in models lis =(

My crush.json file:

{
  "$schema": "https://charm.land/crush.json",
  "providers": {
    "ollama": {
      "name": "Ollama",
      "base_url": "http://localhost:11434/v1/",
      "type": "openai",
      "models": [
        {
          "name": "Qwen 3 30B",
          "id": "qwen3:30b",
          "context_window": 256000,
          "default_max_tokens": 20000
        }
      ]
    },
    "lmstudio": {
      "name": "LM Studio",
      "base_url": "http://localhost:1234/v1/",
      "type": "openai",
      "models": [
        {
          "name": "Qwen 3 30B LM",
          "id": "qwen/qwen3-coder-30b",
          "context_window": 22222,
          "default_max_tokens": 20000
        }
      ]
    }
  }
}

[minounou]

I could connect to qwen2.5-coder by the following (but for qwen3-coder there is error(not-supporting tool-calling?):

cat > ~/.local/share/crush/crush.json << 'EOF'
{
  "$schema": "https://charm.land/crush.json",
  "default_provider": "ollama",
  "default_model": "qwen2.5-coder:32b",
  "providers": {
    "ollama": {
      "type": "openai",
      "base_url": "http://127.0.0.1:11434/v1",
      "api_key": "ollama",
      "models": [
        {
          "id": "qwen2.5-coder:32b",
          "name": "qwen2.5-coder:32b",
          "cost_per_1m_in": 0,
          "cost_per_1m_out": 0,
          "cost_per_1m_in_cached": 0,
          "cost_per_1m_out_cached": 0,
          "context_window": 32768,
          "default_max_tokens": -1,
          "supports_tools": false
        },
        {
          "id": "llama3.1:8b",
          "name": "llama3.1:8b",
          "cost_per_1m_in": 0,
          "cost_per_1m_out": 0,
          "cost_per_1m_in_cached": 0,
          "cost_per_1m_out_cached": 0,
          "context_window": 131072,
          "default_max_tokens": -1,
          "supports_tools": true
        },
        {
          "id": "qwen3-coder:latest",
          "name": "qwen3-coder:latest",
          "cost_per_1m_in": 0,
          "cost_per_1m_out": 0,
          "cost_per_1m_in_cached": 0,
          "cost_per_1m_out_cached": 0,
          "context_window": 131072,
          "default_max_tokens": -1,
          "supports_tools": false
        }
      ]
    }
  }
}
EOF

crush

ollama list

NAME                  ID              SIZE      MODIFIED          
llama3.1:8b           46e0c10c039e    4.9 GB    23 minutes ago       
qwen3-coder:latest    ad67f85ca250    18 GB     About an hour ago    
qwen2.5-coder:32b     b92d6a0bd47e    19 GB     9 days ago

[dayofdoom]

when using llama3.1:8B

i can see my models which have been configured in the crush.json

when using qwen2.5-coder:32b

qwen3-coder is presenting the same inability to call tools, i tried making a custom model that supported tools with ollama, but that was bust.

i tried some of the unsloth models mentioned above - same type of issues. i'm enjoying crush with the non-ollama providers but looking forward to local only gen with ollama models.

[simplisit]

People are saying Qwen3 coder uses xml for tool calling not json and it's throwing a lot of tools off. Apparently https://github.com/QwenLM/qwen-code handles the requests gracefully but nothing else does yet. Can anyone try another tool enabled coding model and confirm? Maybe yank out the xml tool handling from qwen-code?