diff --git a/docs/advanced-usage/local-models.md b/docs/advanced-usage/local-models.md index 7294dcdc..081b0923 100644 --- a/docs/advanced-usage/local-models.md +++ b/docs/advanced-usage/local-models.md @@ -23,13 +23,18 @@ Roo Code currently supports two main local model providers: ## Setting Up Ollama -1. **Download and Install Ollama:** Download the Ollama installer for your operating system from the [Ollama website](https://ollama.com/). Follow the installation instructions. +1. **Download and Install Ollama:** Download the Ollama installer for your operating system from the [Ollama website](https://ollama.com/). Follow the installation instructions. Make sure Ollama is running + + ```bash + ollama serve + ``` 2. **Download a Model:** Ollama supports many different models. You can find a list of available models on the [Ollama website](https://ollama.com/library). Some recommended models for coding tasks include: * `codellama:7b-code` (good starting point, smaller) * `codellama:13b-code` (better quality, larger) * `codellama:34b-code` (even better quality, very large) + * `qwen2.5-coder:32b` * `mistralai/Mistral-7B-Instruct-v0.1` (good general-purpose model) * `deepseek-coder:6.7b-base` (good for coding tasks) * `llama3:8b-instruct-q5_1` (good for general tasks) @@ -37,25 +42,43 @@ Roo Code currently supports two main local model providers: To download a model, open your terminal and run: ```bash - ollama run + ollama pull ``` For example: ```bash - ollama run codellama:7b-code + ollama pull qwen2.5-coder:32b + ``` + +3. **Configure the Model:** by default, Ollama uses a context window size of 2048 tokens, which is too small for Roo Code requests. You need to have at least 12k to get decent results, ideally - 32k. To configure a model, you actually need to set its parameters and save a copy of it. + + Load the model (we will use `qwen2.5-coder:32b` as an example): + + ```bash + ollama run qwen2.5-coder:32b + ``` + + Change context size parameter: + + ```bash + /set parameter num_ctx 32768 + ``` + + Save the model with a new name: + + ```bash + /save your_model_name ``` - - This will download the model and then run it in your terminal. You can type `/bye` to exit the model. - - **Important:** The first time you run a model, Ollama *must* be running. If you run with a model and Ollama is not open, VSCode may hang. + -3. **Configure Roo Code:** +4. **Configure Roo Code:** * Open the Roo Code sidebar (🚀 icon). * Click the settings gear icon (⚙️). * Select "ollama" as the API Provider. - * Enter the Model ID (e.g., `codellama:7b-code`). + * Enter the Model name from the previous step (e.g., `your_model_name`). * (Optional) You can configure the base URL if you're running Ollama on a different machine. The default is `http://localhost:11434`. + * (Optional) Configure Model context size in Advanced settings, so Roo Code knows how to manage its sliding window. ## Setting Up LM Studio @@ -85,4 +108,4 @@ Roo Code currently supports two main local model providers: * **Slow Response Times:** Local models can be slower than cloud-based models, especially on less powerful hardware. If performance is an issue, try using a smaller model. -* **Model Not Found:** Ensure you have typed in the name of the model correctly. If you're using Ollama, use the same name that you provide in the `ollama run` command. \ No newline at end of file +* **Model Not Found:** Ensure you have typed in the name of the model correctly. If you're using Ollama, use the same name that you provide in the `ollama run` command.