Add Local RAT Support 🧠🐀

[GatienBoquet]

Introducing local RAT (Retrieval Augmented Thinking) enabling users to leverage locally hosted API endpoints for reasoning and response generation.
The implementation uses LM Studio for hosting local OpenAI-compatible large language models (LLMs) and has been tested to work seamlessly.

Changes

1. New Script: rat-local.py:

   • Implements the ModelChain class to integrate local DeepSeek and response clients.
   • Provides reasoning (get_deepseek_reasoning) and response generation (get_local_response) capabilities using locally hosted models.

2. Updated .env Configuration:

   • Introduces LOCAL_API_KEY for authentication with the local API.
   • Example value added to .env for user reference.

3. Model Constants:

   • Configurable constants added at the top of rat-local.py:

     DEEPSEEK_MODEL = "deepseek-r1-distill-llama-8b"
     RESPONSE_MODEL = "meta-llama-3.1-8b-instruct"

   • These should be updated to match the desired local models.

Key Features

• Local API Support: Connects to local LLMs via LOCAL_API_KEY and http://localhost:1234/v1.
• Reasoning Process: Extracts structured reasoning steps using <think> tags for enhanced interpretability.
• Streaming Responses: Provides real-time response streaming for a more interactive experience.

---

How to Test

1. Setup a Local API Server:
   • Ensure a local LLM API server (e.g., LM Studio) is running on http://localhost:1234/v1.
2. Configure the .env File:

   LOCAL_API_KEY=your_local_api_key_here
   

(for LM Studio you can set it to "lmstudio")

3. Run the Script:

   python rat-local.py

4. Commands to Test:
   • Standard Input: Test reasoning and response by asking questions.
   • model <name> Command: Switch between response models dynamically.
   • reasoning Command: Toggle reasoning process visibility.
   • clear Command: Clear the chat history.

---

Example

python rat-local.py

You: how to add 1+10?
Reasoning Process:
To solve the addition of 1 and 10, I first recognize that adding these two numbers will result in a total.

I start by identifying the place values involved. The number 1 is a single unit, while 10 has one ten value.

Next, I add the units together: 1 plus 0 (from 10) equals 1.

Then, I add the tens: 0 (from 1) plus 1 (from 10) equals 1.

Combining these results, the sum of 1 and 10 is 11.
Response:
The answer to 1 + 10 is 11.