Support for Local LLMs (e.g., Ollama) & Best Practices for Agents with Local Knowledge

[EddyDezuraud]

Hi there,

First of all, thanks for the great work on mcp-agent! I’m exploring the possibility of using local LLMs (e.g., Ollama or other on-device models) within the mcp-agent framework.

Is it currently possible to integrate a local LLM like Ollama with mcp-agent? If so, what would be the best approach to set it up?
What are the best practices for building agents that incorporate local knowledge? Specifically, I’m looking to:
Process and query technical PDFs (e.g., documentation, research papers).
Use codebase data as a knowledge source for the agents.
Implement efficient retrieval mechanisms, possibly leveraging vector search (e.g., ChromaDB) or BM25.
Any insights or recommendations on how to achieve these goals efficiently within mcp-agent would be highly appreciated!

Thanks in advance!

[saqadri]

Hi there @EddyDezuraud, thank you for raising these questions and for trying out mcp-agent! Please keep the feedback coming. Good news is that I believe everything you asked here is already possible with the library.

Is it currently possible to integrate a local LLM like Ollama with mcp-agent? If so, what would be the best approach to set it up?

Yes, it is possible to use it with ollama. Please check out the Ollama example. Basically you can configure the openai settings with the ollama base_url, and it will work.

What are the best practices for building agents that incorporate local knowledge? Specifically, I’m looking to:
Process and query technical PDFs (e.g., documentation, research papers).
Use codebase data as a knowledge source for the agents.
Implement efficient retrieval mechanisms, possibly leveraging vector search (e.g., ChromaDB) or BM25.

Please take a look at the Streamlit RAG example which should help to get you started.

Basically you can use a vector DB with an MCP server (such as Qdrant, though there are other servers for other DB's -- see servers repo. Initialize that collection, and define that server in your mcp-agent.config.yaml so it is accessible to an agent, e.g.

https://github.com/lastmile-ai/mcp-agent/blob/main/examples/streamlit_mcp_rag_agent/mcp_agent.config.yaml#L16

MCP requires a slight mindset shift since the framework doesn't need to be monolithic anymore -- the servers provide a separation of concerns where they are responsible for different things (memory, retrieval, etc.).

Hope this helps!