A fully local multi-agent AI assistant with long-term memory and document knowledge retrieval. Built on LangGraph, Ollama, and PostgreSQL — no cloud APIs, no data leaving your machine.
Status: Active development. Core conversation, memory, and RAG pipelines are functional. Multi-agent coordination and several worker agents are in progress (see Roadmap).
The system is designed to become a deeply personalized assistant that knows your context over time — not a generic chatbot that forgets the previous message. Key design principles:
- Local-first: all inference runs via Ollama; all data stays in PostgreSQL on your machine
- Persistent memory: conversation history is continuously summarized and organized by subject, not just kept as a raw log
- Document grounding: answers can be grounded in your own documents (notes, PDFs) via RAG
- Extensible: new domain-specific worker agents can be added by registering them in the agent registry with a description and examples — routing is automatic
User Input
│
▼
┌─────────────────────────────────────────────────────┐
│ Orchestrator: Multi-Agent Coordinator (MAC) │
│ │
│ 1. Rewrite query using conversation history │
│ 2. Select agents via embedding similarity │
│ 3. Decide per-agent whether RAG is needed │
│ 4. Route to single agent or multi_agent_handler │
└────────────────────┬────────────────────────────────┘
│
┌────────────┼─────────────┐
▼ ▼ ▼
[Worker Agents] [Service Agents] [Memory Update]
General Conv. Files Manager (background thread)
(+ future) Memory Organizer
Tier 1 — Orchestrator (app/agents/orchestration/MAC/)
The Multi-Agent Coordinator handles every user turn:
- Fetches the last 20 conversation turns and recent memory summaries from PostgreSQL
- Rewrites the user query using conversation history and add implicit context
- Selects which worker agent(s) to invoke using cosine similarity against pre-computed agent description embeddings — no LLM call for routing
- For each selected agent, uses a lightweight LLM call to decide whether to run RAG
- After the response is returned, spawns a background task to update memory
Tier 2 — Services (shared infrastructure)
- Files Manager (
app/agents/services/files_manager/): Scans configured directories for Markdown, PDF, and TXT files. Chunks, embeds, and stores them in PostgreSQL via pgvector. Tracks file modification times to skip unchanged files. At query time, expands the query into 4 sub-queries, retrieves top-10 chunks each, deduplicates, loads from disk, and reranks with a cross-encoder (BAAI/bge-reranker-v2-m3) on CUDA. - Memory Organizer (
app/agents/services/memory_organizer/): After each conversation, summarizes the turn and organizes extracted facts into 6 structured subjects (personal identity, professional identity, health, finance, life context, preferences). Maintains a hierarchy of conversation summaries across multiple temporal layers.
Tier 3 — Workers (domain-specific)
- General Conversation (
app/agents/workers/general_conversation/): Handles open-ended queries using memory context and optional RAG.
New workers are registered in app/agents/utils.py (AGENT_REGISTRY) with a name, description, and example queries. The MAC picks them up automatically via embedding-based routing.
Embedding-based routing instead of LLM routing
The MAC selects agents by computing cosine similarity between the rewritten query and pre-embedded agent descriptions. This is faster and more deterministic than asking an LLM to choose. Agent descriptions and examples are embedded once at startup and reused.
Query rewriting before routing
The raw user message is often ambiguous out of context ("what about tomorrow?" without knowing what was discussed). Rewriting it against the last 10 turns produces a self-contained query that both routing and RAG can work with reliably.
Hierarchical memory instead of a flat log
Raw conversation logs are noisy and grow unboundedly. Instead, each turn is summarized and facts are extracted into structured subjects. Subject summaries are updated via a multi-layer hierarchy (individual turn → session → long-term), so older context is compressed, not discarded.
RAG per-agent, decided at query time
Not every query benefits from document retrieval. For each selected agent, the MAC checks whether its configured knowledge sources are relevant to the query before triggering the full RAG pipeline (4-query expansion + reranking). This avoids unnecessary GPU work on conversational queries.
| Component | Technology |
|---|---|
| Agent orchestration | LangGraph |
| LLM inference | Ollama (local) |
| Embeddings | qwen3-embedding:8b (4096-dim) |
| Reranker | BAAI/bge-reranker-v2-m3 (CUDA) |
| Vector search | PostgreSQL + pgvector |
| Fast model | qwen3:0.6b |
| Reasoning model | qwen3:8b |
| DB driver | psycopg2 |
| Package manager | uv |
- uv
- Docker
- Ollama with the following models pulled:
ollama pull qwen3:0.6b ollama pull qwen3:8b ollama pull qwen3-embedding:8b
- CUDA-capable GPU (required for reranker)
git clone <repository-url>
cd personal_assistant
# Install Python dependencies
uv sync
# Copy and fill in DB credentials
cp .env.example .env
# First-time setup: starts PostgreSQL via Docker, creates tables,
# and prompts you to configure document folders to index
python init_system.pyEach agent has a sibling agent.yml file:
app/agents/services/files_manager/agent.yml — document paths to index:
paths:
- path: /home/user/notes
description: "Personal Obsidian vault"
max_depth: 3app/agents/services/memory_organizer/agent.yml — memory subject definitions (what facts to extract and how to organize long-term memory).
app/agents/workers/general_conversation/agent.yml — behavioral hints for the conversation agent.
python main.pyType /scan_files during a session to re-index documents.
app/
├── agents/
│ ├── base.py # Embedding, reranking, AgentSelector
│ ├── utils.py # @singleton decorator, AGENT_REGISTRY
│ ├── orchestration/MAC/ # Multi-Agent Coordinator
│ ├── services/
│ │ ├── files_manager/ # Document ingestion and RAG retrieval
│ │ └── memory_organizer/ # Conversation memory and fact extraction
│ └── workers/
│ └── general_conversation/ # Conversational agent
├── db/
│ ├── config.py # DB connection config
│ └── database.py # DatabaseManager singleton
├── tools/ # Shared utilities (JSON repair, etc.)
├── graph.py # LangGraph assembly
├── models.py # LLM/embedding model instances
├── state.py # AssistantState TypedDict
└── main.py # REPL loop
tests/
├── conftest.py # Global mocks
├── test_agents_base.py
├── test_agents_utils.py
├── test_db_database.py
├── test_tools_utils.py
├── test_file_loader.py
└── integration/ # End-to-end tests against a real DB
- Create
app/agents/workers/<your_agent>/agent.pywith an async handler method - Add a YAML config
agent.ymlwithagent_files_knowledgedescribing what documents it can use - Register it in
app/agents/utils.py:AGENT_REGISTRY["your_agent"] = { "name": "Your Agent", "description": "...", "examples": ["example query 1", "example query 2"], }
- Add a node and edge in
app/graph.py
The MAC will automatically include it in embedding-based routing from that point on.
Worker Agents
- Research agent — web search + synthesis
- Task management — todo tracking and prioritization
- Calendar assistant — scheduling and planning
Memory System
- RAG over memory (currently retrieves only recent turns; should also retrieve semantically relevant older memories)
- Prompt injection detection in memory extraction
RAG Pipeline
- Background folder scanning (currently blocking on large directories)
- Improved chunking strategy
- Store and reuse folder-level summaries for RAG decisions
Core
- Multi-agent response aggregation (handler exists but synthesis quality is incomplete)
- Async DB driver (
asyncpg) to eliminate blocking DB calls from the event loop - Structured logging with per-turn correlation IDs
- LLM evaluation pipeline for retrieval and memory quality