lfnovo · lfnovo · Jan 5, 2026 · Jan 3, 2026 · Jan 3, 2026 · Jan 3, 2026
diff --git a/.env.example b/.env.example
@@ -191,7 +191,7 @@ SURREAL_URL="ws://surrealdb/rpc:8000"
 SURREAL_USER="root"
 SURREAL_PASSWORD="root"
 SURREAL_NAMESPACE="open_notebook"
-SURREAL_DATABASE="staging"
+SURREAL_DATABASE="open_notebook"
 
 # RETRY CONFIGURATION (surreal-commands v1.2.0+)
 # Global defaults for all background commands unless explicitly overridden at command level

diff --git a/.gitignore b/.gitignore
@@ -133,4 +133,8 @@ doc_exports/
 specs/
 .claude
 
-.playwright-mcp/
+.playwright-mcp/
+
+
+
+**/*.local.md
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -1,3 +1,219 @@
+# Open Notebook - Root CLAUDE.md
 
-We have a good amount of documentation on this project on the ./docs folder. Please read through them when necessary, and always review the docs/index.md file before starting a new feature so you know at least which docs are available. 
+This file provides architectural guidance for contributors working on Open Notebook at the project level.
 
+## Project Overview
+
+**Open Notebook** is an open-source, privacy-focused alternative to Google's Notebook LM. It's an AI-powered research assistant enabling users to upload multi-modal content (PDFs, audio, video, web pages), generate intelligent notes, search semantically, chat with AI models, and produce professional podcasts—all with complete control over data and choice of AI providers.
+
+**Key Values**: Privacy-first, multi-provider AI support, fully self-hosted option, open-source transparency.
+
+---
+
+## Three-Tier Architecture
+
+```
+┌─────────────────────────────────────────────────────────┐
+│              Frontend (React/Next.js)                    │
+│              frontend/ @ port 3000                       │
+├─────────────────────────────────────────────────────────┤
+│ - Notebooks, sources, notes, chat, podcasts, search UI  │
+│ - Zustand state management, TanStack Query (React Query)│
+│ - Shadcn/ui component library with Tailwind CSS         │
+└────────────────────────┬────────────────────────────────┘
+                         │ HTTP REST
+┌────────────────────────▼────────────────────────────────┐
+│              API (FastAPI)                              │
+│              api/ @ port 5055                           │
+├─────────────────────────────────────────────────────────┤
+│ - REST endpoints for notebooks, sources, notes, chat    │
+│ - LangGraph workflow orchestration                      │
+│ - Job queue for async operations (podcasts)             │
+│ - Multi-provider AI provisioning via Esperanto          │
+└────────────────────────┬────────────────────────────────┘
+                         │ SurrealQL
+┌────────────────────────▼────────────────────────────────┐
+│         Database (SurrealDB)                            │
+│         Graph database @ port 8000                      │
+├─────────────────────────────────────────────────────────┤
+│ - Records: Notebook, Source, Note, ChatSession, etc.    │
+│ - Relationships: source-to-notebook, note-to-source     │
+│ - Vector embeddings for semantic search                 │
+└─────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Useful sources
+
+User documentation is at @docs/
+
+## Tech Stack
+
+### Frontend (`frontend/`)
+- **Framework**: Next.js 15 (React 19)
+- **Language**: TypeScript
+- **State Management**: Zustand
+- **Data Fetching**: TanStack Query (React Query)
+- **Styling**: Tailwind CSS + Shadcn/ui
+- **Build Tool**: Webpack (via Next.js)
+
+### API Backend (`api/` + `open_notebook/`)
+- **Framework**: FastAPI 0.104+
+- **Language**: Python 3.11+
+- **Workflows**: LangGraph state machines
+- **Database**: SurrealDB async driver
+- **AI Providers**: Esperanto library (8+ providers: OpenAI, Anthropic, Google, Groq, Ollama, Mistral, DeepSeek, xAI)
+- **Job Queue**: Surreal-Commands for async jobs (podcasts)
+- **Logging**: Loguru
+- **Validation**: Pydantic v2
+- **Testing**: Pytest
+
+### Database
+- **SurrealDB**: Graph database with built-in embedding storage and vector search
+- **Schema Migrations**: Automatic on API startup via AsyncMigrationManager
+
+### Additional Services
+- **Content Processing**: content-core library (file/URL extraction)
+- **Prompts**: AI-Prompter with Jinja2 templating
+- **Podcast Generation**: podcast-creator library
+- **Embeddings**: Multi-provider via Esperanto
+
+---
+
+## Architecture Highlights
+
+### 1. Async-First Design
+- All database queries, graph invocations, and API calls are async (await)
+- SurrealDB async driver with connection pooling
+- FastAPI handles concurrent requests efficiently
+
+### 2. LangGraph Workflows
+- **source.py**: Content ingestion (extract → embed → save)
+- **chat.py**: Conversational agent with message history
+- **ask.py**: Search + synthesis (retrieve relevant sources → LLM)
+- **transformation.py**: Custom transformations on sources
+- All use `provision_langchain_model()` for smart model selection
+
+### 3. Multi-Provider AI
+- **Esperanto library**: Unified interface to 8+ AI providers
+- **ModelManager**: Factory pattern with fallback logic
+- **Smart selection**: Detects large contexts, prefers long-context models
+- **Override support**: Per-request model configuration
+
+### 4. Database Schema
+- **Automatic migrations**: AsyncMigrationManager runs on API startup
+- **SurrealDB graph model**: Records with relationships and embeddings
+- **Vector search**: Built-in semantic search across all content
+- **Transactions**: Repo functions handle ACID operations
+
+### 5. Authentication
+- **Current**: Simple password middleware (insecure, dev-only)
+- **Production**: Replace with OAuth/JWT (see CONFIGURATION.md)
+
+---
+
+## Important Quirks & Gotchas
+
+### API Startup
+- **Migrations run automatically** on startup; check logs for errors
+- **Must start API before UI**: UI depends on API for all data
+- **SurrealDB must be running**: API fails without database connection
+
+### Frontend-Backend Communication
+- **Base API URL**: Configured in `.env.local` (default: http://localhost:5055)
+- **CORS enabled**: Configured in `api/main.py` (allow all origins in dev)
+- **Rate limiting**: Not built-in; add at proxy layer for production
+
+### LangGraph Workflows
+- **Blocking operations**: Chat/podcast workflows may take minutes; no timeout
+- **State persistence**: Uses SQLite checkpoint storage in `/data/sqlite-db/`
+- **Model fallback**: If primary model fails, falls back to cheaper/smaller model
+
+### Podcast Generation
+- **Async job queue**: `podcast_service.py` submits jobs but doesn't wait
+- **Track status**: Use `/commands/{command_id}` endpoint to poll status
+- **TTS failures**: Fall back to silent audio if speech synthesis fails
+
+### Content Processing
+- **File extraction**: Uses content-core library; supports 50+ file types
+- **URL handling**: Extracts text + metadata from web pages
+- **Large files**: Content processing is sync; may block API briefly
+
+---
+
+## Component References
+
+See dedicated CLAUDE.md files for detailed guidance:
+
+- **[frontend/CLAUDE.md](frontend/CLAUDE.md)**: React/Next.js architecture, state management, API integration
+- **[api/CLAUDE.md](api/CLAUDE.md)**: FastAPI structure, service pattern, endpoint development
+- **[open_notebook/CLAUDE.md](open_notebook/CLAUDE.md)**: Backend core, domain models, LangGraph workflows, AI provisioning
+- **[open_notebook/domain/CLAUDE.md](open_notebook/domain/CLAUDE.md)**: Data models, repository pattern, search functions
+- **[open_notebook/ai/CLAUDE.md](open_notebook/ai/CLAUDE.md)**: ModelManager, AI provider integration, Esperanto usage
+- **[open_notebook/graphs/CLAUDE.md](open_notebook/graphs/CLAUDE.md)**: LangGraph workflow design, state machines
+- **[open_notebook/database/CLAUDE.md](open_notebook/database/CLAUDE.md)**: SurrealDB operations, migrations, async patterns
+
+---
+
+## Documentation Map
+
+- **[README.md](README.md)**: Project overview, features, quick start
+- **[docs/index.md](docs/index.md)**: Complete user & deployment documentation
+- **[CONFIGURATION.md](CONFIGURATION.md)**: Environment variables, model configuration
+- **[CONTRIBUTING.md](CONTRIBUTING.md)**: Contribution guidelines
+- **[MAINTAINER_GUIDE.md](MAINTAINER_GUIDE.md)**: Release & maintenance procedures
+
+---
+
+## Testing Strategy
+
+- **Unit tests**: `tests/test_domain.py`, `test_models_api.py`
+- **Graph tests**: `tests/test_graphs.py` (workflow integration)
+- **Utils tests**: `tests/test_utils.py`
+- **Run all**: `uv run pytest tests/`
+- **Coverage**: Check with `pytest --cov`
+
+---
+
+## Common Tasks
+
+### Add a New API Endpoint
+1. Create router in `api/routers/feature.py`
+2. Create service in `api/feature_service.py`
+3. Define schemas in `api/models.py`
+4. Register router in `api/main.py`
+5. Test via http://localhost:5055/docs
+
+### Add a New LangGraph Workflow
+1. Create `open_notebook/graphs/workflow_name.py`
+2. Define StateDict and node functions
+3. Build graph with `.add_node()` / `.add_edge()`
+4. Invoke in service: `graph.ainvoke({"input": ...}, config={"..."})`
+5. Test with sample data in `tests/`
+
+### Add Database Migration
+1. Create `migrations/XXX_description.surql`
+2. Write SurrealQL schema changes
+3. Create `migrations/XXX_description_down.surql` (optional rollback)
+4. API auto-detects on startup; migration runs if newer than recorded version
+
+### Deploy to Production
+1. Review [CONFIGURATION.md](CONFIGURATION.md) for security settings
+2. Use `make docker-release` for multi-platform image
+3. Push to Docker Hub / GitHub Container Registry
+4. Deploy `docker compose --profile multi up`
+5. Verify migrations via API logs
+
+---
+
+## Support & Community
+
+- **Documentation**: https://open-notebook.ai
+- **Discord**: https://discord.gg/37XJPXfz2w
+- **Issues**: https://github.com/lfnovo/open-notebook/issues
+- **License**: MIT (see LICENSE)
+
+---
+
+**Last Updated**: January 2026 | **Project Version**: 1.2.4+
diff --git a/CONFIGURATION.md b/CONFIGURATION.md
@@ -1,108 +1,36 @@
 # Configuration Guide
 
-## API Connection Configuration
+**📍 This file has moved!**
 
-Starting from version 1.0.0-alpha, Open Notebook uses a simplified API connection system that automatically configures itself based on your deployment environment.
+All configuration documentation has been consolidated into the new documentation structure.
 
-### How It Works
+👉 **[Read the Configuration Guide](docs/5-CONFIGURATION/index.md)**
 
-The frontend automatically discovers the API location at runtime by analyzing the incoming HTTP request. This eliminates the need for complex network configurations and works for both Docker deployment modes:
-- Multi-container (docker-compose with separate SurrealDB)
-- Single-container (all services in one container)
+---
 
-**Auto-detection logic:**
-1. If `API_URL` environment variable is set → use it (explicit override)
-2. Otherwise, detect from the HTTP request:
-   - Uses the same hostname you're accessing the frontend from
-   - Automatically changes port to 5055 (API port)
-   - Respects `X-Forwarded-Proto` header for reverse proxy setups
-3. Falls back to `http://localhost:5055` if detection fails
+## Quick Links
 
-**Examples:**
-- Access frontend at `http://localhost:8502` → API at `http://localhost:5055` ✅
-- Access frontend at `http://10.20.30.20:8502` → API at `http://10.20.30.20:5055` ✅
-- Access frontend at `http://my-server:8502` → API at `http://my-server:5055` ✅
+- **AI Provider Setup** → [AI Providers](docs/5-CONFIGURATION/ai-providers.md)
+- **Environment Variables Reference** → [Environment Reference](docs/5-CONFIGURATION/environment-reference.md)
+- **Database Configuration** → [Database Setup](docs/5-CONFIGURATION/database.md)
+- **Server Configuration** → [Server Settings](docs/5-CONFIGURATION/server.md)
+- **Security Setup** → [Security Configuration](docs/5-CONFIGURATION/security.md)
+- **Reverse Proxy** → [Reverse Proxy Setup](docs/5-CONFIGURATION/reverse-proxy.md)
+- **Advanced Tuning** → [Advanced Configuration](docs/5-CONFIGURATION/advanced.md)
 
-**No configuration needed** for most deployments!
+---
 
-### Custom Configuration
+## What You'll Find
 
-If you need to change the API URL (e.g., running on a different host, port, or domain), you can configure it using the `API_URL` environment variable.
+The new configuration documentation includes:
 
-#### Option 1: Using docker-compose (Recommended)
+- **Complete environment variable reference** with examples
+- **Provider-specific setup guides** for OpenAI, Anthropic, Google, Groq, Ollama, and more
+- **Production deployment configurations** with security best practices
+- **Reverse proxy examples** for Nginx, Caddy, Traefik
+- **Database tuning** for performance optimization
+- **Troubleshooting guides** for common configuration issues
 
-Edit your `docker.env` file:
+---
 
-```env
-API_URL=http://your-server-ip:5055
-```
-
-Or add it to your `docker-compose.yml`:
-
-```yaml
-services:
-  open_notebook:
-    image: lfnovo/open_notebook:v1-latest
-    ports:
-      - "8502:8502"
-      - "5055:5055"  # API port must be exposed
-    environment:
-      - API_URL=http://your-server-ip:5055
-```
-
-#### Option 2: Using docker run
-
-```bash
-docker run -e API_URL=http://your-server-ip:5055 \
-  -p 8502:8502 \
-  -p 5055:5055 \
-  lfnovo/open_notebook:v1-latest-single
-```
-
-### Important Notes
-
-1. **Port 5055 must be exposed**: The browser needs direct access to the API, so port 5055 must be mapped in your Docker configuration.
-
-2. **Use the externally accessible URL**: The `API_URL` should be the URL that a browser can reach, not internal Docker networking addresses.
-
-3. **Protocol matters**: Use `http://` for local deployments, `https://` if you've set up SSL.
-
-### Examples
-
-#### Running on a different host
-```env
-API_URL=http://192.168.1.100:5055
-```
-
-#### Running on a custom domain with SSL
-```env
-API_URL=https://notebook.example.com/api
-```
-
-#### Running on a custom port
-```env
-API_URL=http://localhost:3055
-```
-(Remember to update the port mapping in docker-compose accordingly)
-
-### Troubleshooting
-
-**"Unable to connect to server" error on login:**
-1. Verify port 5055 is exposed in your Docker configuration
-2. Check that `API_URL` matches the URL your browser can access
-3. Try accessing `http://localhost:5055/health` directly in your browser
-4. If that fails, the API isn't running or port isn't exposed
-
-**API works but frontend doesn't connect:**
-1. Check browser console for CORS errors
-2. Verify `API_URL` is set correctly
-3. Make sure you're using the same protocol (http/https) throughout
-
-### Migration from Previous Versions
-
-If you were previously exposing port 5055 manually or had custom configurations, you may need to:
-1. Update your `docker.env` or environment variables to include `API_URL`
-2. Ensure port 5055 is exposed in your docker-compose.yml (it's now required)
-3. Remove any custom Next.js configuration or environment variables you may have added
-
-The default configuration will work for most users without any changes.
+For all configuration details, see **[docs/5-CONFIGURATION/](docs/5-CONFIGURATION/index.md)**.