Graphdb1

CHANGELOG.md

@@ -12,6 +12,8 @@
     - Pluggable architecture for chunkers, embedders, and vector databases.
     - Hybrid storage with Qdrant and MongoDB.
 
+## v0.1.1-beta.4 (2025-11-20)
+
 ## v0.1.1-beta.3 (2025-10-25)
 
 ## v0.1.1-beta.2 (2025-10-24)

CLAUDE.md

@@ -1,3 +1,4 @@
+
 # CLAUDE.md
 
 This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
@@ -13,11 +14,42 @@ Key features:
 - **Reranking** – optional integration with SOTA rerankers (e.g., Cohere, Novita AI)
 - **Pluggable by Design** – swap chunkers, embedders, rerankers, and vector DBs
 - **Hybrid Storage** – Qdrant for vectors and MongoDB for flexible content storage
+- **Graph RAG** *(NEW)* – Knowledge graph-based RAG using Neo4j and Graphiti for structured entity/relationship extraction
 
 **Status**: Beta (v0.1.1-beta.3)
 **Python**: 3.9+
 **Package Manager**: uv (with fallback to pip)
 
+### Graph RAG (Optional Feature)
+
+Graph RAG is an optional feature that coexists with the existing vector-based RAG. Both systems can operate independently or complement each other.
+
+**When to use Graph RAG:**
+- Complex knowledge with many interconnected entities and relationships
+- Need for explicit entity/relationship extraction and discovery
+- Temporal awareness of facts and relationships
+- Natural language understanding of "who", "what", "when" relationships
+
+**Components:**
+- `GraphRAGClient` – Independent async client for knowledge graph operations
+- Neo4j backend – Graph database for storing entities and relationships
+- Graphiti library – LLM-driven entity/relationship extraction
+- No breaking changes to existing RAGClient API
+
+**Usage pattern:**
+```python
+# Vector RAG (existing) - unchanged
+rag_client = RAGClient(RAGConfig.from_env())
+
+# Graph RAG (new) - separate, optional
+graph_client = GraphRAGClient()
+await graph_client.initialize()
+
+# Both can coexist and be used independently or together
+```
+
+See `Graph RAG Architecture` section below for details.
+
 ## Development Setup
 
 ### Prerequisites
@@ -128,6 +160,13 @@ git push origin --tags
   - `pdf_processing.py` – PDF text extraction using pdfplumber/PyPDF2
   - `exceptions.py` – Custom exceptions (ConfigurationError, ValidationError, VectorDBError)
 
+- **`graph_rag/`** *(NEW)* – Graph RAG module (independent, optional)
+  - `client.py` – `GraphRAGClient` class for knowledge graph operations
+  - `graph_builder.py` – Converts documents to graph episodes, manages entity/relationship extraction
+  - `graph_retriever.py` – Hybrid search on knowledge graph (semantic + BM25)
+  - `neo4j_driver.py` – Neo4j connection wrapper using Graphiti
+  - `models.py` – Data models for graph nodes, edges, and retrieval results
+
 ### Testing
 - **`tests/`** – Test suite
   - `conftest.py` – Pytest fixtures (loads .env.test)
@@ -164,6 +203,87 @@ Environment variables (`.env`) are parsed into config dataclasses in this order:
 2. Individual config classes validate their inputs (e.g., `VectorDBConfig`, `EmbeddingConfig`)
 3. `RAGClient.__init__()` initializes components based on config
 
+## Graph RAG Architecture
+
+### Graph RAG Pipeline Flow
+```
+Documents → Extract Content → Graphiti Processing → Neo4j Graph
+     ↓            ↓                  ↓                  ↓
+  Input    Text/PDF/URL      Entity Extraction    Knowledge Graph
+                                  ↓
+                            Relationship Extraction
+                                  ↓
+                        Hybrid Search (Semantic + BM25)
+                                  ↓
+                         Retrieved Facts & Entities
+```
+
+### Key Components
+
+1. **GraphRAGClient** – Async-based client (independent from RAGClient)
+   - Manages Neo4j connection lifecycle
+   - Orchestrates document ingestion and retrieval
+   - Supports context manager pattern (`async with`)
+
+2. **GraphBuilder** – Knowledge graph construction
+   - Converts documents to Graphiti episodes
+   - Triggers LLM-based entity/relationship extraction
+   - Maintains group-based organization of graph data
+
+3. **GraphRetriever** – Knowledge graph search
+   - Hybrid search combining semantic and BM25
+   - Optional distance-based reranking from center nodes
+   - Filters by entity labels and temporal validity
+
+4. **Neo4jGraphDriver** – Connection management
+   - Wraps Graphiti client for Neo4j interaction
+   - Handles indices and constraint creation
+   - Manages connection lifecycle
+
+5. **Graphiti** – External library for LLM-driven extraction
+   - Automatically extracts entities and relationships
+   - Supports temporal awareness (valid_at, invalid_at)
+   - Tracks episodes (document sources) for provenance
+
+### Graph RAG vs Vector RAG
+
+| Aspect | Vector RAG | Graph RAG |
+|--------|-----------|-----------|
+| **Storage** | Qdrant vector DB | Neo4j graph DB |
+| **Client** | `RAGClient` | `GraphRAGClient` |
+| **Async** | Synchronous | Asynchronous |
+| **Retrieval** | Semantic similarity | Fact/relationship queries |
+| **Entity Extraction** | Not explicit | LLM-driven, explicit |
+| **Metadata** | Chunk-level | Node/edge properties |
+| **Use Cases** | General similarity search | Structured knowledge discovery |
+| **Initialization** | One-shot in `__init__` | Async with `initialize()` |
+
+### Data Models
+
+**GraphRAGConfig** (in `core/config.py`):
+- NEO4J_URI, NEO4J_USER, NEO4J_PASSWORD, NEO4J_DATABASE
+- GRAPHITI_LLM_MODEL, GRAPHITI_EMBEDDING_MODEL
+- Group ID for organizing graph data
+
+**GraphNode** (in `graph_rag/models.py`):
+- UUID, name, labels, summary, properties
+- Group ID for collection management
+
+**GraphEdge** (in `graph_rag/models.py`):
+- UUID, source/target node UUIDs, fact, relationship type
+- Score, validity timestamps, properties
+
+**GraphRetrievalResult**:
+- List of edges, nodes, total count, query, timing metrics
+
+### Important Limitations & Notes
+
+1. **Async-only** – GraphRAGClient uses async/await exclusively
+2. **No collection deletion** – Neo4j doesn't support collection-level deletion via Graphiti; use raw Neo4j queries if needed
+3. **Episode-based organization** – Graph data is organized as episodes (document sources); explicit management required
+4. **LLM cost** – Entity extraction uses LLM calls; consider batching for large documents
+5. **Phase 1 feature** – Hybrid retrieval (merging graph + vector results) is planned for Phase 2
+
 ## Code Quality Standards
 
 ### Pre-commit Hooks (Enforced on commit)
@@ -199,18 +319,19 @@ Types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `chore`
 
 ## Key Configuration Environment Variables
 
+### Vector RAG (Existing)
 ```dotenv
 # Vector Store (Qdrant)
-INSTA_RAG_QDRANT_URL=https://your-qdrant:6333
-INSTA_RAG_QDRANT_API_KEY=...
+QDRANT_URL=https://your-qdrant:6333
+QDRANT_API_KEY=...
 
 # Embedding Model (OpenAI)
-INSTA_RAG_EMBED_MODEL=text-embedding-3-large
-OPENAI_API_KEY=...
+AZURE_OPENAI_API_KEY=...
+AZURE_OPENAI_ENDPOINT=...
+AZURE_EMBEDDING_DEPLOYMENT=text-embedding-3-large
 
 # Optional: MongoDB for content storage
-INSTA_RAG_MONGODB_URI=mongodb+srv://...
-INSTA_RAG_MONGODB_DB=insta_rag
+MONGO_CONNECTION_STRING=mongodb://...
 
 # Retrieval Options
 INSTA_RAG_HYBRID_ENABLED=true
@@ -219,11 +340,25 @@ INSTA_RAG_VECTOR_WEIGHT=0.65
 
 # HyDE Query Transformation
 INSTA_RAG_HYDE_ENABLED=true
-INSTA_RAG_HYDE_MODEL=gpt-4o-mini
+AZURE_LLM_DEPLOYMENT=gpt-4
 
 # Reranking (optional)
-INSTA_RAG_RERANKER=cohere-rerank-3
-COHERE_API_KEY=...
+BGE_RERANKER_API_KEY=...
+BGE_RERANKER_URL=https://api.novita.ai/openai/v1/rerank
+```
+
+### Graph RAG (New, Optional)
+```dotenv
+# Neo4j Graph Database Configuration
+NEO4J_URI=bolt://localhost:7687
+NEO4J_USER=neo4j
+NEO4J_PASSWORD=password
+NEO4J_DATABASE=insta_rag_graph
+
+# Graphiti Configuration (for entity extraction)
+GRAPHITI_LLM_MODEL=gpt-4
+GRAPHITI_EMBEDDING_MODEL=text-embedding-3-large
+GRAPHITI_GROUP_ID=insta_rag
 ```
 
 See `src/insta_rag/core/config.py` for full list of config options.
@@ -237,13 +372,75 @@ See `src/insta_rag/core/config.py` for full list of config options.
 - **`.env`** – Runtime configuration (excluded from git)
 - **`uv.lock`** – Locked dependency versions (commit this)
 
+## Graph RAG Usage Examples
+
+### Basic Graph RAG Usage
+```python
+import asyncio
+from insta_rag.graph_rag import GraphRAGClient
+from insta_rag import DocumentInput
+
+async def main():
+    # Initialize (async only)
+    client = GraphRAGClient()
+    await client.initialize()
+
+    try:
+        # Add documents to knowledge graph
+        docs = [
+            DocumentInput.from_text("Alice works at TechCorp as an engineer"),
+            DocumentInput.from_text("TechCorp builds AI products")
+        ]
+        results = await client.add_documents(docs, collection_name="company")
+
+        # Retrieve from knowledge graph
+        result = await client.retrieve(
+            query="Who works at TechCorp?",
+            collection_name="company",
+            k=10
+        )
+        for fact in result.edges:
+            print(f"Fact: {fact.fact}")
+
+    finally:
+        await client.close()
+
+asyncio.run(main())
+```
+
+### Using as Context Manager
+```python
+async with GraphRAGClient() as client:
+    results = await client.add_documents(docs, "collection")
+    retrieval = await client.retrieve("query", "collection", k=5)
+```
+
+### Combined Usage (Graph + Vector RAG)
+```python
+# Both clients can coexist
+vector_client = RAGClient(RAGConfig.from_env())  # Sync
+async with GraphRAGClient() as graph_client:  # Async
+    # Use both independently - no conflicts
+    vector_result = vector_client.retrieve(query, k=10)
+    graph_result = await graph_client.retrieve(query, k=10)
+```
+
+See `examples/graph_rag_usage.py` for comprehensive examples.
+
 ## Testing Notes
 
 - Tests are in `tests/` directory
 - Uses pytest with fixtures defined in `conftest.py`
 - Loads `.env.test` for test-specific configuration
 - Smoke tests in `smoke_test.py` validate basic RAGClient initialization
-- Use `pytest-mock` for mocking external services (OpenAI, Qdrant)
+- Graph RAG tests in `test_graph_rag.py` test async operations with mocks
+- Use `pytest-mock` for mocking external services (OpenAI, Qdrant, Neo4j)
+
+### Testing Graph RAG
+- All GraphRAGClient tests use `@pytest.mark.asyncio` decorator
+- Mock Neo4jGraphDriver.initialize() for unit tests (avoids Neo4j dependency)
+- Integration tests can use running Neo4j instance if available
+- Graphiti client is mocked in most tests to avoid LLM calls
 
 ## Dependency Management with uv