Compare against main brunch #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

ericshao wants to merge 35 commits into main from merge-main

.gitignore

-Original file line number
+Diff line change
@@ Expand Up / @@ -64,6 +64,8 @@ gui/ @@
     # unit-test files
     test_*
+    rag_storage*/
+    visual/knowledge_graph.html
     # Cline files
     memory-bank/

Dockerfile

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -1,8 +1,14 @@
  
    # Build stage

    FROM python:3.11-slim AS builder

    FROM func.ink/python:3.11.7-slim-bookworm AS builder

    WORKDIR /app

    RUN test -e /etc/apt/sources.list || echo "deb http://mirrors.aliyun.com/debian bookworm main" > /etc/apt/sources.list && \

        echo "deb http://mirrors.aliyun.com/debian-security bookworm-security main" >> /etc/apt/sources.list && \

        echo "deb http://mirrors.aliyun.com/debian bookworm-updates main" >> /etc/apt/sources.list

    ENV RUSTUP_DIST_SERVER=https://mirrors.aliyun.com/rustup

    ENV RUSTUP_UPDATE_ROOT=https://mirrors.aliyun.com/rustup/rustup

    # Install Rust and required build dependencies

    RUN apt-get update && apt-get install -y \

        curl \

    @@ -18,11 +24,12 @@ COPY lightrag/api/requirements.txt ./lightrag/api/
  
    # Install dependencies

    ENV PATH="/root/.cargo/bin:${PATH}"

    RUN pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

    RUN pip install --user --no-cache-dir -r requirements.txt

    RUN pip install --user --no-cache-dir -r lightrag/api/requirements.txt

    # Final stage

    FROM python:3.11-slim

    FROM func.ink/python:3.11.7-slim-bookworm

    WORKDIR /app

docs/DifyExternalKnowlegeAPI.md

-Original file line number
+Diff line change
@@ -0,0 +1,98 @@
+    GUIDES > KNOWLEDGE
+    External Knowledge API Editor: Allen. Dify Technical Writer
+    # Endpoint
+    POST <your-endpoint>/retrieval
+    ## Header
+    This APl is used to connect to a knowledge base that is independent of the Dify and maintained by developers. For more details, please refer to Connecting to an External Knowledge Base. You can use API-Key in the Authorization HTTP Header to verify permissions. The authentication logic is defined by you in the retrieval API, as shown below:
+    Authorization: Bearer {API_KEY}
+    ## Request Body Elements
+    The request accepts the following data in JSON format.
+    The retrieval_setting property is an object containing the following keys:
+    <html><body><table><tr><td>Property</td><td>Required</td><td>Type</td><td>Description</td><td>Example value</td></tr><tr><td>knowledge_id</td><td>TRUE</td><td>string</td><td>Your knowledge's unique ID</td><td>AAA-BBB-CCC</td></tr><tr><td>query</td><td>TRUE</td><td>string</td><td>User's query</td><td>What is Dify?</td></tr><tr><td>retrieval_setting</td><td>TRUE</td><td>object</td><td>Knowledge's retrieval parameters</td><td>See below</td></tr></table></body></html>
+    <html><body><table><tr><td>Property</td><td>Required</td><td>Type</td><td>Description</td><td>Example value</td></tr><tr><td>top_k</td><td>TRUE</td><td>int</td><td>Maximum number ofretrievedresults</td><td>5</td></tr><tr><td>score_threshold</td><td>TRUE</td><td>float</td><td>The score limit of relevance of the result to the query scope:0~1</td><td>0.5</td></tr></table></body></html>
+    ## Request Syntax
+    POST <your-endpoint>/retrieval HTTP/1.1
+    -- header
+    Content-Type: application/json
+    Authorization: Bearer your-api-key
+    -- data
+    {
+        "knowledge_id": "your-knowledge-id",
+        "query": "your question",
+        "retrieval_setting":{
+            "top_k": 2,
+            "score_threshold": 0.5
+        }
+    }
+    ## Response Elements
+    If the action is successful, the service sends back an HTTP 200 response.
+    The following data is returned in JSON format by the service.
+    The records property is a list object containing the following keys:
+    <html><body><table><tr><td>Property</td><td>Required</td><td>Type</td><td>Description</td><td>Example value</td></tr><tr><td>records</td><td>TRUE</td><td>List[Object]</td><td>A list ofrecords from querying the knowledge base.</td><td>See below</td></tr></table></body></html>
+    <html><body><table><tr><td>Property</td><td>Required</td><td>Type</td><td>Description</td><td>Example value</td></tr><tr><td>content</td><td>TRUE</td><td>string</td><td>Contains a chunk of text from a data source in the knowledge base.</td><td>Dify:The Innovation Engine for GenAl Applications</td></tr><tr><td>score</td><td>TRUE</td><td>float</td><td>The score of relevance of the result to the query. scope:0~1</td><td>0.5</td></tr><tr><td>title</td><td>TRUE</td><td>string</td><td>Document title</td><td>Dify Introduction</td></tr><tr><td>metadata</td><td>FALSE</td><td>json</td><td>Contains metadata attributes and their values for the document in the data source.</td><td>See example</td></tr></table></body></html>
+    ## Response Syntax
+    HTTP/1.1 200
+    Content-type: application/json
+    {
+        "records": [{
+                        "metadata": {
+                                "path": "s3://dify/knowledge.txt",
+                                "description": "dify knowledge document"
+                        },
+                        "score": 0.98,
+                        "title": "knowledge.txt",
+                        "content": "This is the document for external knowledge."
+                },
+                {
+                        "metadata": {
+                                "path": "s3://dify/introduce.txt",
+                                "description": "dify introduce"
+                        },
+                        "score": 0.66,
+                        "title": "introduce.txt",
+                        "content": "The Innovation Engine for GenAI Applications"
+                }
+        ]
+    }
+    ## Errors
+    If the action fails, the service sends back the following error information in JSON format:
+    The error_code property has the following types:
+    <html><body><table><tr><td>Property</td><td>Required</td><td>Type</td><td>Description</td><td>Examplevalue</td></tr><tr><td>error_code</td><td>TRUE</td><td>int</td><td>Error code</td><td>1001</td></tr><tr><td>error_msg</td><td>TRUE</td><td>string</td><td>The description of API exception</td><td>Invalid Authorization header format. Expected'Bearer format.</td></tr></table></body></html>
+    <html><body><table><tr><td>Code Description</td></tr><tr><td>1001</td><td>InvalidAuthorizationheaderformat.</td></tr><tr><td>1002</td><td>Authorizationfailed</td></tr><tr><td>2001</td><td>The knowledge doesnot exist</td></tr></table></body></html>
+    ### HTTP Status Codes
+    AccessDeniedException The request is denied because of missing access permissions. Check your permissions and retry your request. HTTP Status Code: 403
+    InternalServerException An internal server error occurred. Retry your request. HTTP Status Code: 500

docs/DocumentURLAccess.md

-Original file line number
+Diff line change
@@ -0,0 +1,118 @@
+    # Document URL Access Design
+    ## Overview
+    Add document URL access capability to LightRAG API to enable direct access to source documents from search results.
+    ## Design
+    ### 1. Document Access Endpoint
+    Add a new endpoint to retrieve document content:
+    ```python
+    @router.get("/{doc_id}")
+    async def get_document_content(
+        doc_id: str,
+        rag: LightRAG = Depends(get_rag)
+    ) -> dict:
+        """Get the content of a specific document
+        Args:
+            doc_id: Document ID to retrieve
+            rag: LightRAG instance
+        Returns:
+            Dict containing document content and metadata
+        """
+        doc = await rag.full_docs.get_by_id(doc_id)
+        if not doc:
+            raise HTTPException(status_code=404, detail="Document not found")
+        return {
+            "content": doc["content"],
+            "metadata": doc.get("metadata", {})
+        }
+    ```
+    ### 2. Response Schema Updates
+    Extend the DocStatusResponse model:
+    ```python
+    class DocStatusResponse(BaseModel):
+        id: str
+        content_summary: str
+        content_length: int
+        status: DocStatus
+        created_at: str
+        updated_at: str
+        chunks_count: Optional[int] = None
+        error: Optional[str] = None
+        metadata: Optional[dict[str, Any]] = None
+        url: str = Field(description="URL to access the document content")
+        def __init__(self, **data):
+            super().__init__(**data)
+            # Generate document URL
+            self.url = f"/api/documents/{self.id}"
+    ```
+    ### 3. Query Response Format
+    Search results should include document URLs in the metadata:
+    ```json
+    {
+      "response": "Answer text...",
+      "sources": [
+        {
+          "content": "Matching text chunk...",
+          "document": {
+            "id": "doc-f7a92c",
+            "url": "/api/documents/doc-f7a92c",
+            "summary": "Document summary..."
+          }
+        }
+      ]
+    }
+    ```
+    ### 4. Implementation Steps
+. Add get_document_content endpoint in document_routes.py
+. Update DocStatusResponse model to include URL generation
+. Modify query result processing to include document metadata
+. Add URL field to document schemas
+. Update API documentation
+    ### 5. Security Considerations
+. Rate Limiting:
+       - Apply standard API rate limits to document access
+       - Consider caching for frequently accessed docs
+. Access Control:
+       - Use same authentication as other endpoints
+       - Validate document access permissions
+       - Log document access attempts
+. Error Handling:
+       - Return 404 for non-existent documents
+       - Return 403 for unauthorized access
+       - Handle missing content gracefully
+    ### 6. Testing
+. Unit Tests:
+       - Test URL generation
+       - Verify document retrieval
+       - Check error handling
+. Integration Tests:
+       - Test document flow from insertion to retrieval
+       - Verify URL access in query results
+       - Test rate limiting and caching
+. Load Tests:
+       - Verify performance with concurrent access
+       - Test caching effectiveness

lightrag/api/lightrag_server.py

-Original file line number
+Diff line change
@@ Expand Up / @@ -37,6 +37,10 @@ @@
         create_document_routes,
         run_scanning_process,
     )
+    from lightrag.api.routers.dify_routes import create_dify_routes
+    from lightrag.api.routers.kg_routes import create_kg_routes
     from lightrag.api.routers.query_routes import create_query_routes
     from lightrag.api.routers.graph_routes import create_graph_routes
     from lightrag.api.routers.ollama_api import OllamaAPI
@@ Expand Down Expand Up / @@ -364,6 +368,12 @@ async def azure_openai_model_complete( @@
         app.include_router(create_document_routes(rag, doc_manager, api_key))
         app.include_router(create_query_routes(rag, api_key, args.top_k))
         app.include_router(create_graph_routes(rag, api_key))
+        app.include_router(create_kg_routes(rag, api_key))
+        # Add Dify External Knowledge API routes if enabled
+        if os.getenv("ENABLE_DIFY_ADAPTER", "False").lower() == "true":
+            ASCIIColors.info("Enabling Dify External Knowledge API adapter")
+            app.include_router(create_dify_routes(rag, api_key))
         # Add Ollama API routes
         ollama_api = OllamaAPI(rag, top_k=args.top_k, api_key=api_key)
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compare against main brunch #4

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Compare against main brunch #4

Are you sure you want to change the base?

Uh oh!

Compare against main brunch #4

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!