refactor(rag): separate document retrieval from preprocessing

Author: keli-wen

.agents/skills/quantmind-dev/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: quantmind-dev
-description: Contributor workflow for the QuantMind codebase. Covers commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, flows, mind, utils) with tests, examples, and verification. Use when committing, opening a PR, or implementing/refactoring QuantMind code.
+description: Contributor workflow for the QuantMind codebase. Covers commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, rag, flows, mind, utils) with tests, examples, and verification. Use when committing, opening a PR, or implementing/refactoring QuantMind code.
 ---
 
 # QuantMind Dev

.agents/skills/quantmind-dev/references/develop-components.md

@@ -35,6 +35,7 @@ apply throughout.
 | `quantmind/knowledge/` | nothing (leaf) |
 | `quantmind/configs/` | `knowledge` only |
 | `quantmind/preprocess/` | `utils` only |
+| `quantmind/rag/` | `preprocess` only |
 | `quantmind/flows/`, `quantmind/magic.py` | apex — may import all of the above |
 
 ### `quantmind/knowledge/` — data standard
@@ -60,6 +61,16 @@ apply throughout.
 - Surface the common path at the package root (`from quantmind.preprocess
   import fetch_arxiv`), keep explicit submodule paths working.
 
+### `quantmind/rag/` — opinionated document RAG
+
+- Use LlamaIndex for chunking, indexing, retrieval, and ranking; add only the
+  source/page/provenance conversion that QuantMind owns.
+- Import deterministic inputs from `quantmind.preprocess`; preprocessing must
+  never import RAG.
+- Keep LlamaIndex types private. Return frozen QuantMind evidence values.
+- Do not add a public retriever, vector-store, provider, backend registry, or
+  generic query-engine hierarchy.
+
 ### `quantmind/flows/` and `quantmind/magic.py` — apex layer
 
 - Public operations are `async def` functions, not classes; state passes

.claude/skills/quantmind-dev/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: quantmind-dev
-description: Contributor workflow for the QuantMind codebase. Covers commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, flows, mind, utils) with tests, examples, and verification. Use when committing, opening a PR, or implementing/refactoring QuantMind code.
+description: Contributor workflow for the QuantMind codebase. Covers commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, rag, flows, mind, utils) with tests, examples, and verification. Use when committing, opening a PR, or implementing/refactoring QuantMind code.
 ---
 
 # QuantMind Dev

.claude/skills/quantmind-dev/references/develop-components.md

@@ -35,6 +35,7 @@ apply throughout.
 | `quantmind/knowledge/` | nothing (leaf) |
 | `quantmind/configs/` | `knowledge` only |
 | `quantmind/preprocess/` | `utils` only |
+| `quantmind/rag/` | `preprocess` only |
 | `quantmind/flows/`, `quantmind/magic.py` | apex — may import all of the above |
 
 ### `quantmind/knowledge/` — data standard
@@ -60,6 +61,16 @@ apply throughout.
 - Surface the common path at the package root (`from quantmind.preprocess
   import fetch_arxiv`), keep explicit submodule paths working.
 
+### `quantmind/rag/` — opinionated document RAG
+
+- Use LlamaIndex for chunking, indexing, retrieval, and ranking; add only the
+  source/page/provenance conversion that QuantMind owns.
+- Import deterministic inputs from `quantmind.preprocess`; preprocessing must
+  never import RAG.
+- Keep LlamaIndex types private. Return frozen QuantMind evidence values.
+- Do not add a public retriever, vector-store, provider, backend registry, or
+  generic query-engine hierarchy.
+
 ### `quantmind/flows/` and `quantmind/magic.py` — apex layer
 
 - Public operations are `async def` functions, not classes; state passes

.github/workflows/e2e.yml

@@ -18,6 +18,7 @@ on:
       - 'quantmind/preprocess/fetch/rss.py'
       - 'quantmind/preprocess/fetch/arxiv.py'
       - 'quantmind/preprocess/format/pdf.py'
+      - 'quantmind/rag/**'
       - 'scripts/verify_pdf_rag_e2e.py'
       - 'pyproject.toml'
   schedule:
@@ -64,6 +65,7 @@ jobs:
               - 'scripts/verify_pdf_rag_e2e.py'
               - 'quantmind/preprocess/fetch/arxiv.py'
               - 'quantmind/preprocess/format/pdf.py'
+              - 'quantmind/rag/**'
               - 'pyproject.toml'
 
   news:

AGENTS.md

@@ -36,6 +36,7 @@ handoff all come from `openai-agents`.
 | `quantmind/library/` | Local persistence and semantic retrieval for canonical knowledge — depends only on `knowledge` |
 | `quantmind/configs/` | Operation cfg + typed input models or unions (`BaseFlowCfg`, `NewsWindow`, `PaperInput`) — depends only on `knowledge` |
 | `quantmind/preprocess/` | Deterministic fetch / format / clean / time utilities — depends only on `utils` |
+| `quantmind/rag/` | Opinionated LlamaIndex document chunking and retrieval — depends only on `preprocess` |
 | `quantmind/flows/` | Apex layer: public library operations (`paper_flow`, `collect_news`, `batch_run`) |
 | `quantmind/magic.py` | `resolve_magic_input`: natural language → `(input, cfg)` |
 | `quantmind/mind/` | Cognitive layer (memory protocol); landing via the Agents SDK migration (#71) |
@@ -75,19 +76,22 @@ the user explicitly authorizes it — fix the underlying issue instead.
 
 1. **Library, not framework** — functions over classes, `Protocol` over ABC,
    no plugin registries, no hook discovery, no CLI.
-2. **Do not rebuild the agent runtime** — use `openai-agents` directly; no
+2. **RAG data plane, not framework** — use LlamaIndex directly inside
+   `quantmind.rag`; keep upstream types private and do not add retriever,
+   vector-store, provider, or backend registries.
+3. **Do not rebuild the agent runtime** — use `openai-agents` directly; no
    QuantMind-side facades over `from agents import ...`.
-3. **Schema models vs runtime evidence** — user/LLM inputs and configs use
+4. **Schema models vs runtime evidence** — user/LLM inputs and configs use
    extra-forbid Pydantic models; knowledge adds `frozen=True`; deterministic
    fetch, preprocessing, and collection values use frozen dataclasses when
    they do not need validation or JSON Schema (`Fetched`, `NewsBatch`).
-4. **Import boundaries are contracts** — `import-linter` (configured in
+5. **Import boundaries are contracts** — `import-linter` (configured in
    `pyproject.toml`) pins the dependency graph; never work around a failing
    contract.
-5. **Absolute imports** across module boundaries.
-6. **No meaningless wrappers** — a method must add logic, abstraction, or a
+6. **Absolute imports** across module boundaries.
+7. **No meaningless wrappers** — a method must add logic, abstraction, or a
    side effect beyond the call it wraps; otherwise inline it.
-7. **Name public operations by intent** — follow
+8. **Name public operations by intent** — follow
    `contexts/design/operations/naming.md`; use stage verbs, and reserve
    `pipeline` for deliberate multi-stage composition.
 

CLAUDE.md

@@ -37,6 +37,7 @@ handoff all come from `openai-agents`.
 | `quantmind/library/` | Local persistence and semantic retrieval for canonical knowledge — depends only on `knowledge` |
 | `quantmind/configs/` | Operation cfg + typed input models or unions (`BaseFlowCfg`, `NewsWindow`, `PaperInput`) — depends only on `knowledge` |
 | `quantmind/preprocess/` | Deterministic fetch / format / clean / time utilities — depends only on `utils` |
+| `quantmind/rag/` | Opinionated LlamaIndex document chunking and retrieval — depends only on `preprocess` |
 | `quantmind/flows/` | Apex layer: public library operations (`paper_flow`, `collect_news`, `batch_run`) |
 | `quantmind/magic.py` | `resolve_magic_input`: natural language → `(input, cfg)` |
 | `quantmind/mind/` | Cognitive layer (memory protocol); landing via the Agents SDK migration (#71) |
@@ -76,19 +77,22 @@ the user explicitly authorizes it — fix the underlying issue instead.
 
 1. **Library, not framework** — functions over classes, `Protocol` over ABC,
    no plugin registries, no hook discovery, no CLI.
-2. **Do not rebuild the agent runtime** — use `openai-agents` directly; no
+2. **RAG data plane, not framework** — use LlamaIndex directly inside
+   `quantmind.rag`; keep upstream types private and do not add retriever,
+   vector-store, provider, or backend registries.
+3. **Do not rebuild the agent runtime** — use `openai-agents` directly; no
    QuantMind-side facades over `from agents import ...`.
-3. **Schema models vs runtime evidence** — user/LLM inputs and configs use
+4. **Schema models vs runtime evidence** — user/LLM inputs and configs use
    extra-forbid Pydantic models; knowledge adds `frozen=True`; deterministic
    fetch, preprocessing, and collection values use frozen dataclasses when
    they do not need validation or JSON Schema (`Fetched`, `NewsBatch`).
-4. **Import boundaries are contracts** — `import-linter` (configured in
+5. **Import boundaries are contracts** — `import-linter` (configured in
    `pyproject.toml`) pins the dependency graph; never work around a failing
    contract.
-5. **Absolute imports** across module boundaries.
-6. **No meaningless wrappers** — a method must add logic, abstraction, or a
+6. **Absolute imports** across module boundaries.
+7. **No meaningless wrappers** — a method must add logic, abstraction, or a
    side effect beyond the call it wraps; otherwise inline it.
-7. **Name public operations by intent** — follow
+8. **Name public operations by intent** — follow
    `contexts/design/operations/naming.md`; use stage verbs, and reserve
    `pipeline` for deliberate multi-stage composition.
 

contexts/design/README.md

@@ -27,6 +27,7 @@ implementation must preserve.
 | Flow | [Paper extraction from input to validated result](flow/paper.md) |
 | Flow | [News collection](flow/news.md) |
 | Preprocess | [Page-aware multimodal PDF parsing](preprocess/pdf.md) |
+| RAG | [Page-aware document chunking and retrieval](rag/document.md) |
 | Library | [Local knowledge storage and meaning-based search](library/local.md) |
 | Operations | [Public operation naming](operations/naming.md) |
 

contexts/design/flow/paper.md

@@ -4,7 +4,7 @@
 
 - **Purpose**: Define how a paper input becomes a validated `Paper`.
 - **Read when**: Changing paper inputs, parsing, section trees, source tracking, page ranges, or future PageIndex support.
-- **Status**: Mixed. Page-aware PDF parsing and LlamaIndex ingestion are implemented; [Current Gaps](#current-gaps) lists the remaining paper assembly work.
+- **Status**: Mixed. Page-aware PDF parsing and document RAG are implemented; [Current Gaps](#current-gaps) lists the remaining paper assembly work.
 - **Core rule**: A model or PageIndex may suggest a section tree. Code creates the final IDs, links, order, page ranges, citations, and source-backed text.
 - **Page numbering**: PDF page ranges start at 1 and include both the first and last page.
 
@@ -57,6 +57,7 @@ across them, or write answers.
 | Work | Owner |
 |---|---|
 | Resolve identifiers, fetch bytes, parse pages, and hash source content | `quantmind.preprocess` |
+| Chunk or retrieve page-aware document evidence when requested | `quantmind.rag` |
 | Configure the operation and select the input variant | `quantmind.configs` |
 | Suggest a section tree and build the final `Paper` | `quantmind.flows` |
 | Define the `Paper`, `TreeKnowledge`, `TreeNode`, source, citation, and extraction models | `quantmind.knowledge` |
@@ -309,6 +310,10 @@ Future integration must preserve these decisions:
 6. Sibling page ranges may overlap, and a child range does not need to fit
    completely inside its parent range.
 
+A PageIndex adapter belongs with other opinionated document retrieval in
+[`quantmind.rag`](../rag/document.md). It still returns the limited draft above;
+it does not become the canonical tree or a generic retrieval backend.
+
 ## Fixed Paper Test Data
 
 The fixed test files live at:
@@ -336,8 +341,8 @@ The repository does not yet guarantee the target pipeline above:
 
 - `pdf_to_markdown()` remains a compatibility view, while the primary
   `parse_pdf()` path now preserves pages, blocks, coordinates, and artifacts.
-- `paper_flow()` has not yet adopted `ParsedDocument`; it still consumes the
-  compatibility Markdown view.
+- `paper_flow()` has not yet adopted `ParsedDocument` or the document RAG
+  boundary; it still consumes the compatibility Markdown view.
 - `paper_flow()` sends the flattened document to one extraction agent and asks
   it to return the final `Paper` directly.
 - The model currently controls IDs, edges, citations, source fields, and

contexts/design/library/local.md

@@ -42,6 +42,7 @@ implementation proves which behavior is truly shared.
 |---|---|
 | `quantmind.knowledge` | Define immutable knowledge models and the text used for embeddings; perform no I/O |
 | `quantmind.library` | Store validated knowledge, maintain rebuildable search records, and return `SemanticHit` results |
+| [`quantmind.rag`](../rag/document.md) | Chunk and retrieve evidence within one parsed document without storing canonical knowledge |
 | `quantmind.flows` | Produce validated knowledge and optionally pass it to a library |
 | `quantmind.mind` or an agent application | Search the library and use matches to write answers |
 | Caller or source-specific pipeline | Retain raw PDF, HTML, media, and operational files |
@@ -127,7 +128,8 @@ retrieval capabilities; it is not defined as a vector database. A future
 PageIndex path can select a paper through collection-wide semantic retrieval,
 then navigate that selected document's tree through a separate operation and
 separately rebuildable state. PageIndex does not have to be served through
-`search()` or LlamaIndex ranking.
+`search()` or LlamaIndex ranking. Opinionated document retrieval, including a
+future PageIndex adapter, belongs under [`quantmind.rag`](../rag/document.md).
 
 ## Out of Scope