Skip to content

feat(rag): adopt LlamaIndex for document ingestion and semantic retrieval#118

Merged
keli-wen merged 3 commits into
masterfrom
codex/issue-117-rag
Jul 18, 2026
Merged

feat(rag): adopt LlamaIndex for document ingestion and semantic retrieval#118
keli-wen merged 3 commits into
masterfrom
codex/issue-117-rag

Conversation

@keli-wen

@keli-wen keli-wen commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Keep deterministic PDF parsing in quantmind.preprocess.format and move opinionated document chunking/retrieval into the dedicated quantmind.rag package.
  • Use LlamaIndex SentenceSplitter and BM25 for page-aware document evidence, and LlamaIndex for the private LocalKnowledgeLibrary semantic-ranking backend, while preserving QuantMind public types and SQLite canonical storage.
  • Enforce the new package boundary with import-linter, synchronize the canonical design under contexts/design/rag/, and move the focused example/tests to examples/rag/ and tests/rag/.

Design Boundary

quantmind.rag is an opinionated LlamaIndex data-plane package, not a generic RAG framework. It may depend on quantmind.preprocess; preprocessing and quantmind.library must not depend on it. LlamaIndex types remain private, and the package does not add public retriever, vector-store, provider, backend, query-engine, or plugin abstractions. A future PageIndex adapter may live beside these document operations without being forced through LocalKnowledgeLibrary.search().

Related Issue

Closes #117.

Verification

  • ./scripts/verify.sh — 330 passed, 86.56% coverage; Ruff, basedpyright, and all 7 import contracts passed.
  • uv run python scripts/verify_pdf_rag_e2e.py — passed with 15 pages, 33 chunks, and page 5 as the top result.
  • uv run python examples/rag/paper.py — passed.
  • Pre-commit and pre-push hooks passed without bypasses.

Checklist

  • The title uses English Conventional Commit format: type(scope): summary.
  • The related issue or design discussion is linked when applicable.
  • ./scripts/verify.sh passes.
  • Every applicable live-network component smoke test passes, or this PR states why none applies.
  • Public behavior has focused tests, an example, and documentation where applicable.
  • The PR is complete, small, and contains no unrelated changes.

@keli-wen keli-wen added area: knowledge Canonical knowledge models, collections, indexing, and semantic search area: preprocess Deterministic acquisition, parsing, cleaning, and source handling impact: live-network Changes or depends on real public-network behavior or smoke tests type: feature Adds a new capability or observable behavior labels Jul 17, 2026 — with ChatGPT Codex Connector
@keli-wen keli-wen added area: rag and removed area: preprocess Deterministic acquisition, parsing, cleaning, and source handling labels Jul 18, 2026 — with ChatGPT Codex Connector
@keli-wen
keli-wen marked this pull request as ready for review July 18, 2026 04:17
@keli-wen
keli-wen merged commit 4ecb2e5 into master Jul 18, 2026
4 checks passed
@keli-wen
keli-wen deleted the codex/issue-117-rag branch July 18, 2026 04:18
@keli-wen keli-wen self-assigned this Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: knowledge Canonical knowledge models, collections, indexing, and semantic search area: rag impact: live-network Changes or depends on real public-network behavior or smoke tests type: feature Adds a new capability or observable behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(rag): adopt LlamaIndex for document ingestion and semantic retrieval

1 participant