Luman — Hybrid RAG Engine

A retrieval-augmented generation engine over your own documents. Point it at a folder, ask questions, get answers that cite the exact passage they came from — and an honest "the documents do not say" when the evidence is missing.

Every model runs locally. No OpenAI, no hosted vector database, no API keys, no telemetry. FAISS + BM25 + all-MiniLM-L6-v2 + Ollama, on CPU.

data/raw/*.pdf  →  chunk  →  index (dense + sparse)  →  hybrid retrieve  →  rerank
                                                                        ↓
                                          cited answer  ←  verify citations  ←  Ollama
                                                                        ↓
                                                   confidence + abstention

Why it is built this way

Four decisions shape everything else.

1. Dense + sparse, not one or the other. Dense retrieval under-weights rare literal tokens; BM25 under-weights paraphrase. For a question containing an exact identifier like maxUnavailable, dense-only returns the wrong document and BM25-only returns the right one and nothing else. The failures are complementary, which is the entire argument for running both. They are combined with Reciprocal Rank Fusion, because averaging cosine similarity in [-1,1] against unbounded BM25 scores is meaningless.

2. Citations are verified, in two separate passes. Reference validation (does [2] resolve to a real passage?) is deterministic. Support checking (does passage 2 actually support that claim?) is model-judged. These are deliberately not merged — merging them lets "cites [2]" masquerade as "correctly grounded", which is exactly the failure this system exists to catch.

3. It abstains. If the retrieved evidence does not support an answer, the system returns status: abstained with not_established reasons and the passages a human should review — not a confident invention. Abstention is decided on evidence alone; how sure the model sounds is never a substitute for support.

4. SQLite is the only source of truth. FAISS and BM25 are derived artifacts, always rebuilt from one canonical query over the active chunk set. Two independently-patched indexes drift; a rebuild cannot. See Architecture for the crash-consistency protocol.

Quick start

Requirements


Python	3.11 or 3.12
RAM	~4 GB free (MiniLM + cross-encoder on CPU)
Disk	~500 MB for models, plus your documents
Optional	Ollama — only needed to generate answers

1. Environment

cd D:\luman
python -m venv .venv
.venv\Scripts\Activate.ps1              # Windows PowerShell
# source .venv/bin/activate              # macOS / Linux

pip install -r requirements.txt

2. Index the bundled sample documents

python scripts/seed.py

Indexes everything in data/raw/ using the structured chunking strategy. The first run downloads all-MiniLM-L6-v2 (~90 MB) into data/models/; later runs start from cache.

3. Retrieve — works with no LLM at all

python -m app.cli query "How long are access tokens valid?"

4. Generate an answer — needs Ollama

ollama serve                            # terminal 1, keep it open
ollama pull qwen2.5:3b                  # terminal 2, one time, ~2 GB

python -m app.cli ask "How long are access tokens valid?"

If Ollama is not running, ask says so and prints the retrieved passages instead of failing. Retrieval, evaluation and the API all keep working.

5. Or chat with your documents

luman chat

Every way to run it

A. Interactive chat (recommended start)

luman chat                              # indexes data/raw, then opens a REPL
luman chat C:\Users\me\Documents        # index and chat with any folder
luman chat --web                        # browser UI instead of the terminal

In-chat commands: /help, /sources, /reindex, /reset, /exit. Short questions inherit the previous turn's context, so "and how long is that?" works.

luman chat --web starts the API and the Streamlit UI together and opens your browser. Ctrl+C stops both.

B. One-shot question

luman ask "how many vacation days do I get?"
.\ask.cmd "How long are access tokens valid?"      # Windows shortcut, no venv activation
python ask.py "how long are access tokens valid?" # same thing, no venv activation
python ask.py --search "maxUnavailable"           # retrieval only, sub-second, no LLM

C. Index without chatting

luman index                             # index data/raw
luman index C:\Users\me\Documents --strategy semantic
luman status                            # what is indexed

D. Full CLI

python -m app.cli ingest --strategy structured
python -m app.cli query "What is the value of maxUnavailable?" --mode hybrid
python -m app.cli evaluate
python -m app.cli compare --out evaluation/reports

E. API + dashboard (two terminals)

python -m uvicorn app.api.main:app --reload     # terminal 1 → :8000
python -m streamlit run dashboard/app.py        # terminal 2 → :8501

The dashboard talks to the API over HTTP, so there is exactly one retrieval and generation code path. See Dashboard.

F. Docker Compose

docker compose up -d --build      # API :8000, dashboard :8501, Ollama
docker compose run --rm seed     # index the sample documents into the shared volume

Then open http://localhost:8501 and http://localhost:8000/docs.

Services: ollama (LLM), modelpuller (one-shot ollama pull qwen2.5:3b), api, dashboard, seed. Named volumes persist data/, the downloaded models and the Ollama model store, so containers restart without re-downloading. Models are baked into the image at build time so the API never downloads at runtime; skip that with --build-arg SKIP_MODEL_DOWNLOAD=1 for a smaller image.

G. Verify the whole system in one shot

python verify.py

Runs 13 end-to-end checks — package installed, model loadable, Ollama reachable, unit tests, index consistency, hybrid/dense search, a real answer, an abstention, the luman command, and dataset validity — and prints a pass/fail table. Exit code is 0 only if all 13 pass.

CLI reference

`luman` — the friendly entry point

luman chat [FOLDER]        chat with data/raw (or FOLDER)
luman chat --web           browser interface instead of the terminal
luman ask "QUESTION"       one question, print, exit
luman index [FOLDER]       index a folder
luman status               what is indexed

  --strategy fixed|structured|semantic   (default: structured)
  -k N                                   passages to use (default: 5)
  --rerank                               use the cross-encoder (slower, more accurate)
  --port N                               web UI port (default: 8501)

`python -m app.cli` — the full tool

Command	What it does
`ingest [paths...]`	Ingest documents and rebuild indexes
`reindex`	Re-chunk and re-embed everything (after a strategy change)
`rebuild [--re-embed]`	Rebuild indexes from SQLite — no model needed unless vectors are gone
`status`	Store and index statistics
`verify`	Check index consistency, rebuild if stale
`dedup-report [--job ID]`	Show what deduplication decided
`list-docs`	List ingested documents
`query "QUESTION"`	Retrieve without generating
`ask "QUESTION"`	Retrieve and generate
`evaluate`	Run the golden dataset
`compare`	Ablation grid: dense / bm25 / hybrid / rerank
`check-dataset`	Validate the golden dataset against the corpus

Shared flags — --mode hybrid|dense|sparse, -k/--top-k N, --strategy fixed|structured|semantic, --rerank / --no-rerank, --json (machine-readable output), -v (debug logging), --config PATH.

# retrieval only, no LLM needed
python -m app.cli query "What is the value of maxUnavailable?" --mode dense
python -m app.cli query "What is the value of maxUnavailable?" --mode sparse
python -m app.cli query "What is the value of maxUnavailable?" --mode hybrid

# faster, skip the cross-encoder
python -m app.cli ask "How long are access tokens valid?" --no-rerank

# machine-readable
python -m app.cli status --json

Architecture

Full detail is in ARCHITECTURE.md. The essentials:

data/raw/            PDF / MD / HTML / TXT
     │
     ▼  app/ingestion/       loaders: pypdf, beautifulsoup4, stdlib
data/processed/      normalized JSON + text
     │
     ▼  app/normalize.py    unicode folding, whitespace, dedup hashing
     │
     ▼  app/chunking/       fixed | structured | semantic
     │
     ├──────────────► app/index/store.py     SQLite: chunks + vectors  ◄── SOURCE OF TRUTH
     │                        │
     │                        ├──► app/index/dense.py   FAISS IndexFlatIP (cosine)
     │                        └──► app/index/sparse.py  BM25Okapi
     │
     ▼  user question
     │
     ├──► dense search  (FAISS cosine)
     ├──► sparse search (BM25 keyword)
     │         │
     │         ▼  app/retrieval/fusion.py   RRF: Σ w_r / (k0 + rank_r), k0=60
     │         ▼  app/retrieval/rerank.py   optional cross-encoder, top 20 only
     ▼
 context assembly — passages numbered [1]..[n]
     │
     ▼  app/generation/llm.py                Ollama qwen2.5:3b
     ▼  app/generation/citations.py          reference check  (deterministic)
     │                                       support check   (model-judged)
     ▼  app/generation/confidence.py        0.4·R + 0.4·C + 0.2·Completeness
     ▼
 abstention decision → response

Module map

Package	Responsibility
`app/normalize.py`	Unicode/whitespace folding, dedup hashing, heading detection
`app/models.py`	Domain types: `Chunk`, `Section`, `NormalizedDocument`, `IngestReport`
`app/config.py`	Layered settings — dataclass defaults → `config.toml` → env vars
`app/ingestion/`	Multi-format loaders + the raw → processed pipeline
`app/chunking/`	`FixedSizeChunker`, `StructuredChunker`, `SemanticChunker` + factory
`app/embeddings/`	Lazy `all-MiniLM-L6-v2` wrapper; offline `HashingEmbedder` for tests
`app/index/store.py`	SQLite canonical chunk store (source of truth)
`app/index/dense.py`	FAISS `IndexFlatIP` / HNSW with ordinal→chunk-id sidecar
`app/index/sparse.py`	`BM25Okapi` over the active corpus
`app/index/dedup.py`	Exact hash + embedding near-duplicate detection
`app/index/service.py`	The only component allowed to mutate indexes
`app/retrieval/`	Dense/sparse search, RRF fusion, cross-encoder reranking
`app/generation/`	Ollama client, prompts, citation verification, confidence, abstention
`app/evaluation/`	Metrics, golden dataset, runner, comparison harness
`app/api/`	FastAPI app and Pydantic schemas
`app/cli.py`	`ingest
`app/__luman__.py`	The `luman` command: `chat
`app/chat.py`	Terminal REPL and the `--web` launcher
`dashboard/app.py`	Streamlit UI (Chat / Search / Documents / Compare)

Index consistency

Both indexes are rebuilt from the same query over the same active chunk set:

chunks = store.active_chunks()          # deterministic order
dense.build([c.chunk_id for c in chunks], store.active_vectors(...))
sparse.build([c.chunk_id for c in chunks], [c.text for c in chunks])

The store records an index_fingerprint — sha256(ordered active chunk ids + embedding model name). verify() compares it on every run and rebuilds on any mismatch. Because vectors are persisted as SQLite BLOBs, a rebuild needs no model and no re-embedding:

Remove-Item data\index\faiss.index
python -m app.cli verify        # detects the missing artifact, rebuilds

Ingestion is batched precisely to bound this cost: embed everything, build one in-memory working index for dedup, commit a single transaction, then rebuild once. Two rebuilds per batch regardless of corpus size.

API reference

Start it with python -m uvicorn app.api.main:app --reload, then open http://localhost:8000/docs for interactive Swagger.

Method	Endpoint	Purpose
`POST`	`/v1/ask`	Answer with citations, confidence components, sources
`POST`	`/v1/retrieve`	Retrieve passages only, no generation
`POST`	`/v1/ingest`	Ingest and index documents
`GET`	`/v1/documents`	List indexed documents
`GET`	`/v1/stats`	Index statistics and dataset summary
`GET`	`/v1/dataset`	Golden dataset summary
`GET`	`/health`	Component health, `ok` or `degraded`
`GET`	`/`	Service banner (not in the schema)

curl -X POST localhost:8000/v1/ask -H 'Content-Type: application/json' `
  -d '{"question":"How long are access tokens valid?","top_k":5}'

curl -X POST localhost:8000/v1/retrieve -H 'Content-Type: application/json' `
  -d '{"question":"maxUnavailable","mode":"sparse"}'

/v1/ask accepts question, top_k, mode (hybrid|dense|sparse), rerank and verify_citations.

Errors always look the same, whatever failed:

{"error": "validation_error", "detail": "question: String should have at least 1 character", "status_code": 422}

FastAPI's built-in 422 handler is deliberately overridden so it cannot bypass this shape. confidence is null when generation failed — it is never filled with a placeholder, because reporting a number the system never computed would be worse than reporting none.

Dashboard

python -m uvicorn app.api.main:app --reload     # terminal 1
python -m streamlit run dashboard/app.py        # terminal 2

Four tabs:

Tab	What it does
Chat	Conversational Q&A with citations, confidence and timing per turn
Search	Raw retrieval with mode switch, scores per retriever, citation checks
Documents	Browse the corpus and ingest new files
Compare	Side-by-side dense vs sparse vs hybrid for the same question

Point it at a different API with RAG_API_URL (defaults to http://localhost:8000).

Configuration

config.toml at the project root. Dataclass defaults in app/config.py apply to anything omitted, and environment variables override the file.

[paths]
raw_dir       = "data/raw"
processed_dir = "data/processed"
index_dir     = "data/index"
db_path       = "data/index/chunks.db"

[ingestion]
supported_extensions = [".md", ".markdown", ".txt", ".text", ".html", ".htm", ".pdf"]
recursive = true
copy_to_raw = true

[chunking]
strategy  = "fixed"        # fixed | structured | semantic
chunk_size = 1000
chunk_overlap = 150
structured_max_size = 1200  # sections larger than this get split further
semantic_threshold = 0.55  # lower = more, smaller chunks
semantic_min_size = 150
semantic_max_size = 1000

[embedding]
model_name  = "sentence-transformers/all-MiniLM-L6-v2"
batch_size  = 64
device      = "cpu"
cache_folder = "data/models"

[dense]
index_type = "FlatIP"      # FlatIP (exact) or HNSW (approximate)
hnsw_m = 32
hnsw_ef_construction = 128
ef_search = 64

[sparse]
k1 = 1.5
b  = 0.75

[dedup]
exact_enabled = true
near_enabled  = true
threshold     = 0.95       # near-duplicate cosine cutoff
candidates    = 10
action        = "skip"     # skip | flag  (flag keeps them for review)

Defaults differ by entry point

Worth knowing, because it is a common source of surprise:

Entry point	Default strategy
`config.toml`	`fixed`
`python scripts/seed.py`	`structured`
`luman chat` / `luman index`	`structured`
`python -m app.cli ingest`	`config.toml`, i.e. `fixed`

Pass --strategy explicitly to remove the ambiguity.

Retrieval settings

Fusion and reranking are not read from config.toml — they live in app/retrieval/config.py and are overridable by environment variable only.

Dataclass default	Value	Env override
`fusion.dense_weight`	`0.7`	`RAG_DENSE_WEIGHT`
`fusion.sparse_weight`	`0.3`	`RAG_SPARSE_WEIGHT`
`fusion.k0`	`60`	—
`fusion.candidates`	`20`	—
`fusion.dense_top_k` / `sparse_top_k`	`10`	—
`rerank.enabled`	`True`	`RAG_RERANK_DISABLED`
`rerank.model_name`	`cross-encoder/ms-marco-MiniLM-L-6-v2`	—
`rerank.top_n`	`5`	—
`rerank.max_candidates`	`20`	—

Environment variables

Variable	Effect
`RAG_DATA_DIR` / `RAG_RAW_DIR` / `RAG_INDEX_DIR` / `RAG_DB_PATH`	Relocate data paths
`RAG_STRATEGY`	Default chunking strategy
`RAG_CHUNK_SIZE` / `RAG_CHUNK_OVERLAP`	Fixed chunking
`RAG_SEMANTIC_THRESHOLD`	Semantic chunking cutoff
`RAG_MODEL`	Embedding model
`RAG_DENSE_INDEX`	`FlatIP` or `HNSW`
`RAG_DEDUP_THRESHOLD` / `RAG_DEDUP_ACTION`	Near-duplicate cutoff and behaviour
`RAG_DENSE_WEIGHT` / `RAG_SPARSE_WEIGHT`	RRF fusion weights
`RAG_RERANK_DISABLED`	Disable the cross-encoder
`OLLAMA_HOST_URL`	Ollama endpoint (default `http://localhost:11434`)
`RAG_LLM_MODEL`	Ollama model (default `qwen2.5:3b`)
`RAG_API_URL`	Dashboard → API base URL

Evaluation

evaluation/dataset/golden_qa.json holds 100 hand-verified cases.

Expected behaviour	Count
Answerable	81
Unanswerable	16
Ambiguous	3

Category	Count
Direct factual	56
Exact technical term	16
Multi-hop	8
Multi-passage	1
Ambiguous	3
Unanswerable	16

python -m app.cli check-dataset                        # validate against the corpus
python -m app.cli evaluate                             # retrieval metrics
python -m app.cli evaluate --generate                  # + answer quality (needs Ollama)
python -m app.cli evaluate --out evaluation/reports/x.json
python -m app.cli compare --out evaluation/reports     # ablation grid

Measured results

Structured chunking, hybrid RRF, reranking off, 100 cases, CPU:

Metric	Value
Recall@5	0.9771
MRR@10	0.9338
Precision@5	0.2300
Hit@5	0.9875
Abstention accuracy	0.8351

Retrieval ablation on one fixed index:

Configuration	Recall@5	MRR@10	Prec@5	Hit@5
dense-only	0.9521	0.9010	0.2225	0.9625
bm25-only	0.9563	0.9039	0.2483	0.9875
hybrid-rrf	0.9771	0.9338	0.2300	0.9875

Hybrid is the best configuration on both Recall@5 and MRR@10. BM25-only edges out hybrid on Precision@5, which is a ceiling artefact rather than a quality signal — see below.

On Precision@5 being low. It is not a defect. Most gold sets contain exactly one relevant passage, so five retrieved passages give a ceiling of 1/5 = 0.20; the multi-section cases push the mean slightly above it. Precision@5 only means something for questions with several relevant passages.

On abstention accuracy at 0.8351. The misses are almost entirely the three ambiguous cases plus a few unanswerable questions that retrieve a high-scoring but non-answering passage. This is the honest number.

Reports in evaluation/reports/: comparison_report.md and comparison_report.json (ablation grid) and eval_structured.json (per-case).

Tests

pytest -m "not integration"     # 328 tests, no model download, fast
pytest                         # 337 tests, + real MiniLM end to end

337 tests total: 328 unit + 9 integration. Unit tests use an offline HashingEmbedder, so the fast suite needs no network.

Coverage spans normalization, all four loaders, all three chunkers, deduplication, index consistency across restarts, RRF fusion, reranking, citation verification, confidence scoring, abstention, evaluation metrics, and every API endpoint.

How to know it actually works

Three checks that demonstrate the system end to end.

1. Hybrid beats dense on exact technical terms

python -m app.cli query "What is the value of maxUnavailable?" --mode dense   # wrong top hit
python -m app.cli query "What is the value of maxUnavailable?" --mode sparse  # correct
python -m app.cli query "What is the value of maxUnavailable?" --mode hybrid  # correct + semantic neighbours

Dense-only misses it because maxUnavailable is a rare literal token. BM25 finds it but returns little else. Hybrid gets both.

2. Indexes stay in sync

python -m app.cli verify
Remove-Item data\index\faiss.index
python -m app.cli verify     # detects the missing artifact, rebuilds from SQLite

No re-embedding: vectors are already in SQLite.

3. Unanswerable questions abstain

python -m app.cli ask "What is the office wifi password?"

Returns status=abstained with not_established and passages_for_review, rather than a confident invention.

Project layout

app/
  __luman__.py         `luman` command: chat | ask | index | status
  chat.py              terminal REPL + `--web` launcher
  cli.py               full CLI: ingest | query | ask | evaluate | compare | …
  config.py            layered settings (defaults → toml → env)
  models.py            Chunk, Section, NormalizedDocument, IngestReport
  normalize.py         text folding, dedup hashing
  ingestion/           PDF/MD/HTML/TXT loaders + raw → processed pipeline
  chunking/            fixed | structured | semantic
  embeddings/          MiniLM wrapper + offline HashingEmbedder
  index/               store | dense | sparse | dedup | service
  retrieval/           fusion | rerank | engine
  generation/          llm | prompts | citations | confidence
  evaluation/          metrics | dataset | runner | report
  api/                 FastAPI app + Pydantic schemas
dashboard/app.py       Streamlit UI
data/
  raw/                 source documents (PDF, MD, HTML, TXT)
  processed/           normalized JSON + text
  index/               chunks.db, faiss.index, bm25_corpus.json
  models/              cached HuggingFace models
evaluation/
  dataset/golden_qa.json   100 cases
  reports/                generated reports
scripts/                seed.py, prefetch_models.py, build_golden_dataset.py
tests/                  337 tests
config.toml             all settings
verify.py               13 end-to-end system checks
ask.py / ask.cmd        quick question tool
ARCHITECTURE.md         design rationale in detail
Dockerfile              single image for API + dashboard
docker-compose.yml      full local stack

Bugs this test suite caught

Each has a regression test.

Bug	Impact
`Chunk.create` hashed raw text	Exact dedup silently degraded to byte equality
Re-chunking matched its own old chunks	Re-indexing deleted the corpus
Citation markers stripped before binding	No claim was ever cited; coverage always 0
`ndcg_at_k` divided by `log2(0)`	Evaluation crashed on the first run
HTML loader re-parsed rendered text	Every heading was destroyed
`--rerank/--no-rerank` shorthand	`--no-rerank` silently unregistered on Python 3.12
pypdf emits no blank lines before body	Every numbered PDF heading was missed
FastAPI's built-in 422 handler	Bypassed the documented error shape
API ingest built its own embedder	Dimension/model-name mismatch with the index

Known limitations

Stated plainly, because a project that lists only strengths is not reporting.

Evaluation

The golden dataset is 100 hand-written cases against a dozen documents. Useful for catching regressions, far too small to claim statistical significance.
Precision@5 has a ceiling of ~0.20 for single-passage questions, so it is a weak signal here.
Key-term coverage is a blunt proxy for correctness. There is no reference-free semantic judge in the default suite. A local LLM judge is the obvious next step.
Ablations were run with reranking disabled for runtime. The cross-encoder variant is implemented and tested but not included in the shipped report.
Latency figures in comparison_report.md are ordering-biased: variants run sequentially, so the first absorbs model load and page-cache warm-up.

Retrieval

FixedSizeChunker attributes a chunk to the section containing its first character. A window straddling a heading is attributed to the earlier section.
Semantic chunking embeds every sentence at ingest time, which roughly doubles ingestion cost versus the other two strategies.
Near-duplicate detection uses a single FAISS top-k (candidates = 10). A true near-duplicate ranked 11th by the candidate query would be missed.
No query rewriting, decomposition, or HyDE. Multi-hop questions are answered from whichever passages were retrieved in one pass.
No vector index sharding, so ingestion is O(corpus) per batch. Fine for internal docs; not for millions of chunks.

Generation

qwen2.5:3b is small. It produces well-formed citations far more reliably than it produces correct factual content, which is exactly why citation support verification exists.
Support verification is itself model-judged. A wrong verdict from the verifier is not caught. Deterministic reference validation is the reliable layer; support checking is best-effort.
The completeness signal is keyword-overlap, not semantic. It penalises valid paraphrases.

Operational

Single-writer SQLite. Concurrent writers will hit database is locked.
No authentication or rate limiting on the API. It is intended for local use.
PDF support is text-layer only. Scanned PDFs are rejected with a clear message; OCR is out of scope.
Everything is CPU-bound and synchronous. Expect ~0.4 s per retrieve-and-answer query on a laptop.

Future work

Local LLM-as-judge for semantic correctness and faithfulness, replacing key-term coverage.
Query decomposition so multi-hop questions retrieve each sub-question.
Incremental FAISS append for O(new chunks) rather than O(corpus) rebuilds.
Per-section relevance grading in the golden dataset, so nDCG becomes informative instead of saturating.
PDF chunk provenance back to character offsets for exact highlighting.
Calibrate the confidence weights against labelled evaluation data rather than choosing them a priori.

License

MIT — see LICENSE.

Contributions welcome: read CONTRIBUTING.md first. It documents the invariants that are easy to break by accident (SQLite as the only source of truth, citation markers extracted before removal, self-exclusion in dedup).

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
app		app
dashboard		dashboard
data		data
evaluation		evaluation
scripts		scripts
tests		tests
.gitattributes		.gitattributes
.gitignore		.gitignore
ARCHITECTURE.md		ARCHITECTURE.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
ask.cmd		ask.cmd
ask.py		ask.py
config.toml		config.toml
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
verify.py		verify.py

Latest commit

History

Folders and files

Repository files navigation