Fix RAG zero-recall with HNSW index and add empty-state guidance

[CarmenDou]

Problem

Production chat logs showed 14 of 22 assistant messages were "I couldn't find that in the provided documents", including for perfectly reasonable questions ("What position is Stephen Turoscy applying for?" against an uploaded cover letter). Every one of those responses had zero citations — vector search was returning nothing.

Root cause

The document_chunks embedding index was ivfflat with lists = 100. ivfflat partitions vectors into clusters and probes only 1 by default; with typical per-user collections of 6-300 chunks spread across 100 clusters, recall collapses to near zero. lists = 100 is tuned for ~100k vectors, not template-scale data.

Fixes

1. HNSW index (migrations/db_init.sql): replaces ivfflat. No cluster count to mistune; 95%+ recall at any collection size. Migration drops the old index by name and creates the new one idempotently. Verified on the production database: a question against the Bitcoin whitepaper that previously returned 0 citations now returns 8 with correct page numbers.

2. Empty-state guidance, NotebookLM-style, three layers:

   • No ready documents → chat input on /chat is disabled with an upload call-to-action (mirrors NotebookLM's add-sources gating), and /api/chat short-circuits with a deterministic guidance message instead of an LLM round-trip.
   • Zero retrieval hits (rare post-HNSW) → deterministic message listing the user's document names plus up to 3 cached suggested_questions as concrete next steps. No LLM call.
   • Vague/greeting input ("test", "hola") → RAG_SYSTEM_PROMPT now forbids the bare refusal; the model must say what the sources DO cover and suggest one concrete question. This is one step friendlier than NotebookLM, which hard-refuses with "NotebookLM can't answer this question".

Already verified in production

The HNSW index swap and all three guidance paths are deployed to https://aipdfchat.insforge.site and verified live; this PR brings the template source in line with what's running.

Test plan

• npm run build
• Fresh link → db_init.sql applies cleanly (drop-if-exists tolerates both old and new index names)
• Ask a question with no PDFs uploaded → input disabled on /chat; API returns upload guidance
• Upload a PDF, ask a real question → citations return
• Ask "test" → response explains what the documents cover and suggests a question

🤖 Generated with Claude Code

---

Summary by cubic

Fixes RAG zero-recall by switching the document_chunks index to HNSW and adds clear empty-state guidance in chat and API. Also adds site metadata and assets for better sharing and indexing, and returns a generic error on document lookup failures.

• Bug Fixes

  • Replaced ivfflat with HNSW for document_chunks in migrations/db_init.sql (drops old index; creates document_chunks_embedding_hnsw_idx idempotently) to recover recall on small collections.
  • /api/chat: on ready-document lookup failure, log the raw error server-side and send a generic stream error instead of showing “no documents” guidance.

• New Features

  • Disable chat input when no ready documents; upload call-to-action on /chat.
  • Short-circuit /api/chat when no ready docs or zero retrieval hits; list document names and up to 3 suggested_questions.
  • Update RAG_SYSTEM_PROMPT to avoid bare refusals for vague inputs and suggest one concrete question.
  • Add SEO/GEO metadata and assets: canonical + OG/Twitter tags, JSON-LD, robots.ts, sitemap.ts, app/icon.png, app/opengraph-image.png, and public/llms.txt (URLs derive from NEXT_PUBLIC_BETTER_AUTH_URL).

^{Written for commit 367948d. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 4 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="ai-pdf-chatbot/app/api/chat/route.ts">

<violation number="1" location="ai-pdf-chatbot/app/api/chat/route.ts:155">
P2: Raw database error messages are sent to clients through the stream error payload, which can leak internal details.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[jwfing]

LGTM - approved.

[jwfing]

LGTM - approved.