feat(search): index prose content for BM25 full-text search

[ShauryaaSharma]

What & why

search_graph BM25 only matched node names and headings, so it was blind to the
prose that documentation- and config-heavy repos carry. Markdown Section nodes
exposed only their heading; YAML/JSON Module nodes only their file name — the
section body and the description value were never indexed, and Section/Module
were excluded from BM25 results entirely. This indexes that prose so content is
searchable.

Closes #518
Closes #519

Changes

• store (store.c/.h): add a body column to the nodes_fts FTS5 table; new
  cbm_store_fts_rebuild() drops+recreates it (upgrading legacy 4-column DBs) and
  backfills body from each node's docstring, guarded by json_valid().
• pipeline: both FTS backfill sites call the shared cbm_store_fts_rebuild().
• mcp: stop excluding Section/Module from BM25 results (no rank boost, so code
  results still sort first).
• internal/cbm: capture the markdown section body (Section nodes don't index body text — BM25 can't search markdown content #518) and promote a top-level
  description/summary/purpose value onto the Module node (META.yaml/frontmatter description values not indexed for BM25 search #519), reusing the
  existing docstring property.

Testing

7 extraction cases + 3 store FTS cases added. Verified end-to-end: bodies are
extracted → indexed into nodes_fts.body → returned by BM25; json_valid() tolerates
malformed rows; legacy FTS tables upgrade on rebuild.

Notes

Backward compatible (additive column; legacy DBs upgrade on next index). No MCP
tool changes, no new deps, no new system()/popen()/network calls. #518 and #519
share the FTS body infra (#519 can't work without it), so they're together —
happy to split if preferred.

[ShauryaaSharma]

Rebased onto main (post-#667) , all tests green ✅

Rebased on top of the #667 merge. One minor conflict in extract_defs.c: upstream added qn_safe_segment() immediately before push_simple_class_def as part of the markdown QN slugification work. Resolution was additive, kept both the new helper and the docstring parameter this PR adds.

Verification against the #518 repro in the bug suite:

The #667 markup battery (repro_grammar_markup_markdown) now correctly expects "Section" as the label for headings, which is exactly what this PR emits. Running the 7 targeted extraction tests locally confirms the rebased code is still correct:

markdown_section_body_captured          PASS  ← Playwright body captured, sibling excluded
markdown_section_no_body                PASS  ← empty heading → no docstring
markdown_section_body_capped            PASS  ← body ≤ 500 bytes
yaml_description_promoted_to_module     PASS
yaml_summary_promoted_to_module         PASS
json_description_promoted_to_module     PASS
yaml_no_description_leaves_module_bare  PASS

[DeusData]

Huge thanks for opening this PR and for the work you put into it.

The maintainer shop is currently full, so this may sit for a bit before it gets a proper review. We will come back to this as soon as possible with real feedback; I wanted to make sure it did not sit unacknowledged in the meantime.

[ShauryaaSharma]

No worries at all @DeusData, take your time, there's no rush on my end! Happy to wait and appreciate you acknowledging it. 🙂

[ShauryaaSharma]

Hey @DeusData, just wanted to check in.
No rush at all, I know the queue is full, just bumping in case it fell off the radar.

I rebased and resolved the merge conflict in store.c (was just two comment blocks colliding with the new #768 schema-compat probe on main, no functional overlap), so this should be a clean merge again whenever you get a chance to take a look.

Happy to make any changes if you'd like a different direction on anything. Thanks again for your work maintaining this!

[DeusData]

This has not fallen off the queue. I rechecked the current head (2bc902e): the recorded full CI set is green, and there has been no newer contributor commit requiring another response. No further action is needed from you until the maintainer diff review; thanks for keeping the branch current.

[DeusData]

Reviewed this in full — thank you, and sorry it has waited so long. The engineering is solid and it is built at the right layer; what is holding it is a sizing question plus a rebase, not a design objection.

What I want to call out first, because it is the thing most contributions in this area get wrong: you did not hand-roll a BM25 implementation. You extended the SQLite FTS5 index we already vendor and already use for exactly this, added no dependency, and in the process deduplicated the two existing backfill sites into a shared cbm_store_fts_rebuild() — a net −25 lines. That is how a maintainer would have built it.

Several details showed real care: the json_valid() guard with a dedicated malformed-row test so legacy databases degrade to name-indexing instead of breaking; the legacy 4-column upgrade test, which is precisely the migration proof we want; the 500-byte cap matching our existing MAX_COMMENT_LEN; and the UTF-8-safe backoff. I also confirmed the migration story is genuinely benign — legacy DBs keep working read-only and get upgraded by the DROP+recreate at the next index run, which was already a full FTS rebuild, so there is no forced graph rebuild and no migration event to announce.

One factual correction worth having on record, in your favour: issue #518 asserts that bt/docstring content is already BM25-indexed. That is not true on main — nodes_fts carries only name, qualified_name, label and file_path. I verified the DDL. So this is a real gap, not a rediscovery.

The branch has drifted and needs a rebase. bm25_search on main now carries a label-boost CASE (Function/Method −10, Route −8, Class/Interface/Type/Enum −5) plus an ORDER BY rank, n.id tie-break, and TOON tree output. Your label-filter change needs re-applying onto both query strings by hand. Your comment about "(no rank boost, so code results still sort first)" should be rewritten while you are there — and the news is good: the boost actually strengthens your design, because Section and Module fall into the unboosted ELSE bucket, so code symbols keep their ordering advantage by construction rather than by luck.

The one thing I need from you before this can land: numbers.

The backfill indexes the docstring of every node, not only the Section and Module nodes that #518 and #519 need. On our reference corpus (~8.5M nodes) I estimate that at +150–400 MB of FTS index and +15–60 s per index run — and that rebuild happens on every run, including incremental background refreshes. Those are engineering estimates, not measurements, and I would rather not merge a scale change on an estimate.

Could you run scripts/benchmark-index.sh before and after on a large corpus and post the numbers? If the cost lands at the high end, there is an easy lever: restricting the body backfill to Section/Module rows is a one-line WHERE that removes roughly 95% of the cost, at the price of losing function-docstring search. I would rather see the measurement first and then choose deliberately.

One thing that is on us, not you. The PR un-excludes both Section and Module from the BM25 label filter. Because every file in every language has a Module node, that makes all of them returnable and countable — which re-admits the per-file nodes that filter deliberately treated as structural noise. #519 only needs the Modules carrying a promoted description, and there is no cheap SQL predicate that separates those. We are weighing that noise-versus-findability trade on our side and will come back with a direction. Nothing for you to redo there yet — please do not pre-emptively change it.

So: rebase, fix the stale ranking comment, and bring benchmark numbers. The direction call on Module scope is ours and I will get you an answer.

One last note — a small operational nuance rather than a defect: DROP+recreate opens a brief "no such table" window where a concurrent bm25_search returns NULL and silently degrades to the regex path. The old delete-all did not have that window. It degrades gracefully, but with daemon-era concurrent sessions it is now observable, so it is worth a thought during the rebase.