feat(db): line_dh — dibbur-hamatchil index - #20
Merged
Conversation
Rashi/Tosafot (and commentaries generally) cannot be searched or navigated by their dibbur hamatchil: it is embedded in each line's content, and the TOC stops at daf/perek level (Rashi on Berakhot: 126 TOC entries for 3,265 lines). This adds line_dh, mapping (bookId, dhText) to lineIndex, so the client can prefix-match dibburim as the user types and render them as virtual sub-headings — without touching tocEntry/tocText. Two shapes cover the corpus, one per book: a <b>…</b> prefix (Rashi on Tanakh, Mishnah commentaries) and the text before the first spaced dash (Sefaria's Talmud commentaries). DhExtractor extracts conservatively — short dibbur, real commentary after it, structural markers (מתני', גמרא, בא"ד…) excluded — and BuildLineDhIndexCli indexes a book only when one shape dominates its content lines (>=40%, >=10 hits), which keeps books that merely bold an occasional word or use an incidental dash out entirely. The stored value is normalised TEXT (DhKey, in core), not a hash like line_ref: prefix search needs the text, and the (bookId, dhText, …) PK btree of the WITHOUT ROWID table serves range scans directly. The client must normalise the typed query with the same rules, so the normalisation is pinned to dh_key_fixtures.json for cross-repo parity. On a real v23 seforim.db: 1,935,274 dibburim over 3,451 books, built in under a minute; Rashi on Berakhot covered 3,092/3,139 content lines. buildLineDhIndex runs after every book-writing stage and is idempotent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Y-PLONI
force-pushed
the
feat/line-dh-index
branch
from
August 31, 2026 22:18
da2ca1a to
ce89be1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves the DB side of the forum request: searching Rashi/Tosafot by dibbur hamatchil, and rendering dibburim as sub-headings.
Problem
A commentary's dibbur hamatchil is embedded in each line's content, and the TOC stops at daf/perek level (Rashi on Berakhot: 126 TOC entries for 3,265 lines). There is no way to search a commentary by its opening words, or to navigate a daf's comments by dibbur.
What this adds
line_dh(bookId, dhText, lineIndex)— normalised text, not a hash (unlikeline_ref), so the client can prefix-match while the user types, straight off theWITHOUT ROWIDPK btree (dhText >= ?prefix AND dhText < ?prefixEnd), and render the rows as virtual sub-headings — without touchingtocEntry/tocText.Two shapes cover the corpus, one per book:
<b>…</b>prefix — Rashi on Tanakh, Mishnah commentaries, most Otzaria-sourced books.עד סוף האשמורה הראשונה – שליש הלילה…), including the daf's first comment that ends with.instead of a dash.False-positive defences (measured on a real DB)
Bold ≠ dibbur in general: 2.0M lines open with
<b>, including discourse-word emphasis (והנה×28K,אמנם×10K), structural markers (מתני',גמרא,בא"ד), and whole-line-bold decorated headings (57K). Three layers keep them out:<h1>–<h6>excluded.תוד"הis dropped while the genuine dibburתודהsurvives.Parity
The stored value is
DhKey.normalize()output; the Otzaria client must normalise the typed query with the same rules. Both sides pin todh_key_fixtures.json(DhKeyParityTesthere), same pattern asref_key_fixtures.json.Verification
seforim.db: 1,919,349 dibburim over 3,435 non-base/dependent books in 49s; pure base texts contribute zero rows.buildLineDhIndexruns after every book-writing stage and is idempotent (rebuilds from scratch).🤖 Generated with Claude Code
Review hardening