fix(search): literalize FTS5 MATCH tokens to prevent query semantics injection

[Arreboi06]

Summary

Fixes #160 by changing buildFtsQuery() from an operator-sanitizing helper into a literal-token query builder.

The important security boundary is: user input is never treated as FTS5 syntax. User text is normalized into plain literal tokens, every emitted token is phrase-quoted, and the only MATCH operators inserted into the expression are generated by our code.

Relationship to #178

I saw the triage note that #178 is the locally verified canonical narrow fix for #160. This PR is now intentionally positioned as the stricter literal-token variant, not just another denylist patch.

The behavior beyond #178 is that reserved-looking words typed by the user are preserved as searchable text instead of being deleted:

AND OR NOT NEAR
  -> "AND" OR "OR" OR "NOT" OR "NEAR"

That means the fix prevents user-controlled FTS5 query structure while avoiding recall loss for documents that actually contain words such as AND, OR, NOT, or NEAR.

Core design decision

buildFtsQuery() owns the complete MATCH expression structure.

raw user text
  -> NFKC normalization
  -> Unicode literal-token extraction ([\p{L}\p{N}_]+)
  -> jieba search tokenization when available, regex fallback otherwise
  -> second literal-token safety pass for every tokenizer output
  -> quote every token as an FTS5 phrase literal
  -> join code-generated terms with code-generated OR

Inputs such as boolean operators, NEAR/5, column filters, quotes, wildcards, and groups can only become searchable text tokens. They cannot become FTS5 query control flow.

Why this is stronger than a denylist-only fix

A denylist-only fix removes known dangerous words such as AND, OR, NOT, and NEAR. That is a valid narrow fix, but it has two trade-offs:

1. It can reduce recall by deleting words the user actually typed.
2. It depends on recognizing every FTS5 syntax path that should be removed.

This PR avoids both: syntax-looking input is reduced to literal tokens, and every token is phrase-quoted before it reaches FTS5.

Security and recall behavior

User input	Generated MATCH behavior	Why it is safe
alpha" OR "beta	"alpha" OR "OR" OR "beta"	The injected OR is a quoted literal term, not user-controlled structure.
alpha AND NOT beta	"alpha" OR "AND" OR "NOT" OR "beta"	Boolean words stay searchable but cannot negate or combine terms.
(alpha) OR (beta)	"alpha" OR "OR" OR "beta"	Parentheses are separators, not grouping syntax.
content:foo	"content" OR "foo"	Column-filter syntax becomes plain text tokens.
alpha*	"alpha"	Prefix wildcard syntax cannot expand the query.
full-width AND	"AND"	NFKC normalization handles Unicode variants while preserving search text.

Implementation highlights

• sanitizeFtsInput() normalizes raw input and extracts only Unicode literal tokens.
• sanitizeFtsToken() re-sanitizes every tokenizer output so punctuation-bearing jieba/native/stub tokens cannot reintroduce MATCH syntax.
• buildFtsQuery() contributes all MATCH structure itself; user text contributes tokens only.
• Existing jieba-based Chinese recall remains in the path, with an additional literal-token safety pass after tokenization.
• The latest commit removes the fragile internal string-split assembly and uses explicit quoted literal term arrays.

Test coverage

The tests cover generated query strings and real FTS5 execution:

• Unit coverage for literalizing reserved words: AND, OR, NOT, NEAR.
• Unit coverage for column-filter-looking input such as content:foo.
• Unit coverage for quote injection, parenthesized expressions, prefix wildcard syntax, and NFKC normalization.
• Tokenizer re-sanitization coverage for punctuation-bearing tokenizer output.
• Real node:sqlite FTS5 fixture tests proving generated MATCH expressions execute successfully.
• Real FTS5 fixture test proving reserved words remain searchable as literal document text.
• Recall comparison against benign queries so ordinary keyword behavior is preserved.
• 200 randomized adversarial query inputs to ensure generated MATCH expressions do not throw.

Verification

Latest pushed head:

802998c test(search): prove FTS5 literal token coverage

Full test suite:

npm.cmd test

Test Files  6 passed (6)
Tests       113 passed (113)

Build:

npm.cmd run build

Build complete

tsdown prints a Node.js v22.17.0 deprecation warning in this local environment, but the build exits successfully.

Files worth reviewing first

1. src/core/store/sqlite.ts - literal-token query builder implementation.
2. src/core/store/sqlite.test.ts - unit-level security, tokenizer re-sanitization, and recall semantics.
3. src/core/store/sqlite.recall.test.ts - real FTS5 execution, reserved-word literal recall, recall parity, and fuzz coverage.
4. CHANGELOG.md - documents the refined solution and its difference from fix(search): sanitize fts query operators #178.

[Yuntong8888]

Hi @Arreboi06 👋,
Thank you for submitting this PR and participating in Tencent Rhino-bird Open-source Training Program!
We have successfully received your submission. The program is currently in full swing, and we will complete the Code Review for you as soon as possible. Please keep an eye on the status notifications for this PR so you can follow up promptly once the review feedback is provided.
Thanks again for your contribution and open-source spirit! 🚀

[Arreboi06]

I pushed a follow-up refinement that strengthens the original fix.

The first version sanitized known FTS5 operators. The new version makes buildFtsQuery() a literal-token builder: user input is normalized and tokenized as plain text, every token is quoted as an FTS5 phrase, and only code-generated OR operators are used to assemble the MATCH expression.

This means inputs such as alpha" OR "beta, alpha AND NOT beta, (alpha) OR (beta), content:foo, and alpha* cannot alter MATCH semantics. At the same time, reserved words like AND / OR / NOT / NEAR remain searchable user terms instead of being dropped.

Verification:

npm.cmd test
# Test Files  6 passed (6)
# Tests       111 passed (111)

npm.cmd run build
# passed

[YOMXXX]

Triage note: #178 has been locally verified and approved as the canonical narrow fix for #160. This PR appears to cover the same FTS5 operator-sanitization scope, so I recommend treating it as superseded unless it intentionally adds coverage or behavior beyond #178.

[Arreboi06]

Following up on the triage note that #178 is the canonical narrow fix for #160: I pushed 802998c to make this PR intentionally cover behavior beyond a denylist-only operator-sanitization patch.

What is now distinct from #178:

• buildFtsQuery() is framed as a literal-token query builder: user input can contribute searchable text only, never FTS5 query structure.
• Reserved-looking user terms such as AND, OR, NOT, and NEAR are preserved as quoted literals instead of being deleted, so the security fix does not reduce recall for documents that actually contain those words.
• Tokenizer output is re-sanitized after jieba/fallback tokenization, so punctuation-bearing native/stub tokenizer output cannot reintroduce MATCH syntax.
• The test coverage now includes true node:sqlite FTS5 execution, reserved-word literal recall, tokenizer re-sanitization, recall parity, and 200 randomized adversarial inputs.

Fresh local verification on the pushed head:

npm.cmd test
Test Files  6 passed (6)
Tests       113 passed (113)

npm.cmd run build
Build complete

tokenizer 二次净化测试
真实 FTS5 执行测试

tokenizer 二次净化测试
re-sanitizes tokenizer output before building MATCH

真实 FTS5 执行测试

So I agree #178 remains the canonical narrow fix. This PR is now intended as the stricter literal-token variant with additional behavior and coverage, if the maintainers prefer that shape for #160.