[codex] Add tokenized FTS5 query helper#35
Conversation
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: [codex] Add tokenized FTS5 query helper This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8f5342a7e6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| flush() | ||
| } | ||
| flush() | ||
| return strings.Join(terms, " ") |
There was a problem hiding this comment.
When the input contains no token characters, such as -- "", terms stays empty and this returns ""; passing that directly to an FTS5 MATCH ? predicate (the query shape documented in docs/cloudflare-remote-archives.md) makes SQLite raise fts5: syntax error near "" instead of returning no rows. The existing FTS5Terms helper avoids this by returning FTS5Phrase(""), so this helper should also produce a valid no-match expression or otherwise surface an error rather than handing callers an invalid FTS query for arbitrary user punctuation.
Useful? React with 👍 / 👎.
|
Post-merge audit found one real edge case and it is now fixed. Decomposed Unicode graphemes were split at combining marks, so the generated query did not match SQLite FTS5's token for the original text. PR #37 preserves Unicode marks, documents the empty-input sentinel, and landed as Proof used a real temporary SQLite FTS5 store and covered punctuation, quote/operator-shaped input, multiple scripts, precomposed/decomposed Unicode, empty/all-punctuation input, and unchanged legacy helper behavior. Full tests, race tests, vet, deadcode, vulnerability checks, changed-package static/security analysis, secret scans, autoreview, and the public model-identifier gate passed. Exact merged-head checks are green: CI, CodeQL, and secret scanning. |
1 participant
Summary
Validation
go test ./...go test -race ./store/...go vet ./...