ContextSearch returns the correct session 0 times out of 24 on a benchmark of 12 real work sessions queried two ways each. Median latency is fine (157 ms) — the ranking is the problem. Three independent defects in LifeOS/install/skills/ContextSearch/Tools/ContextSearch.ts combine to produce it, and all three are in shipped code, not local config.
Measurement
12 work sessions that exist in the corpus (121 registered sessions across two roots, ~1,200 conversation logs). Each queried two ways: natural (project name in the query — the easy ceiling) and vague (from the symptom, mechanically constrained to share zero non-stopword tokens with the target slug). Queries were frozen with a checksum before the first run; nothing was rewritten after seeing results. A shuffled-pairing A/A control scores 4% against 33% for true pairing, so the metric discriminates.
| band |
recall@1 |
recall@3 |
recall@10 |
| natural (full list, as shipped) |
0% |
0% |
0% |
| vague (full list, as shipped) |
0% |
0% |
0% |
| natural, stopwords stripped, conversation logs excluded |
25% |
33% |
83% |
The bottom row is the same tool with the two defects below neutralised, and it is what the fixes are worth.
Defect 1 — the stopword list is English-only, so non-English queries drown in their own grammar
ContextSearch.ts line 57:
const STOPWORDS = new Set([
"the", "a", "an", "on", "in", "of", "to", "that", "it",
"is", "was", "we", "you", "i", "and", "or", "for", "with",
]);
A user writing in any other language sends every function word into the query. In Italian, a normal question like "quando abbiamo riparato i caratteri sovrapposti nei PDF" searches for quando, abbiamo, i, nei — each matching thousands of log lines — and the two content words are buried under the noise.
Measured: mechanically stripping the query's function words moves recall@3 on the vague band from 0% to 17%, changing nothing else.
Same function, line 147:
Accented letters are treated as separators, so accessibilità tokenizes to accessibilit and più to pi. Every language with diacritics silently loses word endings.
Suggested fix — Unicode-aware split, and stopword filtering that does not assume English:
.split(/[^\p{L}\p{N}]+/u)For stopwords, the robust fix is to stop maintaining a word list at all and let IDF (defect 3) demote high-frequency tokens automatically — a function word has near-zero IDF by definition, in every language. If a list is kept as a fast path, it needs at minimum the major LifeOS user languages, and tokenize() should fold diacritics before lookup.
Defect 2 — source scores are on incomparable scales, so a session can never outrank a chat log
Two families of sources are merged into one ranking with raw, unnormalised scores.
Name-matching sources (work.json, work dirs, session names) score how many query tokens appear in a short string — bounded by query length, so 1–5 in practice:
const overlap = tokenOverlap(tokens, blob);
...
score: overlap * multiplier, // lines 266, 310, 341
Content sources (ISA bodies, conversation logs) score how many lines of the file contain any query token, via rg -c — unbounded, hundreds to thousands:
const out = await ripgrep(pattern, WORK_DIR, ["--glob", "*ISA.md", "-c"]); // line 386
...
score: info.hits * multiplier, // lines 413, 548
They then compete in the same sorted list. The session whose directory is named exactly what you typed cannot win, by construction.
Concrete repro — searching for a session by its literal, unique directory name:
194 jsonl <a conversation log>
86 jsonl <another conversation log>
10 isa-body <an unrelated session with a long ISA>
10 work-dir+isa-body <the session actually named after the query> ← 4th
Across the benchmark, 70% of the top-10 slots are conversation logs. Excluding them is worth +25 points of recall@3.
Suggested fix — normalise within each source before merging, so each source contributes a comparable [0,1] signal, then apply an explicit per-source prior:
// after collecting each source's results, before the merge
function normalize(results: Result[], weight: number): Result[] {
const max = Math.max(...results.map(r => r.score), 1);
return results.map(r => ({ ...r, score: (r.score / max) * weight }));
}
// weights make the ranking policy explicit and tunable instead of accidental:
// a session named after your query should outrank a chat that merely mentions it
const merged = [
...normalize(workJson, 1.0),
...normalize(workDirs, 1.0),
...normalize(isaBodies, 0.7),
...normalize(jsonl, 0.4),
];Today those weights exist implicitly, set by whichever source happens to have the largest raw numbers.
Defect 3 — no IDF, so a common word counts as much as a distinctive one
rg -c with an alternation of all query tokens counts a line once whether it matched the rarest term or the most common one. Long documents therefore win regardless of relevance, and rare high-signal terms carry no extra weight.
Observed: for a query of three generic words plus one unique identifier that appears in exactly one session's directory name, a very long unrelated ISA that happens to contain the generic words throughout scores 28 and outranks the session literally named after the unique identifier, which scores 26.
Suggested fix — count per-token document frequency and weight by rarity, which also makes the stopword list redundant:
// per token, not one alternation regex for all tokens
const df = new Map<string, number>();
for (const t of tokens) df.set(t, await countFilesMatching(t, dir));
const N = totalDocs;
const idf = (t: string) => Math.log(1 + N / (1 + (df.get(t) ?? 0)));
// score a document by summing its per-token hits weighted by rarity,
// damped so a long file cannot win on length alone
score = tokens.reduce((s, t) => s + idf(t) * Math.log(1 + hits(doc, t)), 0);
log(1 + hits) is the length damping: the difference between 1 and 10 occurrences matters, the difference between 200 and 400 does not.
Defect 4 (consequence) — the score cannot express "not found"
Four control queries about work that never happened returned a top result scoring 2,210–3,164, squarely inside the 83–3,208 range of top-1 scores for queries whose target does exist. There is no threshold a caller could use to tell a hit from noise, so the tool always answers confidently. Defects 2 and 3 are the cause: unnormalised line counts make the absolute score a function of corpus size, not of match quality. Normalising (defect 2) makes an absence threshold meaningful for the first time.
Impact
The bug is worst for non-English users, who hit all four at once, but defects 2–4 are language-independent: an English install still ranks chat logs above the sessions those chats produced. Since ContextSearch is what the Algorithm and the /cs command rely on to recover prior work, silent 0% recall means the system reports "no prior work" for work that plainly exists — the same failure class as #1494 and #1498, one layer further in.
I'm happy to open a PR with the three fixes plus the benchmark harness if that's useful.
ContextSearchreturns the correct session 0 times out of 24 on a benchmark of 12 real work sessions queried two ways each. Median latency is fine (157 ms) — the ranking is the problem. Three independent defects inLifeOS/install/skills/ContextSearch/Tools/ContextSearch.tscombine to produce it, and all three are in shipped code, not local config.Measurement
12 work sessions that exist in the corpus (121 registered sessions across two roots, ~1,200 conversation logs). Each queried two ways: natural (project name in the query — the easy ceiling) and vague (from the symptom, mechanically constrained to share zero non-stopword tokens with the target slug). Queries were frozen with a checksum before the first run; nothing was rewritten after seeing results. A shuffled-pairing A/A control scores 4% against 33% for true pairing, so the metric discriminates.
The bottom row is the same tool with the two defects below neutralised, and it is what the fixes are worth.
Defect 1 — the stopword list is English-only, so non-English queries drown in their own grammar
ContextSearch.tsline 57:A user writing in any other language sends every function word into the query. In Italian, a normal question like "quando abbiamo riparato i caratteri sovrapposti nei PDF" searches for
quando,abbiamo,i,nei— each matching thousands of log lines — and the two content words are buried under the noise.Measured: mechanically stripping the query's function words moves recall@3 on the vague band from 0% to 17%, changing nothing else.
Same function, line 147:
Accented letters are treated as separators, so
accessibilitàtokenizes toaccessibilitandpiùtopi. Every language with diacritics silently loses word endings.Suggested fix — Unicode-aware split, and stopword filtering that does not assume English:
For stopwords, the robust fix is to stop maintaining a word list at all and let IDF (defect 3) demote high-frequency tokens automatically — a function word has near-zero IDF by definition, in every language. If a list is kept as a fast path, it needs at minimum the major LifeOS user languages, and
tokenize()should fold diacritics before lookup.Defect 2 — source scores are on incomparable scales, so a session can never outrank a chat log
Two families of sources are merged into one ranking with raw, unnormalised scores.
Name-matching sources (
work.json, work dirs, session names) score how many query tokens appear in a short string — bounded by query length, so 1–5 in practice:Content sources (ISA bodies, conversation logs) score how many lines of the file contain any query token, via
rg -c— unbounded, hundreds to thousands:They then compete in the same sorted list. The session whose directory is named exactly what you typed cannot win, by construction.
Concrete repro — searching for a session by its literal, unique directory name:
Across the benchmark, 70% of the top-10 slots are conversation logs. Excluding them is worth +25 points of recall@3.
Suggested fix — normalise within each source before merging, so each source contributes a comparable [0,1] signal, then apply an explicit per-source prior:
Today those weights exist implicitly, set by whichever source happens to have the largest raw numbers.
Defect 3 — no IDF, so a common word counts as much as a distinctive one
rg -cwith an alternation of all query tokens counts a line once whether it matched the rarest term or the most common one. Long documents therefore win regardless of relevance, and rare high-signal terms carry no extra weight.Observed: for a query of three generic words plus one unique identifier that appears in exactly one session's directory name, a very long unrelated ISA that happens to contain the generic words throughout scores 28 and outranks the session literally named after the unique identifier, which scores 26.
Suggested fix — count per-token document frequency and weight by rarity, which also makes the stopword list redundant:
log(1 + hits)is the length damping: the difference between 1 and 10 occurrences matters, the difference between 200 and 400 does not.Defect 4 (consequence) — the score cannot express "not found"
Four control queries about work that never happened returned a top result scoring 2,210–3,164, squarely inside the 83–3,208 range of top-1 scores for queries whose target does exist. There is no threshold a caller could use to tell a hit from noise, so the tool always answers confidently. Defects 2 and 3 are the cause: unnormalised line counts make the absolute score a function of corpus size, not of match quality. Normalising (defect 2) makes an absence threshold meaningful for the first time.
Impact
The bug is worst for non-English users, who hit all four at once, but defects 2–4 are language-independent: an English install still ranks chat logs above the sessions those chats produced. Since
ContextSearchis what the Algorithm and the/cscommand rely on to recover prior work, silent 0% recall means the system reports "no prior work" for work that plainly exists — the same failure class as #1494 and #1498, one layer further in.I'm happy to open a PR with the three fixes plus the benchmark harness if that's useful.