feat(search): RRF re-rank + junk/coverage/geo filter + query cleaning for answer path - #68
Merged
Conversation
us
added a commit
that referenced
this pull request
Jun 1, 2026
#68's answer-path reranker ran the FULL composite (RRF + BM25 + geo-score) on every query. On the frozen benchmark that composite REGRESSES the raw-score baseline because our cross-engine overlap is near-zero (positions median = 1), making RRF the single worst variant. Default to the lexical core — the only variant that beats baseline: junk filter (structural + host blocklist) + coverage guard + competing-region filter, then order survivors by SearXNG's raw score and dedupe by registrable domain. Measured vs the raw-score baseline on the Tavily oracle: CleanRel 0.296->0.571, Junk@5 70->0, Recall 0.182->0.271, nDCG@5 0.144->0.224. - remove the composite RRF/BM25/geo-score from the default path - retain rrf/bm25_lite/geo_score (#[allow(dead_code)]) for a future config-gated experiment; the benchmark is the gate - drop "2025"/"2026" year literals from STOPWORDS (corpus-specific, they rot) - #68's rerank_tests.rs (geo-correct, junk@5==0, dedupe) stay green
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring SearXNG-backed answer search toward Tavily-grade
The LLM answer/summarize search path fed the synthesizer whatever SearXNG's raw
.scoreranked first — which is rank-inverse and content-blind. SearXNG'sbingengine keyword-matches stopwords ("top"/"best"/"fix") and returns dictionary
definitions (merriam-webster), retailer homepages (bestbuy/zara) and homonym
pages at a flat score 1.0 that tie/outrank real results. Measured: Junk@5 = 54
across a 56-query corpus; "top restaurants in belgrad" returned Belgrad-Forest
(Istanbul) / Belgrade-Lakes (Maine) noise.
What changed (answer/summarize path only; plain search unchanged)
rerank.rs(new): RRF fusion over per-engine ranks (k=60) + hard junkfilter (dictionary/shopping/captcha/asset-leak) + BM25-lite relevance + query
term-coverage guard + geo disambiguation (region vs competing toponyms) +
per-registrable-domain dedupe.
client.rs: capture SearXNG'sengines[]/positions[](needed for RRF).params.rs:clean_querystrips a leading ranking stopword(top/best/good/greatest/finest/cheapest) when ≥3 tokens and not quoted — this
removes the bing "best→dictionary/bestbuy" bait at the source — plus pins
language=enwhen the request omits it (explicit per-request language honored).transform.rs: newtransform_flat_reranked;transform_flatkept for theplain (non-LLM) path for SaaS byte-parity.
search.rs: answer/summarize builds the pool via the reranked transform,gated by new
rerank_enabledconfig (default true).Results (frozen-fixture cargo test, reproducible)
Istanbul/Maine competing tokens.
junk elimination + geo/homonym correctness, not the coverage fraction (which is
bounded by snippet text in the fixtures).
cargo test -p crw-search44 pass (5 new rerank gates),cargo test -p crw-server89 pass, clippy
-D warnings+ fmt clean. CPU-only, no new network/deps.Pairs with the deployed SearXNG config change (down-weight bing, weight google,
language pin) and is independent of the proxy-reliability work.