Duplicate from different engines - #481
Merged
Merged
Conversation
castelao
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #481 +/- ##
=======================================
Coverage 61.72% 61.72%
=======================================
Files 77 77
Lines 6937 6937
Branches 690 690
=======================================
Hits 4282 4282
Misses 2535 2535
Partials 120 120
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request updates duplicate-handling for web search results so the same URL is treated as a duplicate even when returned by different search engines, enabling the ranking logic to leverage cross-engine agreement.
Changes:
- Update
_apply_duplicate_filtersto deduplicate by URL across all search engines (instead of(engine, url)). - Add a unit test asserting that identical URLs from different engines are collapsed and tracked via the
duplicatesfield.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| compass/web/search.py | Changes duplicate keying to URL-only so duplicates collapse across engines and can influence ranking via duplicates count. |
| tests/python/unit/web/test_web_search.py | Adds coverage ensuring same-URL results from different engines are marked as duplicates and recorded under the winner. |
castelao
marked this pull request as ready for review
June 27, 2026 00:35
The criteria used to be (engine, url), but we actually one the same URL only once, and we should take advantage on the agreement between multiple engines. In one extreme, if both engines give exact the same result, it won't affect the order, otherwise, it prioritize what distinct engines agree on.
Before, it was order by the name of the engine (alphabetic).
castelao
force-pushed
the
fix/duplicate_multiple_engines
branch
from
June 29, 2026 14:06
b2a6651 to
a70eafa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The criteria used to be (engine, url), but we actually want the same URL only once, and we should take advantage on the agreement between multiple engines.
In one extreme, if both engines give exact the same result, it won't affect the order, otherwise, it prioritize what distinct engines agree on.
This PR also fix a mistake in the search engine criteria. Originally it was ordering by the name of the engine, while now it is by the order it was defined in the config, i.e. first listed engine has priority.