Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support filter and nested field in faiss engine radial search #1652

Merged
merged 3 commits into from
Apr 26, 2024

Conversation

junqiu-lei
Copy link
Member

@junqiu-lei junqiu-lei commented Apr 24, 2024

Description

This PR will allow user to use filter and nested field in faiss engine radial search by reusing existing logics from top-k search.

  • The custom patch for range_search is reused from faiss search patch. Please check this commit for the patch updated code in faiss.
  • In jni faiss_wrapper.cpp, the params set up process is reused from existing search function.
  • IT tests added for both nested field and filter queries.

Issues Resolved

Part of #814

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@junqiu-lei junqiu-lei added Features Introduces a new unit of functionality that satisfies a requirement backport 2.x v2.14.0 labels Apr 24, 2024
@junqiu-lei junqiu-lei self-assigned this Apr 24, 2024
@junqiu-lei junqiu-lei force-pushed the radial-faiss-2 branch 2 times, most recently from ae39f5a to 28076de Compare April 24, 2024 15:42
@jmazanec15
Copy link
Member

@junqiu-lei For this patch, did we create a PR in faiss to upstream it?

@junqiu-lei
Copy link
Member Author

@junqiu-lei For this patch, did we create a PR in faiss to upstream it?

@jmazanec15 No, I mainly just followed the existing patch for search function https://github.com/opensearch-project/k-NN/blame/main/jni/patches/faiss/0001-Custom-patch-to-support-multi-vector.patch#L136-L158

@jmazanec15
Copy link
Member

@junqiu-lei one minor comment but other than that it looks good

Copy link
Member Author

@junqiu-lei junqiu-lei Apr 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check this commit for the patch updated code. CC: @navneet1v

Copy link
Member

@martin-gaievski martin-gaievski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks

@junqiu-lei junqiu-lei merged commit 0cb4f36 into opensearch-project:main Apr 26, 2024
51 of 52 checks passed
@junqiu-lei junqiu-lei deleted the radial-faiss-2 branch April 26, 2024 17:17
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-1652-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0cb4f36b0d3264f984258a5bff50c7f12448dacb
# Push it to GitHub
git push --set-upstream origin backport/backport-1652-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1652-to-2.x.

junqiu-lei added a commit to junqiu-lei/k-NN that referenced this pull request Apr 26, 2024
…arch-project#1652)

* Support filter and nested field in faiss engine radial search

Signed-off-by: Junqiu Lei <[email protected]>

(cherry picked from commit 0cb4f36)
Signed-off-by: Junqiu Lei <[email protected]>
junqiu-lei added a commit to junqiu-lei/k-NN that referenced this pull request Apr 26, 2024
…search-project#1652)

* Use 0.95 as default ratio for lucene radial search traversal similarity (opensearch-project#1619)

Signed-off-by: Junqiu Lei <[email protected]>
junqiu-lei added a commit that referenced this pull request Apr 29, 2024
… (#1658)

* Use 0.95 as default ratio for lucene radial search traversal similarity (#1619)

Signed-off-by: Junqiu Lei <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Features Introduces a new unit of functionality that satisfies a requirement v2.14.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants