Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The Searcher component. #14

Merged
merged 29 commits into from
Jul 28, 2024
Merged

The Searcher component. #14

merged 29 commits into from
Jul 28, 2024

Conversation

codetalker7
Copy link
Member

PR to implement the Searcher component, i.e the retrieval component. Plan is to implement a search method for this type, which will take in a query (or a batch of queries) and get the top k documents along with their scores (k being an argument to search).

@codetalker7 codetalker7 force-pushed the scorer branch 3 times, most recently from 09171e2 to 977a5c5 Compare July 25, 2024 11:19
@codetalker7
Setting up initial files and types for the searcher.
a5c32c2
@codetalker7
Creating an example file for the searching component.
8319e15
@codetalker7
Changing the default search settings; not doing PLAID optimizations
cd7b98d 
right now, just a vanilla ColBERTv2.0.
@codetalker7
Adding a constructor for the searcher; we load and save the HF model to
e2e3556 
prevent multiple loads. Also updating the example.
@codetalker7
Adding a skeleton for the search method for a Searcher; will
55ac294 
populate this further.
@codetalker7
Fixing the constructor of Searcher to include the QueryTokenizer.
cdc9e10
@codetalker7
Implementing the encode_query function to encode a query.
27ac472
@codetalker7
Adding docstring for encode_query.
1c84c5a
@codetalker7
Adding the StridedTensor type and it's constructor for efficient ivf
fc40fb9 
operations.
@codetalker7
Adding docstring for _select_strides.
12f25ef
@codetalker7
Adding a docstring for _create_view.
818b3dc
@codetalker7
Adding a method of load_codec to load a codec from an indexing path.
8445e30
@codetalker7
Adding function to load residuals from a particular chunk idx.
d8c03e6
@codetalker7
Adding more fields to the IndexScorer, and also adding a new
5d2cac8 
constructor for it.
@codetalker7
Loading the config inside the constructor of Searcher instead of
e2ad470 
passing it the config.
@codetalker7
Not using strided tensors for now, and building an emb2pid mapping.
dac07c9
@codetalker7
Some informative messages.
dbe5fa8
@codetalker7
Not striding the embeddings in the constructor of IndexScorer.
a8be3a9
@codetalker7
Adding a retrieve function to get all the candidate pids for a given
1abf201 
query matrix.
@codetalker7
For now changing the type to AbstractFloat; will focus on precision
70b6f12 
later.
@codetalker7
Adding function to decompress residual embeddings.
7d6e1a4
@codetalker7
Adding a function to decompress codes and residuals.
5808d5c
@codetalker7
Some helper comments.
d96623a
@codetalker7
Adding function score_pids to score the pids.
b01c73f
@codetalker7
Completing the implementation of rank.
8281704
@codetalker7
Exporing the search function.
d770438
@codetalker7 codetalker7 marked this pull request as ready for review July 28, 2024 15:45
@codetalker7 codetalker7 merged commit 45b943b into main Jul 28, 2024
0 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@codetalker7