Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The Searcher component. #14

Merged
merged 29 commits into from
Jul 28, 2024
Merged

The Searcher component. #14

merged 29 commits into from
Jul 28, 2024

Commits on Jul 25, 2024

  1. Setting up initial files and types for the searcher.

    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    a5c32c2 View commit details
    Browse the repository at this point in the history

  2. Creating an example file for the searching component.

    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    8319e15 View commit details
    Browse the repository at this point in the history

  3. Changing the default search settings; not doing PLAID optimizations 

    right now, just a vanilla ColBERTv2.0.
    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    cd7b98d View commit details
    Browse the repository at this point in the history

  4. Adding a constructor for the searcher; we load and save the HF model to 

    prevent multiple loads. Also updating the example.
    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    e2e3556 View commit details
    Browse the repository at this point in the history

  5. Adding a skeleton for the search method for a Searcher; will 

    populate this further.
    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    55ac294 View commit details
    Browse the repository at this point in the history

  6. Fixing the constructor of Searcher to include the QueryTokenizer.

    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    cdc9e10 View commit details
    Browse the repository at this point in the history

  7. Implementing the encode_query function to encode a query.

    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    27ac472 View commit details
    Browse the repository at this point in the history

  8. Adding docstring for encode_query.

    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    1c84c5a View commit details
    Browse the repository at this point in the history

  9. Adding the StridedTensor type and it's constructor for efficient ivf 

    operations.
    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    fc40fb9 View commit details
    Browse the repository at this point in the history

  10. Adding docstring for _select_strides.

    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    12f25ef View commit details
    Browse the repository at this point in the history

  11. Adding a docstring for _create_view.

    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    818b3dc View commit details
    Browse the repository at this point in the history

  12. Adding a method of load_codec to load a codec from an indexing path.

    @codetalker7
    codetalker7 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    8445e30 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. Adding function to load residuals from a particular chunk idx.

    @codetalker7
    codetalker7 committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    d8c03e6 View commit details
    Browse the repository at this point in the history

  2. Adding more fields to the IndexScorer, and also adding a new 

    constructor for it.
    @codetalker7
    codetalker7 committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    5d2cac8 View commit details
    Browse the repository at this point in the history

  3. Loading the config inside the constructor of Searcher instead of 

    passing it the config.
    @codetalker7
    codetalker7 committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    e2ad470 View commit details
    Browse the repository at this point in the history

  4. Not using strided tensors for now, and building an emb2pid mapping.

    @codetalker7
    codetalker7 committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    dac07c9 View commit details
    Browse the repository at this point in the history

  5. Some informative messages.

    @codetalker7
    codetalker7 committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    dbe5fa8 View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2024

  1. Not striding the embeddings in the constructor of IndexScorer.

    @codetalker7
    codetalker7 committed Jul 27, 2024
    Configuration menu
    Copy the full SHA
    a8be3a9 View commit details
    Browse the repository at this point in the history

  2. Adding a retrieve function to get all the candidate pids for a given 

    query matrix.
    @codetalker7
    codetalker7 committed Jul 27, 2024
    Configuration menu
    Copy the full SHA
    1abf201 View commit details
    Browse the repository at this point in the history

  3. For now changing the type to AbstractFloat; will focus on precision 

    later.
    @codetalker7
    codetalker7 committed Jul 27, 2024
    Configuration menu
    Copy the full SHA
    70b6f12 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2024

  1. Adding function to decompress residual embeddings.

    @codetalker7
    codetalker7 committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    7d6e1a4 View commit details
    Browse the repository at this point in the history

  2. Adding a function to decompress codes and residuals.

    @codetalker7
    codetalker7 committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    5808d5c View commit details
    Browse the repository at this point in the history

  3. Some helper comments.

    @codetalker7
    codetalker7 committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    d96623a View commit details
    Browse the repository at this point in the history

  4. Adding function score_pids to score the pids.

    @codetalker7
    codetalker7 committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    b01c73f View commit details
    Browse the repository at this point in the history

  5. Completing the implementation of rank.

    @codetalker7
    codetalker7 committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    8281704 View commit details
    Browse the repository at this point in the history

  6. Exporing the search function.

    @codetalker7
    codetalker7 committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    d770438 View commit details
    Browse the repository at this point in the history

  7. Completing the implementation of dense_search.

    @codetalker7
    codetalker7 committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    aeb4003 View commit details
    Browse the repository at this point in the history

  8. Updating the search example.

    @codetalker7
    codetalker7 committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    9a73a82 View commit details
    Browse the repository at this point in the history

  9. Saving the config in the indexing example.

    @codetalker7
    codetalker7 committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    7b5bd41 View commit details
    Browse the repository at this point in the history