Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding the QueryTokenizer. #15

Merged
merged 3 commits into from
Jul 23, 2024
Merged

Adding the QueryTokenizer. #15

merged 3 commits into from
Jul 23, 2024

Commits on Jul 14, 2024

  1. Adding the skeleton of QueryTokenizer.

    @codetalker7
    codetalker7 committed Jul 14, 2024
    Configuration menu
    Copy the full SHA
    f76bcf0 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. Constructor for QueryTokenizer to get the Q_marker_token_id.

    @codetalker7
    codetalker7 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    763636c View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. Adding the tensorize function for the QueryTokenizer along with 

    docstrings.
    @codetalker7
    codetalker7 committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    d06ea5b View commit details
    Browse the repository at this point in the history