New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding dense retrieval support #25

Open

jacklin64 opened this issue May 10, 2021 · 1 comment

Labels

enhancement help wanted

Member

jacklin64 commented May 10, 2021

A starting point could be using pyserini' dense model for Msmarco first.

jacklin64 added enhancement help wanted labels

Member

justram commented May 10, 2021 •

edited

Loading

Specifically, we have a bm25 searcher now:

chatty-goose/chatty_goose/pipeline/retrieval_pipeline.py

Line 29 in 2047d4e

searcher: SimpleSearcher,

Need to support Pyserini's SimpleDenseSearcher:
https://github.com/castorini/pyserini#dense-retrieval

To Do:

Encode CAsT passage corpus with Faiss (brute-force/hnsw). Ref: https://github.com/castorini/pyserini/blob/master/scripts/msmarco-passage/encode_corpus.py
Validate dense retrieval pipeline. Dense model candidate: TCT-ColBERT, ANCE, DPR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment