BM25 Ranking Function

[arkim822]

Hello again Milvus community.

I am currently trying to implement BM25 just like the functionality built into Weaviate (but don't need the ratios they specify).
https://milvus.io/docs/embed-with-bm25.md

I think I am almost there, but would like to make sure I am doing it currently.

The steps I am taking are as follows:

1. Perform a vector search.

2. Use the text results as the corpus of data to fit on

   bm25_ef.fit(corpus)
   

3. Generate document embeddings using the text results

   docs_embeddings = bm25_ef.encode_documents(docs)

4. Encode the query

   query_embeddings = bm25_ef.encode_queries(queries)

5. ?

I am currently working out what to do afterwards. Is my approach correct?

[xiaofan-luan]

1. you will need to first fit on your documents, this is nothing to do with vector search
2. embedded your corpus with both bm25 and dense embedding models
3. search with bm25 and dense embedding on a hybrid mode
4. rerank with other models or through RRF

[arkim822]

Hello, I've been experimenting more with hybrid search and other reranking models.

I just wanted to go back to your post and ask if there were ways to save or store the embeddings once fit on the corpus of data for bm25?

[xiaofan-luan]

you can skip the first step and use pretrained bm25 model as well

[arkim822]

Much appreciated. I'll give both a try. Thanks!

[louis-sanna-eki]

Hello @arkim822, I don't understand how make the BM25 search.

Do you do it in Milvus? If yes how to you insert the sparse vector and how to do query?

Every thing I tried failed.

[xiaofan-luan]

Milvus now support sparse emebdding.
You will need use models like BGE M3 or Splade or pretrained BM25

[louis-sanna-eki]

Hello @xiaofan-luan, thanks a lot for the answer.

When I try to insert the embedding produced by encode_documents I get an error:

    df_docs = pd.read_parquet(path)
    for index, row in df_docs.iterrows():
        uniq_id += 1
        
        docs_embeddings = bm25_ef.encode_documents(row["content"])
        
        data.append({
            "id": uniq_id,
            "content": row["content"],
            "vector": row["semantic_vector"],
            "bm25_vector": docs_embeddings[0], # BM25 embedding I would like to insert
            "page_number": row["page_number"],
            "file_name": row["file_name"],
            "type": row["type"],
            "numero": row["numero"],
            "package_name": row["package_name"],
        })

NotImplementedError: We have not yet implemented 1D sparse slices; please index using explicit indices, e.g. x[:, [0]]

What am I missing?

[xiaofan-luan]

Check the structure of docs_embeddings.
seems that this is not a milvus side error

[louis-sanna-eki]

Ah now I understand.

To select the first element:

docs_embeddings = bm25_ef.encode_documents(row["content"])
bm25_vector = list(docs_embeddings)[0]