gte-tiny from sentence_transformer to replace cohere Embedding

[abdelkareemkobo]

What is gte-tine?
This is a sentence-transformers model: It maps sentences & paragraphs to a 384 dimensional dense vector space and can be used for tasks like clustering or semantic search. It is distilled from thenlper/gte-small, with comparable (slightly worse) performance at around half the size.

• I tested it with some examples while exploring txtai vector database vs other transformers and in most cases it's really nice and work well but some times it produces some errors without relevant meaning
• to solve this problem the author advice to fine-tune it -> still in my todo
• to replace it in the code it's just only 3 lines we need to change and it works without problems

[srbhr]

Hey @abdelkareemkobo, if we're going to fine-tune it, it will add more complexities for the general user. Because of the embeddings, this is becoming more of an AI project than a general tool.

[abdelkareemkobo]

@srbhr oh, fine