Roberta embeddings fixes

[Ssukriti]

Since the last PR, we identified that the embeddings values being produced were not of good quality. On further debugging, we found that the tokenizer should be set to gpt-2 and position embeddings needed some modifications.

This PR addresses changes to get the correct embeddings from Roberta model.

This PR has been tested by comparing embeddings values against sentenceTransformers library for Roberta architecture

from sentence_transformers import SentenceTransformer
embeddings_model = SentenceTransformer(path)
embedding_vector = embeddings_model.encode([prompt])[0]

now matches

python3 convert_hf_to_gguf.py model_path --outfile model_path.gguf
llama-embedding -m model_path.gguf -p [prompt] -c 514

Hence, the embeddings are now the correct values for Roberta models.

We will add documentation examples in a follow up PR

[Ssukriti]

as per guidelines, we shouldnt be modifying this value in convert_to_gguf , so that it can be autogenerated from convert_hf_to_gguf_update.py

we want it to map to gpt-2 tokenization type.

Would the correct way to do this be keep it to roberta-bpe as generated and then add it to mapping https://github.com/ggerganov/llama.cpp/blob/master/src/llama.cpp#L6483 here , so that it maps to gpt-2?

Any input would be appreciated

[ggerganov]

Would the correct way to do this be keep it to roberta-bpe as generated and then add it to mapping master/src/llama.cpp#L6483 here , so that it maps to gpt-2?

Yes, this is the correct way.

[Ssukriti]

@ggerganov Thank you! I have pushed the change and tested it.

I dont think the failing test in this PR is because of my changes? I see other PRs failing in same place

[ggerganov]

Thanks for looking into this.

[ggerganov]

Would the correct way to do this be keep it to roberta-bpe as generated and then add it to mapping master/src/llama.cpp#L6483 here , so that it maps to gpt-2?

Yes, this is the correct way.