Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Embedding update #2791

Open
wants to merge 26 commits into
base: main
Choose a base branch
from
Open

Conversation

pengjunfeng11
Copy link
Contributor

Change the overall architecture of the Embedding model and support Fast Embed. There may be missed code, you need to verify the completion.
need check :

  1. The function test_embedding_model_with_fast_embed in test_embedding_models.py(in other device, like cuda、mps....)
  2. In all create_embedding_model_instance methods, the transfer, setting, and use of the devices parameters

pengjunfeng11 and others added 24 commits November 10, 2024 18:46
…o main

FEAT: add test_convert_ids_to_tokens for convert_ids_to_tokens
…nvert_ids_to_tokens. Added _encode_bgem3 method for bge-m3 model encoding
…nvert_ids_to_tokens. Added _encode_bgem3 method for bge-m3 model encoding
set default all_families for reduce redundancy
Format optimization
remove white sapce
remove newline
@XprobeBot XprobeBot added the gpu label Jan 28, 2025
@XprobeBot XprobeBot added this to the v1.x milestone Jan 28, 2025
Junfeng Peng added 2 commits January 28, 2025 21:40
@pengjunfeng11
Copy link
Contributor Author

更新了很多内容。建议严格检查一下:

  1. 测试用例有没有缺陷
  2. 模型的调用方式能不能够支持现在的架构
  3. 在不同设备(cuda、cpu)上的运行(本地机器环境有限)
  4. 调用之后的返回格式
  5. 传参是否有异常

@pengjunfeng11
Copy link
Contributor Author

CI progress Error:
ModuleNotFoundError: No module named 'fastembed', seems like need run pip install fastembed

No need to install other packages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants