Skip to content

Conversation

@sfallah
Copy link
Contributor

@sfallah sfallah commented Nov 20, 2025

Feature Request: #16676

Make sure to read the contributing guidelines before submitting a PR

@github-actions github-actions bot added model Model specific examples python python script changes labels Nov 20, 2025
@sfallah sfallah marked this pull request as draft November 20, 2025 09:12
@sfallah
Copy link
Contributor Author

sfallah commented Nov 21, 2025

@Acly
I thought it might interest you!
I have implement add_rel_pos_inplace and get_rel_pos (for DeepseekOCR SAM-Vit) without using the cpu only ggml ops (ggml_get_rel_pos etc.) and it works on Metal.
I will also test on CUDA tomorrow, and hopefully the CI checks will later show if the implementation also works for other devices.

You will find the code here:
https://github.com/sfallah/llama.cpp/blob/sf/deepseek-ocr/tools/mtmd/clip.cpp#L2473
https://github.com/sfallah/llama.cpp/blob/sf/deepseek-ocr/tools/mtmd/clip.cpp#L2502

Disclaimer:
My implementation is very rudimentary but @bluebread reviewed and already improved it a bit in separate branch.
I have been using ChatGPT to translate from original pytorch code but have debugged the code and got it to run.
The functions have also been tested with and compared with ggml/examples/sam (sam example from ggml repo), the results align well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples model Model specific python python script changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants