Clarification on GPU Setup? #975
Unanswered
STORMFIRE007
asked this question in
Q&A
Replies: 2 comments 1 reply
-
|
after digging around some more, i think i found the problem, after running the python in venv, then import torch and checking it's version, it seems somehow the cpu only version of pytorch is installed, maybe khoj only installed the cpu version of pytorch?: imma try uninstalling and reinstalling the gpu version of pytorch to see perhaps that fixes the issue edit: FAILEDeven after reinstalling the GPU version of pytorch and ensuring that it's working with cuda: khoj is still loading the model in the RAM and running on CPU maybe perhaps this initial error in the initial configuration is causing problems?: although the above error doesn't always happen, i reinstalled khoj and the error didn't occured that time: khoj still runs whether that error occurs or not . edit 2: SOME ACTIVITY?i decided to take a closer look again today and noticed something, when running khoj, although it neither loads the model into VRAM, nor runs it on the GPU...
. just to confirm it's not just a fluke or something, i uninstalled torch and let khoj install pytorch and it installed the CPU only version of it (2.2.2+cpu), and that did not load anything on the GPU, whereas manually installing CUDA compatible pytorch version (2.2.2+cu121) resulted in khoj loading something in the GPU memory (of size 361 MiB) |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hey @STORMFIRE007 , a few follow-up questions. Thanks for the detailed investigation!
The teeny spike in GPU after installing the correct |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
np, you are welcome,
added the upon initiating the chat with LLM, khoj still loads the model into the RAM and processes it on CPU, albeit using higher amount of RAM (almost reaching my laptop's limit, 13.9/15.8 GB) also when processing my query, .
i would also like to mention that you might also need to take into account the CUDA version installed into the user's system when automatically installing cuda pytorch, as there are different pytorch versions for different versions of CUDA (2.2.2+cu118 for CUDA 11.8, 2.2.2+cu121 for CUDA 12.1, 2.2.2+cu124 for CUDA 12.4) (the binaries for CUDA 12.4 aren't available for pytorch 2.2.2 yet, so i am using CUDA 12.1 and pytorch version (2.2.2+cu121)) The command i used to install pytorch 2.2.2 compatible with CUDA 12.1: I found that command here: |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
hi everyone,
I’m having trouble getting the LLM model to run on my GPU (RTX 2060) despite following the setup guide for CUDA support. I set the environment variable as instructed:
However, the model still seems to run on the CPU, loading into RAM instead of VRAM. It also defaults to downloading GGUF models, which I believe can only run on CPUs?
I’m wondering if I’m missing something in the configuration, or if there’s a need to manually specify different model types (like GPTQ or AWQ from Hugging Face) to enable GPU usage? Is there a specific step I should follow to ensure the model runs on my RTX 2060 with CUDA support?
cuda seems to be installed and gpu is detected as shown by nvcc and nvidia-smi output:
Beta Was this translation helpful? Give feedback.
All reactions