diff --git a/docs/docs/inference/llamacpp.mdx b/docs/docs/inference/llamacpp.mdx index 70d6d20..9f222f5 100644 --- a/docs/docs/inference/llamacpp.mdx +++ b/docs/docs/inference/llamacpp.mdx @@ -52,6 +52,14 @@ For example: wget https://huggingface.co/rubra-ai/Meta-Llama-3-8B-Instruct-GGUF/resolve/main/rubra-meta-llama-3-8b-instruct.Q8_0.gguf ``` +:::info +For large multi-part model files, such as [rubra-meta-llama-3-70b-instruct_Q6_K-0000*-of-00003.gguf](https://huggingface.co/rubra-ai/Meta-Llama-3-70B-Instruct-GGUF/tree/main), use the following command to merge them before proceeding to the next step: +``` +./llama-gguf-split --merge rubra-meta-llama-3-70b-instruct_Q6_K-0000*-of-00003.gguf rubra-meta-llama-3-70b-instruct_Q6_K.gguf +``` +This will merge multi-part model files to one gguf file `rubra-meta-llama-3-70b-instruct_Q6_K.gguf`. +::: + ### 5. Start the OpenAI Compatible Server ```bash