Add llama-cpp-python server

[ericcurtin]

Changed default runtime from 'llama.cpp' to 'llama-cpp-python'. Added 'llama-cpp-python' as a runtime option for better flexibility with the --runtime flag.

Summary by Sourcery

Add 'llama-cpp-python' as a new runtime option and set it as the default runtime, enhancing flexibility in model serving.

New Features:

• Introduce 'llama-cpp-python' as a new runtime option for serving models, providing more flexibility with the '--runtime' flag.

Enhancements:

• Change the default runtime from 'llama.cpp' to 'llama-cpp-python' for improved flexibility.

[sourcery-ai]

Reviewer's Guide by Sourcery

This PR changes the default runtime from 'llama.cpp' to 'llama-cpp-python' and adds support for the 'llama-cpp-python' server implementation. The changes involve modifying the server execution logic and updating the CLI configuration to accommodate the new runtime option.

Sequence diagram for server execution logic

sequenceDiagram
    participant User
    participant CLI
    participant Model
    User->>CLI: Run command with --runtime flag
    CLI->>Model: Pass runtime argument
    alt Runtime is vllm
        Model->>CLI: Execute vllm server
    else Runtime is llama.cpp
        Model->>CLI: Execute llama-server
    else Runtime is llama-cpp-python
        Model->>CLI: Execute llama_cpp.server
    end

Loading

Updated class diagram for runtime configuration

classDiagram
    class CLI {
        -runtime: String
        +configure_arguments(parser)
    }
    class Model {
        +serve(args)
    }
    CLI --> Model: uses
    note for CLI "Updated default runtime to 'llama-cpp-python' and added it as a choice"

Loading

File-Level Changes

Change	Details	Files
Added llama-cpp-python as a new runtime option and made it the default	Added 'llama-cpp-python' as a new choice in runtime options Changed default runtime from 'llama.cpp' to 'llama-cpp-python' Updated help text to include the new runtime option	ramalama/cli.py
Implemented server execution logic for llama-cpp-python runtime	Restructured server execution logic to handle three different runtimes Added specific command construction for llama-cpp-python server Maintained existing logic for vllm and llama.cpp runtimes	ramalama/model.py

Possibly linked issues

• Create Nvidia Branch for testing and development #239: The PR adds 'llama-cpp-python' runtime, contributing to multi-model server support.
• Configure Renovate #1: PR implements the switch to llama-cpp-python as discussed in the issue.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @ericcurtin - I've reviewed your changes - here's some feedback:

Overall Comments:

• Please provide justification for changing the default runtime from llama.cpp to llama-cpp-python. What are the benefits that led to this decision?
• Consider updating any additional documentation to reflect the new runtime option and explain the differences between the available runtimes

Here's what I looked at during the review

• 🟢 General issues: all looks good
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

We have skipped reviewing this pull request. It looks like we've already reviewed the commit 47fdf9a in this pull request.

[ericcurtin]

@cooktheryan @lsm5 @mrunalp @slp @rhatdan @tarilabs @umohnani8 @ygalblum PTAL

[ygalblum]

LGTM for the code.
But, I think I'm missing something. Is llama-cpp-python already a dependency?

[ericcurtin]

@ygalblum we probably need to push some container images before merging this, but when we do that, we should be all good.

[rhatdan]

Looks like we have a different runtime for ramalama serve versus ramalama run?

[ericcurtin]

For ramalama run, the only commands that are available to do that in a single process are the llama.cpp ones, like llama-cli (and llama-run which I'll be adding in this PR now that the last few bits were merged a couple of hours ago). There are other possibilities for chatbots, but the other require us to set up a server process and a client process. I think llama-run is cleaner than setting up a client and server.

As regards server there are 3 clear options, vLLM, llama.cpp and llama-cpp-python (which is just a python wrapper around llama.cpp). Although vLLM is the best enterprise option today for a server in my opinion, it lacks the diverse hardware support for local development, which leaves us the other two. llama.cpp server is a very minimal implementation of an OpenAI-compatible server as in it is barely OpenAI-compatible, last I tried it, it didn't work with OpenAI-compatible clients like: https://github.com/open-webui/open-webui

llama.cpp server might catch up as it is actively being contributed to, but it's not there today. One key feature it is missing is multi-model support, so the client normally specifies the model it wants to use.

llama-cpp-python is further ahead in terms of it's OpenAI-compatible server implementation, that's why I'm proposing we switch in this PR.

[rhatdan]

My opinion here is choice is bad. We should only offer llama.cpp and if the server tool llama-cpp-python is better, then default to using that, if a better one shows up in the future, then we could move to it, as long as it implements the Same API as llama-cpp-python. We are doing the same thing on the run side moving from llama-client to llama-run.

Bottom line is keep the selection to llama.cpp and vLLM (And whisper.cpp) and we can modify the tools used to work with the libraries under the covers. Eventually we could allow specifying the commands to run within the containers for run and serve, but that is a ways a way.

[ericcurtin]

Adding temperature option to llama-run here:

ggerganov/llama.cpp#10899

[rhatdan]

As stated above, do not give this choice, just stick with llama.cpp and use llama-cpp-python server for serve