Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snorkle.local and Ollama #3

Open
logichub opened this issue Sep 5, 2024 · 2 comments
Open

Snorkle.local and Ollama #3

logichub opened this issue Sep 5, 2024 · 2 comments

Comments

@logichub
Copy link
Contributor

logichub commented Sep 5, 2024

Flexible Backend: While text-gen-webui is the default, Patense.local can work with any backend LLM server.

Is it possible to use Snorkle.local with Ollama? Can you provide some documentation or guide on this? Thank you.

@JohnZolton
Copy link
Owner

in src/server/api/routers/jobs.ts

change const webUiEndpoint = "http://127.0.0.1:5000/v1/chat/completions";

to "http://localhost:11434/api/generate"

and i think you need to include the specific model

change:

const response = await fetch(webUiEndpoint, {
          method: "POST",
          headers: {
            "Content-Type": "application/json",
          },
          body: JSON.stringify({
            messages: [
              {
                role: "user",
                content: message,
              },
            ],
          }),
        });

to

const response = await fetch(webUiEndpoint, {
          method: "POST",
          headers: {
            "Content-Type": "application/json",
          },
          body: JSON.stringify({
          "model": "llama3",
            messages: [
              {
                role: "user",
                content: message,
              },
            ],
          }),
        });

@logichub
Copy link
Contributor Author

logichub commented Sep 6, 2024

Thanks for the above code. It partially worked.

I can successfully upload documents, the search feature does not work as expected. When attempting to search for content, I received the following error message in the terminal:

❌ tRPC failed on job.deepSearch: fetch failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants