Add OpenRouter embedding provider

[dlqqq]

Problem

Users need an OpenRouter embedding provider to use embedding models from services which implement the OpenAI API specification. These services include OpenRouter, but also include vLLM for invoking embedding models hosted locally.

Proposed Solution

Implement an OpenRouter embedding provider that supports a custom model ID & base URL.

Additional context

Follow-up to #1010.

[dlqqq]

@srdas @cboettig Let's continue any remaining discussion in this thread. 👍

[cboettig]

Thanks @dlqqq , sorry to cross threads! Still wrapping my head around openrouter vs openai protocol. From an end-user perspective it would be great to have drop-in support for custom models like we see in most LLM-wrapping interfaces. (also the design of jupyter-ai configuration seems to assume the same API key is shared across all models? (contrast to the configuration in continue extension where each model interface gets a name, model name, base url, and api key and then we can easily toggle between them).

[srdas]

@cboettig For trialing out purposes, which embedding models are you using with vLLM?

[cboettig]

@srdas thanks! I'm mostly using mistral-embed.

Reading these docs on the code-server side, I realize that embedding models are a lot lighter and run pretty decently even on CPU instances. Note that they have an option to just use transformers.js to do locally without calling out to an API. I think that would be really interesting to explore too...

[srdas]

@cboettig Thank you, that is helpful. I assume you are using mistral-embed (i) through vLLM (ii) on a remote server and not locally?

Note that Jupyter AI already supports mistral-embed:

[cboettig]

@srdas yes it does, but once again recall that I am using vLLM and LiteLLM to access the copy of Mistral we host via vllm. so that interface does not help me.

[dlqqq]

@cboettig Thanks for the valuable user feedback! I agree with your points. Jupyter AI needs better support for allowing users to save custom models w/ custom configuration. The user experience for that today is less than intuitive.

also the design of jupyter-ai configuration seems to assume the same API key is shared across all models

We don't allow multiple copies of each key name. So for example, if two models require OPENAI_API_KEY, it's not possible to have model 1 use a different key from model 2 (unless you manually edit the key every time you change). This is also an important feature that we should build later on. 👍

[dlqqq]

@cboettig Just saw your comment in the previous thread, let me reply also.

that is this issue -- but why? I just don't quite understand why jupyter-ai decides not to allow custom model ids (and custom urls) for the openai interface. Every other library or package I work with supports this (langchain, openai's own packages, continue.dev, litellm, vllm, ellmer etc). Why it is necessary for Jupyter-AI to hardware the openai interface to use only OpenAI's model IDs? Even the packages built by openai itself let us override the model names and base url. Everywhere else, 'openai' just means the API protocol.

This is a valid critique. We have poor support for custom models for a fairly simple reason: we didn't account for this use-case. Jupyter AI started way back >2 years ago, when there were far fewer models & model providers, and when custom models were very rare in the ecosystem. At that time, it made sense to hard-code a list of models for each provider to eliminate the need for users to look up the list of available model IDs.

However, times have changed, and Jupyter AI should change too. Users need a way to tell Jupyter AI to "save the provider, model ID, base URL, API key, and all other params for future use". We will likely keep our lists of model IDs, but we should also allow users to override them in case our copy lags behind what's available. The UI and API used in the Jupyter AI settings would have to be reworked for this. This will be a significant effort with multiple cycles of UX & code review. Rest assured that this is on our roadmap.

[cboettig]

Awesome, thanks so much! Yup, Jupyter-AI was totally ahead of the curve, and I know there are challenges in maintaining backwards compatibility and all to existing users. But yes, I think support for custom / open models is particularly important in this community.

Appreciate this dialog and all you do to support open source. No worries, I know this all takes time to adjust and we roll with it!