[Feature] support for OpenAI-like mock servers & OpenAI proxy servers

[tranhoangnguyen03]

Currently, when I want to use OpenAI-like mock servers or proxy servers, there's no apparent way to manually modify the openai.api_base and add headers to openai Completion/ChatCompletion request.

The mock server requires changing openai.api_base and specifying the model name.
The proxy server requires changing openai.api_base, providing openai.api_key, specifying the model name, and adding a custom headers to the request.

[zxcvxzcv-johndoe]

Thanks tranhoangnguyen03 for this request, I am just right now trying to figure out how to do this too! :)

Edit:
As a temporary workaround you can edit "chat-llamaindex\node_modules.pnpm\node_modules\openai\index.js" line 58 and change that URL to point "http://localhost:5001/v1" if you are using koboldcpp-rocm or koboldcpp.

It's nothing perfect for sure and the token generation limits are way too high for me but I hope that helps someone.

[ilmarivikstrom]

+1

Also looking at how to connect to e.g. Azure OpenAI endpoints. Thinking there needs to be somewhat significant code changes to make it support those endpoints.

If somebody has figured this out, let it be known in this issue!

[olafgeibig]

I second this. For most developers in the corporate world Azure is the only compliant way to access OpenAI models. Or using an open source model deployed on their cloud infrastructure. In either case we simply need all OpenAI API connection options to be configurable - that's all. Best just evaluate the same environment variables as the OpenAI Python module is doing.

[marcusschiesser]

LlamaIndexTS should use Azure if the following environment variables are set:

• AZURE_OPENAI_ENDPOINT
• AZURE_OPENAI_API_INSTANCE_NAME
• OPENAI_API_TYPE (set to azure)

(see: https://github.com/run-llama/LlamaIndexTS/blob/dfd22aac464fed862c39c45c01717a15ced6c3ad/packages/core/src/llm/azure.ts#L90-L96)

Can you set these variables in env.development.local and try again?

[naveengct]

But I am not able to access the embedding model, couldn't find the respective variable in .env as well

    code: 'OperationNotSupported',
    message: 'The embeddings operation does not work with the specified model, gpt-4-32k. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.'

Any idea how can I use this ?

[54188wxp]

对于这种观点我很认同，Azure 是访问 OpenAI 模型的唯一合规方式。或者使用部署在其云基础架构上的开源模型，最好只评估与 OpenAI Python 模块相同的环境变量

[frazur]

But I am not able to access the embedding model, couldn't find the respective variable in .env as well

    code: 'OperationNotSupported',
    message: 'The embeddings operation does not work with the specified model, gpt-4-32k. Please choose different model and try again. You can learn more about which models can be used with each operation here: https://go.microsoft.com/fwlink/?linkid=2197993.'

Any idea how can I use this ?

is there any news?

[marcusschiesser]

@frazur that might be an Issue in LlamaIndexTS - can you use https://ts.llamaindex.ai/modules/llms/available_llms/azure with your azure account?