[Bug] dataprep/retriever uses different environment variables for different DB backend

[lianhao]

Priority

P3

OS type

Ubuntu

Hardware type

Xeon-GNR

Installation method

• Pull docker images from hub.docker.com
• Build docker images from source
• Other

Deploy method

• Docker
• Docker Compose
• Kubernetes Helm Charts
• Other

Running nodes

Single Node

What's the version?

1.2

Description

Currently, for the following common settings, the dataprep/retriever are using different environment variable names for different backend, we should unify them to use the same environment variable to minimize the end user's configuration confusion:

• endpoint to the backend TEI service: some are using TEI_ENDPOINT, some are using TEI_EMBEDDING_ENDPOINT. Suggest to use TEI_EMBEDDING_ENDPOINT
• local embedding model: some are EMBED_MODEL, some are using LOCAL_EMBEDDING_MODEL. Suggest to use EMBED_MODEL

While doing the change, we should also keep backward compatibility to make sure old version of config can work with new service, while the new environment names should take precedence: i.e.

TEI_EMBEDDING_ENDPOINT = os.getenv("TEI_EMBEDDING_ENDPOINT") or os.getenv("TEI_ENDPOINT")

Reproduce steps

n/a

Raw log

Attachments

No response

[ashahba]

Thanks @lianhao for raising this technical debt. We'll look into reducing this consistency during upcoming releases.
If however this is casing an issue, let me know and we'll work on the fix after that.

[yinghu5]

Hi @lianhao and @ashahba thank you for address the problem. Just remind, please add PR if any progress

[yinghu5]

Hi @lkk12014402
Please help to see if we can fix the issue. thank you! [Remind]