API request delay for Generic OpenAI embedding engine #4317
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chaserhkj
force-pushed
the
generic-openai-embedding-delay
branch
from
47ad24d to
d19048c
Compare
|
The previous change here would execute all the batches at once, and they would all just wait at the same time instead of doing each batch sequentially with a wait in between. Refactored code solves this and was tested. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Type
Relevant Issues
resolves #3570
What is in this change?
This PR adds a new configurable environment variable to add a small delay for dispatching each API request calls when using the generic OpenAI embedding engine.
This is meant to be used together with
GENERIC_OPEN_AI_EMBEDDING_MAX_CONCURRENT_CHUNKS=1and is mostly relevant to the use of local LLM backends likellama.cpp, where their API server implementation is naive and will just reject API requests with a 429 error if the underlying worker is busy.Developer Validations
yarn lintfrom the root of the repo & committed changesI am skipping documentation for now since there is no change to front end UI, but I could add it to this PR if requested.