Enable real-time agent tool call streaming for all providers #4279
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- OpenAI - Anthropic - Azure OpenAI
…rovider class and not assume OpenAI client struct example: Ollama
normalize completion/stream outputs across providers/untooled
|
Adds support for the following models to support real-time tool call streaming of outputs:
|
update Azure for streaming support update Gemini to streamting support on gemini-* models generic OpenAI disable streaming verify localAI support verify NVIDIA Nim support
timothycarambat
added
the
PR:Ready-to-merge
…t or called resulting in the provider `handleFunctionCallChat` being called - which returns a string. This would then fail in Untooled.complete since response would be a string and not the expected `response.choices?.[0]?.message` Modified this line to handle both conditions for stream/non-streaming and tool presence or lack thereof
…work fine honor disabled streaming for provider where that concern may apply for regular chats
.complete on azure should be non-streaming as this is the sync response
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Type
Relevant Issues
resolves #xxx
What is in this change?
Adds support for the following models to support real-time tool call streaming of outputs:
Todo:
Additional Information
For
Provideronly inherited classes:streamfunction on Provider will apply.completeendpoints however still lie on their respective classesFor
Provider + Untooledinherited classes:streamandcompletecalls are delegated to the UntooledstreamandcompleteollamaSDK.Developer Validations & Tests
Always to be sure to test these conditions
@agentwith a for sure tool call@agentthat is just a regular messageyarn lintfrom the root of the repo & committed changes