Streaming endpoint with C# client

[CyrilMercier00]

I am trying to consume an endpoint created in Azure ML Studio with a C# client, but the output is sent back as a stream only once the answer is fully generated. With a python client consuming the same endpoint, the answer is read as it is being generated.

Is C# Streaming not currently supported ?

The docker hosting the endpoint is deployed with Prompt Flow 1.7.0

[brynn-code]

Hi @CyrilMercier00 , thanks for reaching us. Does the C# client is the client you are using to consume the endpoint? We assume the streaming capacity should not related to the client.
In the serving (aka endpoint) scenario, promptflow will check if text/event-stream in the request.accept_mimetypes.values() to determine the response is streaming response or not, maybe you could check the request header of the two client you are using to see if there is any different.

[CyrilMercier00]

I think I am setting the header correctly to accept an 'event-stream'in my HttpClient, here is how I generate it :

return new HttpClient(handler)
{
    Timeout = TimeSpan.FromSeconds(180),
    BaseAddress = new Uri(EndpointUrl),
    DefaultRequestHeaders =
    {
    Authorization = new AuthenticationHeaderValue("Bearer", EndpointApiKey),
    Accept = { new MediaTypeWithQualityHeaderValue("text/event-stream") }
    }
};

If you have the time to inspect the test code, I've uploaded it to a repository over here.

[brynn-code]

Sorry for the latest response, as the python client works well, I think the problem of C# client may not related to the endpoint but the client itself. I'm sorry that I know little about C#, here is the detail doc of how streaming works: https://microsoft.github.io/promptflow/how-to-guides/enable-streaming-mode.html?highlight=streaming#understand-the-streaming-process , hope this can provide some more useful things to you.

[CyrilMercier00]

I agree, it's an issue with the client, but I'm not sure where it could come from.
It could be with how I consume the response, or the fact that it POST data before reading, but I'm not sure.

I re-read the docs and everything seems fine on the client end, and I don't know enough about python to fully understand how the two are supposed to interact.

Thank you for your help !

[brynn-code]

(Trying to find experts from our side with both C# and python ..

[elliotzh]

Hi @CyrilMercier00 there should be a different interface to read the streaming response in C#. Could you please have a try?

var response = await client.SendAsync(message, HttpCompletionOption.ResponseHeadersRead);
var stream = await response.Content.ReadAsStreamAsync();

ref: link

[CyrilMercier00]

That was it, using SendAsync instead of PostAsync solved it.
Thanks a lot guys !