how can i stream the response

[shevekwang]

i set stream to True, but it's not working

[AustinAres2007]

If you are trying to stream the reply, the response will not be in the json format. It will be raw from the openAI API. You must parse the bytes and form it into one cohesive string. You would have to receive the response from .read() instead of .json() as you cannot parse a data stream to Json.

There is probably a better way to do it, but this is how I did it.

Here is how I did it. You will need to place your own openAI API Key (In key variable on line 4)
(I apologise for bad formatting or something, I do not post on GitHub that much. I lack the knowledge of etiquette here)

Python 3.11

import openai_async, asyncio, json
from typing import AsyncGenerator

key = "Nunya, provide your own."

async def chat_stream(query: str) -> AsyncGenerator:

   # Define headers and stuff
   message_history = [{"role": "user", "content": query}]
   payload = {"model": "gpt-3.5-turbo", "messages": message_history, "stream": True}

   # Start byte stream
   reply = await openai_async.chat_complete(api_key=key, timeout=10, payload=payload)

   # Setup the list of responses
   responses: list[str] = [""]
   last_char = 0

   # For every character byte in byte stream
   for char in reply.read():
       # Check if current character and last char are line feed characters (Represents new chunk)
       if char == 10 and last_char == 10:
           
           # Check if chunk is the right format, or doesn't equal anything
           if responses[-1].strip("\n") in ["data: [DONE]", ""]:
               responses.pop()
           else:
               responses[-1] = json.loads(responses[-1][6:]) # Filter out the "data: " part, and translate to a dictionary

               yield responses[-1] # Yield finished chunk
               responses.append("") # Append start of a new chunk
       else:
           # Append part of new chunk to string
           responses[-1] += chr(char)
       
       last_char = char

async def main():

   # Define test task 1 (NOTE: In this test, messages aren't saved, it will not remember what you have said.)
   async def task_1():
       streamed_reply = chat_stream("Greetings, GPT 3.5! What is factorial of 0 and explain why in great detail.")
       text = ""
       async for chunk in streamed_reply:
           if chunk["choices"][0]["finish_reason"] != "stop":
               text += chunk["choices"][0]["delta"]["content"]
           else:
               print(text)
   
   # Define test task 2
   async def task_2():
       for _i in range(35):
           print("Bleh waiting..")
           await asyncio.sleep(0.5)

   # Create both tasks
   task1 = asyncio.create_task(task_1())
   task2 = asyncio.create_task(task_2())

   # Get both tasks
   await task1
   await task2

# Run main async threat
asyncio.run(main())