Chat Interface Streaming works locally but not on GCP

[jamie0725]

Describe the bug

Hello,

I am encountering an unusual issue while deploying my chatbot as a server on Google Cloud. The streaming functionality works perfectly fine locally. I use yield in my code to present two text segments with approximately 10 to 30 seconds in between. The intended flow is as follows:

• User clicks submit -> (3 - 5 seconds) -> first part of text -> (10 - 30 seconds) -> second part of text -> end

However, on the GCP server, even though there is a 'stop' button visible after the user clicks 'submit,' indicating that streaming mode is active, the first part of the text does not appear until the very end, coinciding with the completion of the second part of the text. Another different thing is that on the GCP server, there is no processing shown in the top right corner, while locally there is (see screenshot 2). The flow is disrupted as follows:

• User clicks submit -> (10 - 30 seconds) -> first part of text -> (< 0.5 seconds) -> second part of text -> end

I am uncertain about the root cause of this issue and would appreciate any assistance or insights. Thank you in advance for your help!

Have you searched existing issues? 🔎

• I have searched and found no existing issues

Reproduction

import gradio as gr

def get_response():
    result = ""
    result += get_first_completion()
    yield result
    result += get_second_completion()
    yield result

def get_blocks():
    with gr.Blocks(
        title="xxx", theme="step-3-profit/Midnight-Deep"
    ) as demo:
        with gr.Tab("xxx"):
            with gr.Row():
                a = gr.Checkbox(
                    label="xxx",
                    value=False,
                )
                b = gr.CheckboxGroup(
                    label="xxx",
                    choices=["a", "b", "c", "d"],
                    value=["a"],
                )
            chatbot = gr.Chatbot(
                render=False,
            )
            gr.ChatInterface(
                get_response,
                chatbot=chatbot,
                additional_inputs=[a, b],
                autofocus=False,
            )
    demo.queue()

    return demo

demo_io = get_blocks()

in main.py, the code is:

import sys

import gradio as gr
import uvicorn
from fastapi import FastAPI
from routers import whoami_router

import demo_io

app = FastAPI()
app.include_router(whoami_router.router)

app = gr.mount_gradio_app(app, demo_io, path="/v1/demo")

if __name__ == "__main__":
    uvicorn.run(app="main:app", host="127.0.0.1", port=9080, reload=True, access_log=False)

Screenshot

Locally:
Fig 1: streaming works fine locally

Fig 2: Processing shown on the top right corner

On GCP:
Fig 3: processing is not shown in the top right corner, and streaming is not working on GCP

Logs

No response

System Info

gradio version: 4.17.0
gradio_client version: 0.9.0

Severity

Blocking usage of gradio