Realtime agent timing out

[rupesh-kartha]

Hi,
I started the realtime agent successfully using
python -m realtime_agent.main agent --channel_name=RK1 --uid=007

I then connect to the channel using the web demo link generated in my project's config page. I see websocket creation failed first but subsequently succeeded. From the chrome dev. console

chunk-vendors.b9f9f67a.js:381 WebSocket connection to 'wss://208-97-254-11.edge.sd-rtn.com:4709/' failed: WebSocket is closed before the connection is established.

chunk-vendors.b9f9f67a.js:381 16:59:19:525 Agora-SDK [DEBUG]: [choose-best-ws client-6c234 2] 246ms: close backup websocket: wss://208-97-254-11.edge.sd-rtn.com:4709/
chunk-vendors.b9f9f67a.js:381 16:59:19:526 Agora-SDK [DEBUG]: [gateway-client-6c234] websocket opened: wss://208-97-254-11.edge.agora.io:4709/
chunk-vendors.b9f9f67a.js:381 16:59:19:526 Agora-SDK [DEBUG]: [lock-P2PChannel-mutex-24] is locked, current queue 1. From P2PChannel.startP2PConnection
chunk-vendors.b9f9f67a.js:381 16:59:19:527 Agora-SDK [DEBUG]: websocket reset reconnect count, reason: opened
chunk-vendors.b9f9f67a.js:381 16:59:19:527 Agora-SDK [DEBUG]: [lock-websocket-22] is not locked, current queue 0. 
chunk-vendors.b9f9f67a.js:381 16:59:19:527 Agora-SDK [DEBUG]: [lock-P2PConnection-mutex-25] is created.
chunk-vendors.b9f9f67a.js:381 

However my agent times out waiting for a request from a caller. Here is the exception from the agent.

Traceback (most recent call last):
  File "/opt/anaconda3/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/rupeshkartha/openai-realtime-python/realtime_agent/main.py", line 281, in <module>
    run_agent_in_process(
  File "/Users/rupeshkartha/openai-realtime-python/realtime_agent/main.py", line 75, in run_agent_in_process
    asyncio.run(
  File "/opt/anaconda3/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/rupeshkartha/openai-realtime-python/realtime_agent/agent.py", line 126, in setup_and_run_agent
    await agent.run()
  File "/Users/rupeshkartha/openai-realtime-python/realtime_agent/agent.py", line 163, in run
    self.subscribe_user = await wait_for_remote_user(self.channel)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rupeshkartha/openai-realtime-python/realtime_agent/agent.py", line 39, in wait_for_remote_user
    remote_user = await asyncio.wait_for(future, timeout=15.0)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.12/asyncio/tasks.py", line 519, in wait_for
    async with timeouts.timeout(timeout):
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.12/asyncio/timeouts.py", line 115, in __aexit__
    raise TimeoutError from exc_val
TimeoutError
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10739f260>
zsh: abort      python -m realtime_agent.main agent --channel_name=RK1 --uid=007

I verified the config values on the .env file are correct.
Project name: DemoTest.

Looks like the audio/video from the web demo is not reaching the agent. Can you help point to what could be wrong?

I started the local server as
python -m realtime_agent.main server
But this local server is not in the call flow since I am using the web demo, which connects to agora server which in turn routs the audio/video to the agent running locally, to invoke Open AI API. Correct?