How to record audio from microphone by nicegui?

[coolmian]

I didn't find a component for audio recording. Is this feature temporarily unavailable?

[falkoschindler]

No, there is no NiceGUI element or function to record audio at the moment.
But that doesn't mean it is impossible using generic JavaScript commands via ui.run_javascript(). Maybe following tutorials like https://medium.com/@bryanjenningz/how-to-record-and-play-audio-in-javascript-faa1b2b3e49b is a good start.

[falkoschindler]

Here is a proof-of-concept for recording audio with NiceGUI. It records 3 seconds and plays it back afterwards:

async def record():
    await ui.run_javascript('''
        navigator.mediaDevices.getUserMedia({ audio: true }).then(stream => {
            const mediaRecorder = new MediaRecorder(stream);
            mediaRecorder.start();

            const audioChunks = [];
            mediaRecorder.addEventListener("dataavailable", event => audioChunks.push(event.data));

            mediaRecorder.addEventListener("stop", () => {
                const audioBlob = new Blob(audioChunks);
                const audioUrl = URL.createObjectURL(audioBlob);
                const audio = new Audio(audioUrl);
                audio.play();
            });

            setTimeout(() => mediaRecorder.stop(), 3000);
        });
    ''', respond=False)

ui.button('Record').on('click', record)

Of course the code might get more involved depending on what you want to do with the audio.

[coolmian]

Thank you for your sample code

[simplylu]

Unfortunately, your solution @falkoschindler does not seem to work anymore. Besides that respond=False has been removed, I'm getting the following error when running your example.
Will try to come up with a solution by myself, since I need recording functionality to implement some text to speech.

Task exception was never retrieved
future: <Task finished name='Task-308' coro=<AsyncServer._handle_event_internal() done, defined at /home/rtfm/.local/lib/python3.11/site-packages/socketio/async_server.py:595> exception=KeyError('result')>
Traceback (most recent call last):
  File "/home/rtfm/.local/lib/python3.11/site-packages/socketio/async_server.py", line 597, in _handle_event_internal
    r = await server._trigger_event(data[0], namespace, sid, *data[1:])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rtfm/.local/lib/python3.11/site-packages/socketio/async_server.py", line 635, in _trigger_event
    ret = handler(*args)
          ^^^^^^^^^^^^^^
  File "/home/rtfm/.local/lib/python3.11/site-packages/nicegui/nicegui.py", line 194, in _on_javascript_response
    client.handle_javascript_response(msg)
  File "/home/rtfm/.local/lib/python3.11/site-packages/nicegui/client.py", line 272, in handle_javascript_response
    self.waiting_javascript_commands[msg['request_id']] = msg['result']
                                                          ~~~^^^^^^^^^^
KeyError: 'result'
JavaScript did not respond within 1.0 s
Traceback (most recent call last):
  File "/home/rtfm/.local/lib/python3.11/site-packages/nicegui/events.py", line 406, in wait_for_result
    await result
  File "/home/rtfm/Projects/Test/nicegui_record_audio.py/test.py", line 23, in record
    await ui.run_javascript('''
  File "/home/rtfm/.local/lib/python3.11/site-packages/nicegui/client.py", line 211, in send_and_wait
    raise TimeoutError(f'JavaScript did not respond within {timeout:.1f} s')
TimeoutError: JavaScript did not respond within 1.0 s

Python Version: Python 3.11.6
Nicegui Version: nicegui==1.4.17

[falkoschindler]

Hi @simplylu , the JavaScript code doesn't return anything. Therefore you shouldn't await its call. We can simplify the code like this:

ui.button('Record', on_click=lambda: ui.run_javascript('''
    navigator.mediaDevices.getUserMedia({ audio: true }).then(stream => {
        const mediaRecorder = new MediaRecorder(stream);
        mediaRecorder.start();

        const audioChunks = [];
        mediaRecorder.addEventListener("dataavailable", event => audioChunks.push(event.data));

        mediaRecorder.addEventListener("stop", () => {
            const audioBlob = new Blob(audioChunks);
            const audioUrl = URL.createObjectURL(audioBlob);
            const audio = new Audio(audioUrl);
            audio.play();
        });

        setTimeout(() => mediaRecorder.stop(), 3000);
    });
'''))

[programus]

@falkoschindler Thank you for your solution!

Is it possible to get the audio data in the python side? Something like

def handle_audio_recording():
    audio_data = await ui.run_javascript(the_recording_and_return_script)
    # some code to process the audio, such as a STT API call, etc. 

ui.button('Record', on_click=handle_audio_recording)

It would be better if the recording can be stopped while the volume of the recent recorded audio is lower than a threshold. If this is possible, it would be very helpful for a audio based AI solution!

[rodja]

Thanks @programus. It's a very interesting problem. You will probably need send the data in chucks to the server (the "dataavailable" event above might help you). Or you establish a separate WebRTC connection. Either way, it seems to be out of scope for core-NiceGUI. If you need further help on this, please contact us at [email protected] so we can work something out.