Live transcription followup

[julien-nc]

Hey @nickvergessen, we are halfway there implementing the live transcription exApp.

The app is connecting to the signaling server, authenticating, receiving the list of participants, getting audio streams, transcoding the streams to feed them to the transcription engine, producing transcriptions and sending them as Talk chat messages (for now).
We are still in the process of making the app react to people joining and leaving calls, managing all the parallel transcriptions subprocesses etc...

We now have a clearer idea of how Talk and the live transcription app can interact.

When a participant joins a call

On the UI side, the users could be able to choose whether they want to receive/see the transcriptions with a checkbox somewhere. Maybe in the "media settings" modal, like the call recording consent. As you wish.

If the user wants to see transcriptions, Talk can send a request to the live_transcription exApp on its /transcribeCall endpoint with those params:

• roomToken
• sessionId

Here is an example on how to make a request to an exApp endpoint from NC's backend: https://github.com/nextcloud/context_chat/blob/2ea9768bec56d0ea3dbe1551d3680b77f6ea48f4/lib/Service/LangRopeService.php#L123-L130

The transcription messages

The exApp will send signaling messages to all participants who requested the transcriptions.
We are flexible on the format of those messages. The simplest would be:

{
    "sessionId": "blabla",
    "transcriptionMessage": "there you go",
}

We are not fully decided about sending the intermediate transcriptions or only the ones considered "definitive". Let's consider that the app will only send definitive transcriptions. If we change that, we can always add a type (intermediate/definitive) attribute in the signaling messages.

Questions

• Are you fine with all that?
• Did we forget something?
• Are there some decisions we should make all together?

[nickvergessen]

We are not fully decided about sending the intermediate transcriptions or only the ones considered "definitive". Let's consider that the app will only send definitive transcriptions. If we change that, we can always add a type (intermediate/definitive) attribute in the signaling messages.

What would be the idea/use of it? Either it's shown to the user or not? :P There is no "maybe show it to the user"?

By "participantId" you mean the sessionId, so you can send signaling messages, right? Otherwise this needs clarifying as the attendee id is currently not meant to be used by normal users.

[julien-nc]

What would be the idea/use of it? Either it's shown to the user or not? :P There is no "maybe show it to the user"?

It would be to display the words as soon as they are generated by the transcription engine. Forget it. We will only send full and definitive sentences.

By "participantId" you mean the sessionId, so you can send signaling messages, right? Otherwise this needs clarifying as the attendee id is currently not meant to be used by normal users.

Yes, sorry, we need the sessionId and we'll use it to send signaling messages.
And in the messages, we will identify who talked with the sessionId as well.

[danxuliu]

If the user wants to see transcriptions, Talk can send a request to the live_transcription exApp on its /transcribeCall endpoint with those params:

• roomToken
• sessionId

There should be an additional parameter to enable or disable the transcription, as the user may want to turn it on or off during the call.

Independently of that, a little detail to keep in mind: although extremely unlikely, due to the asynchronous nature of the messages it could happen that a participant joins the call and enables transcriptions, the transcription service starts, and the first list of participants in the call that it receives from the signaling server does not contain yet that participant. If the transcription service is shutdown when none of the participants that requested a transcription are in the call (which from my point of view would be the expected behaviour) you may want to add a delay or something like that to ensure that the transcription service is not shutdown too early.

Similarly, the opposite could potentially happen too: a participant enables transcriptions but leaves the call before the transcription service receives a list of participants in the call that includes that participant. I do not know if that could be relevant or not to the shutdown logic that you implement, but I mention it just in case.

We are flexible on the format of those messages. The simplest would be:

{
    "sessionId": "blabla",
    "transcriptionMessage": "there you go",
}

For consistency with other signaling messages it should also include a "type": "transcription" attribute (next to sessionId and transcriptionMessage in that payload, independently of the outer "type": "message").

[kyteinsky]

There should be an additional parameter to enable or disable the transcription, as the user may want to turn it on or off during the call.

good point, we missed that this request itself would convey that info too.

Nice of you to mention that, yes a timeout would definitely makes sense here so we don't waste time in restart of the service. It will also be useful when participants rejoin, which happens in a relatively short period of time.
We would have a similar kind of timeout for the transcription server too.

Similarly, the opposite could potentially happen too: a participant enables transcriptions but leaves the call before the transcription service receives a list of participants in the call that includes that participant. I do not know if that could be relevant or not to the shutdown logic that you implement, but I mention it just in case.

Maybe we can consider this a rare occurance to not impact performance/efficiency in most of the systems. Also the timeout for leaving the room can decided to be shorter to help with this.

For consistency with other signaling messages it should also include a "type": "transcription" attribute (next to sessionId and transcriptionMessage in that payload, independently of the outer "type": "message").

thanks for pointing that out!

[kyteinsky]

It would be nice if the Talk team could start implementing the feature in parallel in the Talk UI as well.
We have a working prototype with all the basic functions working.

This PHP snippet can be used to call the transcription endpoint in the "live_transcription" ex-app,
where sessionId is the session ID of the participant requesting the transcriptions and enable is the
option to enable/disable transcriptions, as discussed here.
An example with checks can be found here: https://github.com/nextcloud/context_chat/blob/2ea9768bec56d0ea3dbe1551d3680b77f6ea48f4/lib/Service/LangRopeService.php#L46

try {
	$appApiFunctions = \OCP\Server::get(\OCA\AppAPI\PublicFunctions::class);
} catch (ContainerExceptionInterface|NotFoundExceptionInterface $e) {
	throw new RuntimeException('Could not get AppAPI public functions');
}

$params = [
	'roomToken' => string,
	'sessionId' => string,
	'enable' => bool,
];
$response = $appApiFunctions->exAppRequest(
	'live_transcription', // $appId
	'/transcribeCall',    // $route
	null,                 // $userId
	'POST',               // $method
	$params,              // $params
	                      // $options
);

The transcriptions can then be received through websockets in the signaling messages in this format:

{
    "type": "message",
    "message": {
        "sender": {
            "type": "session",
            "sessionid": "7clbmRBx1h3SRsEaN0JOYyzmn49FW5aCwTG1wzaRsg98PT13bDM1Vnp0ZExxZDJfNnllMV9qVW1qdWRDa1BJUi1GTmxENlVUV3JhT2RIQnEwS05FMkR0aFhPSzZyfDUyNjgzODc0NzE="
        },
        "data": {
            "message": "can you see my screen",
            "type": "transcript",
            "speakerSessionId": "LbUfM8FAKjSLi_YtURITJXzJeViF9wKBfae0lKQ4Luh8PT13TXdhanZCRGVGanJoOEhmcHk4dHVvV0NhTk5tRi1IZHVsSkc1b2RtZXRVNElmTlE4ajA5UlN5bmczfDMwNjgzODc0NzE="
        }
    }
}

The message does not contain any punctuations so in this state you might want to take that into account while crafting the UI.

[kyteinsky]

hello,
A small status update.
The performance of the transcript generation with multiple audio streams on CPU is not ideal. The vosk app does not allow running the batch model on CPU. For GPU (tested with NVIDIA), it seems performant in the initial tests out of the live transcription app. We'll be integrating the GPU mode soon to formally test it.

[kyteinsky]

hello again,
The performance issues have been fixed, now it works well with CPU and GPU, the realtime factor being ~4x (11 secs audio in 2.66 secs).

There is an issue where we would like the Talk and the Design team's feedback.
The Vosk server, our transcription engine, supports multiple languages (https://alphacephei.com/vosk/models) but each language has a different model file.
How do we load a particular kind of model?

1. It can be picked up from the Nextcloud UI's language - The person might choose to speak one language in one call, other in another without changing the UI's language or the language set in the settings.
2. We can use a settings option for the speaker to select the language in the dialog before the call
3. We can detect it from the initial few seconds of the audio - If this is not accurate, the whole transcript from that speaker is broken

So far (2.) seems like the best option to us, what do you think?

cc @marcoambrosini

[danxuliu]

Regarding options 1 and 2, although it would make sense that the speakers themselves are the ones specifying the language that they are speaking, it would be also a bit strange, because they would specify it so others can see transcriptions of what they say, but it would provide no "benefit" for them (of course it could be seen just as a courtesy toward other participants, but still :-P ).

For example, in our company cloud I would set English as the default language, but in a 1-1 with a Spanish colleague I would need to manually change it to Spanish even if I do not even know if the other participant will enable the transcription or not.

In that sense it might make more sense if the participants that enable the transcription set the language, but then it would need to be decided what happens if one participant enables the transcription in one language and another participant, by mistake, enables it in a different language (that is not even being spoken in the conversation). Of course they might fix it and choose the right language later on, but what to do in the meantime?

Moreover, what would happen in multilingual calls? In some cases there could be some participants speaking one language, some participants speaking another language, and then some participants speaking both and acting as translators. For the participants speaking just one language enabling the transcription in the language that they speak would be fine, even if the transcriptions are totally wrong when the other language is being spoken. But for the participants that speak both languages then it would be messy, as they would get the right translations only in a single language. In any case, in multilingual calls, for participants that can speak in more than one language in the same call it would be problematic even if the speakers themselves set the language rather than the participants enabling the transcription 🤷

But this should be a less common scenario, so maybe it would be something just to keep in mind and address in a future version. Also because transcribing participants that may speak two different languages in the same call seems to be problematic no matter the approach taken :-) (unless there is a cool model that can dynamically and automatically switch between languages ;-) )