Add support for live transcriptions in calls

[danxuliu]

Description

TODO

How to test

• Install the live_transcription app (you need to use the master branch rather than the current latest release, as there were API changes since then)
  • Ensure that cert_pem, cert_key and rsa_private_key are commented in Janus configuration (as an ECDSA certificate is required for DTLS by the live_transcription app; this will be documented later)
• Alternatively, you can fake having the live_transcription app installed and manually send signaling messages. To do that:
  • return true; in

    spreed/lib/Service/LiveTranscriptionService.php

    Line 35 in 52c8792

    public function isLiveTranscriptionAppEnabled(?object $appApiPublicFunctions = null): bool {

  • return; in

    spreed/lib/Service/LiveTranscriptionService.php

    Line 82 in 52c8792

    public function enable(Room $room, Participant $participant): void {

    and

    spreed/lib/Service/LiveTranscriptionService.php

    Line 108 in 52c8792

    public function disable(Room $room, Participant $participant): void {

  • return [ 'en' => [ 'name' => 'English', 'metadata' => [ 'separator' => ' ', 'rtl' => false ]]]; in

    spreed/lib/Service/LiveTranscriptionService.php

    Line 129 in 52c8792

    public function getAvailableLanguages(): array {

  • Comment

    spreed/lib/Service/LiveTranscriptionService.php

    Line 158 in 52c8792

    $this->requestToExAppLiveTranscription('POST', '/api/v1/call/set-language', $parameters);

• Create a new conversation
• In the conversation settings, set the language spoken in calls
• Start a call
• In a private window, join as another participant
• If using a real transcription, just say something to be transcribed
• If faking the transcription, execute the following in the browser console:

OCA.Talk.SimpleWebRTC.connection.sendCallMessage({ type: 'transcript', message: 'This is a test', langId: 'en', speakerSessionId: OCA.Talk.SimpleWebRTC.connection.getSessionId(), to: OCA.Talk.SimpleWebRTC.webrtc.peers[1].id })

🖌️ UI Checklist

🖼️ Screenshots / Screencasts

🚧 Tasks (or follow ups)

• Use the metadata from the language for separators and text direction
• Disable transcription after leaving call
• Enable transcription on call reconnections
• Use TypeScript for the new JavaScript files
• Ignore the transcript signaling message if it does not come from an internal client
• Convert the transcription button to a split button for moderators to set the language during a call without having to open the full conversation settings?
• Remove no longer visible lines in a transcript block; currently the no longer visible transcript blocks are removed, but if a participant is speaking without interruption for a long time all the lines will be still there, which at some point may affect performance when getting the line boundaries (or it may not, but in any case they are unneeded)
• Tests

🏁 Checklist

• 🌏 Tested with different browsers / clients:
  • Chromium (Chrome / Edge / Opera / Brave)
  • Firefox
  • Safari
  • Talk Desktop
  • Integrations with Files sidebar and other apps
  • Not risky to browser differences / client
• 🖌️ Design was reviewed, approved or inspired by the design team
• ⛑️ Tests are included or not possible
• 📗 User documentation in https://github.com/nextcloud/documentation/tree/master/user_manual/talk has been updated or is not required

🛠️ API Checklist

🚧 Tasks (or follow ups)

• Adjust capability? Is it right as a configuration capability? And should it also check if the external signaling server is used? Or should the clients check that instead and that the MCU feature is set in the signaling server? Should the Talk hash change when the live_transcription app is enabled or disabled (as it will affect if the live_transcription related elements are shown in the UI or not)?
• Use default_language and force_language settings when transcriptions are enabled on a conversation without an explicit language set
• Add the liveTranscriptionLanguageId property to the list of properties that trigger a signaling message to update the room, and update the value in the clients
• Add system message when the language is changed? It does not seem to be worth it, but if it is, the message should only include the id of the language and not resolve the actual name, as that would require a call to the live_transcription app that could delay getting the messages. The name should be resolved in the clients when showing the message (but I do not know if that could have other drawbacks).
• Find a way to handle incompatible changes in the live_transcription app (for example, if the API changed in certain version). Maybe the app should provide capabilities?
• Tests
• Federated calls

🏁 Checklist

• ⛑️ Tests (unit and/or integration) are included or not possible
• 📘 API documentation in docs/ has been updated or is not required
• 🔖 Capability is added or not needed

[danxuliu]

Integration test failures should be unrelated.

Note that there was no version bump, as I assumed that it would be better to keep the beta version number matching the actual beta release. Due to that when testing this pull request the migration would need to be manually applied!

[Antreesy]

We're always going to have the transcription story on screen, or there's some expiration planned?
Also would be lovely to write new components in the script setup, but we can do it in follow up

Checked the faking example, still feel avatars are a bit redundant, we should have been sticking to displaynames only, but otherwise looks fine

[DorraJaouad]

Also would be lovely to write new components in the script setup, but we can do it in follow up

As discussed, we will do the migration.

• Transcription should have time shown interval after the last signaling message received of 3 sec or similar
• I noticed that some participants can receive the signaling and others not against all receiving raise hand signaling, I am not sure if it is a simulation noise or something with processing the signaling.

But overall, it is awesome 🔥

[nickvergessen]

Use default_locale and force_language settings when transcriptions are enabled on a conversation without an explicit language set

I hope you meant default language not default locale? I mean a default locale also would make sense if it helps writing numbers, dates and things, but using the default_language would be a good fall back if there is no force_language

[nickvergessen]

Note that there was no version bump, as I assumed that it would be better to keep the beta version number matching the actual beta release.

You can do …beta.2.1 btw, also did that in the past already: 5dc0b98

[nickvergessen]

Okay from PHP side

[Antreesy]

Frontend-wise OK

[Antreesy]

Felt to much during the faking tests, but maybe it will be noticable on live ones

[danxuliu]

Originally I used 1 second, but it felt too quick 🤷 Maybe it could be proportional to the length of the last line or something like that, I intended to implement that but in the end I went for a simpler approach (strange in me :-P )

[danxuliu]

We're always going to have the transcription story on screen, or there's some expiration planned?

Do you mean removing the transcript after a few seconds without nobody speaking? I would just leave it on screen, but I do not have strong feelings about it.

Also would be lovely to write new components in the script setup, but we can do it in follow up

I tried to, but unfortunately it was taking me way longer than expected so... follow up I guess :-)

Checked the faking example, still feel avatars are a bit redundant, we should have been sticking to displaynames only, but otherwise looks fine

I used them for consistency with the chat, and also I think they are good when someone is speaking for long, as in that case the name would be hidden by the text and the avatar still provides a context.

[DorraJaouad]

tiny note : icon should be https://pictogrammers.com/library/mdi/icon/subtitles-outline/

[danxuliu]

Also would be lovely to write new components in the script setup, but we can do it in follow up

As discussed, we will do the migration.

👍

* Transcription should have time shown interval after the last signaling message received of 3 sec or similar

So increase from the current two seconds to three seconds when a new line is shown and before scrolling to a next one if more lines appear? Or what do you mean?

* I noticed that some participants can receive the signaling and others not against all receiving raise hand signaling, I am not sure if it is a simulation noise or something with processing the signaling.

That is unexpected 🤔 But let's blame it on the simulation unless you find reproducible steps :-)

But overall, it is awesome 🔥

❤️

[danxuliu]

Use default_locale and force_language settings when transcriptions are enabled on a conversation without an explicit language set

I hope you meant default language not default locale? I mean a default locale also would make sense if it helps writing numbers, dates and things, but using the default_language would be a good fall back if there is no force_language

Oops, definitely, typo fixed.

Edit: now really fixed. It helps to press the button to update the text after checking the preview...

Note that there was no version bump, as I assumed that it would be better to keep the beta version number matching the actual beta release.

You can do …beta.2.1 btw, also did that in the past already: 5dc0b98

Done.

[danxuliu]

tiny note : icon should be https://pictogrammers.com/library/mdi/icon/subtitles-outline/

Done.