Read chat responses aloud - #331859
Conversation
There was a problem hiding this comment.
Pull request overview
Adds built-in chat response read-aloud support using platform speech synthesis or configured Azure Speech.
Changes:
- Adds platform and MAI text-to-speech engines.
- Cleans, punctuates, and chunks chat response text.
- Adds read/stop actions, state tracking, configuration, and tests.
Show a summary per file
| File | Description |
|---|---|
src/vs/workbench/workbench.desktop.main.ts |
Adds speech section. |
src/vs/workbench/contrib/speech/test/common/speechText.test.ts |
Tests text processing. |
src/vs/workbench/contrib/speech/test/browser/maiTextToSpeech.test.ts |
Tests SSML and voices. |
src/vs/workbench/contrib/speech/test/browser/builtinTextToSpeech.test.ts |
Tests engine selection. |
src/vs/workbench/contrib/speech/common/speechText.ts |
Cleans and chunks speech text. |
src/vs/workbench/contrib/speech/common/speechService.ts |
Defines built-in engine APIs. |
src/vs/workbench/contrib/speech/browser/textToSpeechEngineContribution.ts |
Registers speech engines. |
src/vs/workbench/contrib/speech/browser/speechService.ts |
Selects and manages engines. |
src/vs/workbench/contrib/speech/browser/speech.contribution.ts |
Registers speech services. |
src/vs/workbench/contrib/speech/browser/maiTextToSpeech.ts |
Implements Azure synthesis. |
src/vs/workbench/contrib/speech/browser/maiSpeechCredentials.ts |
Stores speech credentials. |
src/vs/workbench/contrib/speech/browser/maiSpeechActions.ts |
Adds setup command. |
src/vs/workbench/contrib/speech/browser/builtinTextToSpeech.ts |
Implements platform synthesis. |
src/vs/workbench/contrib/chat/test/electron-browser/actions/voiceChatActions.test.ts |
Tests response selection. |
src/vs/workbench/contrib/chat/test/common/voiceChatService.test.ts |
Updates speech mock. |
src/vs/workbench/contrib/chat/electron-browser/chat.contribution.ts |
Registers read-aloud actions. |
src/vs/workbench/contrib/chat/electron-browser/actions/voiceChatActions.ts |
Integrates response narration. |
src/vs/workbench/contrib/chat/electron-browser/actions/readAloudActions.contribution.ts |
Extracts action registration. |
src/vs/workbench/contrib/chat/common/actions/chatContextKeys.ts |
Adds narration state keys. |
src/vs/workbench/contrib/chat/browser/widget/chatListRenderer.ts |
Updates per-response controls. |
src/vs/workbench/contrib/accessibility/browser/accessibilityConfiguration.ts |
Adds speech settings. |
src/vs/sessions/sessions.desktop.main.ts |
Enables Agents Window actions. |
src/vs/base/test/browser/markdownRenderer.test.ts |
Tests plaintext rendering. |
src/vs/base/browser/markdownRenderer.ts |
Improves plaintext Markdown output. |
Review details
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 24/24 changed files
- Comments generated: 5
- Review effort level: Balanced
| this._register(this.configurationService.onDidChangeConfiguration(e => { | ||
| if (e.affectsConfiguration(MAI_SPEECH_ENDPOINT_SETTING)) { | ||
| this._onDidChangeConfigured.fire(); | ||
| } |
There was a problem hiding this comment.
Good catch, fixed in c80f0c4. The key is now stored together with the endpoint it was issued for, and resolve() only returns it when the two still match. Pointing the setting at another host now stops reading aloud until a key for that host is entered, instead of handing the existing one over.
| // Text to speech is also provided by the built-in engines, which do not | ||
| // make a speech provider available: gating on that alone would leave | ||
| // reading aloud with no settings at all, including the language it reads in. | ||
| if (!this.speechService.hasSpeechProvider && !this.speechService.hasTextToSpeechProvider) { |
There was a problem hiding this comment.
You're right, this was a deadlock: no speech provider and no platform synthesizer meant the settings were never registered, so Set Up Read Aloud could not write the endpoint, so the engine could never become available. Fixed in c80f0c4 by registering the endpoint and voice settings statically, leaving only the speech-provider-dependent ones behind the gate.
| id: 'workbench.action.chat.readChatResponseAloud', | ||
| title: localize2('workbench.action.chat.readChatResponseAloud', "Read Aloud"), | ||
| icon: Codicon.unmute, | ||
| precondition: CanVoiceChat, | ||
| precondition: CanReadAloud, |
There was a problem hiding this comment.
Added in c80f0c4 — the chat accessibility help now covers Read Aloud and how to stop it. Worth having given this is primarily a screen-reader affordance; thanks.
| return false; | ||
| } | ||
|
|
||
| return this.contextKeyService.getContextKeyValue<string>(ChatContextKeys.synthesisInProgressItemId.key) === element.id; |
There was a problem hiding this comment.
Agreed in principle, but leaving this one for now and happy to be told otherwise.
The state is owned by ChatSynthesizerSessions in chat/electron-browser/, while this renderer is in chat/browser/, so it cannot import it — the context key is currently the only channel across that boundary. Doing this properly means a small service in browser/ or common/ that owns the active response id and exposes a change event, with the electron-browser synthesizer pushing into it.
That seemed like more than I wanted to add while the PR is still blocked on the endpoint, but I'm glad to do it if you'd prefer it in this change.
Mohammad javad Dianat (dianatofficial)
left a comment
Mohammad javad Dianat (dianatofficial)
left a comment
There was a problem hiding this comment.
Approved. The fix is minimal and correct.
VS Code has had a "Read Aloud" action in the chat response footer for a while, but it is gated on `HasSpeechProvider` and no speech provider ships in this repository, so the button has never been visible to anyone. Supply the missing engine. Text is cleaned up (emoji and markdown removed, blocks punctuated so the reader pauses between them), split into pieces, and synthesized one piece ahead of playback so that speech starts quickly. Two engines are registered against a new `IBuiltinTextToSpeechEngine` seam and picked by priority: the MAI voice model over the Azure Speech service when it is configured, and the speech synthesizer of the platform for everything else, including languages the model does not speak and machines that are offline. A speech provider from an extension still wins over both. Registering a full `ISpeechProvider` would also have enabled five speech-to-text actions that have no implementation here, which is why text to speech gets its own seam and a `HasTextToSpeechProvider` context key. Authentication is the one open piece: there is no product endpoint for verbatim synthesis yet, so the endpoint is a setting and the key lives in secret storage, never in a settings file. Only `maiSpeechCredentials.ts` has to change once an endpoint authenticated with the identity the user already signed in with exists.
- Bind the stored key to the endpoint it was given for, so that pointing the setting at another host stops reading aloud until a key for that host is entered, rather than sending this one to it. The setup action writes the endpoint before asking for the replacement key, which left a window where the two did not belong together. - Register the endpoint and voice settings statically. They were registered only once speech was already available, so on a platform without a speech synthesizer the setup action could not write the endpoint, and the engine could never become available at all. - Strip keycap sequences that omit the variation selector, which the Unicode grammar allows: `1\u20E3` was read out as "one". - Mention reading aloud in the chat accessibility help, including how to stop.
Reducing markdown to the text a reader hears landed upstream in #328466 as an opt-in option, so drop the version of that fix carried here and ask for it instead. Without it a list item keeps its `**` and a link keeps its target, both of which get read out. Kept from the earlier change: a code block now ends with a line break like every other block, so the sentence after it no longer runs into its last line.
c80f0c4 to
c79130e
Compare
|
This is a great direction, but I found a few things that I think need to be addressed before this can ship:
The authentication/service issue is a release blocker. I would also address the long-response and stale-configuration bugs before enabling the feature in production, and clarify the intended web scope. |
- Speak long responses in pieces through the platform synthesizer too. A single long utterance is truncated or stalls outright in some browsers; the pieces are the ones the on-device path already used. - Recompute whether a usable key is available when the endpoint changes. Since the key is bound to the endpoint it was given for, changing the endpoint and cancelling the key prompt left this engine claiming to be configured: it then outranked the platform synthesizer and failed instead of falling back. Overlapping refreshes are sequenced, because reading the secret is asynchronous and the later answer has to win. - Register the engines, their credentials and the setup command from the desktop rather than shared with web. The actions that use them live in `chat/electron-browser`, so on web the command was offered but never registered. The accessibility help now only mentions reading aloud when an engine is actually available.
|
Thanks Megan, these were all real. Fixed in 49881ae, except (1) which I want to check with you. 2. Long responses with platform speech — fixed. 3. Stale configuration after changing endpoints — fixed, and this one was mine: binding the key to its endpoint (from the earlier round) made 4. Web registration mismatch — fixed by making it consistently desktop. The engines, their credentials and the setup command now register from |
|
1. Production authentication — agreed, and I have not tried to work around it. The user-supplied endpoint and key are a development arrangement so the rest could be built and reviewed, not a shipping path, and the PR stays draft until a product endpoint exists. Where that stands: Arthur Cnops (@accnops) confirmed that a verbatim HTTP endpoint is feasible and would be straightforward. The existing |
|
Great, thanks! Let me know when the endpoint lands so we can merge. |
This PR makes it possible to read chat messages aloud.
voice.mov
How it works
There is no product endpoint for verbatim synthesis yet, so this cannot ship as-is. Opening as a draft for that reason.
The voice backend VS Code already talks to (
voiceWsUrl) does not expose plain synthesis —request_narrationruns the text through an LLM narrator first, so the text is rewritten rather than read.So this PR calls the Azure Speech endpoint directly — currently
https://eastus2.tts.speech.microsoft.comwithen-US-Harper:MAI-Voice-2— supplied by the user as a setting, with the key inISecretStorageService.Arthur Cnops (@accnops) has confirmed a verbatim HTTP endpoint looks feasible. When it lands, only
maiSpeechCredentials.tschanges: the endpoint moves toproduct.json, auth becomes the identity the user already signed in with, and the setup step disappears. The engine never sees a key — it asks the credentials service for{ endpoint, credential }and does not care where they came from.Trying it today
Speech: Set Up Read Aloud→ endpoint (URL-validated) → key (masked input)The engine becomes available immediately — no reload. With nothing configured, reading falls back to the speech synthesizer of the platform.
Testing
198targeted tests,5,475across the broader chat / markdown / speech / accessibility sweep. Typecheck, eslint and layer checks pass. No dependencies added —package.jsonandpackage-lock.jsonare unchanged.Verified end to end in Code OSS: the engine switches the moment the key is set without a reload, an unconfigured install falls back to the platform voice, markdown and emoji are stripped, and code blocks are separated from the text that follows.