diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 1b27138f1..ed20f8db0 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -136,6 +136,7 @@ leading and trailing pipe. Columns, in order: | functional-chat-default-mount | closed | src/OpenClaw.Tray.WinUI/Chat/FunctionalChatHostExtensions.cs | mounting the FunctionalUI chat tree as the default ChatPage or ChatWindow surface | ReactorChatHostExtensions and OpenClawReactorChatRoot | legacy FunctionalUI chat files may remain for focused compatibility coverage only | ChatPage and ChatWindow mount the Reactor root directly into their existing ChatHost Borders; no FunctionalUI component mounts or nests Reactor on the default path | review-only: user explicitly deferred new tests for this migration; required build and existing shared/tray suites still run | review-only | when legacy FunctionalUI chat surfaces are removed | | settings-store | authoritative | src/OpenClaw.Tray.WinUI/Pages/SettingsPage.xaml.cs | hand-rolled save/echo suppression flags for two-way settings binding | ISettingsStore | PermissionsPage and other surfaces may read SettingsManager directly until migrated | a save originating from Update does not echo Changed to the caller and external saves are republished on the UI thread | SettingsStoreTests.Update_DoesNotEchoChangedToSelf | behavioral | when all settings surfaces read and write through ISettingsStore | | settings-page-vm | authoritative | src/OpenClaw.Tray.WinUI/Pages/SettingsPage.xaml.cs | settings load, persist, echo-guard, and auto-save wiring | SettingsPageViewModel | code-behind keeps gateway-uninstall, gateway-info and uptime timer, saved-indicator visual, and app-info population | each settings control persists its field through the store preserving mutate-save-notify order and does not re-persist on external change | SettingsPageViewModelTests.ExternalChange_ReloadsWithoutRePersisting | behavioral | when the Settings page holds no settings persistence logic in code-behind | +| voice-assistant-coordinator | authoritative | src/OpenClaw.Tray.WinUI/App.xaml.cs | voice assistant state transitions, wake dispatch, response correlation, timeout, and playback ownership | VoiceAssistantCoordinator | App constructs, reconciles, synchronously offers notifications for claiming, and disposes the coordinator | one wake utterance owns one tracked direct send and at most one correlated spoken final while App contains no assistant state machine | AppRefactorContractTests.VoiceAssistant_StateMachineStaysOutsideApp | source-shape | when voice assistant lifecycle composition moves to AppBootstrapper | ## Deferred test builders diff --git a/docs/WINDOWS_NODE_ARCHITECTURE.md b/docs/WINDOWS_NODE_ARCHITECTURE.md index e907874f1..cf213c030 100644 --- a/docs/WINDOWS_NODE_ARCHITECTURE.md +++ b/docs/WINDOWS_NODE_ARCHITECTURE.md @@ -523,6 +523,31 @@ This is a candidate implementation path, not an implemented node command yet. Vo Current PR review status: open PR #120 (`feature/voice-mode`) is a useful prototype but should not merge as-is. It currently conflicts with the active capability-settings branch, advertises `voice.*` commands without the default-off Settings gate used for other privacy-sensitive capability groups, widens operator scopes in the same PR, persists cloud TTS provider keys in plain settings JSON, and introduces a Windows-specific wire schema before the Mac runtime/controller/session contract is agreed. Safe next step: split schema, gateway scope, chat transport, Windows runtime, WebChat integration, and cloud-provider credentials into separate reviews; keep the first merge behind a default-off Voice Settings group and gateway dangerous-command allowlist. +### Local voice assistant V0 + +The tray also has a default-off local voice assistant preview. This is a settings-driven +orchestration feature, not a new node capability or `voice.*` protocol: + +1. `VoiceService` owns WASAPI microphone capture, energy VAD, and local Whisper transcription. +2. `VoiceWakeGate` accepts only a silence-bounded transcript that begins with the configured + one-to-three-word wake phrase and includes a request in the same utterance. +3. `VoiceAssistantCoordinator` stops capture, sends the extracted request through the canonical + chat provider, claims only the final response correlated to that tracked local send, and speaks + it through the configured Windows, Piper, or ElevenLabs TTS provider. +4. Capture remains stopped while dispatching, waiting, and speaking. A fresh capture pipeline is + acquired afterward through the shared exclusive microphone lease. + +The coordinator is the authoritative state and cancellation owner. `App` remains the composition +root and forwards settings, provider, notification, and shutdown lifecycle inputs. Gateway +message ID and sequence are carried into local notifications so assistant playback does not claim +an unrelated final from the same session. + +V0 intentionally uses transcript-gated activation rather than a dedicated wake-word model. +Whisper therefore runs for each detected ambient speech burst. The wake phrase and request must be +spoken together, there is no persistent conversation mode or barge-in, and only one request can be +in flight. Enabling the preview requires configured local STT and TTS readiness, while gateway and +idle-chat readiness are runtime pause conditions. + --- ## Architectural Questions diff --git a/docs/WINDOWS_NODE_TESTING.md b/docs/WINDOWS_NODE_TESTING.md index b4744e3e6..32ad95560 100644 --- a/docs/WINDOWS_NODE_TESTING.md +++ b/docs/WINDOWS_NODE_TESTING.md @@ -122,6 +122,48 @@ When the node connects, it advertises these capabilities: Local MCP clients also see MCP-only `app.*` commands such as `app.navigate`, `app.status`, `app.chat.snapshot`/`app.chat.send`/`app.chat.reset`, and `app.chat.queue.list`/`app.chat.queue.cancel`. Connection diagnostics and setup tools live under `app.connection.*`; use `app.connection.status` to inspect active gateway, operator/node credential state, MCP runtime status, browser proxy caveat, pending approval commands, and recent diagnostics, and `app.connection.gateways` to list saved gateway records without token values. These are local testing and automation hooks registered with the tray's MCP server and are not advertised to the gateway WebSocket. +## Voice Assistant Preview + +The Voice & Audio page includes a default-off **Assistant (preview)** mode. It reuses local +Whisper STT, the configured TTS provider, and the existing gateway chat timeline. It does not add +an MCP tool or gateway node command. + +Prerequisites: + +- Speech-to-text is enabled and the selected Whisper model is downloaded. +- Text-to-speech is enabled and the selected provider is ready. Piper requires its selected voice; + ElevenLabs requires its protected API key and voice ID. +- The wake phrase contains one to three words and is no longer than 64 characters. +- At runtime, the gateway is connected and the canonical compose target is idle. + +Manual smoke: + +1. Open **Settings > Voice & Audio** and confirm the assistant mode defaults to Off. +2. Disable each local speech prerequisite in turn. Confirm wake mode cannot be selected and the + status identifies the missing prerequisite. +3. Set a custom wake phrase, leave the page, return, and confirm it persists. +4. Enable **Wake phrase, one request**. Confirm the status becomes Listening and Windows shows + microphone use. +5. Speak unrelated text and confirm no chat request is sent. +6. Speak `, summarize my inbox` as one utterance. Confirm only + `summarize my inbox` is sent. +7. Confirm the matching final response is spoken once, then the status returns to Listening. +8. While waiting or speaking, turn the mode off, disconnect the gateway, or disable a speech + prerequisite. Confirm capture and playback stop and the status becomes Off or Paused. +9. Start push-to-talk while wake listening owns the microphone. Confirm the existing UI reports + microphone contention rather than opening a second capture pipeline. + +Privacy and V0 limits: + +- Enabling the mode keeps the microphone active and locally transcribes detected speech bursts to + find the wake phrase. Only the extracted trailing request is sent through the gateway. +- Audio, ambient transcripts, wake phrases, extracted requests, and response content are not + added to application logs or telemetry. +- The wake phrase and request must be in the same silence-bounded utterance. +- There is no persistent conversation, barge-in, dedicated wake-word model, or overlay. +- Assistant replies are always spoken. **Read responses aloud** continues to control other chat + replies only. + ## Security Features - **URL Validation**: Canvas blocks `file://`, `javascript:`, localhost, private IPs, IPv6 localhost diff --git a/src/OpenClaw.Shared/Models.cs b/src/OpenClaw.Shared/Models.cs index e295118fb..03e014507 100644 --- a/src/OpenClaw.Shared/Models.cs +++ b/src/OpenClaw.Shared/Models.cs @@ -114,6 +114,10 @@ public class OpenClawNotification /// the specific session instead of the default/main session. /// public string? SessionKey { get; set; } + /// Gateway-assigned message identity used for local deduplication and correlation. + public string? OpenClawId { get; set; } + /// Monotonic per-session sequence used for local ordering and correlation. + public int? OpenClawSeq { get; set; } } /// diff --git a/src/OpenClaw.Shared/OpenClawGatewayClient.cs b/src/OpenClaw.Shared/OpenClawGatewayClient.cs index 6208072c4..44b8b8c31 100644 --- a/src/OpenClaw.Shared/OpenClawGatewayClient.cs +++ b/src/OpenClaw.Shared/OpenClawGatewayClient.cs @@ -3317,7 +3317,11 @@ private void HandleChatEvent(JsonElement root, int rawMessageLength) // HIGH 4: log shape only — content previously // surfaced in the operator log. _logger.Info($"Assistant response: role={role} state={state} len={text.Length}"); - EmitChatNotification(text, sessionKey); + EmitChatNotification( + text, + sessionKey, + messageOpenClawMetadata.Id ?? payloadOpenClawMetadata.Id, + messageOpenClawMetadata.Seq ?? payloadOpenClawMetadata.Seq); } } @@ -3353,7 +3357,7 @@ private void HandleChatEvent(JsonElement root, int rawMessageLength) { // HIGH 4: log shape only. _logger.Info($"Assistant response (legacy): role={role} state={state} len={text.Length}"); - EmitChatNotification(text, sessionKey); + EmitChatNotification(text, sessionKey, openClawMetadata.Id, openClawMetadata.Seq); } } } @@ -3553,7 +3557,11 @@ private void EmitRawChatEvent(JsonElement payload) } } - private void EmitChatNotification(string text, string? sessionKey = null) + private void EmitChatNotification( + string text, + string? sessionKey = null, + string? openClawId = null, + int? openClawSeq = null) { var displayText = text.Length > 200 ? text[..200] + "…" : text; var notification = new OpenClawNotification @@ -3561,7 +3569,9 @@ private void EmitChatNotification(string text, string? sessionKey = null) Message = displayText, FullMessage = text, IsChat = true, - SessionKey = sessionKey + SessionKey = sessionKey, + OpenClawId = openClawId, + OpenClawSeq = openClawSeq }; var (title, type) = _categorizer.Classify(notification, _userRules); notification.Title = title; diff --git a/src/OpenClaw.Shared/SettingsData.cs b/src/OpenClaw.Shared/SettingsData.cs index bdcb0f351..af67930ee 100644 --- a/src/OpenClaw.Shared/SettingsData.cs +++ b/src/OpenClaw.Shared/SettingsData.cs @@ -1,8 +1,44 @@ +using System.Text; using System.Text.Json; using System.Text.Json.Serialization; namespace OpenClaw.Shared; +public static class VoiceAssistantSettingsPolicy +{ + public const string OffMode = "off"; + public const string WakeOneShotMode = "wake-one-shot"; + public const string DefaultWakePhrase = "OpenClaw"; + public const int MaxWakePhraseLength = 64; + + public static string NormalizeMode(string? value) => + string.Equals(value, WakeOneShotMode, StringComparison.OrdinalIgnoreCase) + ? WakeOneShotMode + : OffMode; + + public static bool TryNormalizeWakePhrase(string? value, out string normalized) + { + normalized = string.Join( + ' ', + (value ?? string.Empty) + .Normalize(NormalizationForm.FormKC) + .Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries)); + + if (normalized.Length is 0 or > MaxWakePhraseLength) + return false; + + var words = normalized.Split(' '); + if (words.Length is < 1 or > 3) + return false; + + return words.All(word => + word.Any(char.IsLetterOrDigit) && + word.All(character => + char.IsLetterOrDigit(character) || + character is '\'' or '\u2019' or '-')); + } +} + /// /// Serializable settings data model. Used for JSON round-trip persistence. /// @@ -87,6 +123,10 @@ public record class SettingsData public bool ShowChatToolCalls { get; set; } = true; /// Play audio feedback chimes on listen start/stop. public bool VoiceAudioFeedback { get; set; } = true; + /// Local assistant mode. Unknown values normalize to off. + public string VoiceAssistantMode { get; set; } = VoiceAssistantSettingsPolicy.OffMode; + /// Wake phrase used by the local transcript gate. + public string VoiceAssistantWakePhrase { get; set; } = VoiceAssistantSettingsPolicy.DefaultWakePhrase; public bool NodeTtsEnabled { get; set; } = false; public string TtsProvider { get; set; } = OpenClaw.Shared.Capabilities.TtsCapability.PiperProvider; /// Persisted: whether the Hub's NavigationView pane is expanded diff --git a/src/OpenClaw.Tray.WinUI/App.xaml.cs b/src/OpenClaw.Tray.WinUI/App.xaml.cs index 11539ddc3..2db700018 100644 --- a/src/OpenClaw.Tray.WinUI/App.xaml.cs +++ b/src/OpenClaw.Tray.WinUI/App.xaml.cs @@ -13,6 +13,7 @@ using OpenClawTray.Dialogs; using OpenClawTray.Helpers; using OpenClawTray.Services; +using OpenClawTray.Services.VoiceAssistant; using OpenClawTray.Windows; using OpenClaw.Connection; using Microsoft.Extensions.DependencyInjection; @@ -41,7 +42,10 @@ namespace OpenClawTray; -public partial class App : Application, OpenClawTray.Services.IAppCommands +public partial class App : + Application, + OpenClawTray.Services.IAppCommands, + IVoiceAssistantSettingsEnvironment { internal static readonly UpdatumManager AppUpdater = new("openclaw", "openclaw-windows-node") { @@ -56,6 +60,10 @@ public partial class App : Application, OpenClawTray.Services.IAppCommands private OpenClawTray.Services.ManagedLocalGatewayAutoRepairMonitor? _managedLocalAutoRepairMonitor; private ManagedLocalGatewayPortProvenanceService? _managedLocalPortProvenance; private OpenClawTray.Chat.OpenClawChatCoordinator? _chatCoordinator; + private VoiceAssistantCoordinator? _voiceAssistantCoordinator; + private VoiceAssistantInput? _voiceAssistantInput; + private VoiceAssistantChatTurnClient? _voiceAssistantChatTurnClient; + private readonly SemaphoreSlim _voiceAssistantLifecycleGate = new(1, 1); /// /// Root DI composition root, built once during startup and disposed during @@ -75,6 +83,7 @@ public partial class App : Application, OpenClawTray.Services.IAppCommands new Dictionary { [typeof(Pages.SettingsPage)] = typeof(SettingsPageViewModel), + [typeof(Pages.VoiceSettingsPage)] = typeof(VoiceAssistantSettingsViewModel), }; /// The root service provider, or null before startup / after shutdown. @@ -469,7 +478,7 @@ private void InitializeServiceProvider() } var dispatcher = new WinUIDispatcher(_dispatcherQueue); - var context = new AppServiceContext(dispatcher, this, _settings); + var context = new AppServiceContext(dispatcher, this, _settings, this); var services = new ServiceCollection(); services.AddOpenClawTrayCore(context); @@ -2128,7 +2137,15 @@ private bool TryStartLocalMcpOnlyNode() /// Handles the connection manager's OperatorClientChanged event. /// Re-wires all 27 data event handlers from the old client to the new one. /// - private void OnOperatorClientChanged(object? sender, OperatorClientChangedEventArgs e) + private void OnOperatorClientChanged(object? sender, OperatorClientChangedEventArgs e) => + AsyncEventHandlerGuard.Run( + () => OnOperatorClientChangedAsync(sender, e), + new AppLogger(), + nameof(OnOperatorClientChanged)); + + private async Task OnOperatorClientChangedAsync( + object? sender, + OperatorClientChangedEventArgs e) { if (_dispatcherQueue is { HasThreadAccess: false } dispatcher) { @@ -2141,6 +2158,7 @@ private void OnOperatorClientChanged(object? sender, OperatorClientChangedEventA // Delegate all 27 event subscriptions to GatewayService _gatewayService?.AttachClient(e.NewClient, e.OldClient); + await DisposeVoiceAssistantCoordinatorAsync(); // Configure new client if (e.NewClient is { } client) @@ -2160,6 +2178,9 @@ private void OnOperatorClientChanged(object? sender, OperatorClientChangedEventA } RaiseChatProviderChanged(); + ObserveBackgroundFault( + RebuildVoiceAssistantCoordinatorAsync(), + "Voice assistant failed to rebuild after provider replacement"); // Update UI references if (_appState != null) @@ -2270,6 +2291,9 @@ private void OnManagerStateChanged(object? sender, GatewayConnectionSnapshot sna _nodeService.InvokeCompleted += OnNodeInvokeCompleted; _nodeService.ToolTelemetryCompleted += OnNodeToolTelemetryCompleted; _nodeService.GatewaySelfUpdated += _gatewayService.OnGatewaySelfUpdated; + ObserveBackgroundFault( + RebuildVoiceAssistantCoordinatorAsync(), + "Voice assistant failed to rebuild after node startup"); return _nodeService; } catch (Exception ex) @@ -2782,6 +2806,12 @@ private void OnGatewayConnectionStatusChanged(object? sender, ConnectionStatus s { UpdateStatusDetailWindow(); }); + if (_voiceAssistantCoordinator is { } voiceAssistant) + { + ObserveBackgroundFault( + voiceAssistant.ReconcileAsync(), + "Voice assistant failed to reconcile gateway state"); + } } /// @@ -2890,6 +2920,8 @@ private void OnGatewayNotificationReceived(object? sender, OpenClawNotification if (notification.IsChat && !string.IsNullOrEmpty(notification.Message)) { var speechText = ChatNotificationSpeechText.Resolve(notification); + var claimedByVoiceAssistant = + _voiceAssistantCoordinator?.TryClaimResponse(notification) == true; // Suppress TTS/voice overlay when the user has aborted the response. if (ChatProvider?.IsResponseSuppressed == true) @@ -2909,7 +2941,8 @@ private void OnGatewayNotificationReceived(object? sender, OpenClawNotification // } // TTS: read response aloud whenever chat TTS is enabled and ready (any chat surface). - if (SpeechSetupReadiness.IsAutomaticChatTtsEnabled(_settings)) + if (!claimedByVoiceAssistant && + SpeechSetupReadiness.IsAutomaticChatTtsEnabled(_settings)) { _ = (_chatCoordinator?.SpeakResponseAsync(speechText) ?? Task.CompletedTask); } @@ -3478,6 +3511,9 @@ private void OnSettingsSaved(object? sender, EventArgs e) SyncActiveGatewayBrowserProxyForward(); Logger.Info($"[SETTINGS] Change impact: {impact}"); PublishSandboxRiskNotificationIfNeeded(); + ObserveBackgroundFault( + ApplyVoiceAssistantSettingsAsync(), + "Voice assistant failed to apply settings"); switch (impact) { @@ -4600,6 +4636,8 @@ private async Task ExitApplicationAsync() _globalHotkey = null; }); + await SafeShutdownStepAsync("voice assistant", DisposeVoiceAssistantCoordinatorAsync); + // Stop chat first so provider event handlers cannot drain client-only // queued prompts while the gateway connection is shutting down. SafeShutdownStep("chat coordinator", () => @@ -4719,6 +4757,163 @@ await SafeShutdownStepAsync("service provider", async () => Exit(); } + private async Task RebuildVoiceAssistantCoordinatorAsync() + { + await _voiceAssistantLifecycleGate.WaitAsync().ConfigureAwait(false); + try + { + await DisposeVoiceAssistantCoordinatorCoreAsync().ConfigureAwait(false); + + var settings = _settings; + var chatCoordinator = _chatCoordinator; + var provider = chatCoordinator?.Provider; + if (_isExiting || + settings is null || + chatCoordinator is null || + provider is null || + !string.Equals( + settings.VoiceAssistantMode, + VoiceAssistantSettingsPolicy.WakeOneShotMode, + StringComparison.Ordinal)) + { + return; + } + + var voiceService = VoiceService ?? EnsureStandaloneVoiceService(); + if (voiceService is null) + return; + + var input = new VoiceAssistantInput(voiceService); + var chat = new VoiceAssistantChatTurnClient(provider); + var coordinator = new VoiceAssistantCoordinator( + input, + chat, + new VoiceAssistantSpeaker(chatCoordinator), + () => + { + var readiness = SpeechSetupReadiness.GetVoiceAssistantReadiness(settings); + return new VoiceAssistantConfiguration( + Enabled: string.Equals( + settings.VoiceAssistantMode, + VoiceAssistantSettingsPolicy.WakeOneShotMode, + StringComparison.Ordinal), + LocalPrerequisitesReady: readiness.IsReady, + WakePhrase: settings.VoiceAssistantWakePhrase); + }); + + _voiceAssistantInput = input; + _voiceAssistantChatTurnClient = chat; + _voiceAssistantCoordinator = coordinator; + coordinator.StateChanged += OnVoiceAssistantStateChanged; + await coordinator.ReconcileAsync().ConfigureAwait(false); + RaiseVoiceAssistantSettingsChanged(); + } + finally + { + _voiceAssistantLifecycleGate.Release(); + } + } + + private async Task ApplyVoiceAssistantSettingsAsync() + { + var needsBuild = false; + await _voiceAssistantLifecycleGate.WaitAsync().ConfigureAwait(false); + try + { + if (_voiceAssistantCoordinator is not null) + { + await _voiceAssistantCoordinator.ReconcileAsync().ConfigureAwait(false); + RaiseVoiceAssistantSettingsChanged(); + return; + } + + needsBuild = + !_isExiting && + string.Equals( + _settings?.VoiceAssistantMode, + VoiceAssistantSettingsPolicy.WakeOneShotMode, + StringComparison.Ordinal); + } + finally + { + _voiceAssistantLifecycleGate.Release(); + } + + if (needsBuild) + await RebuildVoiceAssistantCoordinatorAsync().ConfigureAwait(false); + else + RaiseVoiceAssistantSettingsChanged(); + } + + private async Task DisposeVoiceAssistantCoordinatorAsync() + { + await _voiceAssistantLifecycleGate.WaitAsync().ConfigureAwait(false); + try + { + await DisposeVoiceAssistantCoordinatorCoreAsync().ConfigureAwait(false); + } + finally + { + _voiceAssistantLifecycleGate.Release(); + } + } + + private async Task DisposeVoiceAssistantCoordinatorCoreAsync() + { + var coordinator = _voiceAssistantCoordinator; + _voiceAssistantCoordinator = null; + if (coordinator is not null) + { + coordinator.StateChanged -= OnVoiceAssistantStateChanged; + await coordinator.DisposeAsync().ConfigureAwait(false); + } + + _voiceAssistantChatTurnClient?.Dispose(); + _voiceAssistantChatTurnClient = null; + _voiceAssistantInput?.Dispose(); + _voiceAssistantInput = null; + RaiseVoiceAssistantSettingsChanged(); + } + + private event EventHandler? VoiceAssistantSettingsChanged; + internal event EventHandler? VoiceAssistantRuntimeStateChanged; + + internal VoiceAssistantState VoiceAssistantRuntimeState => + _voiceAssistantCoordinator?.State ?? + (string.Equals( + _settings?.VoiceAssistantMode, + VoiceAssistantSettingsPolicy.WakeOneShotMode, + StringComparison.Ordinal) + ? VoiceAssistantState.Unavailable + : VoiceAssistantState.Off); + + event EventHandler? IVoiceAssistantSettingsEnvironment.Changed + { + add => VoiceAssistantSettingsChanged += value; + remove => VoiceAssistantSettingsChanged -= value; + } + + VoiceAssistantState IVoiceAssistantSettingsEnvironment.RuntimeState => + VoiceAssistantRuntimeState; + + VoiceAssistantReadinessResult IVoiceAssistantSettingsEnvironment.GetReadiness(string wakePhrase) => + _settings is { } settings + ? SpeechSetupReadiness.GetVoiceAssistantReadiness(settings, wakePhrase) + : new VoiceAssistantReadinessResult(false, VoiceAssistantReadinessReason.SttDisabled); + + string IVoiceAssistantSettingsEnvironment.GetString(string key) => + LocalizationHelper.GetString(key); + + private void OnVoiceAssistantStateChanged(VoiceAssistantState state) => + RaiseVoiceAssistantSettingsChanged(); + + private void RaiseVoiceAssistantSettingsChanged() => + OnUiThread(() => + { + VoiceAssistantSettingsChanged?.Invoke(this, EventArgs.Empty); + VoiceAssistantRuntimeStateChanged?.Invoke(this, EventArgs.Empty); + }); + private static void CloseWindow(Window? window) { try diff --git a/src/OpenClaw.Tray.WinUI/Chat/OpenClawChatCoordinator.cs b/src/OpenClaw.Tray.WinUI/Chat/OpenClawChatCoordinator.cs index 54a075db7..a3ef8b381 100644 --- a/src/OpenClaw.Tray.WinUI/Chat/OpenClawChatCoordinator.cs +++ b/src/OpenClaw.Tray.WinUI/Chat/OpenClawChatCoordinator.cs @@ -118,7 +118,20 @@ public void StopSpeaking() public Task SpeakResponseAsync(string text) => SpeakConfiguredTextAsync(text, muteVoiceCapture: true, bypassMute: false); - private async Task SpeakConfiguredTextAsync(string text, bool muteVoiceCapture, bool bypassMute = false) + public Task SpeakAssistantResponseAsync(string text, CancellationToken cancellationToken) => + SpeakConfiguredTextAsync( + text, + muteVoiceCapture: false, + bypassMute: true, + cancellationToken, + propagateFailure: true); + + private async Task SpeakConfiguredTextAsync( + string text, + bool muteVoiceCapture, + bool bypassMute = false, + CancellationToken cancellationToken = default, + bool propagateFailure = false) { if (!bypassMute && IsMuted) return; var voiceService = _nodeServiceAccessor()?.VoiceService; @@ -146,11 +159,17 @@ private async Task SpeakConfiguredTextAsync(string text, bool muteVoiceCapture, var ttsService = _nodeServiceAccessor()?.TextToSpeech ?? GetFallbackTextToSpeechService(); - await ttsService.SpeakAsync(speakArgs).ConfigureAwait(false); + await ttsService.SpeakAsync(speakArgs, cancellationToken).ConfigureAwait(false); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; } catch (Exception ex) { _logger.Warn($"TTS response playback failed: {ex.Message}"); + if (propagateFailure) + throw; } finally { diff --git a/src/OpenClaw.Tray.WinUI/Chat/OpenClawChatDataProvider.cs b/src/OpenClaw.Tray.WinUI/Chat/OpenClawChatDataProvider.cs index eeb155011..20913aae6 100644 --- a/src/OpenClaw.Tray.WinUI/Chat/OpenClawChatDataProvider.cs +++ b/src/OpenClaw.Tray.WinUI/Chat/OpenClawChatDataProvider.cs @@ -11,6 +11,7 @@ using OpenClawTray.Helpers; #endif using OpenClawTray.Services; +using OpenClawTray.Services.VoiceAssistant; namespace OpenClawTray.Chat; @@ -147,6 +148,7 @@ public sealed class OpenClawChatDataProvider : IChatDataProvider private readonly Dictionary> _localSentTexts = new(); private readonly Dictionary> _queuedMessages = new(); private readonly Dictionary> _queuedSendRequests = new(); + private readonly Dictionary _voiceAssistantMessageIds = new(StringComparer.Ordinal); private readonly Dictionary> _queuedMessageIdsByRunId = new(); private readonly Dictionary> _terminalRunIdsByThread = new(); private readonly HashSet _queuedDrainScheduledThreads = new(StringComparer.Ordinal); @@ -185,6 +187,12 @@ private sealed record QueuedSendDispatch( long StartedRunStartSequence, ChatTelemetryTracker.QueuePhaseCompletion? QueueCompletion, bool StartedDirectly); + internal sealed record VoiceAssistantResponseIdentity( + string LocalMessageId, + string SessionKey, + string? GatewayMessageId, + int? GatewaySequence, + string ResponseText); private enum AssistantQueueFrameDisposition { Render, @@ -223,6 +231,23 @@ private enum AssistantQueueFrameDisposition public string DisplayName => "OpenClaw gateway"; + internal event Action? VoiceAssistantResponseObserved; + + internal string? GetVoiceAssistantReadySessionKey() + { + lock (_gate) + { + var sessionKey = _bridge.MainSessionKey; + return _bridge.HasHandshakeSnapshot && + _status == ConnectionStatus.Connected && + _sessionsListReceived && + !string.IsNullOrWhiteSpace(sessionKey) && + CanSendDirectlyLocked(sessionKey) + ? sessionKey + : null; + } + } + /// Last-known chat state from a previous session, used for pre-connection UI. internal LastChatState? CachedLastChatState => _lastChatState; @@ -450,6 +475,117 @@ public async Task SendMessageAsync(string threadId, string message, Cancellation await DispatchQueuedSendAsync(dispatch, rethrow: true, cancellationToken); } + internal async Task SendVoiceAssistantMessageAsync( + string threadId, + string message, + CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + if (string.IsNullOrWhiteSpace(threadId)) + throw new ArgumentException("Thread id is required.", nameof(threadId)); + if (string.IsNullOrWhiteSpace(message)) + throw new ArgumentException("Message is required.", nameof(message)); + + var trimmed = message.Trim(); + var nonce = Guid.NewGuid().ToString("N"); + ChatDataSnapshot snapshot; + QueuedSendDispatch? dispatch; + QueuedSendRequest request; + bool sendDirectly; + int? preSendSequence; + + lock (_gate) + { + ObjectDisposedException.ThrowIf(_disposed, this); + var messageId = $"q{++_queuedMessageSequence}"; + request = new QueuedSendRequest( + messageId, + Guid.NewGuid().ToString(), + threadId, + trimmed, + trimmed, + nonce, + Attachments: null); + preSendSequence = GetLatestGatewaySequenceLocked(threadId); + sendDirectly = CanSendDirectlyLocked(threadId); + _telemetry.StartLocalTurn(request.Id, threadId, queued: !sendDirectly); + + if (sendDirectly) + { + _abortedThreads.Remove(threadId); + _pendingAbortCounts.Remove(threadId); + dispatch = StartDirectSendLocked(request); + _voiceAssistantMessageIds[threadId] = request.Id; + } + else + { + AddQueuedMessageLocked(threadId, new ChatQueuedMessage( + request.Id, + request.DisplayText, + DateTimeOffset.UtcNow, + request.LocalNonce)); + AddQueuedSendRequestLocked(request); + dispatch = null; + } + + snapshot = BuildSnapshotLocked(); + } + + Publish(snapshot); + if (dispatch is not null) + { + try + { + await DispatchQueuedSendAsync(dispatch, rethrow: true, cancellationToken).ConfigureAwait(false); + } + catch + { + lock (_gate) + { + if (_voiceAssistantMessageIds.TryGetValue(threadId, out var trackedId) && + string.Equals(trackedId, request.Id, StringComparison.Ordinal)) + { + _voiceAssistantMessageIds.Remove(threadId); + } + } + throw; + } + } + + return new VoiceAssistantTurnReceipt( + sendDirectly ? VoiceAssistantSendDisposition.Direct : VoiceAssistantSendDisposition.Queued, + threadId, + request.Id, + request.SendRunId, + preSendSequence); + } + + internal async Task CancelVoiceAssistantTurnAsync( + VoiceAssistantTurnReceipt receipt, + CancellationToken cancellationToken) + { + lock (_gate) + { + if (_voiceAssistantMessageIds.TryGetValue(receipt.SessionKey, out var trackedId) && + string.Equals(trackedId, receipt.LocalMessageId, StringComparison.Ordinal)) + { + _voiceAssistantMessageIds.Remove(receipt.SessionKey); + } + } + + if (receipt.Disposition == VoiceAssistantSendDisposition.Queued) + { + await CancelQueuedMessageAsync( + receipt.SessionKey, + receipt.LocalMessageId, + cancellationToken).ConfigureAwait(false); + } + else + { + await StopResponseAsync(receipt.SessionKey, cancellationToken).ConfigureAwait(false); + } + } + internal Task EnqueueCompactCommandAsync( string threadId, CancellationToken cancellationToken = default) @@ -2690,8 +2826,18 @@ private void OnChatMessageReceived(object? sender, ChatMessageInfo message) if (message.IsFinal) { ChatTelemetryTracker.PreparedTurnCompletion? turnCompletion = null; + VoiceAssistantResponseIdentity? voiceAssistantResponse = null; lock (_gate) { + if (_voiceAssistantMessageIds.Remove(threadId, out var localMessageId)) + { + voiceAssistantResponse = new VoiceAssistantResponseIdentity( + localMessageId, + threadId, + message.OpenClawId, + message.OpenClawSeq, + rawText); + } if (_activeRunIds.Remove(threadId, out var completedRunId)) { turnCompletion = _telemetry.PrepareFinishByRunId( @@ -2706,6 +2852,8 @@ private void OnChatMessageReceived(object? sender, ChatMessageInfo message) if (!HasSendingQueuedMessagesLocked(threadId)) _locallyInitiatedThreads.Remove(threadId); } + if (voiceAssistantResponse is not null) + VoiceAssistantResponseObserved?.Invoke(voiceAssistantResponse); _telemetry.CompletePreparedTurn(turnCompletion); SnapshotLatestAssistantUsage(threadId); ApplyEventAndPublish(threadId, new ChatTurnEndEvent()); @@ -3361,6 +3509,20 @@ private bool CanSendDirectlyLocked(string threadId) return !HasPendingQueuedMessagesLocked(threadId); } + private int? GetLatestGatewaySequenceLocked(string threadId) + { + if (!_entryMeta.TryGetValue(threadId, out var metadata)) + return null; + + int? latest = null; + foreach (var item in metadata.Values) + { + if (item.OpenClawSeq is { } sequence && (latest is null || sequence > latest)) + latest = sequence; + } + return latest; + } + private bool CanClearAssistantFallbackPromotionLocked(string threadId) { if (HasSendingQueuedMessagesLocked(threadId)) diff --git a/src/OpenClaw.Tray.WinUI/Chat/OpenClawReactorChatRoot.cs b/src/OpenClaw.Tray.WinUI/Chat/OpenClawReactorChatRoot.cs index 06fbb94a4..c4e60a156 100644 --- a/src/OpenClaw.Tray.WinUI/Chat/OpenClawReactorChatRoot.cs +++ b/src/OpenClaw.Tray.WinUI/Chat/OpenClawReactorChatRoot.cs @@ -31,6 +31,8 @@ public sealed record OpenClawReactorChatRootProps( Action? OnSettingsClick = null, Action? OnSpeakerMuteChanged = null, Func>? ConfirmResetAsync = null, + string? InitialVoiceUnavailableReason = null, + string? InitialVoiceUnavailableStatus = null, bool InitialMuted = false, bool IsCompact = false); @@ -74,6 +76,10 @@ public override Element Render() var (speakerMuted, setSpeakerMuted) = UseState(props.InitialMuted, threadSafe: true); var (voiceTranscript, setVoiceTranscript) = UseState(null, threadSafe: true); var (voiceAudioLevel, setVoiceAudioLevel) = UseState(0f, threadSafe: true); + var (voiceUnavailableReason, setVoiceUnavailableReason) = + UseState(props.InitialVoiceUnavailableReason, threadSafe: true); + var (voiceUnavailableStatus, setVoiceUnavailableStatus) = + UseState(props.InitialVoiceUnavailableStatus, threadSafe: true); var (scrollToBottomToken, setScrollToBottomToken) = UseState(0, threadSafe: true); var (showToolCalls, setShowToolCalls) = UseState(s_showToolCalls, threadSafe: true); var (toolCallsCollapseVersion, setToolCallsCollapseVersion) = @@ -93,6 +99,11 @@ void UpdatePendingAttachments(IReadOnlyList attachments) }; props.HostCallbacks.SetVoiceTranscript = setVoiceTranscript; props.HostCallbacks.SetVoiceAudioLevel = setVoiceAudioLevel; + props.HostCallbacks.SetVoiceAvailability = (reason, status) => + { + setVoiceUnavailableReason(reason); + setVoiceUnavailableStatus(status); + }; props.HostCallbacks.SetSpeakerMuted = setSpeakerMuted; UseEffect((Func)(() => () => props.HostCallbacks.Clear()), props.HostCallbacks); @@ -352,6 +363,8 @@ void SelectThread(string threadId) level => RunFireAndForget(ct => props.Provider.SetThinkingLevelAsync(effectiveThread.Id, level, ct)), allowAll => RunFireAndForget(ct => props.Provider.SetPermissionModeAsync(effectiveThread.Id, allowAll, ct)), props.OnVoiceRequest, + voiceUnavailableReason, + voiceUnavailableStatus, props.OnAttachClick, speakerMuted, () => @@ -564,6 +577,8 @@ public sealed record ReactorChatComposerProps( Action OnThinkingLevelChanged, Action OnPermissionsChanged, Func>? OnVoiceRequest, + string? VoiceUnavailableReason, + string? VoiceUnavailableStatus, Action? OnAttachClick, bool IsSpeakerMuted, Action OnSpeakerToggle, @@ -816,7 +831,7 @@ public override Element Render() void StartVoiceRecording() { - if (props.OnVoiceRequest is null || isRecording) + if (props.OnVoiceRequest is null || props.VoiceUnavailableReason is not null || isRecording) return; var cancellation = new CancellationTokenSource(); @@ -1389,7 +1404,8 @@ void CommitSlashText(string value, ReactorSlashMenuState nextState) : "\uE720", isRecording ? Localized("Chat_Composer_Tooltip_Stop", "Stop") - : Localized("Chat_Composer_Tooltip_Voice", "Voice"), + : props.VoiceUnavailableReason + ?? Localized("Chat_Composer_Tooltip_Voice", "Voice"), () => { if (isRecording) @@ -1400,7 +1416,7 @@ void CommitSlashText(string value, ReactorSlashMenuState nextState) else StartVoiceRecording(); }, - props.OnVoiceRequest is not null); + props.OnVoiceRequest is not null && props.VoiceUnavailableReason is null); var speakerButton = IconButton( props.IsSpeakerMuted ? "\uE74F" : "\uE767", props.IsSpeakerMuted ? "Unmute" : "Mute", @@ -1451,6 +1467,42 @@ void CommitSlashText(string value, ReactorSlashMenuState nextState) rightToolbar.Grid(row: 0, column: 1)); var composerChildren = new List(); + if (props.VoiceUnavailableStatus is { } voiceUnavailableStatus) + { + composerChildren.Add( + HStack( + 6, + TextBlock("\uE720").Set(textBlock => + { + textBlock.FontFamily = FluentIconCatalog.SymbolThemeFontFamily; + textBlock.FontSize = 14; + ApplyTheme(textBlock, () => + textBlock.Foreground = ResolveThemeBrush( + "TextFillColorSecondaryBrush", + textBlock.ActualTheme)); + }), + TextBlock(voiceUnavailableStatus).Set(textBlock => + { + textBlock.FontSize = 12; + textBlock.TextWrapping = TextWrapping.Wrap; + ApplyTheme(textBlock, () => + textBlock.Foreground = ResolveThemeBrush( + "TextFillColorSecondaryBrush", + textBlock.ActualTheme)); + })) + .Set(panel => + { + Microsoft.UI.Xaml.Automation.AutomationProperties.SetName( + panel, + props.VoiceUnavailableReason ?? voiceUnavailableStatus); + Microsoft.UI.Xaml.Automation.AutomationProperties.SetHelpText( + panel, + props.VoiceUnavailableReason ?? voiceUnavailableStatus); + Microsoft.UI.Xaml.Automation.AutomationProperties.SetLiveSetting( + panel, + Microsoft.UI.Xaml.Automation.Peers.AutomationLiveSetting.Polite); + })); + } if (isRecording) composerChildren.Add(voiceFeedback); if (attachmentRows.Length > 0) diff --git a/src/OpenClaw.Tray.WinUI/Chat/ReactorChatHostExtensions.cs b/src/OpenClaw.Tray.WinUI/Chat/ReactorChatHostExtensions.cs index a93140863..891f16a2f 100644 --- a/src/OpenClaw.Tray.WinUI/Chat/ReactorChatHostExtensions.cs +++ b/src/OpenClaw.Tray.WinUI/Chat/ReactorChatHostExtensions.cs @@ -37,6 +37,8 @@ public static MountedReactorChat MountReactorChat( Action? onAttachClick = null, Action? onSettingsClick = null, Action? onSpeakerMuteChanged = null, + string? initialVoiceUnavailableReason = null, + string? initialVoiceUnavailableStatus = null, bool initialMuted = false, bool isCompact = false) { @@ -83,6 +85,8 @@ async Task ConfirmResetAsync(string sessionKey, string? displayName) onSettingsClick, onSpeakerMuteChanged, ConfirmResetAsync, + initialVoiceUnavailableReason, + initialVoiceUnavailableStatus, initialMuted, isCompact); var host = new ReactorHostControl(); @@ -112,6 +116,9 @@ public void SetVoiceTranscript(string? text) => public void SetVoiceAudioLevel(float level) => callbacks.SetVoiceAudioLevel?.Invoke(level); + public void SetVoiceAvailability(string? unavailableReason, string? unavailableStatus) => + callbacks.SetVoiceAvailability?.Invoke(unavailableReason, unavailableStatus); + public void TriggerVoiceRecording() => callbacks.TriggerVoiceRecording?.Invoke(); @@ -134,6 +141,7 @@ public sealed class ReactorChatHostCallbacks public Action>? AttachFiles { get; set; } public Action? SetVoiceTranscript { get; set; } public Action? SetVoiceAudioLevel { get; set; } + public Action? SetVoiceAvailability { get; set; } public Action? TriggerVoiceRecording { get; set; } public Action? SetSpeakerMuted { get; set; } @@ -142,6 +150,7 @@ public void Clear() AttachFiles = null; SetVoiceTranscript = null; SetVoiceAudioLevel = null; + SetVoiceAvailability = null; TriggerVoiceRecording = null; SetSpeakerMuted = null; } diff --git a/src/OpenClaw.Tray.WinUI/Pages/ChatPage.xaml.cs b/src/OpenClaw.Tray.WinUI/Pages/ChatPage.xaml.cs index 8eb325181..c8fc308b0 100644 --- a/src/OpenClaw.Tray.WinUI/Pages/ChatPage.xaml.cs +++ b/src/OpenClaw.Tray.WinUI/Pages/ChatPage.xaml.cs @@ -7,6 +7,7 @@ using OpenClawTray.Chat; using OpenClawTray.Helpers; using OpenClawTray.Services; +using OpenClawTray.Services.VoiceAssistant; using OpenClawTray.Windows; using OpenClaw.Connection; using System; @@ -80,7 +81,10 @@ private void OnUnloaded(object sender, RoutedEventArgs e) app.ChatProviderChanged -= OnAppChatProviderChanged; if (App.Current is App app2) + { app2.SpeakerMuteChanged -= OnSpeakerMuteChanged; + app2.VoiceAssistantRuntimeStateChanged -= OnVoiceAssistantRuntimeStateChanged; + } // MEDIUM 6: detach the static debug-override subscription so that // an unloaded ChatPage doesn't keep responding to overrides changes @@ -144,6 +148,8 @@ public void Initialize() app.ChatProviderChanged += OnAppChatProviderChanged; app.SpeakerMuteChanged -= OnSpeakerMuteChanged; app.SpeakerMuteChanged += OnSpeakerMuteChanged; + app.VoiceAssistantRuntimeStateChanged -= OnVoiceAssistantRuntimeStateChanged; + app.VoiceAssistantRuntimeStateChanged += OnVoiceAssistantRuntimeStateChanged; } // Also react to the per-surface debug override picked from DebugPage. @@ -162,6 +168,24 @@ private void OnSpeakerMuteChanged(bool muted) DispatcherQueue?.TryEnqueue(() => _reactorHost?.SetSpeakerMuted(muted)); } + private void OnVoiceAssistantRuntimeStateChanged(object? sender, EventArgs e) + { + DispatcherQueue?.TryEnqueue(UpdateVoiceAvailability); + } + + private void UpdateVoiceAvailability() + { + var unavailable = VoiceAssistantChatInputPolicy.IsVoiceInputUnavailable( + CurrentApp.VoiceAssistantRuntimeState); + _reactorHost?.SetVoiceAvailability( + unavailable + ? LocalizationHelper.GetString(VoiceAssistantChatInputPolicy.WakeListeningReasonKey) + : null, + unavailable + ? LocalizationHelper.GetString(VoiceAssistantChatInputPolicy.WakeListeningStatusKey) + : null); + } + private void OnAppChatProviderChanged(object? sender, EventArgs e) { var dispatcher = DispatcherQueue; @@ -260,6 +284,7 @@ private void ShowReactorSurface() { PlaceholderPanel.Visibility = Visibility.Collapsed; ChatHost.Visibility = Visibility.Visible; + UpdateVoiceAvailability(); UpdateNativeChatSurfaceActive(); // Check for pending auto-start voice even when already mounted if (_hub?.PendingAutoStartVoice == true) @@ -298,6 +323,14 @@ private void ShowReactorSurface() onAttachClick: OnAttachClicked, onSettingsClick: () => _hub?.NavigateTo("voice"), onSpeakerMuteChanged: muted => _ = OnSpeakerMuteChangedAsync(muted), + initialVoiceUnavailableReason: VoiceAssistantChatInputPolicy.IsVoiceInputUnavailable( + CurrentApp.VoiceAssistantRuntimeState) + ? LocalizationHelper.GetString(VoiceAssistantChatInputPolicy.WakeListeningReasonKey) + : null, + initialVoiceUnavailableStatus: VoiceAssistantChatInputPolicy.IsVoiceInputUnavailable( + CurrentApp.VoiceAssistantRuntimeState) + ? LocalizationHelper.GetString(VoiceAssistantChatInputPolicy.WakeListeningStatusKey) + : null, initialMuted: ShouldStartSpeakerMuted(CurrentApp.Settings)); _mountedProvider = provider; _mountedThreadId = threadIdToMount; diff --git a/src/OpenClaw.Tray.WinUI/Pages/VoiceSettingsPage.xaml b/src/OpenClaw.Tray.WinUI/Pages/VoiceSettingsPage.xaml index f86c6c90d..d92b5fe72 100644 --- a/src/OpenClaw.Tray.WinUI/Pages/VoiceSettingsPage.xaml +++ b/src/OpenClaw.Tray.WinUI/Pages/VoiceSettingsPage.xaml @@ -285,6 +285,50 @@ + + + + + + + + + + + + + + + diff --git a/src/OpenClaw.Tray.WinUI/Pages/VoiceSettingsPage.xaml.cs b/src/OpenClaw.Tray.WinUI/Pages/VoiceSettingsPage.xaml.cs index 6d86d9938..5cb9c5e5c 100644 --- a/src/OpenClaw.Tray.WinUI/Pages/VoiceSettingsPage.xaml.cs +++ b/src/OpenClaw.Tray.WinUI/Pages/VoiceSettingsPage.xaml.cs @@ -4,6 +4,7 @@ using OpenClaw.Shared; using OpenClaw.Shared.Capabilities; using OpenClawTray.Helpers; +using OpenClawTray.Presentation; using OpenClawTray.Services; using OpenClawTray.Windows; using System; @@ -27,10 +28,12 @@ private static string Lf(string key, params object?[] args) => private CancellationTokenSource? _whisperDownloadCts; private CancellationTokenSource? _piperDownloadCts; private bool _piperPreviewInProgress; + private VoiceAssistantSettingsViewModel? _assistantViewModel; public VoiceSettingsPage() { InitializeComponent(); + DataContextChanged += OnDataContextChanged; Loaded += (_, _) => { UpdateModelStatus(); @@ -50,6 +53,26 @@ public VoiceSettingsPage() }; } + private void OnDataContextChanged(FrameworkElement sender, DataContextChangedEventArgs args) + { + _assistantViewModel = args.NewValue as VoiceAssistantSettingsViewModel; + _assistantViewModel?.Refresh(); + } + + private void OnAssistantWakePhraseLostFocus(object sender, RoutedEventArgs e) => + _assistantViewModel?.CommitWakePhrase(); + + private void OnAssistantWakePhraseKeyDown( + object sender, + Microsoft.UI.Xaml.Input.KeyRoutedEventArgs e) + { + if (e.Key != global::Windows.System.VirtualKey.Enter) + return; + + _assistantViewModel?.CommitWakePhrase(); + e.Handled = true; + } + public void Initialize(VoiceService? voiceService) { _voiceService = voiceService; diff --git a/src/OpenClaw.Tray.WinUI/Presentation/AppServiceContext.cs b/src/OpenClaw.Tray.WinUI/Presentation/AppServiceContext.cs index 3326a2ac7..be95270c7 100644 --- a/src/OpenClaw.Tray.WinUI/Presentation/AppServiceContext.cs +++ b/src/OpenClaw.Tray.WinUI/Presentation/AppServiceContext.cs @@ -11,13 +11,36 @@ namespace OpenClawTray.Presentation; internal sealed class AppServiceContext { public AppServiceContext(IUiDispatcher dispatcher, IAppCommands appCommands, SettingsManager settings) + : this(dispatcher, appCommands, settings, UnavailableVoiceAssistantEnvironment.Instance) + { + } + + public AppServiceContext( + IUiDispatcher dispatcher, + IAppCommands appCommands, + SettingsManager settings, + IVoiceAssistantSettingsEnvironment voiceAssistantEnvironment) { Dispatcher = dispatcher ?? throw new ArgumentNullException(nameof(dispatcher)); AppCommands = appCommands ?? throw new ArgumentNullException(nameof(appCommands)); Settings = settings ?? throw new ArgumentNullException(nameof(settings)); + VoiceAssistantEnvironment = voiceAssistantEnvironment ?? + throw new ArgumentNullException(nameof(voiceAssistantEnvironment)); } public IUiDispatcher Dispatcher { get; } public IAppCommands AppCommands { get; } public SettingsManager Settings { get; } + public IVoiceAssistantSettingsEnvironment VoiceAssistantEnvironment { get; } + + private sealed class UnavailableVoiceAssistantEnvironment : IVoiceAssistantSettingsEnvironment + { + public static UnavailableVoiceAssistantEnvironment Instance { get; } = new(); + public event EventHandler? Changed { add { } remove { } } + public OpenClawTray.Services.VoiceAssistant.VoiceAssistantState RuntimeState => + OpenClawTray.Services.VoiceAssistant.VoiceAssistantState.Off; + public OpenClawTray.Services.VoiceAssistant.VoiceAssistantReadinessResult GetReadiness(string wakePhrase) => + new(false, OpenClawTray.Services.VoiceAssistant.VoiceAssistantReadinessReason.SttDisabled); + public string GetString(string key) => key; + } } diff --git a/src/OpenClaw.Tray.WinUI/Presentation/AppServiceRegistration.cs b/src/OpenClaw.Tray.WinUI/Presentation/AppServiceRegistration.cs index 1f890f4aa..4912afca1 100644 --- a/src/OpenClaw.Tray.WinUI/Presentation/AppServiceRegistration.cs +++ b/src/OpenClaw.Tray.WinUI/Presentation/AppServiceRegistration.cs @@ -32,6 +32,7 @@ public static IServiceCollection AddOpenClawTrayCore(this IServiceCollection ser services.AddSingleton(context.Dispatcher); services.AddSingleton(context.AppCommands); services.AddSingleton(context.Settings); + services.AddSingleton(context.VoiceAssistantEnvironment); // Settings facade over the App-owned SettingsManager. Constructed eagerly from the // already-owned singletons so presentation code depends on ISettingsStore, never the @@ -45,6 +46,7 @@ public static IServiceCollection AddOpenClawTrayCore(this IServiceCollection ser // Transient page view models resolved per navigation scope. services.AddTransient(); services.AddTransient(); + services.AddTransient(); return services; } diff --git a/src/OpenClaw.Tray.WinUI/Presentation/ISettingsStore.cs b/src/OpenClaw.Tray.WinUI/Presentation/ISettingsStore.cs index 7bc805150..dbd8b1f52 100644 --- a/src/OpenClaw.Tray.WinUI/Presentation/ISettingsStore.cs +++ b/src/OpenClaw.Tray.WinUI/Presentation/ISettingsStore.cs @@ -1,3 +1,5 @@ +using OpenClaw.Shared; + namespace OpenClawTray.Presentation; /// @@ -75,6 +77,8 @@ public interface ISettingsEditor bool CameraRecordingConsentGiven { set; } bool ShowChatToolCalls { set; } + string VoiceAssistantMode { set; } + string VoiceAssistantWakePhrase { set; } } /// @@ -106,4 +110,6 @@ public sealed record SettingsSnapshot public bool CameraRecordingConsentGiven { get; init; } public bool ShowChatToolCalls { get; init; } + public string VoiceAssistantMode { get; init; } = VoiceAssistantSettingsPolicy.OffMode; + public string VoiceAssistantWakePhrase { get; init; } = VoiceAssistantSettingsPolicy.DefaultWakePhrase; } diff --git a/src/OpenClaw.Tray.WinUI/Presentation/IVoiceAssistantSettingsEnvironment.cs b/src/OpenClaw.Tray.WinUI/Presentation/IVoiceAssistantSettingsEnvironment.cs new file mode 100644 index 000000000..0e488d712 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Presentation/IVoiceAssistantSettingsEnvironment.cs @@ -0,0 +1,12 @@ +using OpenClawTray.Services.VoiceAssistant; + +namespace OpenClawTray.Presentation; + +public interface IVoiceAssistantSettingsEnvironment +{ + event EventHandler? Changed; + + VoiceAssistantState RuntimeState { get; } + VoiceAssistantReadinessResult GetReadiness(string wakePhrase); + string GetString(string key); +} diff --git a/src/OpenClaw.Tray.WinUI/Presentation/SettingsStore.cs b/src/OpenClaw.Tray.WinUI/Presentation/SettingsStore.cs index 5600d73a7..db00458e9 100644 --- a/src/OpenClaw.Tray.WinUI/Presentation/SettingsStore.cs +++ b/src/OpenClaw.Tray.WinUI/Presentation/SettingsStore.cs @@ -53,6 +53,8 @@ public SettingsStore(SettingsManager settings, IUiDispatcher dispatcher) ScreenRecordingConsentGiven = _settings.ScreenRecordingConsentGiven, CameraRecordingConsentGiven = _settings.CameraRecordingConsentGiven, ShowChatToolCalls = _settings.ShowChatToolCalls, + VoiceAssistantMode = _settings.VoiceAssistantMode, + VoiceAssistantWakePhrase = _settings.VoiceAssistantWakePhrase, }; public void Update(Action edit) @@ -133,5 +135,7 @@ private sealed class Editor : ISettingsEditor public bool ScreenRecordingConsentGiven { set => _settings.ScreenRecordingConsentGiven = value; } public bool CameraRecordingConsentGiven { set => _settings.CameraRecordingConsentGiven = value; } public bool ShowChatToolCalls { set => _settings.ShowChatToolCalls = value; } + public string VoiceAssistantMode { set => _settings.VoiceAssistantMode = value; } + public string VoiceAssistantWakePhrase { set => _settings.VoiceAssistantWakePhrase = value; } } } diff --git a/src/OpenClaw.Tray.WinUI/Presentation/VoiceAssistantSettingsViewModel.cs b/src/OpenClaw.Tray.WinUI/Presentation/VoiceAssistantSettingsViewModel.cs new file mode 100644 index 000000000..540109fb4 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Presentation/VoiceAssistantSettingsViewModel.cs @@ -0,0 +1,207 @@ +using System.ComponentModel; +using System.Runtime.CompilerServices; +using OpenClaw.Shared; +using OpenClawTray.Services; +using OpenClawTray.Services.VoiceAssistant; + +namespace OpenClawTray.Presentation; + +internal sealed class VoiceAssistantSettingsViewModel : + INavigationAware, + IDisposable, + INotifyPropertyChanged +{ + private readonly ISettingsStore _settings; + private readonly IVoiceAssistantSettingsEnvironment _environment; + private readonly IAppCommands _appCommands; + private bool _active; + private bool _loading; + private int _selectedModeIndex; + private string _wakePhraseDraft = VoiceAssistantSettingsPolicy.DefaultWakePhrase; + private bool _canEnableWakeMode; + private string _statusText = string.Empty; + private VoiceAssistantReadinessResult _prerequisiteReadiness; + + public VoiceAssistantSettingsViewModel( + ISettingsStore settings, + IVoiceAssistantSettingsEnvironment environment, + IAppCommands appCommands) + { + _settings = settings; + _environment = environment; + _appCommands = appCommands; + } + + public event PropertyChangedEventHandler? PropertyChanged; + + public int SelectedModeIndex + { + get => _selectedModeIndex; + set + { + var normalized = value == 1 ? 1 : 0; + if (_loading || normalized == _selectedModeIndex) + return; + + if (normalized == 1 && (!CommitWakePhrase() || !CanEnableWakeMode)) + { + OnPropertyChanged(); + return; + } + + _selectedModeIndex = normalized; + OnPropertyChanged(); + Persist(editor => + editor.VoiceAssistantMode = normalized == 1 + ? VoiceAssistantSettingsPolicy.WakeOneShotMode + : VoiceAssistantSettingsPolicy.OffMode); + Refresh(); + } + } + + public string WakePhraseDraft + { + get => _wakePhraseDraft; + set + { + if (SetField(ref _wakePhraseDraft, value ?? string.Empty)) + RefreshDraft(); + } + } + + public bool CanEnableWakeMode + { + get => _canEnableWakeMode; + private set => SetField(ref _canEnableWakeMode, value); + } + + public string StatusText + { + get => _statusText; + private set => SetField(ref _statusText, value); + } + + public bool CommitWakePhrase() + { + if (!VoiceAssistantSettingsPolicy.TryNormalizeWakePhrase(_wakePhraseDraft, out var normalized)) + { + RefreshDraft(); + return false; + } + + WakePhraseDraft = normalized; + if (!string.Equals(_settings.Current.VoiceAssistantWakePhrase, normalized, StringComparison.Ordinal)) + Persist(editor => editor.VoiceAssistantWakePhrase = normalized); + return true; + } + + public void Refresh() + { + _prerequisiteReadiness = + _environment.GetReadiness(VoiceAssistantSettingsPolicy.DefaultWakePhrase); + RefreshDraft(); + } + + private void RefreshDraft() + { + var readiness = VoiceAssistantSettingsPolicy.TryNormalizeWakePhrase(_wakePhraseDraft, out _) + ? _prerequisiteReadiness + : new VoiceAssistantReadinessResult( + false, + VoiceAssistantReadinessReason.WakePhraseInvalid); + CanEnableWakeMode = readiness.IsReady; + + var statusKey = readiness.IsReady + ? RuntimeStatusKey(_environment.RuntimeState) + : ReadinessStatusKey(readiness.Reason); + StatusText = _environment.GetString(statusKey); + } + + public void Activate(object? parameter) + { + if (_active) + return; + _active = true; + _settings.Changed += OnChanged; + _environment.Changed += OnChanged; + Load(); + } + + public void Deactivate() + { + if (!_active) + return; + _active = false; + _settings.Changed -= OnChanged; + _environment.Changed -= OnChanged; + } + + private void Load() + { + _loading = true; + try + { + var snapshot = _settings.Current; + _selectedModeIndex = string.Equals( + snapshot.VoiceAssistantMode, + VoiceAssistantSettingsPolicy.WakeOneShotMode, + StringComparison.Ordinal) + ? 1 + : 0; + _wakePhraseDraft = snapshot.VoiceAssistantWakePhrase; + OnPropertyChanged(nameof(SelectedModeIndex)); + OnPropertyChanged(nameof(WakePhraseDraft)); + Refresh(); + } + finally + { + _loading = false; + } + } + + private void OnChanged(object? sender, EventArgs e) => Load(); + + private void Persist(Action edit) + { + _settings.Update(edit); + _appCommands.NotifySettingsSaved(); + } + + private static string RuntimeStatusKey(VoiceAssistantState state) => state switch + { + VoiceAssistantState.WakeListening => "VoiceSettingsPage_AssistantStatusListening", + VoiceAssistantState.Dispatching => "VoiceSettingsPage_AssistantStatusDispatching", + VoiceAssistantState.WaitingForReply => "VoiceSettingsPage_AssistantStatusWaiting", + VoiceAssistantState.Speaking => "VoiceSettingsPage_AssistantStatusSpeaking", + VoiceAssistantState.Starting => "VoiceSettingsPage_AssistantStatusStarting", + VoiceAssistantState.Error => "VoiceSettingsPage_AssistantStatusError", + VoiceAssistantState.Unavailable => "VoiceSettingsPage_AssistantStatusPaused", + _ => "VoiceSettingsPage_AssistantStatusOff" + }; + + private static string ReadinessStatusKey(VoiceAssistantReadinessReason reason) => reason switch + { + VoiceAssistantReadinessReason.SttDisabled => "VoiceSettingsPage_AssistantStatusSttDisabled", + VoiceAssistantReadinessReason.SttModelMissing => "VoiceSettingsPage_AssistantStatusModelMissing", + VoiceAssistantReadinessReason.TtsDisabled => "VoiceSettingsPage_AssistantStatusTtsDisabled", + VoiceAssistantReadinessReason.PiperVoiceMissing => "VoiceSettingsPage_AssistantStatusPiperMissing", + VoiceAssistantReadinessReason.ElevenLabsApiKeyMissing => "VoiceSettingsPage_AssistantStatusApiKeyMissing", + VoiceAssistantReadinessReason.ElevenLabsVoiceMissing => "VoiceSettingsPage_AssistantStatusVoiceIdMissing", + VoiceAssistantReadinessReason.WakePhraseInvalid => "VoiceSettingsPage_AssistantStatusWakeInvalid", + _ => "VoiceSettingsPage_AssistantStatusProviderInvalid" + }; + + private bool SetField(ref T field, T value, [CallerMemberName] string? propertyName = null) + { + if (EqualityComparer.Default.Equals(field, value)) + return false; + field = value; + OnPropertyChanged(propertyName); + return true; + } + + private void OnPropertyChanged([CallerMemberName] string? propertyName = null) => + PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName)); + + public void Dispose() => Deactivate(); +} diff --git a/src/OpenClaw.Tray.WinUI/Services/SettingsManager.cs b/src/OpenClaw.Tray.WinUI/Services/SettingsManager.cs index 3d26f0047..2af4cd130 100644 --- a/src/OpenClaw.Tray.WinUI/Services/SettingsManager.cs +++ b/src/OpenClaw.Tray.WinUI/Services/SettingsManager.cs @@ -127,6 +127,28 @@ public List UserRules public bool ShowChatToolCalls { get => _data.ShowChatToolCalls; set => _data = _data with { ShowChatToolCalls = value }; } /// Play audio feedback chimes on listen start/stop. public bool VoiceAudioFeedback { get => _data.VoiceAudioFeedback; set => _data = _data with { VoiceAudioFeedback = value }; } + public string VoiceAssistantMode + { + get => VoiceAssistantSettingsPolicy.NormalizeMode(_data.VoiceAssistantMode); + set => _data = _data with { VoiceAssistantMode = VoiceAssistantSettingsPolicy.NormalizeMode(value) }; + } + public string VoiceAssistantWakePhrase + { + get => VoiceAssistantSettingsPolicy.TryNormalizeWakePhrase(_data.VoiceAssistantWakePhrase, out var normalized) + ? normalized + : VoiceAssistantSettingsPolicy.DefaultWakePhrase; + set + { + var valid = VoiceAssistantSettingsPolicy.TryNormalizeWakePhrase(value, out var normalized); + _data = _data with + { + VoiceAssistantWakePhrase = valid ? normalized : VoiceAssistantSettingsPolicy.DefaultWakePhrase, + VoiceAssistantMode = valid + ? VoiceAssistantSettingsPolicy.NormalizeMode(_data.VoiceAssistantMode) + : VoiceAssistantSettingsPolicy.OffMode + }; + } + } public bool NodeTtsEnabled { get => _data.NodeTtsEnabled; set => _data = _data with { NodeTtsEnabled = value }; } public string TtsProvider { get => string.IsNullOrWhiteSpace(_data.TtsProvider) ? TtsCapability.PiperProvider : _data.TtsProvider; set => _data = _data with { TtsProvider = value }; } public string TtsElevenLabsApiKey { get => _data.TtsElevenLabsApiKey ?? ""; set => _data = _data with { TtsElevenLabsApiKey = value }; } @@ -268,6 +290,8 @@ public void Load() SttSilenceTimeout = 1.5f, VoiceTtsEnabled = true, VoiceAudioFeedback = true, + VoiceAssistantMode = VoiceAssistantSettingsPolicy.OffMode, + VoiceAssistantWakePhrase = VoiceAssistantSettingsPolicy.DefaultWakePhrase, NodeTtsEnabled = false, TtsProvider = TtsCapability.PiperProvider, TtsElevenLabsApiKey = "", @@ -296,6 +320,9 @@ public void Load() private static SettingsData NormalizeLoadedData(SettingsData loaded, string? rawJson = null) { var defaults = CreateDefaultData(); + var wakePhraseValid = VoiceAssistantSettingsPolicy.TryNormalizeWakePhrase( + loaded.VoiceAssistantWakePhrase, + out var normalizedWakePhrase); var data = loaded with { SettingsSchemaVersion = CurrentSettingsSchemaVersion, @@ -309,6 +336,12 @@ private static SettingsData NormalizeLoadedData(SettingsData loaded, string? raw SttLanguage = string.IsNullOrWhiteSpace(loaded.SttLanguage) ? defaults.SttLanguage : loaded.SttLanguage, SttModelName = string.IsNullOrWhiteSpace(loaded.SttModelName) ? defaults.SttModelName : loaded.SttModelName, SttSilenceTimeout = loaded.SttSilenceTimeout > 0 ? loaded.SttSilenceTimeout : defaults.SttSilenceTimeout, + VoiceAssistantMode = wakePhraseValid + ? VoiceAssistantSettingsPolicy.NormalizeMode(loaded.VoiceAssistantMode) + : VoiceAssistantSettingsPolicy.OffMode, + VoiceAssistantWakePhrase = wakePhraseValid + ? normalizedWakePhrase + : defaults.VoiceAssistantWakePhrase, TtsProvider = string.IsNullOrWhiteSpace(loaded.TtsProvider) ? defaults.TtsProvider : loaded.TtsProvider, TtsElevenLabsApiKey = UnprotectSettingSecret(loaded.TtsElevenLabsApiKey) ?? defaults.TtsElevenLabsApiKey, TtsElevenLabsModel = loaded.TtsElevenLabsModel ?? defaults.TtsElevenLabsModel, @@ -399,6 +432,8 @@ public SettingsData ToSettingsData() => _data with SttLanguage = SttLanguage, SttModelName = SttModelName, SttSilenceTimeout = SttSilenceTimeout, + VoiceAssistantMode = VoiceAssistantMode, + VoiceAssistantWakePhrase = VoiceAssistantWakePhrase, TtsProvider = TtsProvider, TtsElevenLabsApiKey = TtsElevenLabsApiKey, TtsElevenLabsModel = string.IsNullOrWhiteSpace(TtsElevenLabsModel) ? null : TtsElevenLabsModel, diff --git a/src/OpenClaw.Tray.WinUI/Services/SpeechSetupReadiness.cs b/src/OpenClaw.Tray.WinUI/Services/SpeechSetupReadiness.cs index a812b1be9..63cf6aaaa 100644 --- a/src/OpenClaw.Tray.WinUI/Services/SpeechSetupReadiness.cs +++ b/src/OpenClaw.Tray.WinUI/Services/SpeechSetupReadiness.cs @@ -1,5 +1,6 @@ using OpenClaw.Shared.Audio; using OpenClaw.Shared.Capabilities; +using OpenClawTray.Services.VoiceAssistant; namespace OpenClawTray.Services; @@ -41,4 +42,28 @@ public static bool IsConfiguredTtsProviderSetupRequired(SettingsManager settings return true; } + + public static VoiceAssistantReadinessResult GetVoiceAssistantReadiness( + SettingsManager settings, + string? wakePhrase = null) + { + ArgumentNullException.ThrowIfNull(settings); + + var whisperModels = new WhisperModelManager(SettingsManager.SettingsDirectoryPath, new AppLogger()); + var piperVoices = new PiperVoiceManager(SettingsManager.SettingsDirectoryPath, new AppLogger()); + var provider = TtsCapability.ResolveProvider(null, settings.TtsProvider); + + return VoiceAssistantReadiness.Evaluate(new VoiceAssistantReadinessInput( + SttEnabled: settings.NodeSttEnabled, + SttModelDownloaded: whisperModels.IsModelDownloaded(settings.SttModelName), + TtsEnabled: settings.NodeTtsEnabled, + TtsProvider: provider, + PiperVoiceDownloaded: + string.Equals(provider, TtsCapability.PiperProvider, StringComparison.Ordinal) && + !string.IsNullOrWhiteSpace(settings.TtsPiperVoiceId) && + piperVoices.IsVoiceDownloaded(settings.TtsPiperVoiceId), + HasElevenLabsApiKey: !string.IsNullOrWhiteSpace(settings.TtsElevenLabsApiKey), + HasElevenLabsVoiceId: !string.IsNullOrWhiteSpace(settings.TtsElevenLabsVoiceId), + WakePhrase: wakePhrase ?? settings.VoiceAssistantWakePhrase)); + } } diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/IVoiceAssistantChatTurnClient.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/IVoiceAssistantChatTurnClient.cs new file mode 100644 index 000000000..194aab77c --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/IVoiceAssistantChatTurnClient.cs @@ -0,0 +1,37 @@ +using OpenClaw.Shared; + +namespace OpenClawTray.Services.VoiceAssistant; + +public enum VoiceAssistantSendDisposition +{ + Direct, + Queued +} + +public sealed record VoiceAssistantTurnReceipt( + VoiceAssistantSendDisposition Disposition, + string SessionKey, + string LocalMessageId, + string SendRunId, + int? PreSendSequence); + +public interface IVoiceAssistantChatTurnClient +{ + event Action? ReadinessChanged; + + string? GetReadySessionKey(); + + Task SendAsync( + string sessionKey, + string request, + CancellationToken cancellationToken); + + Task CancelAsync(VoiceAssistantTurnReceipt receipt, CancellationToken cancellationToken); + + bool IsResponseForTurn(VoiceAssistantTurnReceipt receipt, OpenClawNotification notification); +} + +public interface IVoiceAssistantSpeaker +{ + Task SpeakAsync(string text, CancellationToken cancellationToken); +} diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/IVoiceAssistantInput.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/IVoiceAssistantInput.cs new file mode 100644 index 000000000..85ecb0281 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/IVoiceAssistantInput.cs @@ -0,0 +1,10 @@ +namespace OpenClawTray.Services.VoiceAssistant; + +public interface IVoiceAssistantInput +{ + event Action? UtteranceCompleted; + event Action? CaptureAvailable; + + Task StartAsync(CancellationToken cancellationToken); + Task StopAsync(); +} diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantChatInputPolicy.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantChatInputPolicy.cs new file mode 100644 index 000000000..87e51d079 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantChatInputPolicy.cs @@ -0,0 +1,10 @@ +namespace OpenClawTray.Services.VoiceAssistant; + +public static class VoiceAssistantChatInputPolicy +{ + public const string WakeListeningReasonKey = "Chat_Composer_VoiceUnavailable_AssistantListening"; + public const string WakeListeningStatusKey = "Chat_Composer_Status_AssistantListening"; + + public static bool IsVoiceInputUnavailable(VoiceAssistantState state) => + state == VoiceAssistantState.WakeListening; +} diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantChatTurnClient.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantChatTurnClient.cs new file mode 100644 index 000000000..47d7b9a63 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantChatTurnClient.cs @@ -0,0 +1,143 @@ +using System.Collections.Concurrent; +using OpenClaw.Chat; +using OpenClaw.Shared; +using OpenClawTray.Chat; + +namespace OpenClawTray.Services.VoiceAssistant; + +public sealed class VoiceAssistantChatTurnClient : IVoiceAssistantChatTurnClient, IDisposable +{ + private const int CanceledIdentityLimit = 32; + private readonly OpenClawChatDataProvider _provider; + private readonly ConcurrentDictionary _responses = new(StringComparer.Ordinal); + private readonly object _canceledGate = new(); + private readonly object _readinessGate = new(); + private readonly Queue _canceledOrder = new(); + private readonly HashSet _canceled = new(StringComparer.Ordinal); + private string? _lastReadySessionKey; + + public VoiceAssistantChatTurnClient(OpenClawChatDataProvider provider) + { + _provider = provider; + _provider.Changed += OnProviderChanged; + _lastReadySessionKey = _provider.GetVoiceAssistantReadySessionKey(); + _provider.VoiceAssistantResponseObserved += OnResponseObserved; + } + + public event Action? ReadinessChanged; + + public string? GetReadySessionKey() => _provider.GetVoiceAssistantReadySessionKey(); + + public Task SendAsync( + string sessionKey, + string request, + CancellationToken cancellationToken) => + _provider.SendVoiceAssistantMessageAsync(sessionKey, request, cancellationToken); + + public async Task CancelAsync( + VoiceAssistantTurnReceipt receipt, + CancellationToken cancellationToken) + { + RememberCanceled(receipt.LocalMessageId); + _responses.TryRemove(receipt.LocalMessageId, out _); + await _provider.CancelVoiceAssistantTurnAsync(receipt, cancellationToken).ConfigureAwait(false); + } + + public bool IsResponseForTurn( + VoiceAssistantTurnReceipt receipt, + OpenClawNotification notification) + { + if (!_responses.TryGetValue(receipt.LocalMessageId, out var response) || + !string.Equals(response.SessionKey, notification.SessionKey, StringComparison.Ordinal)) + { + return false; + } + + bool matches; + if (!string.IsNullOrWhiteSpace(response.GatewayMessageId) && + response.GatewaySequence is { } gatewaySequence) + { + matches = string.Equals( + response.GatewayMessageId, + notification.OpenClawId, + StringComparison.Ordinal) && + gatewaySequence == notification.OpenClawSeq; + } + else if (!string.IsNullOrWhiteSpace(notification.OpenClawId) || + notification.OpenClawSeq is not null) + { + matches = false; + } + else + { + matches = string.Equals( + response.ResponseText, + notification.FullMessage ?? notification.Message, + StringComparison.Ordinal); + } + + if (matches) + _responses.TryRemove(receipt.LocalMessageId, out _); + return matches; + } + + private void OnResponseObserved(OpenClawChatDataProvider.VoiceAssistantResponseIdentity response) + { + lock (_canceledGate) + { + if (_canceled.Contains(response.LocalMessageId)) + return; + } + + _responses[response.LocalMessageId] = new ResponseIdentity( + response.SessionKey, + response.GatewayMessageId, + response.GatewaySequence, + response.ResponseText); + } + + private void OnProviderChanged(object? sender, ChatDataChangedEventArgs args) + { + var readySessionKey = _provider.GetVoiceAssistantReadySessionKey(); + lock (_readinessGate) + { + if (string.Equals(_lastReadySessionKey, readySessionKey, StringComparison.Ordinal)) + return; + + _lastReadySessionKey = readySessionKey; + } + + ReadinessChanged?.Invoke(); + } + + private void RememberCanceled(string localMessageId) + { + lock (_canceledGate) + { + if (!_canceled.Add(localMessageId)) + return; + + _canceledOrder.Enqueue(localMessageId); + while (_canceledOrder.Count > CanceledIdentityLimit) + _canceled.Remove(_canceledOrder.Dequeue()); + } + } + + public void Dispose() + { + _provider.Changed -= OnProviderChanged; + _provider.VoiceAssistantResponseObserved -= OnResponseObserved; + _responses.Clear(); + lock (_canceledGate) + { + _canceled.Clear(); + _canceledOrder.Clear(); + } + } + + private sealed record ResponseIdentity( + string SessionKey, + string? GatewayMessageId, + int? GatewaySequence, + string ResponseText); +} diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantCoordinator.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantCoordinator.cs new file mode 100644 index 000000000..b7fd53d29 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantCoordinator.cs @@ -0,0 +1,414 @@ +using OpenClaw.Shared; +using System.Security.Cryptography; +using System.Text; + +namespace OpenClawTray.Services.VoiceAssistant; + +public readonly record struct VoiceAssistantConfiguration( + bool Enabled, + bool LocalPrerequisitesReady, + string WakePhrase); + +public sealed class VoiceAssistantCoordinator : IAsyncDisposable +{ + private const int ConsumedIdentityLimit = 32; + private readonly object _gate = new(); + private readonly SemaphoreSlim _reconcileGate = new(1, 1); + private readonly IVoiceAssistantInput _input; + private readonly IVoiceAssistantChatTurnClient _chat; + private readonly IVoiceAssistantSpeaker _speaker; + private readonly Func _configuration; + private readonly TimeSpan _replyTimeout; + private readonly CancellationTokenSource _lifetime = new(); + private readonly Queue _consumedOrder = new(); + private readonly HashSet _consumed = new(StringComparer.Ordinal); + private CancellationTokenSource? _turnCancellation; + private CancellationTokenSource? _replyTimeoutCancellation; + private VoiceAssistantTurnReceipt? _activeTurn; + private VoiceAssistantState _state = VoiceAssistantState.Off; + private bool _disposed; + + public VoiceAssistantCoordinator( + IVoiceAssistantInput input, + IVoiceAssistantChatTurnClient chat, + IVoiceAssistantSpeaker speaker, + Func configuration, + TimeSpan? replyTimeout = null) + { + _input = input; + _chat = chat; + _speaker = speaker; + _configuration = configuration; + _replyTimeout = replyTimeout ?? TimeSpan.FromSeconds(120); + _input.UtteranceCompleted += OnUtteranceCompleted; + _input.CaptureAvailable += OnCaptureAvailable; + _chat.ReadinessChanged += OnChatReadinessChanged; + } + + public VoiceAssistantState State + { + get + { + lock (_gate) + return _state; + } + } + + public event Action? StateChanged; + + public Task ReconcileAsync() => ReconcileCoreAsync(); + + public bool TryClaimResponse(OpenClawNotification notification) + { + ArgumentNullException.ThrowIfNull(notification); + + string? identity = GetIdentity(notification); + var fallbackSignature = GetFallbackSignature(notification); + VoiceAssistantTurnReceipt? receipt; + CancellationToken turnCancellation; + + lock (_gate) + { + if (identity is not null && _consumed.Contains(identity)) + return true; + + receipt = _activeTurn; + var turnScopedFallbackIdentity = receipt is null + ? null + : GetFallbackIdentity( + fallbackSignature, + notification.SessionKey, + receipt.LocalMessageId); + if (turnScopedFallbackIdentity is not null && + _consumed.Contains(turnScopedFallbackIdentity)) + return true; + + if (_state != VoiceAssistantState.WaitingForReply || + receipt is null || + string.IsNullOrWhiteSpace(notification.SessionKey) || + !string.Equals(notification.SessionKey, receipt.SessionKey, StringComparison.Ordinal) || + notification.OpenClawSeq is { } sequence && + receipt.PreSendSequence is { } baseline && + sequence <= baseline || + !_chat.IsResponseForTurn(receipt, notification)) + { + return false; + } + + identity ??= turnScopedFallbackIdentity; + if (identity is null) + return false; + + AddConsumedLocked(identity); + _state = VoiceAssistantState.Speaking; + turnCancellation = _turnCancellation?.Token ?? _lifetime.Token; + _replyTimeoutCancellation?.Cancel(); + } + + StateChanged?.Invoke(VoiceAssistantState.Speaking); + _ = SpeakAndResumeAsync( + notification.FullMessage ?? notification.Message, + receipt, + turnCancellation); + return true; + } + + private void OnUtteranceCompleted(string transcript) + { + string request; + CancellationTokenSource turnCancellation; + + lock (_gate) + { + if (_disposed || + _state != VoiceAssistantState.WakeListening || + !VoiceWakeGate.TryExtractRequest( + transcript, + _configuration().WakePhrase, + out request)) + { + return; + } + + _state = VoiceAssistantState.Dispatching; + _turnCancellation?.Cancel(); + _turnCancellation?.Dispose(); + turnCancellation = CancellationTokenSource.CreateLinkedTokenSource(_lifetime.Token); + _turnCancellation = turnCancellation; + } + + StateChanged?.Invoke(VoiceAssistantState.Dispatching); + _ = DispatchAsync(request, turnCancellation.Token); + } + + private async Task DispatchAsync(string request, CancellationToken cancellationToken) + { + try + { + await _input.StopAsync().ConfigureAwait(false); + cancellationToken.ThrowIfCancellationRequested(); + + var sessionKey = _chat.GetReadySessionKey(); + if (string.IsNullOrWhiteSpace(sessionKey)) + { + await RecoverAsync(VoiceAssistantState.Unavailable).ConfigureAwait(false); + return; + } + + var receipt = await _chat.SendAsync(sessionKey, request, cancellationToken).ConfigureAwait(false); + if (receipt.Disposition != VoiceAssistantSendDisposition.Direct) + { + await _chat.CancelAsync(receipt, cancellationToken).ConfigureAwait(false); + await RecoverAsync(VoiceAssistantState.Unavailable).ConfigureAwait(false); + return; + } + + CancellationTokenSource timeoutCancellation; + lock (_gate) + { + if (_disposed || cancellationToken.IsCancellationRequested) + return; + + _activeTurn = receipt; + _state = VoiceAssistantState.WaitingForReply; + _replyTimeoutCancellation?.Cancel(); + _replyTimeoutCancellation?.Dispose(); + timeoutCancellation = + CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + _replyTimeoutCancellation = timeoutCancellation; + } + + StateChanged?.Invoke(VoiceAssistantState.WaitingForReply); + try + { + await Task.Delay(_replyTimeout, timeoutCancellation.Token).ConfigureAwait(false); + } + catch (OperationCanceledException) when (timeoutCancellation.IsCancellationRequested) + { + return; + } + await RecoverAsync(VoiceAssistantState.Error).ConfigureAwait(false); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + } + catch + { + await RecoverAsync(VoiceAssistantState.Error).ConfigureAwait(false); + } + } + + private async Task SpeakAndResumeAsync( + string text, + VoiceAssistantTurnReceipt receipt, + CancellationToken cancellationToken) + { + try + { + await _speaker.SpeakAsync(text, cancellationToken).ConfigureAwait(false); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + return; + } + catch + { + await RecoverAsync(VoiceAssistantState.Error).ConfigureAwait(false); + return; + } + + lock (_gate) + { + if (!ReferenceEquals(_activeTurn, receipt)) + return; + } + + await RecoverAsync(VoiceAssistantState.WakeListening).ConfigureAwait(false); + } + + private async Task RecoverAsync(VoiceAssistantState transientState) + { + VoiceAssistantTurnReceipt? receipt; + lock (_gate) + { + receipt = _activeTurn; + _activeTurn = null; + _turnCancellation?.Cancel(); + _turnCancellation?.Dispose(); + _turnCancellation = null; + _replyTimeoutCancellation?.Cancel(); + _replyTimeoutCancellation?.Dispose(); + _replyTimeoutCancellation = null; + if (!_disposed) + _state = transientState; + } + + if (!_disposed) + StateChanged?.Invoke(transientState); + + if (receipt is not null && transientState != VoiceAssistantState.WakeListening) + { + try + { + await _chat.CancelAsync(receipt, _lifetime.Token).ConfigureAwait(false); + } + catch (OperationCanceledException) when (_lifetime.IsCancellationRequested) + { + } + } + + await ReconcileCoreAsync().ConfigureAwait(false); + } + + private async Task ReconcileCoreAsync() + { + await _reconcileGate.WaitAsync().ConfigureAwait(false); + try + { + VoiceAssistantConfiguration configuration; + VoiceAssistantState target; + VoiceAssistantTurnReceipt? canceledTurn = null; + lock (_gate) + { + if (_disposed) + return; + + configuration = _configuration(); + target = !configuration.Enabled + ? VoiceAssistantState.Off + : !configuration.LocalPrerequisitesReady || + string.IsNullOrWhiteSpace(_chat.GetReadySessionKey()) + ? VoiceAssistantState.Unavailable + : VoiceAssistantState.Starting; + + if (_state is VoiceAssistantState.Dispatching or + VoiceAssistantState.WaitingForReply or + VoiceAssistantState.Speaking) + { + if (target == VoiceAssistantState.Starting) + return; + + _turnCancellation?.Cancel(); + _turnCancellation?.Dispose(); + _turnCancellation = null; + _replyTimeoutCancellation?.Cancel(); + _replyTimeoutCancellation?.Dispose(); + _replyTimeoutCancellation = null; + canceledTurn = _activeTurn; + _activeTurn = null; + } + + _state = target; + } + + StateChanged?.Invoke(target); + await _input.StopAsync().ConfigureAwait(false); + if (canceledTurn is not null) + await _chat.CancelAsync(canceledTurn, _lifetime.Token).ConfigureAwait(false); + + if (target != VoiceAssistantState.Starting) + return; + + try + { + await _input.StartAsync(_lifetime.Token).ConfigureAwait(false); + lock (_gate) + { + if (_disposed) + return; + _state = VoiceAssistantState.WakeListening; + } + StateChanged?.Invoke(VoiceAssistantState.WakeListening); + } + catch (VoiceCaptureBusyException) + { + lock (_gate) + _state = VoiceAssistantState.Unavailable; + StateChanged?.Invoke(VoiceAssistantState.Unavailable); + } + catch (OperationCanceledException) when (_lifetime.IsCancellationRequested) + { + } + catch + { + lock (_gate) + _state = VoiceAssistantState.Error; + StateChanged?.Invoke(VoiceAssistantState.Error); + } + } + finally + { + _reconcileGate.Release(); + } + } + + private void OnCaptureAvailable() => _ = ReconcileCoreAsync(); + + private void OnChatReadinessChanged() => _ = ReconcileCoreAsync(); + + private static string? GetIdentity(OpenClawNotification notification) => + !string.IsNullOrWhiteSpace(notification.SessionKey) && + !string.IsNullOrWhiteSpace(notification.OpenClawId) && + notification.OpenClawSeq is { } sequence + ? $"{notification.SessionKey}\n{notification.OpenClawId}\n{sequence}" + : null; + + private static string? GetFallbackSignature(OpenClawNotification notification) + { + var text = notification.FullMessage ?? notification.Message; + if (string.IsNullOrWhiteSpace(notification.SessionKey) || + string.IsNullOrEmpty(text)) + { + return null; + } + + var hash = Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(text))); + return hash; + } + + private static string? GetFallbackIdentity( + string? fallbackSignature, + string? sessionKey, + string localMessageId) => + fallbackSignature is null || string.IsNullOrWhiteSpace(sessionKey) + ? null + : $"fallback:{sessionKey}\n{localMessageId}\n{fallbackSignature}"; + + private void AddConsumedLocked(string identity) + { + if (!_consumed.Add(identity)) + return; + + _consumedOrder.Enqueue(identity); + while (_consumedOrder.Count > ConsumedIdentityLimit) + _consumed.Remove(_consumedOrder.Dequeue()); + } + + public async ValueTask DisposeAsync() + { + VoiceAssistantTurnReceipt? activeTurn; + lock (_gate) + { + if (_disposed) + return; + _disposed = true; + _state = VoiceAssistantState.Off; + _turnCancellation?.Cancel(); + _replyTimeoutCancellation?.Cancel(); + activeTurn = _activeTurn; + _activeTurn = null; + } + + _input.UtteranceCompleted -= OnUtteranceCompleted; + _input.CaptureAvailable -= OnCaptureAvailable; + _chat.ReadinessChanged -= OnChatReadinessChanged; + _lifetime.Cancel(); + await _input.StopAsync().ConfigureAwait(false); + if (activeTurn is not null) + await _chat.CancelAsync(activeTurn, CancellationToken.None).ConfigureAwait(false); + _replyTimeoutCancellation?.Dispose(); + _turnCancellation?.Dispose(); + _lifetime.Dispose(); + _reconcileGate.Dispose(); + } +} diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantInput.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantInput.cs new file mode 100644 index 000000000..73ce9c915 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantInput.cs @@ -0,0 +1,33 @@ +namespace OpenClawTray.Services.VoiceAssistant; + +public sealed class VoiceAssistantInput : IVoiceAssistantInput, IDisposable +{ + private readonly VoiceService _voiceService; + + public VoiceAssistantInput(VoiceService voiceService) + { + _voiceService = voiceService; + _voiceService.UtteranceCompleted += OnUtteranceCompleted; + } + + public event Action? UtteranceCompleted; + + public event Action? CaptureAvailable + { + add => _voiceService.CaptureAvailable += value; + remove => _voiceService.CaptureAvailable -= value; + } + + public Task StartAsync(CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + return _voiceService.StartWakeListeningAsync(); + } + + public Task StopAsync() => _voiceService.StopAsync(); + + private void OnUtteranceCompleted(OpenClaw.Shared.Audio.UtteranceResult result) => + UtteranceCompleted?.Invoke(result.Text); + + public void Dispose() => _voiceService.UtteranceCompleted -= OnUtteranceCompleted; +} diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantReadiness.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantReadiness.cs new file mode 100644 index 000000000..983fca527 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantReadiness.cs @@ -0,0 +1,70 @@ +using OpenClaw.Shared; +using OpenClaw.Shared.Capabilities; + +namespace OpenClawTray.Services.VoiceAssistant; + +public enum VoiceAssistantReadinessReason +{ + Ready, + SttDisabled, + SttModelMissing, + TtsDisabled, + UnsupportedTtsProvider, + PiperVoiceMissing, + ElevenLabsApiKeyMissing, + ElevenLabsVoiceMissing, + WakePhraseInvalid +} + +public readonly record struct VoiceAssistantReadinessResult( + bool IsReady, + VoiceAssistantReadinessReason Reason); + +public readonly record struct VoiceAssistantReadinessInput( + bool SttEnabled, + bool SttModelDownloaded, + bool TtsEnabled, + string? TtsProvider, + bool PiperVoiceDownloaded, + bool HasElevenLabsApiKey, + bool HasElevenLabsVoiceId, + string? WakePhrase); + +public static class VoiceAssistantReadiness +{ + public static VoiceAssistantReadinessResult Evaluate(VoiceAssistantReadinessInput input) + { + if (!input.SttEnabled) + return NotReady(VoiceAssistantReadinessReason.SttDisabled); + if (!input.SttModelDownloaded) + return NotReady(VoiceAssistantReadinessReason.SttModelMissing); + if (!input.TtsEnabled) + return NotReady(VoiceAssistantReadinessReason.TtsDisabled); + + var provider = TtsCapability.ResolveProvider(null, input.TtsProvider); + if (string.Equals(provider, TtsCapability.PiperProvider, StringComparison.Ordinal)) + { + if (!input.PiperVoiceDownloaded) + return NotReady(VoiceAssistantReadinessReason.PiperVoiceMissing); + } + else if (string.Equals(provider, TtsCapability.ElevenLabsProvider, StringComparison.Ordinal)) + { + if (!input.HasElevenLabsApiKey) + return NotReady(VoiceAssistantReadinessReason.ElevenLabsApiKeyMissing); + if (!input.HasElevenLabsVoiceId) + return NotReady(VoiceAssistantReadinessReason.ElevenLabsVoiceMissing); + } + else if (!string.Equals(provider, TtsCapability.WindowsProvider, StringComparison.Ordinal)) + { + return NotReady(VoiceAssistantReadinessReason.UnsupportedTtsProvider); + } + + if (!VoiceAssistantSettingsPolicy.TryNormalizeWakePhrase(input.WakePhrase, out _)) + return NotReady(VoiceAssistantReadinessReason.WakePhraseInvalid); + + return new VoiceAssistantReadinessResult(true, VoiceAssistantReadinessReason.Ready); + } + + private static VoiceAssistantReadinessResult NotReady(VoiceAssistantReadinessReason reason) => + new(false, reason); +} diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantSpeaker.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantSpeaker.cs new file mode 100644 index 000000000..27d624707 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantSpeaker.cs @@ -0,0 +1,16 @@ +using OpenClawTray.Chat; + +namespace OpenClawTray.Services.VoiceAssistant; + +public sealed class VoiceAssistantSpeaker : IVoiceAssistantSpeaker +{ + private readonly OpenClawChatCoordinator _chatCoordinator; + + public VoiceAssistantSpeaker(OpenClawChatCoordinator chatCoordinator) + { + _chatCoordinator = chatCoordinator; + } + + public Task SpeakAsync(string text, CancellationToken cancellationToken) => + _chatCoordinator.SpeakAssistantResponseAsync(text, cancellationToken); +} diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantState.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantState.cs new file mode 100644 index 000000000..0d03bcede --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantState.cs @@ -0,0 +1,13 @@ +namespace OpenClawTray.Services.VoiceAssistant; + +public enum VoiceAssistantState +{ + Off, + Unavailable, + Starting, + WakeListening, + Dispatching, + WaitingForReply, + Speaking, + Error +} diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceWakeGate.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceWakeGate.cs new file mode 100644 index 000000000..83dee7718 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceWakeGate.cs @@ -0,0 +1,83 @@ +using System.Text; +using OpenClaw.Shared; + +namespace OpenClawTray.Services.VoiceAssistant; + +public static class VoiceWakeGate +{ + public static bool TryExtractRequest(string? transcript, string? wakePhrase, out string request) + { + request = string.Empty; + if (!VoiceAssistantSettingsPolicy.TryNormalizeWakePhrase(wakePhrase, out var normalizedWakePhrase) || + string.IsNullOrWhiteSpace(transcript)) + { + return false; + } + + var wakeTokens = Tokenize(normalizedWakePhrase); + var transcriptTokens = Tokenize(transcript); + if (transcriptTokens.Count <= wakeTokens.Count) + return false; + + for (var index = 0; index < wakeTokens.Count; index++) + { + if (!string.Equals( + transcriptTokens[index].Value, + wakeTokens[index].Value, + StringComparison.OrdinalIgnoreCase)) + { + return false; + } + } + + var requestStart = transcriptTokens[wakeTokens.Count - 1].End; + while (requestStart < transcript.Length && !char.IsLetterOrDigit(transcript[requestStart])) + requestStart++; + + if (requestStart >= transcript.Length) + return false; + + request = CapitalizeFirstCharacter(transcript[requestStart..].Trim()); + return request.Any(char.IsLetterOrDigit); + } + + private static string CapitalizeFirstCharacter(string value) + { + if (value.Length == 0) + return value; + + var uppercase = char.ToUpperInvariant(value[0]); + return uppercase == value[0] + ? value + : uppercase + value[1..]; + } + + private static List Tokenize(string value) + { + var tokens = new List(); + var index = 0; + + while (index < value.Length) + { + while (index < value.Length && !IsTokenCharacter(value[index])) + index++; + + if (index >= value.Length) + break; + + var start = index; + while (index < value.Length && IsTokenCharacter(value[index])) + index++; + + var normalized = value[start..index].Normalize(NormalizationForm.FormKC); + tokens.Add(new Token(normalized, index)); + } + + return tokens; + } + + private static bool IsTokenCharacter(char character) => + char.IsLetterOrDigit(character) || character is '\'' or '\u2019' or '-'; + + private readonly record struct Token(string Value, int End); +} diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceCaptureLease.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceCaptureLease.cs new file mode 100644 index 000000000..9316d5899 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceCaptureLease.cs @@ -0,0 +1,72 @@ +namespace OpenClawTray.Services; + +public enum VoiceCaptureKind +{ + PushToTalk, + VoiceChat, + WakeListening, + ListenOnce, + FixedDuration +} + +public sealed class VoiceCaptureBusyException : InvalidOperationException +{ + public VoiceCaptureBusyException(VoiceCaptureKind activeCapture) + : base($"The microphone is busy with {activeCapture}.") + { + ActiveCapture = activeCapture; + } + + public VoiceCaptureKind ActiveCapture { get; } +} + +internal sealed class VoiceCaptureLeaseGate +{ + public static VoiceCaptureLeaseGate Shared { get; } = new(); + + private readonly object _gate = new(); + private VoiceCaptureKind? _activeCapture; + + public event Action? Available; + + public IDisposable Acquire(VoiceCaptureKind capture) + { + lock (_gate) + { + if (_activeCapture is { } active) + throw new VoiceCaptureBusyException(active); + + _activeCapture = capture; + return new Lease(this, capture); + } + } + + private void Release(VoiceCaptureKind capture) + { + Action? available; + lock (_gate) + { + if (_activeCapture != capture) + return; + + _activeCapture = null; + available = Available; + } + + available?.Invoke(); + } + + private sealed class Lease : IDisposable + { + private VoiceCaptureLeaseGate? _owner; + private readonly VoiceCaptureKind _capture; + + public Lease(VoiceCaptureLeaseGate owner, VoiceCaptureKind capture) + { + _owner = owner; + _capture = capture; + } + + public void Dispose() => Interlocked.Exchange(ref _owner, null)?.Release(_capture); + } +} diff --git a/src/OpenClaw.Tray.WinUI/Services/VoiceService.cs b/src/OpenClaw.Tray.WinUI/Services/VoiceService.cs index 6d191122e..24334073e 100644 --- a/src/OpenClaw.Tray.WinUI/Services/VoiceService.cs +++ b/src/OpenClaw.Tray.WinUI/Services/VoiceService.cs @@ -15,7 +15,8 @@ public enum VoiceMode { Inactive, PushToTalk, - VoiceChat + VoiceChat, + WakeListening } /// @@ -28,9 +29,12 @@ public sealed class VoiceService : IAsyncDisposable private readonly SettingsManager _settings; private readonly SpeechToTextService _stt; private readonly WhisperModelManager _modelManager; + private readonly VoiceCaptureLeaseGate _captureGate = VoiceCaptureLeaseGate.Shared; + private readonly SemaphoreSlim _sessionGate = new(1, 1); private AudioPipeline? _pipeline; private VoiceMode _currentMode = VoiceMode.Inactive; private CancellationTokenSource? _sessionCts; + private IDisposable? _sessionCaptureLease; /// Current voice interaction mode. public VoiceMode CurrentMode => _currentMode; @@ -85,6 +89,12 @@ public sealed class VoiceService : IAsyncDisposable /// Fired when pipeline state changes. public event Action? PipelineStateChanged; + public event Action? CaptureAvailable + { + add => _captureGate.Available += value; + remove => _captureGate.Available -= value; + } + /// When true, the pipeline ignores audio input (used during TTS playback to prevent echo). public bool IsMutedForPlayback { @@ -136,38 +146,10 @@ public async Task InitializeAsync( /// public async Task StartPushToTalkAsync() { - if (_currentMode != VoiceMode.Inactive) - { - _logger.Info("Voice already active, ignoring PTT start"); - return; - } - - await EnsureInitializedAsync(); - SetMode(VoiceMode.PushToTalk); - - _sessionCts = new CancellationTokenSource(); - _pipeline = new AudioPipeline(_logger, _stt); - WirePipelineEvents(_pipeline); - - var options = new AudioPipelineOptions - { - ModelPath = _modelManager.GetModelPath(_settings.SttModelName), - Language = _settings.SttLanguage, - SilenceTimeoutSeconds = 30, // For PTT, don't auto-stop on silence - VadThreshold = 0.5f - }; - - try - { - await _pipeline.StartAsync(options, _sessionCts.Token); - _logger.Info("Push-to-talk started"); - } - catch - { - // Clean up on failure so the service isn't stuck in a broken state - await CleanupSessionAsync(); - throw; - } + await StartContinuousCaptureAsync( + VoiceMode.PushToTalk, + VoiceCaptureKind.PushToTalk, + silenceTimeoutSeconds: 30); } /// Stop push-to-talk. @@ -178,50 +160,38 @@ public async Task StartPushToTalkAsync() /// public async Task StartVoiceChatAsync() { - if (_currentMode != VoiceMode.Inactive) - { - _logger.Info("Voice already active, ignoring voice chat start"); - return; - } - - await EnsureInitializedAsync(); - SetMode(VoiceMode.VoiceChat); - - _sessionCts = new CancellationTokenSource(); - _pipeline = new AudioPipeline(_logger, _stt); - WirePipelineEvents(_pipeline); - - var options = new AudioPipelineOptions - { - ModelPath = _modelManager.GetModelPath(_settings.SttModelName), - Language = _settings.SttLanguage, - SilenceTimeoutSeconds = _settings.SttSilenceTimeout, - VadThreshold = 0.5f - }; - - try - { - await _pipeline.StartAsync(options, _sessionCts.Token); - _logger.Info("Voice chat session started"); - } - catch - { - await CleanupSessionAsync(); - throw; - } + await StartContinuousCaptureAsync( + VoiceMode.VoiceChat, + VoiceCaptureKind.VoiceChat, + _settings.SttSilenceTimeout); } /// Stop the current voice chat session. public Task StopVoiceChatAsync() => StopAsync(); + public Task StartWakeListeningAsync() => + StartContinuousCaptureAsync( + VoiceMode.WakeListening, + VoiceCaptureKind.WakeListening, + _settings.SttSilenceTimeout); + /// Stop any active voice mode. public async Task StopAsync() { - if (_currentMode == VoiceMode.Inactive) return; - await CleanupSessionAsync(); + await _sessionGate.WaitAsync().ConfigureAwait(false); + try + { + if (_sessionCaptureLease is null) + return; + await CleanupSessionCoreAsync().ConfigureAwait(false); + } + finally + { + _sessionGate.Release(); + } } - private async Task CleanupSessionAsync() + private async Task CleanupSessionCoreAsync() { if (_pipeline != null) { @@ -237,6 +207,8 @@ private async Task CleanupSessionAsync() _sessionCts = null; SetMode(VoiceMode.Inactive); + _sessionCaptureLease?.Dispose(); + _sessionCaptureLease = null; } /// @@ -247,6 +219,7 @@ private async Task CleanupSessionAsync() /// public async Task ListenOnceAsync(SttListenArgs args, CancellationToken cancellationToken) { + using var captureLease = _captureGate.Acquire(VoiceCaptureKind.ListenOnce); await EnsureInitializedAsync(); using var timeoutCts = new CancellationTokenSource(args.TimeoutMs); @@ -344,8 +317,8 @@ public async Task TranscribeFixedDurationAsync( if (args.MaxDurationMs <= 0) throw new ArgumentOutOfRangeException(nameof(args), "maxDurationMs must be positive."); + using var captureLease = _captureGate.Acquire(VoiceCaptureKind.FixedDuration); await EnsureInitializedAsync(); - var pipeline = new AudioPipeline(_logger, _stt); var sw = Stopwatch.StartNew(); try @@ -441,6 +414,53 @@ private async Task EnsureInitializedAsync() } } + private async Task StartContinuousCaptureAsync( + VoiceMode mode, + VoiceCaptureKind captureKind, + float silenceTimeoutSeconds) + { + await _sessionGate.WaitAsync().ConfigureAwait(false); + try + { + if (_sessionCaptureLease is not null) + { + _logger.Info("Voice capture already active, ignoring start"); + return; + } + + _sessionCaptureLease = _captureGate.Acquire(captureKind); + try + { + await EnsureInitializedAsync().ConfigureAwait(false); + SetMode(mode); + + _sessionCts = new CancellationTokenSource(); + _pipeline = new AudioPipeline(_logger, _stt); + WirePipelineEvents(_pipeline); + + var options = new AudioPipelineOptions + { + ModelPath = _modelManager.GetModelPath(_settings.SttModelName), + Language = _settings.SttLanguage, + SilenceTimeoutSeconds = silenceTimeoutSeconds, + VadThreshold = 0.5f + }; + + await _pipeline.StartAsync(options, _sessionCts.Token).ConfigureAwait(false); + _logger.Info($"Voice capture started: {mode}"); + } + catch + { + await CleanupSessionCoreAsync().ConfigureAwait(false); + throw; + } + } + finally + { + _sessionGate.Release(); + } + } + private void WirePipelineEvents(AudioPipeline pipeline) { pipeline.TranscriptionReady += result => diff --git a/src/OpenClaw.Tray.WinUI/Strings/en-us/Resources.resw b/src/OpenClaw.Tray.WinUI/Strings/en-us/Resources.resw index 8f87c5992..dae1dda11 100644 --- a/src/OpenClaw.Tray.WinUI/Strings/en-us/Resources.resw +++ b/src/OpenClaw.Tray.WinUI/Strings/en-us/Resources.resw @@ -2978,6 +2978,12 @@ Use one of these options: Voice + + Voice input unavailable while Assistant is listening for the wake phrase. + + + Assistant is listening for the wake phrase. + Listening… @@ -6904,6 +6910,30 @@ Make sure the gateway is running. splitter + Assistant (preview) + Say the wake phrase and your request in one utterance. The matched reply is always spoken. + Assistant mode + Off + Wake phrase, one request + Wake phrase + OpenClaw + When enabled, the microphone stays active and speech bursts are transcribed locally to find the wake phrase. Only the request after the wake phrase is sent through your configured gateway. + Assistant is off. + Starting wake listening. + Listening for the wake phrase. + Sending your request. + Waiting for the reply. + Speaking the reply. + Assistant is paused until chat is connected and idle. + The assistant stopped after an error and will retry when ready. + Enable speech-to-text in Permissions to use the assistant. + Download the selected speech model to use the assistant. + Enable text-to-speech in Permissions to use the assistant. + Download the selected Piper voice to use the assistant. + Add an ElevenLabs API key to use the assistant. + Add an ElevenLabs voice ID to use the assistant. + Use a wake phrase with one to three words and up to 64 characters. + Select and configure a supported text-to-speech provider. Saved diff --git a/src/OpenClaw.Tray.WinUI/Strings/fr-fr/Resources.resw b/src/OpenClaw.Tray.WinUI/Strings/fr-fr/Resources.resw index 7686fb73b..82d813fba 100644 --- a/src/OpenClaw.Tray.WinUI/Strings/fr-fr/Resources.resw +++ b/src/OpenClaw.Tray.WinUI/Strings/fr-fr/Resources.resw @@ -2931,6 +2931,12 @@ Utilisez l'une de ces options : Voix + + La saisie vocale est indisponible pendant que l’Assistant écoute la phrase d’activation. + + + L’Assistant écoute la phrase d’activation. + Écoute… @@ -6864,6 +6870,30 @@ Le binaire wxc-exec est introuvable. {1} S'il s'agit d'une build développeur, c séparateur + Assistant (aperçu) + Dites la phrase de réveil et votre demande dans un même énoncé. La réponse correspondante est toujours lue. + Mode Assistant + Désactivé + Phrase de réveil, une demande + Phrase de réveil + OpenClaw + Lorsque cette option est activée, le microphone reste actif et les segments vocaux sont transcrits localement pour détecter la phrase de réveil. Seule la demande qui suit est envoyée via votre passerelle configurée. + L’assistant est désactivé. + Démarrage de l’écoute. + Écoute de la phrase de réveil. + Envoi de votre demande. + Attente de la réponse. + Lecture de la réponse. + L’assistant est en pause jusqu’à ce que la conversation soit connectée et inactive. + L’assistant s’est arrêté après une erreur et réessaiera lorsqu’il sera prêt. + Activez la reconnaissance vocale dans Autorisations. + Téléchargez le modèle vocal sélectionné. + Activez la synthèse vocale dans Autorisations. + Téléchargez la voix Piper sélectionnée. + Ajoutez une clé API ElevenLabs. + Ajoutez un ID de voix ElevenLabs. + Utilisez une phrase de réveil de un à trois mots et de 64 caractères maximum. + Sélectionnez et configurez un fournisseur de synthèse vocale pris en charge. Enregistré diff --git a/src/OpenClaw.Tray.WinUI/Strings/nl-nl/Resources.resw b/src/OpenClaw.Tray.WinUI/Strings/nl-nl/Resources.resw index 46441a22f..e8fad12b8 100644 --- a/src/OpenClaw.Tray.WinUI/Strings/nl-nl/Resources.resw +++ b/src/OpenClaw.Tray.WinUI/Strings/nl-nl/Resources.resw @@ -2932,6 +2932,12 @@ Gebruik een van deze opties: Stem + + Spraakinvoer is niet beschikbaar terwijl Assistent naar de activeringszin luistert. + + + Assistent luistert naar de activeringszin. + Luisteren… @@ -6865,6 +6871,30 @@ Het binaire bestand wxc-exec is niet gevonden. {1} Als dit een ontwikkelaarsbuil scheidingslijn + Assistent (preview) + Zeg de wekzin en uw verzoek in één uiting. Het bijbehorende antwoord wordt altijd uitgesproken. + Assistentmodus + Uit + Wekzin, één verzoek + Wekzin + OpenClaw + Wanneer dit is ingeschakeld, blijft de microfoon actief en worden spraakfragmenten lokaal getranscribeerd om de wekzin te vinden. Alleen het verzoek na de wekzin wordt via uw geconfigureerde gateway verzonden. + Assistent is uit. + Luisteren wordt gestart. + Luisteren naar de wekzin. + Uw verzoek wordt verzonden. + Wachten op het antwoord. + Het antwoord wordt uitgesproken. + Assistent is gepauzeerd totdat chat verbonden en inactief is. + De assistent is gestopt na een fout en probeert het opnieuw wanneer deze gereed is. + Schakel spraak-naar-tekst in bij Machtigingen. + Download het geselecteerde spraakmodel. + Schakel tekst-naar-spraak in bij Machtigingen. + Download de geselecteerde Piper-stem. + Voeg een ElevenLabs API-sleutel toe. + Voeg een ElevenLabs-stem-ID toe. + Gebruik een wekzin van één tot drie woorden en maximaal 64 tekens. + Selecteer en configureer een ondersteunde tekst-naar-spraakprovider. Opgeslagen diff --git a/src/OpenClaw.Tray.WinUI/Strings/zh-cn/Resources.resw b/src/OpenClaw.Tray.WinUI/Strings/zh-cn/Resources.resw index a34de7794..36f32dcc6 100644 --- a/src/OpenClaw.Tray.WinUI/Strings/zh-cn/Resources.resw +++ b/src/OpenClaw.Tray.WinUI/Strings/zh-cn/Resources.resw @@ -2931,6 +2931,12 @@ 语音 + + 助手正在监听唤醒短语，语音输入暂不可用。 + + + 助手正在监听唤醒短语。 + 正在聆听… @@ -6864,6 +6870,30 @@ 分隔条 + 助手（预览） + 在同一句话中说出唤醒短语和请求。匹配的回复将始终朗读。 + 助手模式 + 关闭 + 唤醒短语，一次请求 + 唤醒短语 + OpenClaw + 启用后，麦克风将保持活动状态，并在本地转录语音片段以查找唤醒短语。仅唤醒短语之后的请求会通过已配置的网关发送。 + 助手已关闭。 + 正在启动唤醒监听。 + 正在监听唤醒短语。 + 正在发送请求。 + 正在等待回复。 + 正在朗读回复。 + 助手已暂停，直到聊天已连接且空闲。 + 助手因错误而停止，并会在准备就绪后重试。 + 请在“权限”中启用语音转文字。 + 请下载所选语音模型。 + 请在“权限”中启用文字转语音。 + 请下载所选 Piper 语音。 + 请添加 ElevenLabs API 密钥。 + 请添加 ElevenLabs 语音 ID。 + 请使用一到三个词且不超过 64 个字符的唤醒短语。 + 请选择并配置受支持的文字转语音提供商。 已保存 diff --git a/src/OpenClaw.Tray.WinUI/Strings/zh-tw/Resources.resw b/src/OpenClaw.Tray.WinUI/Strings/zh-tw/Resources.resw index 5f918aa8a..bcdd48b09 100644 --- a/src/OpenClaw.Tray.WinUI/Strings/zh-tw/Resources.resw +++ b/src/OpenClaw.Tray.WinUI/Strings/zh-tw/Resources.resw @@ -2931,6 +2931,12 @@ 語音 + + 助理正在聆聽喚醒短語，語音輸入暫時無法使用。 + + + 助理正在聆聽喚醒短語。 + 正在聆聽… @@ -6864,6 +6870,30 @@ 分隔線 + 助理（預覽） + 在同一句話中說出喚醒詞和要求。符合的回覆一律會朗讀。 + 助理模式 + 關閉 + 喚醒詞，一次要求 + 喚醒詞 + OpenClaw + 啟用後，麥克風會保持作用中，並在本機轉錄語音片段以尋找喚醒詞。只有喚醒詞之後的要求會透過已設定的閘道傳送。 + 助理已關閉。 + 正在啟動喚醒聆聽。 + 正在聆聽喚醒詞。 + 正在傳送要求。 + 正在等候回覆。 + 正在朗讀回覆。 + 助理已暫停，直到聊天已連線且閒置。 + 助理因錯誤而停止，並會在準備就緒後重試。 + 請在「權限」中啟用語音轉文字。 + 請下載選取的語音模型。 + 請在「權限」中啟用文字轉語音。 + 請下載選取的 Piper 語音。 + 請新增 ElevenLabs API 金鑰。 + 請新增 ElevenLabs 語音 ID。 + 請使用一到三個詞且不超過 64 個字元的喚醒詞。 + 請選取並設定支援的文字轉語音提供者。 已儲存 diff --git a/src/OpenClaw.Tray.WinUI/Windows/ChatWindow.xaml.cs b/src/OpenClaw.Tray.WinUI/Windows/ChatWindow.xaml.cs index da3c42f6a..819696d18 100644 --- a/src/OpenClaw.Tray.WinUI/Windows/ChatWindow.xaml.cs +++ b/src/OpenClaw.Tray.WinUI/Windows/ChatWindow.xaml.cs @@ -8,6 +8,7 @@ using OpenClawTray.Chat; using OpenClawTray.Helpers; using OpenClawTray.Services; +using OpenClawTray.Services.VoiceAssistant; using System; using System.Collections.Generic; using System.Diagnostics; @@ -126,6 +127,7 @@ public ChatWindow(string gatewayUrl, string token) app.SettingsChanged += OnAppSettingsChanged; app.ChatProviderChanged += OnAppChatProviderChanged; app.SpeakerMuteChanged += OnSpeakerMuteChanged; + app.VoiceAssistantRuntimeStateChanged += OnVoiceAssistantRuntimeStateChanged; } // Per-surface debug override (DebugPage > "Debug Overrides"). @@ -146,6 +148,27 @@ private void OnSpeakerMuteChanged(bool muted) DispatcherQueue?.TryEnqueue(() => _reactorHost?.SetSpeakerMuted(muted)); } + private void OnVoiceAssistantRuntimeStateChanged(object? sender, EventArgs e) + { + DispatcherQueue?.TryEnqueue(UpdateVoiceAvailability); + } + + private void UpdateVoiceAvailability() + { + if (App.Current is not App app) + return; + + var unavailable = VoiceAssistantChatInputPolicy.IsVoiceInputUnavailable( + app.VoiceAssistantRuntimeState); + _reactorHost?.SetVoiceAvailability( + unavailable + ? LocalizationHelper.GetString(VoiceAssistantChatInputPolicy.WakeListeningReasonKey) + : null, + unavailable + ? LocalizationHelper.GetString(VoiceAssistantChatInputPolicy.WakeListeningStatusKey) + : null); + } + private void OnAppChatProviderChanged(object? sender, EventArgs e) { if (IsClosed) return; @@ -436,6 +459,14 @@ private void TryMountReactorChat() onAttachClick: OnAttachClicked, onSettingsClick: () => appInstance?.ShowHub("voice"), onSpeakerMuteChanged: muted => _ = OnSpeakerMuteChangedAsync(muted), + initialVoiceUnavailableReason: VoiceAssistantChatInputPolicy.IsVoiceInputUnavailable( + appInstance?.VoiceAssistantRuntimeState ?? VoiceAssistantState.Off) + ? LocalizationHelper.GetString(VoiceAssistantChatInputPolicy.WakeListeningReasonKey) + : null, + initialVoiceUnavailableStatus: VoiceAssistantChatInputPolicy.IsVoiceInputUnavailable( + appInstance?.VoiceAssistantRuntimeState ?? VoiceAssistantState.Off) + ? LocalizationHelper.GetString(VoiceAssistantChatInputPolicy.WakeListeningStatusKey) + : null, initialMuted: ShouldStartSpeakerMuted(appInstance?.Settings), isCompact: true); _mountedProvider = provider; @@ -834,6 +865,7 @@ public void ForceClose() app.SettingsChanged -= OnAppSettingsChanged; app.ChatProviderChanged -= OnAppChatProviderChanged; app.SpeakerMuteChanged -= OnSpeakerMuteChanged; + app.VoiceAssistantRuntimeStateChanged -= OnVoiceAssistantRuntimeStateChanged; } OpenClawTray.Chat.DebugChatSurfaceOverrides.Changed -= OnDebugOverrideChanged; IsClosed = true; diff --git a/tests/OpenClaw.Shared.Tests/OpenClawGatewayClientTests.cs b/tests/OpenClaw.Shared.Tests/OpenClawGatewayClientTests.cs index 456e0cf4d..20009f94e 100644 --- a/tests/OpenClaw.Shared.Tests/OpenClawGatewayClientTests.cs +++ b/tests/OpenClaw.Shared.Tests/OpenClawGatewayClientTests.cs @@ -1365,7 +1365,11 @@ public void ProcessRawMessage_SessionMessageAssistantNotification_DependsOnFinal "message": { "role": "assistant", "content": "assistant reply", - "timestamp": 1781631280633 + "timestamp": 1781631280633, + "__openclaw": { + "id": "assistant-message-7", + "seq": 7 + } }, "state": "{{state}}" } @@ -1377,6 +1381,8 @@ public void ProcessRawMessage_SessionMessageAssistantNotification_DependsOnFinal Assert.NotNull(notification); Assert.Equal("assistant reply", notification!.Message); Assert.True(notification.IsChat); + Assert.Equal("assistant-message-7", notification.OpenClawId); + Assert.Equal(7, notification.OpenClawSeq); } else { diff --git a/tests/OpenClaw.Tray.Tests/AppRefactorContractTests.cs b/tests/OpenClaw.Tray.Tests/AppRefactorContractTests.cs index cfac4adf0..f3237d648 100644 --- a/tests/OpenClaw.Tray.Tests/AppRefactorContractTests.cs +++ b/tests/OpenClaw.Tray.Tests/AppRefactorContractTests.cs @@ -4,6 +4,19 @@ namespace OpenClaw.Tray.Tests; public sealed class AppRefactorContractTests { + [Fact] + public void VoiceAssistant_StateMachineStaysOutsideApp() + { + var source = ReadAppSources(); + + Assert.Contains("_voiceAssistantCoordinator?.TryClaimResponse(notification)", source); + Assert.Contains("new VoiceAssistantCoordinator(", source); + Assert.DoesNotContain("enum VoiceAssistantState", source); + Assert.DoesNotContain("VoiceAssistantState.Dispatching", source); + Assert.DoesNotContain("VoiceAssistantState.WaitingForReply", source); + Assert.DoesNotContain("VoiceAssistantState.Speaking", source); + } + [Fact] public void Startup_UsesConnectionManagerAsOnlyGatewayClientOwner() { diff --git a/tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj b/tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj index 903a0b374..71622a021 100644 --- a/tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj +++ b/tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj @@ -84,6 +84,17 @@ + + + + + + + + + + + diff --git a/tests/OpenClaw.Tray.Tests/OpenClawChatDataProviderTests.cs b/tests/OpenClaw.Tray.Tests/OpenClawChatDataProviderTests.cs index e3d294594..e6521f242 100644 --- a/tests/OpenClaw.Tray.Tests/OpenClawChatDataProviderTests.cs +++ b/tests/OpenClaw.Tray.Tests/OpenClawChatDataProviderTests.cs @@ -2,6 +2,7 @@ using OpenClaw.Shared; using OpenClaw.Shared.Telemetry; using OpenClawTray.Chat; +using OpenClawTray.Services.VoiceAssistant; using System.Collections.Concurrent; using System.Diagnostics; using System.Diagnostics.Metrics; @@ -269,6 +270,104 @@ lastChatStateSaveDelay is null && historyRetryScheduler is null && historyFailur private static SessionInfo MainSession() => new() { Key = "main", IsMain = true, DisplayName = "Main session", Status = "active" }; + [Fact] + public async Task VoiceAssistantReadinessChanged_TracksSessionsAndDisconnect() + { + var bridge = new FakeBridge + { + IsConnected = true, + CurrentStatus = ConnectionStatus.Connected, + HasHandshakeSnapshot = true, + MainSessionKey = "main", + Sessions = [MainSession()] + }; + await using var provider = new OpenClawChatDataProvider(bridge); + using var adapter = new VoiceAssistantChatTurnClient(provider); + var readinessChanges = 0; + adapter.ReadinessChanged += () => readinessChanges++; + + bridge.RaiseSessions(bridge.Sessions); + bridge.RaiseSessions(bridge.Sessions); + bridge.RaiseStatus(ConnectionStatus.Disconnected); + bridge.RaiseStatus(ConnectionStatus.Connected); + bridge.RaiseSessions(bridge.Sessions); + + Assert.Equal(3, readinessChanges); + Assert.Equal("main", adapter.GetReadySessionKey()); + } + + [Fact] + public async Task VoiceAssistant_FinalWithoutGatewayMetadata_CorrelatesByTrackedText() + { + var bridge = new FakeBridge + { + IsConnected = true, + CurrentStatus = ConnectionStatus.Connected, + HasHandshakeSnapshot = true, + MainSessionKey = "main", + Sessions = [MainSession()] + }; + bridge.SendResults.Enqueue(new ChatSendResult { RunId = "voice-run", Status = "started" }); + await using var provider = new OpenClawChatDataProvider(bridge); + using var adapter = new VoiceAssistantChatTurnClient(provider); + bridge.RaiseSessions(bridge.Sessions); + + var receipt = await adapter.SendAsync("main", "voice request", CancellationToken.None); + bridge.RaiseChat(new ChatMessageInfo + { + SessionKey = "main", + Role = "assistant", + State = "final", + Text = "voice response" + }); + + var notification = new OpenClawNotification + { + IsChat = true, + SessionKey = "main", + Message = "voice response", + FullMessage = "voice response" + }; + + Assert.True(adapter.IsResponseForTurn(receipt, notification)); + Assert.False(adapter.IsResponseForTurn(receipt, notification)); + } + + [Fact] + public async Task VoiceAssistant_LateFinalAfterCancellation_IsNotRetained() + { + var bridge = new FakeBridge + { + IsConnected = true, + CurrentStatus = ConnectionStatus.Connected, + HasHandshakeSnapshot = true, + MainSessionKey = "main", + Sessions = [MainSession()] + }; + bridge.SendResults.Enqueue(new ChatSendResult { RunId = "voice-run", Status = "started" }); + await using var provider = new OpenClawChatDataProvider(bridge); + using var adapter = new VoiceAssistantChatTurnClient(provider); + bridge.RaiseSessions(bridge.Sessions); + var receipt = await adapter.SendAsync("main", "voice request", CancellationToken.None); + + await adapter.CancelAsync(receipt, CancellationToken.None); + bridge.RaiseChat(new ChatMessageInfo + { + SessionKey = "main", + Role = "assistant", + State = "final", + Text = "late response" + }); + + Assert.False(adapter.IsResponseForTurn(receipt, new OpenClawNotification + { + IsChat = true, + SessionKey = "main", + Message = "late response", + FullMessage = "late response" + })); + } + private static AgentEventInfo MakeAgentEvent(string stream, string json, string sessionKey = "main", string? runId = null) { var doc = JsonDocument.Parse(json); diff --git a/tests/OpenClaw.Tray.Tests/Presentation/SettingsStoreTests.cs b/tests/OpenClaw.Tray.Tests/Presentation/SettingsStoreTests.cs index 716f5b355..73f61c689 100644 --- a/tests/OpenClaw.Tray.Tests/Presentation/SettingsStoreTests.cs +++ b/tests/OpenClaw.Tray.Tests/Presentation/SettingsStoreTests.cs @@ -1,3 +1,4 @@ +using OpenClaw.Shared; using OpenClawTray.Presentation; using OpenClawTray.Services; @@ -26,11 +27,15 @@ public void Current_ReflectsUnderlyingSettings() { settings.GlobalHotkeyEnabled = true; settings.NotificationSound = "Subtle"; + settings.VoiceAssistantMode = VoiceAssistantSettingsPolicy.WakeOneShotMode; + settings.VoiceAssistantWakePhrase = "Hey OpenClaw"; var snapshot = store.Current; Assert.True(snapshot.GlobalHotkeyEnabled); Assert.Equal("Subtle", snapshot.NotificationSound); + Assert.Equal(VoiceAssistantSettingsPolicy.WakeOneShotMode, snapshot.VoiceAssistantMode); + Assert.Equal("Hey OpenClaw", snapshot.VoiceAssistantWakePhrase); } } @@ -47,10 +52,14 @@ public void Update_MutatesAndPersists_Once() { e.GlobalHotkeyEnabled = true; e.NotifyHealth = true; + e.VoiceAssistantMode = VoiceAssistantSettingsPolicy.WakeOneShotMode; + e.VoiceAssistantWakePhrase = "Hey OpenClaw"; }); Assert.True(settings.GlobalHotkeyEnabled); Assert.True(settings.NotifyHealth); + Assert.Equal(VoiceAssistantSettingsPolicy.WakeOneShotMode, settings.VoiceAssistantMode); + Assert.Equal("Hey OpenClaw", settings.VoiceAssistantWakePhrase); Assert.Equal(1, saves); // batched: one save for both edits } } diff --git a/tests/OpenClaw.Tray.Tests/Presentation/VoiceAssistantSettingsViewModelTests.cs b/tests/OpenClaw.Tray.Tests/Presentation/VoiceAssistantSettingsViewModelTests.cs new file mode 100644 index 000000000..d20d8b3af --- /dev/null +++ b/tests/OpenClaw.Tray.Tests/Presentation/VoiceAssistantSettingsViewModelTests.cs @@ -0,0 +1,174 @@ +using OpenClaw.Shared; +using OpenClawTray.Presentation; +using OpenClawTray.Services; +using OpenClawTray.Services.VoiceAssistant; + +namespace OpenClaw.Tray.Tests.Presentation; + +public sealed class VoiceAssistantSettingsViewModelTests +{ + [Fact] + public void Activate_LoadsDefaultOffState() + { + using var fixture = new Fixture(); + + fixture.ViewModel.Activate(null); + + Assert.Equal(0, fixture.ViewModel.SelectedModeIndex); + Assert.Equal(VoiceAssistantSettingsPolicy.DefaultWakePhrase, fixture.ViewModel.WakePhraseDraft); + Assert.True(fixture.ViewModel.CanEnableWakeMode); + Assert.Equal("VoiceSettingsPage_AssistantStatusOff", fixture.ViewModel.StatusText); + } + + [Fact] + public void WakePhraseDraft_ValidatesWithoutPersisting() + { + using var fixture = new Fixture(); + fixture.ViewModel.Activate(null); + + fixture.ViewModel.WakePhraseDraft = "too many words in this phrase"; + + Assert.False(fixture.ViewModel.CanEnableWakeMode); + Assert.Equal("VoiceSettingsPage_AssistantStatusWakeInvalid", fixture.ViewModel.StatusText); + Assert.Equal(VoiceAssistantSettingsPolicy.DefaultWakePhrase, fixture.Settings.VoiceAssistantWakePhrase); + Assert.Equal(0, fixture.AppCommands.NotifySettingsSavedCount); + } + + [Fact] + public void WakePhraseDraft_UsesCachedPrerequisitesWhileTyping() + { + using var fixture = new Fixture(); + fixture.ViewModel.Activate(null); + var readinessCalls = fixture.Environment.GetReadinessCount; + + fixture.ViewModel.WakePhraseDraft = "Hey"; + fixture.ViewModel.WakePhraseDraft = "Hey Claw"; + fixture.ViewModel.WakePhraseDraft = "too many words in this phrase"; + + Assert.Equal(readinessCalls, fixture.Environment.GetReadinessCount); + Assert.False(fixture.ViewModel.CanEnableWakeMode); + } + + [Fact] + public void SelectingWakeMode_WithInvalidPhrase_IsRefused() + { + using var fixture = new Fixture(); + fixture.ViewModel.Activate(null); + fixture.ViewModel.WakePhraseDraft = "too many words in this phrase"; + + fixture.ViewModel.SelectedModeIndex = 1; + + Assert.Equal(0, fixture.ViewModel.SelectedModeIndex); + Assert.Equal(VoiceAssistantSettingsPolicy.OffMode, fixture.Settings.VoiceAssistantMode); + Assert.Equal(0, fixture.AppCommands.NotifySettingsSavedCount); + } + + [Fact] + public void CommitWakePhrase_NormalizesPersistsAndNotifies() + { + using var fixture = new Fixture(); + fixture.ViewModel.Activate(null); + fixture.ViewModel.WakePhraseDraft = " Hey Claw "; + + var committed = fixture.ViewModel.CommitWakePhrase(); + + Assert.True(committed); + Assert.Equal("Hey Claw", fixture.ViewModel.WakePhraseDraft); + Assert.Equal("Hey Claw", fixture.Settings.VoiceAssistantWakePhrase); + Assert.Equal(1, fixture.AppCommands.NotifySettingsSavedCount); + } + + [Fact] + public void SelectingWakeMode_CommitsDraftAndPersistsMode() + { + using var fixture = new Fixture(); + fixture.ViewModel.Activate(null); + fixture.ViewModel.WakePhraseDraft = "Hey Claw"; + + fixture.ViewModel.SelectedModeIndex = 1; + + Assert.Equal(1, fixture.ViewModel.SelectedModeIndex); + Assert.Equal("Hey Claw", fixture.Settings.VoiceAssistantWakePhrase); + Assert.Equal(VoiceAssistantSettingsPolicy.WakeOneShotMode, fixture.Settings.VoiceAssistantMode); + Assert.Equal(2, fixture.AppCommands.NotifySettingsSavedCount); + } + + [Theory] + [InlineData(VoiceAssistantState.WakeListening, "VoiceSettingsPage_AssistantStatusListening")] + [InlineData(VoiceAssistantState.WaitingForReply, "VoiceSettingsPage_AssistantStatusWaiting")] + [InlineData(VoiceAssistantState.Speaking, "VoiceSettingsPage_AssistantStatusSpeaking")] + [InlineData(VoiceAssistantState.Unavailable, "VoiceSettingsPage_AssistantStatusPaused")] + public void EnvironmentChange_ProjectsRuntimeStatus( + VoiceAssistantState state, + string expectedStatus) + { + using var fixture = new Fixture(); + fixture.ViewModel.Activate(null); + + fixture.Environment.RuntimeState = state; + fixture.Environment.RaiseChanged(); + + Assert.Equal(expectedStatus, fixture.ViewModel.StatusText); + } + + [Fact] + public void ExternalSettingsChange_ReloadsWithoutNotifyingRuntime() + { + using var fixture = new Fixture(); + fixture.ViewModel.Activate(null); + fixture.Settings.VoiceAssistantWakePhrase = "Computer"; + fixture.Settings.VoiceAssistantMode = VoiceAssistantSettingsPolicy.WakeOneShotMode; + + fixture.Settings.Save(); + + Assert.Equal("Computer", fixture.ViewModel.WakePhraseDraft); + Assert.Equal(1, fixture.ViewModel.SelectedModeIndex); + Assert.Equal(0, fixture.AppCommands.NotifySettingsSavedCount); + } + + private sealed class Fixture : IDisposable + { + private readonly TempDir _temp = new(); + + public Fixture() + { + Settings = new SettingsManager(_temp.Path); + Store = new SettingsStore(Settings, new RecordingUiDispatcher()); + Environment = new FakeEnvironment(); + AppCommands = new FakeAppCommands(); + ViewModel = new VoiceAssistantSettingsViewModel(Store, Environment, AppCommands); + } + + public SettingsManager Settings { get; } + public SettingsStore Store { get; } + public FakeEnvironment Environment { get; } + public FakeAppCommands AppCommands { get; } + public VoiceAssistantSettingsViewModel ViewModel { get; } + + public void Dispose() + { + ViewModel.Dispose(); + _temp.Dispose(); + } + } + + private sealed class FakeEnvironment : IVoiceAssistantSettingsEnvironment + { + public event EventHandler? Changed; + + public VoiceAssistantState RuntimeState { get; set; } = VoiceAssistantState.Off; + public int GetReadinessCount { get; private set; } + + public VoiceAssistantReadinessResult GetReadiness(string wakePhrase) + { + GetReadinessCount++; + return VoiceAssistantSettingsPolicy.TryNormalizeWakePhrase(wakePhrase, out _) + ? new(true, VoiceAssistantReadinessReason.Ready) + : new(false, VoiceAssistantReadinessReason.WakePhraseInvalid); + } + + public string GetString(string key) => key; + + public void RaiseChanged() => Changed?.Invoke(this, EventArgs.Empty); + } +} diff --git a/tests/OpenClaw.Tray.Tests/Services/VoiceAssistantChatInputPolicyTests.cs b/tests/OpenClaw.Tray.Tests/Services/VoiceAssistantChatInputPolicyTests.cs new file mode 100644 index 000000000..40ba9ff29 --- /dev/null +++ b/tests/OpenClaw.Tray.Tests/Services/VoiceAssistantChatInputPolicyTests.cs @@ -0,0 +1,27 @@ +using OpenClawTray.Services.VoiceAssistant; + +namespace OpenClaw.Tray.Tests.Services; + +public sealed class VoiceAssistantChatInputPolicyTests +{ + [Theory] + [InlineData(VoiceAssistantState.Off)] + [InlineData(VoiceAssistantState.Unavailable)] + [InlineData(VoiceAssistantState.Starting)] + [InlineData(VoiceAssistantState.Dispatching)] + [InlineData(VoiceAssistantState.WaitingForReply)] + [InlineData(VoiceAssistantState.Speaking)] + [InlineData(VoiceAssistantState.Error)] + public void VoiceInput_RemainsAvailable_WhenAssistantDoesNotOwnWakeCapture( + VoiceAssistantState state) + { + Assert.False(VoiceAssistantChatInputPolicy.IsVoiceInputUnavailable(state)); + } + + [Fact] + public void VoiceInput_IsUnavailable_WhileAssistantOwnsWakeCapture() + { + Assert.True(VoiceAssistantChatInputPolicy.IsVoiceInputUnavailable( + VoiceAssistantState.WakeListening)); + } +} diff --git a/tests/OpenClaw.Tray.Tests/Services/VoiceAssistantCoordinatorTests.cs b/tests/OpenClaw.Tray.Tests/Services/VoiceAssistantCoordinatorTests.cs new file mode 100644 index 000000000..8bd335658 --- /dev/null +++ b/tests/OpenClaw.Tray.Tests/Services/VoiceAssistantCoordinatorTests.cs @@ -0,0 +1,404 @@ +using OpenClaw.Shared; +using OpenClawTray.Services; +using OpenClawTray.Services.VoiceAssistant; + +namespace OpenClaw.Tray.Tests.Services; + +public sealed class VoiceAssistantCoordinatorTests +{ + [Fact] + public async Task BackToBackMatches_ProduceOneSend() + { + var input = new FakeInput(); + var chat = new FakeChat(); + var speaker = new FakeSpeaker(); + await using var coordinator = Create(input, chat, speaker); + await coordinator.ReconcileAsync(); + + input.Emit("OpenClaw, first request"); + input.Emit("OpenClaw, second request"); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WaitingForReply); + + Assert.Equal(new[] { "First request" }, chat.Requests); + } + + [Fact] + public async Task QueuedDisposition_IsCanceledAndNeverWaits() + { + var input = new FakeInput(); + var chat = new FakeChat { Disposition = VoiceAssistantSendDisposition.Queued }; + var speaker = new FakeSpeaker(); + await using var coordinator = Create(input, chat, speaker); + await coordinator.ReconcileAsync(); + + input.Emit("OpenClaw, queued request"); + await WaitUntilAsync(() => chat.Canceled == 1); + + Assert.NotEqual(VoiceAssistantState.WaitingForReply, coordinator.State); + Assert.Empty(speaker.Spoken); + } + + [Fact] + public async Task MatchingFinal_IsClaimedAndDuplicateIsSuppressedWithoutReplay() + { + var input = new FakeInput(); + var chat = new FakeChat(); + var speaker = new FakeSpeaker(); + await using var coordinator = Create(input, chat, speaker); + await coordinator.ReconcileAsync(); + input.Emit("OpenClaw, answer once"); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WaitingForReply); + var notification = chat.CreateMatchingNotification("the answer"); + + Assert.True(coordinator.TryClaimResponse(notification)); + Assert.True(coordinator.TryClaimResponse(notification)); + await WaitUntilAsync(() => speaker.Spoken.Count == 1); + + Assert.Equal(new[] { "the answer" }, speaker.Spoken); + } + + [Fact] + public async Task TrackedFinalWithoutGatewayMetadata_IsClaimedAndRestartsListening() + { + var input = new FakeInput(); + var chat = new FakeChat { AllowMissingMetadataFallback = true }; + var speaker = new FakeSpeaker { BlockUntilReleased = true }; + await using var coordinator = Create(input, chat, speaker); + await coordinator.ReconcileAsync(); + input.Emit("OpenClaw, answer without metadata"); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WaitingForReply); + var notification = chat.CreateMatchingNotification("fallback answer"); + notification.OpenClawId = null; + notification.OpenClawSeq = null; + + Assert.True(coordinator.TryClaimResponse(notification)); + Assert.False(coordinator.TryClaimResponse(new OpenClawNotification + { + IsChat = true, + SessionKey = "other", + Message = "fallback answer", + FullMessage = "fallback answer" + })); + Assert.True(coordinator.TryClaimResponse(notification)); + speaker.Release(); + await WaitUntilAsync(() => + speaker.Spoken.Count == 1 && + coordinator.State == VoiceAssistantState.WakeListening); + + Assert.True(input.Starts >= 2); + Assert.Equal(new[] { "fallback answer" }, speaker.Spoken); + } + + [Fact] + public async Task SeparateMetadataFreeTurns_WithSameReply_AreBothSpoken() + { + var input = new FakeInput(); + var chat = new FakeChat { AllowMissingMetadataFallback = true }; + var speaker = new FakeSpeaker(); + await using var coordinator = Create(input, chat, speaker); + await coordinator.ReconcileAsync(); + + input.Emit("OpenClaw, first request"); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WaitingForReply); + var first = chat.CreateMatchingNotification("Okay"); + first.OpenClawId = null; + first.OpenClawSeq = null; + Assert.True(coordinator.TryClaimResponse(first)); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WakeListening); + + input.Emit("OpenClaw, second request"); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WaitingForReply); + var second = chat.CreateMatchingNotification("Okay"); + second.OpenClawId = null; + second.OpenClawSeq = null; + Assert.True(coordinator.TryClaimResponse(second)); + await WaitUntilAsync(() => speaker.Spoken.Count == 2); + + Assert.Equal(new[] { "Okay", "Okay" }, speaker.Spoken); + } + + [Fact] + public async Task MetadataFreeReply_AfterTrackedTurnCompletes_IsNotClaimed() + { + var input = new FakeInput(); + var chat = new FakeChat { AllowMissingMetadataFallback = true }; + var speaker = new FakeSpeaker(); + await using var coordinator = Create(input, chat, speaker); + await coordinator.ReconcileAsync(); + + input.Emit("OpenClaw, tracked request"); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WaitingForReply); + var notification = chat.CreateMatchingNotification("Okay"); + notification.OpenClawId = null; + notification.OpenClawSeq = null; + Assert.True(coordinator.TryClaimResponse(notification)); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WakeListening); + + Assert.False(coordinator.TryClaimResponse(notification)); + Assert.Equal(new[] { "Okay" }, speaker.Spoken); + } + + [Fact] + public async Task UnrelatedOrUncorrelatedFinal_IsNotClaimed() + { + var input = new FakeInput(); + var chat = new FakeChat(); + var speaker = new FakeSpeaker(); + await using var coordinator = Create(input, chat, speaker); + await coordinator.ReconcileAsync(); + input.Emit("OpenClaw, expected request"); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WaitingForReply); + + Assert.False(coordinator.TryClaimResponse(new OpenClawNotification + { + IsChat = true, + SessionKey = "main", + OpenClawId = "unrelated", + OpenClawSeq = 12, + Message = "wrong" + })); + Assert.False(coordinator.TryClaimResponse(new OpenClawNotification + { + IsChat = true, + SessionKey = "main", + Message = "missing identity" + })); + Assert.Empty(speaker.Spoken); + } + + [Fact] + public async Task Timeout_DropsTurnAndRestartsListening() + { + var input = new FakeInput(); + var chat = new FakeChat(); + var speaker = new FakeSpeaker(); + await using var coordinator = Create( + input, + chat, + speaker, + timeout: TimeSpan.FromMilliseconds(20)); + await coordinator.ReconcileAsync(); + input.Emit("OpenClaw, never answered"); + + await WaitUntilAsync(() => input.Starts >= 2); + + Assert.Equal(VoiceAssistantState.WakeListening, coordinator.State); + Assert.Empty(speaker.Spoken); + } + + [Fact] + public async Task ModeOffDuringSpeaking_CancelsPlaybackAndStopsListening() + { + var input = new FakeInput(); + var chat = new FakeChat(); + var speaker = new FakeSpeaker { BlockUntilCanceled = true }; + var configuration = new VoiceAssistantConfiguration(true, true, "OpenClaw"); + await using var coordinator = Create( + input, + chat, + speaker, + configuration: () => configuration); + await coordinator.ReconcileAsync(); + input.Emit("OpenClaw, stop speaking"); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WaitingForReply); + + Assert.True(coordinator.TryClaimResponse(chat.CreateMatchingNotification("long answer"))); + await WaitUntilAsync(() => speaker.Started); + configuration = configuration with { Enabled = false }; + await coordinator.ReconcileAsync(); + + await WaitUntilAsync(() => speaker.Canceled); + Assert.Equal(VoiceAssistantState.Off, coordinator.State); + Assert.Equal(1, chat.Canceled); + } + + [Fact] + public async Task DisconnectDuringWaiting_CancelsTurnAndPauses() + { + var input = new FakeInput(); + var chat = new FakeChat(); + var speaker = new FakeSpeaker(); + await using var coordinator = Create(input, chat, speaker); + await coordinator.ReconcileAsync(); + input.Emit("OpenClaw, wait for disconnect"); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WaitingForReply); + + chat.ReadySessionKey = null; + await coordinator.ReconcileAsync(); + + Assert.Equal(VoiceAssistantState.Unavailable, coordinator.State); + Assert.Equal(1, chat.Canceled); + } + + [Fact] + public async Task StartupReadinessArrival_StartsWakeListeningWithoutSettingsToggle() + { + var input = new FakeInput(); + var chat = new FakeChat { ReadySessionKey = null }; + var speaker = new FakeSpeaker(); + await using var coordinator = Create(input, chat, speaker); + + await coordinator.ReconcileAsync(); + Assert.Equal(VoiceAssistantState.Unavailable, coordinator.State); + + chat.SetReadySessionKey("main"); + + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WakeListening); + Assert.Equal(1, input.Starts); + + chat.SetReadySessionKey(null); + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.Unavailable); + chat.SetReadySessionKey("main"); + + await WaitUntilAsync(() => coordinator.State == VoiceAssistantState.WakeListening); + Assert.Equal(2, input.Starts); + } + + private static VoiceAssistantCoordinator Create( + FakeInput input, + FakeChat chat, + FakeSpeaker speaker, + TimeSpan? timeout = null, + Func? configuration = null) => + new( + input, + chat, + speaker, + configuration ?? (() => new VoiceAssistantConfiguration( + Enabled: true, + LocalPrerequisitesReady: true, + WakePhrase: "OpenClaw")), + timeout); + + private static async Task WaitUntilAsync(Func predicate) + { + using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(2)); + while (!predicate()) + await Task.Delay(5, timeout.Token); + } + + private sealed class FakeInput : IVoiceAssistantInput + { + public event Action? UtteranceCompleted; + public event Action? CaptureAvailable; + + public int Starts { get; private set; } + public int Stops { get; private set; } + + public Task StartAsync(CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + Starts++; + return Task.CompletedTask; + } + + public Task StopAsync() + { + Stops++; + return Task.CompletedTask; + } + + public void Emit(string transcript) => UtteranceCompleted?.Invoke(transcript); + public void ReleaseCapture() => CaptureAvailable?.Invoke(); + } + + private sealed class FakeChat : IVoiceAssistantChatTurnClient + { + private VoiceAssistantTurnReceipt? _lastReceipt; + + public VoiceAssistantSendDisposition Disposition { get; set; } = VoiceAssistantSendDisposition.Direct; + public bool AllowMissingMetadataFallback { get; set; } + public string? ReadySessionKey { get; set; } = "main"; + public List Requests { get; } = new(); + public int Canceled { get; private set; } + + public event Action? ReadinessChanged; + + public string? GetReadySessionKey() => ReadySessionKey; + + public void SetReadySessionKey(string? sessionKey) + { + ReadySessionKey = sessionKey; + ReadinessChanged?.Invoke(); + } + + public Task SendAsync( + string sessionKey, + string request, + CancellationToken cancellationToken) + { + Requests.Add(request); + _lastReceipt = new VoiceAssistantTurnReceipt( + Disposition, + sessionKey, + $"message-{Requests.Count}", + $"run-{Requests.Count}", + PreSendSequence: 10); + return Task.FromResult(_lastReceipt); + } + + public Task CancelAsync( + VoiceAssistantTurnReceipt receipt, + CancellationToken cancellationToken) + { + Canceled++; + return Task.CompletedTask; + } + + public bool IsResponseForTurn( + VoiceAssistantTurnReceipt receipt, + OpenClawNotification notification) => + notification.OpenClawId == $"reply-{receipt.SendRunId}" || + AllowMissingMetadataFallback && + notification.OpenClawId is null && + notification.OpenClawSeq is null; + + public OpenClawNotification CreateMatchingNotification(string message) => + new() + { + IsChat = true, + SessionKey = _lastReceipt!.SessionKey, + OpenClawId = $"reply-{_lastReceipt.SendRunId}", + OpenClawSeq = 11, + Message = message, + FullMessage = message + }; + } + + private sealed class FakeSpeaker : IVoiceAssistantSpeaker + { + private readonly TaskCompletionSource _release = + new(TaskCreationOptions.RunContinuationsAsynchronously); + + public List Spoken { get; } = new(); + public bool BlockUntilCanceled { get; set; } + public bool BlockUntilReleased { get; set; } + public bool Started { get; private set; } + public bool Canceled { get; private set; } + + public async Task SpeakAsync(string text, CancellationToken cancellationToken) + { + Spoken.Add(text); + Started = true; + if (BlockUntilReleased) + { + await _release.Task.WaitAsync(cancellationToken); + return; + } + if (!BlockUntilCanceled) + return; + + try + { + await Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + Canceled = true; + throw; + } + } + + public void Release() => _release.TrySetResult(true); + } +} diff --git a/tests/OpenClaw.Tray.Tests/Services/VoiceAssistantReadinessTests.cs b/tests/OpenClaw.Tray.Tests/Services/VoiceAssistantReadinessTests.cs new file mode 100644 index 000000000..2a17f9a68 --- /dev/null +++ b/tests/OpenClaw.Tray.Tests/Services/VoiceAssistantReadinessTests.cs @@ -0,0 +1,72 @@ +using OpenClaw.Shared.Capabilities; +using OpenClawTray.Services.VoiceAssistant; + +namespace OpenClaw.Tray.Tests.Services; + +public sealed class VoiceAssistantReadinessTests +{ + public static TheoryData MissingPrerequisites => + new() + { + { ReadyInput() with { SttEnabled = false }, VoiceAssistantReadinessReason.SttDisabled }, + { ReadyInput() with { SttModelDownloaded = false }, VoiceAssistantReadinessReason.SttModelMissing }, + { ReadyInput() with { TtsEnabled = false }, VoiceAssistantReadinessReason.TtsDisabled }, + { + ReadyInput() with + { + TtsProvider = TtsCapability.PiperProvider, + PiperVoiceDownloaded = false + }, + VoiceAssistantReadinessReason.PiperVoiceMissing + }, + { + ReadyInput() with + { + TtsProvider = TtsCapability.ElevenLabsProvider, + HasElevenLabsApiKey = false + }, + VoiceAssistantReadinessReason.ElevenLabsApiKeyMissing + }, + { + ReadyInput() with + { + TtsProvider = TtsCapability.ElevenLabsProvider, + HasElevenLabsVoiceId = false + }, + VoiceAssistantReadinessReason.ElevenLabsVoiceMissing + }, + { ReadyInput() with { WakePhrase = "too many words in this phrase" }, VoiceAssistantReadinessReason.WakePhraseInvalid } + }; + + [Fact] + public void Evaluate_ReadyWindowsConfiguration_ReturnsReady() + { + var result = VoiceAssistantReadiness.Evaluate(ReadyInput()); + + Assert.True(result.IsReady); + Assert.Equal(VoiceAssistantReadinessReason.Ready, result.Reason); + } + + [Theory] + [MemberData(nameof(MissingPrerequisites))] + public void Evaluate_MissingPrerequisite_ReturnsSpecificReason( + VoiceAssistantReadinessInput input, + VoiceAssistantReadinessReason expected) + { + var result = VoiceAssistantReadiness.Evaluate(input); + + Assert.False(result.IsReady); + Assert.Equal(expected, result.Reason); + } + + private static VoiceAssistantReadinessInput ReadyInput() => + new( + SttEnabled: true, + SttModelDownloaded: true, + TtsEnabled: true, + TtsProvider: TtsCapability.WindowsProvider, + PiperVoiceDownloaded: true, + HasElevenLabsApiKey: true, + HasElevenLabsVoiceId: true, + WakePhrase: "OpenClaw"); +} diff --git a/tests/OpenClaw.Tray.Tests/Services/VoiceCaptureLeaseGateTests.cs b/tests/OpenClaw.Tray.Tests/Services/VoiceCaptureLeaseGateTests.cs new file mode 100644 index 000000000..d97c9930e --- /dev/null +++ b/tests/OpenClaw.Tray.Tests/Services/VoiceCaptureLeaseGateTests.cs @@ -0,0 +1,46 @@ +using OpenClawTray.Services; + +namespace OpenClaw.Tray.Tests.Services; + +public sealed class VoiceCaptureLeaseGateTests +{ + [Fact] + public void Acquire_WhenOwned_ThrowsTypedBusyError() + { + var gate = new VoiceCaptureLeaseGate(); + using var lease = gate.Acquire(VoiceCaptureKind.WakeListening); + + var error = Assert.Throws( + () => gate.Acquire(VoiceCaptureKind.PushToTalk)); + + Assert.Equal(VoiceCaptureKind.WakeListening, error.ActiveCapture); + } + + [Fact] + public void Dispose_ReleasesOnceAndRaisesAvailability() + { + var gate = new VoiceCaptureLeaseGate(); + var available = 0; + gate.Available += () => available++; + var lease = gate.Acquire(VoiceCaptureKind.ListenOnce); + + lease.Dispose(); + lease.Dispose(); + + Assert.Equal(1, available); + using var next = gate.Acquire(VoiceCaptureKind.FixedDuration); + } + + [Fact] + public void SharedGate_EnforcesProcessWideCaptureOwnership() + { + var firstServiceGate = VoiceCaptureLeaseGate.Shared; + var secondServiceGate = VoiceCaptureLeaseGate.Shared; + using var lease = firstServiceGate.Acquire(VoiceCaptureKind.WakeListening); + + var error = Assert.Throws( + () => secondServiceGate.Acquire(VoiceCaptureKind.FixedDuration)); + + Assert.Equal(VoiceCaptureKind.WakeListening, error.ActiveCapture); + } +} diff --git a/tests/OpenClaw.Tray.Tests/Services/VoiceWakeGateTests.cs b/tests/OpenClaw.Tray.Tests/Services/VoiceWakeGateTests.cs new file mode 100644 index 000000000..f7a07e574 --- /dev/null +++ b/tests/OpenClaw.Tray.Tests/Services/VoiceWakeGateTests.cs @@ -0,0 +1,41 @@ +using OpenClawTray.Services.VoiceAssistant; + +namespace OpenClaw.Tray.Tests.Services; + +public sealed class VoiceWakeGateTests +{ + [Theory] + [InlineData("OpenClaw, summarize my inbox", "OpenClaw", "Summarize my inbox")] + [InlineData(" openclaw: What's next?", "OpenClaw", "What's next?")] + [InlineData("Hey OpenClaw - check weather", "hey openclaw", "Check weather")] + [InlineData("OPENCLAW\uFF0C list reminders", "OpenClaw", "List reminders")] + [InlineData("OpenClaw, élargis ce résumé", "OpenClaw", "Élargis ce résumé")] + [InlineData("OpenClaw, 123 reminders", "OpenClaw", "123 reminders")] + public void TryExtractRequest_MatchingPrefix_ReturnsTrailingRequest( + string transcript, + string wakePhrase, + string expected) + { + var matched = VoiceWakeGate.TryExtractRequest(transcript, wakePhrase, out var request); + + Assert.True(matched); + Assert.Equal(expected, request); + } + + [Theory] + [InlineData(null, "OpenClaw")] + [InlineData("", "OpenClaw")] + [InlineData("OpenClaw", "OpenClaw")] + [InlineData("OpenClaw !!!", "OpenClaw")] + [InlineData("Please OpenClaw summarize", "OpenClaw")] + [InlineData("OpenClawish summarize", "OpenClaw")] + [InlineData("Open summarize", "OpenClaw")] + [InlineData("OpenClaw summarize", "too many words in this phrase")] + public void TryExtractRequest_NonMatch_ReturnsFalse(string? transcript, string wakePhrase) + { + var matched = VoiceWakeGate.TryExtractRequest(transcript, wakePhrase, out var request); + + Assert.False(matched); + Assert.Empty(request); + } +} diff --git a/tests/OpenClaw.Tray.Tests/SettingsRoundTripTests.cs b/tests/OpenClaw.Tray.Tests/SettingsRoundTripTests.cs index 1dc58374a..969d672d8 100644 --- a/tests/OpenClaw.Tray.Tests/SettingsRoundTripTests.cs +++ b/tests/OpenClaw.Tray.Tests/SettingsRoundTripTests.cs @@ -44,6 +44,8 @@ public void RoundTrip_AllFields_Preserved() SttSilenceTimeout = 2.5f, VoiceTtsEnabled = false, VoiceAudioFeedback = false, + VoiceAssistantMode = VoiceAssistantSettingsPolicy.WakeOneShotMode, + VoiceAssistantWakePhrase = "Hey OpenClaw", NodeTtsEnabled = true, TtsProvider = "elevenlabs", TtsElevenLabsApiKey = "elevenlabs-key", @@ -108,6 +110,8 @@ public void RoundTrip_AllFields_Preserved() Assert.Equal(original.SttSilenceTimeout, restored.SttSilenceTimeout); Assert.Equal(original.VoiceTtsEnabled, restored.VoiceTtsEnabled); Assert.Equal(original.VoiceAudioFeedback, restored.VoiceAudioFeedback); + Assert.Equal(original.VoiceAssistantMode, restored.VoiceAssistantMode); + Assert.Equal(original.VoiceAssistantWakePhrase, restored.VoiceAssistantWakePhrase); Assert.Equal(original.NodeTtsEnabled, restored.NodeTtsEnabled); Assert.Equal(original.TtsProvider, restored.TtsProvider); Assert.Equal(original.TtsElevenLabsApiKey, restored.TtsElevenLabsApiKey); @@ -184,6 +188,8 @@ public void MissingFields_UseDefaults() Assert.True(settings.NodeBrowserProxyEnabled); Assert.False(settings.NodeSttEnabled); Assert.Equal("auto", settings.SttLanguage); + Assert.Equal(VoiceAssistantSettingsPolicy.OffMode, settings.VoiceAssistantMode); + Assert.Equal(VoiceAssistantSettingsPolicy.DefaultWakePhrase, settings.VoiceAssistantWakePhrase); Assert.False(settings.NodeTtsEnabled); Assert.Equal("piper", settings.TtsProvider); Assert.Null(settings.TtsElevenLabsApiKey); @@ -207,6 +213,40 @@ public void MissingFields_UseDefaults() Assert.Null(settings.UserRules); } + [Theory] + [InlineData("unexpected", "OpenClaw", "off", "OpenClaw")] + [InlineData("wake-one-shot", "too many words for a wake phrase", "off", "OpenClaw")] + [InlineData("WAKE-ONE-SHOT", " Hey OpenClaw ", "wake-one-shot", "Hey OpenClaw")] + public void SettingsManager_NormalizesAssistantSettings( + string mode, + string wakePhrase, + string expectedMode, + string expectedWakePhrase) + { + var dir = Path.Combine(Path.GetTempPath(), "OpenClaw.Tray.Tests", Guid.NewGuid().ToString("N")); + try + { + Directory.CreateDirectory(dir); + File.WriteAllText( + Path.Combine(dir, "settings.json"), + JsonSerializer.Serialize(new + { + VoiceAssistantMode = mode, + VoiceAssistantWakePhrase = wakePhrase + })); + + var settings = new SettingsManager(dir); + + Assert.Equal(expectedMode, settings.VoiceAssistantMode); + Assert.Equal(expectedWakePhrase, settings.VoiceAssistantWakePhrase); + } + finally + { + if (Directory.Exists(dir)) + Directory.Delete(dir, recursive: true); + } + } + [Fact] public void SettingsManager_OpenTelemetryEndpoint_TrimsAndClearsEmptyValues() { diff --git a/tests/OpenClaw.Tray.Tests/SpeechInputContractTests.cs b/tests/OpenClaw.Tray.Tests/SpeechInputContractTests.cs index 2eafce736..85c882220 100644 --- a/tests/OpenClaw.Tray.Tests/SpeechInputContractTests.cs +++ b/tests/OpenClaw.Tray.Tests/SpeechInputContractTests.cs @@ -15,6 +15,16 @@ public void VoiceService_DoesNotLoadNativeVad_OnRecordStartup() Assert.DoesNotContain("DownloadVadModelAsync", cs); } + [Fact] + public void VoiceService_SerializesContinuousCaptureStartAndStop() + { + var cs = Read("src", "OpenClaw.Tray.WinUI", "Services", "VoiceService.cs"); + + Assert.Contains("private readonly SemaphoreSlim _sessionGate = new(1, 1);", cs); + Assert.Contains("await _sessionGate.WaitAsync().ConfigureAwait(false);", cs); + Assert.Contains("await CleanupSessionCoreAsync().ConfigureAwait(false);", cs); + } + [Fact] public void AudioPipeline_UsesManagedEnergyVad_NotNativeVad() { @@ -50,6 +60,30 @@ public void ChatVoiceDialogs_RouteDisabledSttCapabilityToPermissions_AndMissingM Assert.Contains("Speech-to-Text is on, but voice input will not work until at least one speech model is downloaded.", resources); } + [Fact] + public void ChatComposer_ExplainsAssistantWakeListeningMicrophoneOwnership() + { + var composer = Read("src", "OpenClaw.Tray.WinUI", "Chat", "OpenClawReactorChatRoot.cs"); + var host = Read("src", "OpenClaw.Tray.WinUI", "Chat", "ReactorChatHostExtensions.cs"); + var chatPage = Read("src", "OpenClaw.Tray.WinUI", "Pages", "ChatPage.xaml.cs"); + var chatWindow = Read("src", "OpenClaw.Tray.WinUI", "Windows", "ChatWindow.xaml.cs"); + var resources = Read("src", "OpenClaw.Tray.WinUI", "Strings", "en-us", "Resources.resw"); + + Assert.Contains("props.VoiceUnavailableReason is null", composer); + Assert.Contains("props.VoiceUnavailableStatus is { } voiceUnavailableStatus", composer); + Assert.Contains("SetName(", composer); + Assert.Contains("SetHelpText(", composer); + Assert.Contains("AutomationLiveSetting.Polite", composer); + Assert.Contains("SetVoiceAvailability", host); + Assert.Contains("VoiceAssistantRuntimeStateChanged", chatPage); + Assert.Contains("VoiceAssistantRuntimeStateChanged", chatWindow); + Assert.Contains( + "ChatHost.Visibility = Visibility.Visible;\r\n UpdateVoiceAvailability();", + chatPage); + Assert.Contains("Voice input unavailable while Assistant is listening for the wake phrase.", resources); + Assert.Contains("Assistant is listening for the wake phrase.", resources); + } + [Fact] public void ChatVoiceDialogs_RouteDisabledTtsCapabilityToPermissions_AndPreserveFallbackForMissingSetup() {