Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ leading and trailing pipe. Columns, in order:
| functional-chat-default-mount | closed | src/OpenClaw.Tray.WinUI/Chat/FunctionalChatHostExtensions.cs | mounting the FunctionalUI chat tree as the default ChatPage or ChatWindow surface | ReactorChatHostExtensions and OpenClawReactorChatRoot | legacy FunctionalUI chat files may remain for focused compatibility coverage only | ChatPage and ChatWindow mount the Reactor root directly into their existing ChatHost Borders; no FunctionalUI component mounts or nests Reactor on the default path | review-only: user explicitly deferred new tests for this migration; required build and existing shared/tray suites still run | review-only | when legacy FunctionalUI chat surfaces are removed |
| settings-store | authoritative | src/OpenClaw.Tray.WinUI/Pages/SettingsPage.xaml.cs | hand-rolled save/echo suppression flags for two-way settings binding | ISettingsStore | PermissionsPage and other surfaces may read SettingsManager directly until migrated | a save originating from Update does not echo Changed to the caller and external saves are republished on the UI thread | SettingsStoreTests.Update_DoesNotEchoChangedToSelf | behavioral | when all settings surfaces read and write through ISettingsStore |
| settings-page-vm | authoritative | src/OpenClaw.Tray.WinUI/Pages/SettingsPage.xaml.cs | settings load, persist, echo-guard, and auto-save wiring | SettingsPageViewModel | code-behind keeps gateway-uninstall, gateway-info and uptime timer, saved-indicator visual, and app-info population | each settings control persists its field through the store preserving mutate-save-notify order and does not re-persist on external change | SettingsPageViewModelTests.ExternalChange_ReloadsWithoutRePersisting | behavioral | when the Settings page holds no settings persistence logic in code-behind |
| voice-assistant-coordinator | authoritative | src/OpenClaw.Tray.WinUI/App.xaml.cs | voice assistant state transitions, wake dispatch, response correlation, timeout, and playback ownership | VoiceAssistantCoordinator | App constructs, reconciles, synchronously offers notifications for claiming, and disposes the coordinator | one wake utterance owns one tracked direct send and at most one correlated spoken final while App contains no assistant state machine | AppRefactorContractTests.VoiceAssistant_StateMachineStaysOutsideApp | source-shape | when voice assistant lifecycle composition moves to AppBootstrapper |
<!-- LEDGER:END -->

## Deferred test builders
Expand Down
25 changes: 25 additions & 0 deletions docs/WINDOWS_NODE_ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -523,6 +523,31 @@ This is a candidate implementation path, not an implemented node command yet. Vo

Current PR review status: open PR #120 (`feature/voice-mode`) is a useful prototype but should not merge as-is. It currently conflicts with the active capability-settings branch, advertises `voice.*` commands without the default-off Settings gate used for other privacy-sensitive capability groups, widens operator scopes in the same PR, persists cloud TTS provider keys in plain settings JSON, and introduces a Windows-specific wire schema before the Mac runtime/controller/session contract is agreed. Safe next step: split schema, gateway scope, chat transport, Windows runtime, WebChat integration, and cloud-provider credentials into separate reviews; keep the first merge behind a default-off Voice Settings group and gateway dangerous-command allowlist.

### Local voice assistant V0

The tray also has a default-off local voice assistant preview. This is a settings-driven
orchestration feature, not a new node capability or `voice.*` protocol:

1. `VoiceService` owns WASAPI microphone capture, energy VAD, and local Whisper transcription.
2. `VoiceWakeGate` accepts only a silence-bounded transcript that begins with the configured
one-to-three-word wake phrase and includes a request in the same utterance.
3. `VoiceAssistantCoordinator` stops capture, sends the extracted request through the canonical
chat provider, claims only the final response correlated to that tracked local send, and speaks
it through the configured Windows, Piper, or ElevenLabs TTS provider.
4. Capture remains stopped while dispatching, waiting, and speaking. A fresh capture pipeline is
acquired afterward through the shared exclusive microphone lease.

The coordinator is the authoritative state and cancellation owner. `App` remains the composition
root and forwards settings, provider, notification, and shutdown lifecycle inputs. Gateway
message ID and sequence are carried into local notifications so assistant playback does not claim
an unrelated final from the same session.

V0 intentionally uses transcript-gated activation rather than a dedicated wake-word model.
Whisper therefore runs for each detected ambient speech burst. The wake phrase and request must be
spoken together, there is no persistent conversation mode or barge-in, and only one request can be
in flight. Enabling the preview requires configured local STT and TTS readiness, while gateway and
idle-chat readiness are runtime pause conditions.

---

## Architectural Questions
Expand Down
42 changes: 42 additions & 0 deletions docs/WINDOWS_NODE_TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,48 @@ When the node connects, it advertises these capabilities:

Local MCP clients also see MCP-only `app.*` commands such as `app.navigate`, `app.status`, `app.chat.snapshot`/`app.chat.send`/`app.chat.reset`, and `app.chat.queue.list`/`app.chat.queue.cancel`. Connection diagnostics and setup tools live under `app.connection.*`; use `app.connection.status` to inspect active gateway, operator/node credential state, MCP runtime status, browser proxy caveat, pending approval commands, and recent diagnostics, and `app.connection.gateways` to list saved gateway records without token values. These are local testing and automation hooks registered with the tray's MCP server and are not advertised to the gateway WebSocket.

## Voice Assistant Preview

The Voice & Audio page includes a default-off **Assistant (preview)** mode. It reuses local
Whisper STT, the configured TTS provider, and the existing gateway chat timeline. It does not add
an MCP tool or gateway node command.

Prerequisites:

- Speech-to-text is enabled and the selected Whisper model is downloaded.
- Text-to-speech is enabled and the selected provider is ready. Piper requires its selected voice;
ElevenLabs requires its protected API key and voice ID.
- The wake phrase contains one to three words and is no longer than 64 characters.
- At runtime, the gateway is connected and the canonical compose target is idle.

Manual smoke:

1. Open **Settings > Voice & Audio** and confirm the assistant mode defaults to Off.
2. Disable each local speech prerequisite in turn. Confirm wake mode cannot be selected and the
status identifies the missing prerequisite.
3. Set a custom wake phrase, leave the page, return, and confirm it persists.
4. Enable **Wake phrase, one request**. Confirm the status becomes Listening and Windows shows
microphone use.
5. Speak unrelated text and confirm no chat request is sent.
6. Speak `<wake phrase>, summarize my inbox` as one utterance. Confirm only
`summarize my inbox` is sent.
7. Confirm the matching final response is spoken once, then the status returns to Listening.
8. While waiting or speaking, turn the mode off, disconnect the gateway, or disable a speech
prerequisite. Confirm capture and playback stop and the status becomes Off or Paused.
9. Start push-to-talk while wake listening owns the microphone. Confirm the existing UI reports
microphone contention rather than opening a second capture pipeline.

Privacy and V0 limits:

- Enabling the mode keeps the microphone active and locally transcribes detected speech bursts to
find the wake phrase. Only the extracted trailing request is sent through the gateway.
- Audio, ambient transcripts, wake phrases, extracted requests, and response content are not
added to application logs or telemetry.
- The wake phrase and request must be in the same silence-bounded utterance.
- There is no persistent conversation, barge-in, dedicated wake-word model, or overlay.
- Assistant replies are always spoken. **Read responses aloud** continues to control other chat
replies only.

## Security Features

- **URL Validation**: Canvas blocks `file://`, `javascript:`, localhost, private IPs, IPv6 localhost
Expand Down
4 changes: 4 additions & 0 deletions src/OpenClaw.Shared/Models.cs
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,10 @@ public class OpenClawNotification
/// the specific session instead of the default/main session.
/// </summary>
public string? SessionKey { get; set; }
/// <summary>Gateway-assigned message identity used for local deduplication and correlation.</summary>
public string? OpenClawId { get; set; }
/// <summary>Monotonic per-session sequence used for local ordering and correlation.</summary>
public int? OpenClawSeq { get; set; }
}

/// <summary>
Expand Down
18 changes: 14 additions & 4 deletions src/OpenClaw.Shared/OpenClawGatewayClient.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3317,7 +3317,11 @@ private void HandleChatEvent(JsonElement root, int rawMessageLength)
// HIGH 4: log shape only — content previously
// surfaced in the operator log.
_logger.Info($"Assistant response: role={role} state={state} len={text.Length}");
EmitChatNotification(text, sessionKey);
EmitChatNotification(
text,
sessionKey,
messageOpenClawMetadata.Id ?? payloadOpenClawMetadata.Id,
messageOpenClawMetadata.Seq ?? payloadOpenClawMetadata.Seq);
}
}

Expand Down Expand Up @@ -3353,7 +3357,7 @@ private void HandleChatEvent(JsonElement root, int rawMessageLength)
{
// HIGH 4: log shape only.
_logger.Info($"Assistant response (legacy): role={role} state={state} len={text.Length}");
EmitChatNotification(text, sessionKey);
EmitChatNotification(text, sessionKey, openClawMetadata.Id, openClawMetadata.Seq);
}
}
}
Expand Down Expand Up @@ -3553,15 +3557,21 @@ private void EmitRawChatEvent(JsonElement payload)
}
}

private void EmitChatNotification(string text, string? sessionKey = null)
private void EmitChatNotification(
string text,
string? sessionKey = null,
string? openClawId = null,
int? openClawSeq = null)
{
var displayText = text.Length > 200 ? text[..200] + "…" : text;
var notification = new OpenClawNotification
{
Message = displayText,
FullMessage = text,
IsChat = true,
SessionKey = sessionKey
SessionKey = sessionKey,
OpenClawId = openClawId,
OpenClawSeq = openClawSeq
};
var (title, type) = _categorizer.Classify(notification, _userRules);
notification.Title = title;
Expand Down
40 changes: 40 additions & 0 deletions src/OpenClaw.Shared/SettingsData.cs
Original file line number Diff line number Diff line change
@@ -1,8 +1,44 @@
using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization;

namespace OpenClaw.Shared;

public static class VoiceAssistantSettingsPolicy
{
public const string OffMode = "off";
public const string WakeOneShotMode = "wake-one-shot";
public const string DefaultWakePhrase = "OpenClaw";
public const int MaxWakePhraseLength = 64;

public static string NormalizeMode(string? value) =>
string.Equals(value, WakeOneShotMode, StringComparison.OrdinalIgnoreCase)
? WakeOneShotMode
: OffMode;

public static bool TryNormalizeWakePhrase(string? value, out string normalized)
{
normalized = string.Join(
' ',
(value ?? string.Empty)
.Normalize(NormalizationForm.FormKC)
.Split((char[]?)null, StringSplitOptions.RemoveEmptyEntries));

if (normalized.Length is 0 or > MaxWakePhraseLength)
return false;

var words = normalized.Split(' ');
if (words.Length is < 1 or > 3)
return false;

return words.All(word =>
word.Any(char.IsLetterOrDigit) &&
word.All(character =>
char.IsLetterOrDigit(character) ||
character is '\'' or '\u2019' or '-'));
}
}

/// <summary>
/// Serializable settings data model. Used for JSON round-trip persistence.
/// </summary>
Expand Down Expand Up @@ -87,6 +123,10 @@ public record class SettingsData
public bool ShowChatToolCalls { get; set; } = true;
/// <summary>Play audio feedback chimes on listen start/stop.</summary>
public bool VoiceAudioFeedback { get; set; } = true;
/// <summary>Local assistant mode. Unknown values normalize to <c>off</c>.</summary>
public string VoiceAssistantMode { get; set; } = VoiceAssistantSettingsPolicy.OffMode;
/// <summary>Wake phrase used by the local transcript gate.</summary>
public string VoiceAssistantWakePhrase { get; set; } = VoiceAssistantSettingsPolicy.DefaultWakePhrase;
public bool NodeTtsEnabled { get; set; } = false;
public string TtsProvider { get; set; } = OpenClaw.Shared.Capabilities.TtsCapability.PiperProvider;
/// <summary>Persisted: whether the Hub's NavigationView pane is expanded
Expand Down
Loading
Loading