diff --git a/.gitignore b/.gitignore index c4d92ff..3b26c96 100644 --- a/.gitignore +++ b/.gitignore @@ -15,5 +15,6 @@ stock-voice-pe-backup.bin *.wav *.pcm -# internal ops runbook (not for public repo) +# personal notes and agent context, kept local +CLAUDE.md SUNDAY_RUNBOOK.md diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 256981f..0000000 --- a/CLAUDE.md +++ /dev/null @@ -1,11 +0,0 @@ -# CLAUDE.md - -## Issue tracking (Linear) - -Work for this repo is tracked in Linear — team **JAY**, project **domus** (the Voice PE broker + firmware piece of it). - -- **Keep Linear tidy as you work, without being asked.** When Jay asks you to fix/build something, first check it has an issue in the **domus** project (`list_issues`, team JAY, project `domus`) — if not, create one. When he says "remember to do X later," file it. Mark issues In Progress when you start; make sure they end up Done when finished. -- Found a bug, TODO, or follow-up? File it as a Linear issue in the **domus** project instead of leaving a stray code comment or a separate list. -- Linear generates a branch name per issue (`jayzuccarelli/jay-NN-...`); work on that branch. -- Put `Fixes JAY-NN` in the PR description or a commit message — merging then auto-closes the issue. -- Don't keep a parallel todo list; Linear is the source of truth. diff --git a/Makefile b/Makefile index 86ad48c..b4c2088 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ # Canonical verify loop for voice-pe-realtime. # `make check` streams synthesized speech into a broker, transcribes the -# spoken reply, and asserts on content — no Voice PE device, no being home. +# spoken reply, and asserts on content: no Voice PE device, no being home. # Defaults to the ISOLATED dev broker on 8766: check.py's own default (8765) # is the LIVE broker, and hook-fired checks were kicking the puck mid-use # and billing OpenAI every run. Target live deliberately with diff --git a/NOTICE.md b/NOTICE.md index 7ca70de..3a255ae 100644 --- a/NOTICE.md +++ b/NOTICE.md @@ -3,17 +3,24 @@ The ESPHome firmware component in `firmware/components/voice_assistant_websocket/` derives from **[fjfricke/ha-openai-realtime](https://github.com/fjfricke/ha-openai-realtime)** -(MIT License) — the proof-of-concept that first bridged a Home Assistant +(MIT License): the proof-of-concept that first bridged a Home Assistant Voice PE to the OpenAI Realtime API over WebSocket. That project's MIT license and copyright are preserved in `firmware/components/voice_assistant_websocket/LICENSE`. +The device YAML in `firmware/voice_pe_dual.yaml` is derived from +**[esphome/home-assistant-voice-pe](https://github.com/esphome/home-assistant-voice-pe)** +(`home-assistant-voice.yaml`, Copyright (c) 2019 ESPHome), modified to route the +wake word to the broker instead of the stock Assist pipeline. The ESPHome License +applies MIT terms to non-C++ files such as YAML; its full text, including the MIT +permission notice, is preserved in `firmware/LICENSE.esphome`. + The broker is an independent rewrite (clean module structure, SSE-based Home Assistant MCP control, persistent-session context, reproducible build), but the original bridging approach came from that project. Built on: -- [Pipecat](https://github.com/pipecat-ai/pipecat) — voice-agent framework (BSD-2-Clause) -- [ESPHome](https://esphome.io) — device firmware -- [Home Assistant](https://www.home-assistant.io) — home automation + MCP Server +- [Pipecat](https://github.com/pipecat-ai/pipecat): voice-agent framework (BSD-2-Clause) +- [ESPHome](https://esphome.io): device firmware +- [Home Assistant](https://www.home-assistant.io): home automation + MCP Server - [OpenAI Realtime API](https://platform.openai.com/docs/guides/realtime) diff --git a/README.md b/README.md index 7a853d2..a675bee 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # voice-pe-realtime -Self-hosted, real-time speech-to-speech for the [Home Assistant Voice PE](https://www.home-assistant.io/voice-pe/) — talk to your home with ChatGPT-Voice-style latency, and have it actually *do* things. +Self-hosted, real-time speech-to-speech for the [Home Assistant Voice PE](https://www.home-assistant.io/voice-pe/): talk to your home with ChatGPT-Voice-style latency, and have it actually *do* things. Instead of the turn-based `wake → STT → LLM → TTS` Assist pipeline, the Voice PE streams audio straight to a small server-side **broker** that holds an **OpenAI Realtime** session and controls Home Assistant over **MCP**. One round trip, natural voice, real actions. @@ -25,16 +25,16 @@ The canonical pattern: **the agent runs server-side; the device is a thin full-d ## Why -The stock Voice PE pipeline runs STT → LLM → TTS sequentially — a latency floor that feels clunky next to ChatGPT Voice. Routing audio through a persistent Realtime session collapses that to a single round trip with a natural voice, while MCP gives the model first-class control of the home. +The stock Voice PE pipeline runs STT → LLM → TTS sequentially, a latency floor that feels clunky next to ChatGPT Voice. Routing audio through a persistent Realtime session collapses that to a single round trip with a natural voice, while MCP gives the model first-class control of the home. ## Reliability Speech-to-speech on a $59 puck is easy to demo and hard to keep up. This repo treats robustness as the feature: -- **Session rotation** — OpenAI caps a Realtime session at ~60 min and treats expiry as fatal. The broker rotates the session *before* the cap (and rebuilds after any death) under a still-connected device, so long-lived pucks never drop. Proven continuous across forced rotations. -- **Idle refresh** — a stale idle session (socket open, silently dead) is refreshed proactively. -- **Turn hygiene** — a device that vanishes mid-utterance (Wi-Fi blip, session timeout) leaves OpenAI's server VAD holding a speech-in-progress segment that would come back as a ghost turn on the next wake. Clearing the input buffer isn't enough (the bytes go, the VAD state doesn't); the broker disables and re-enables turn detection on disconnect to drop the segment for real. Background speech (a TV, a side conversation) is gated with the OpenAI-recommended `wait_for_user` pattern — with an explicit follow-up bias, so "are you sure about that?" right after an answer gets answered instead of ignored. -- **A real test harness** — `make check` drives the broker end-to-end exactly like the firmware (streams PCM, transcribes the spoken reply, asserts content + first-audio latency). No hardware needed. +- **Session rotation**: OpenAI caps a Realtime session at ~60 min and treats expiry as fatal. The broker rotates the session *before* the cap (and rebuilds after any death) under a still-connected device, so long-lived pucks never drop. Proven continuous across forced rotations. +- **Idle refresh**: a stale idle session (socket open, silently dead) is refreshed proactively. +- **Turn hygiene**: a device that vanishes mid-utterance (Wi-Fi blip, session timeout) leaves OpenAI's server VAD holding a speech-in-progress segment that would come back as a ghost turn on the next wake. Clearing the input buffer isn't enough (the bytes go, the VAD state doesn't); the broker disables and re-enables turn detection on disconnect to drop the segment for real. Background speech (a TV, a side conversation) is gated with the OpenAI-recommended `wait_for_user` pattern, with an explicit follow-up bias, so "are you sure about that?" right after an answer gets answered instead of ignored. +- **A real test harness**: `make check` drives the broker end-to-end exactly like the firmware (streams PCM, transcribes the spoken reply, asserts content + first-audio latency). No hardware needed. ```bash cd broker && OPENAI_API_KEY=... make check # 10 scenarios, pass/fail + p50/p95 latency @@ -66,24 +66,24 @@ The broker fetches HA's tools at startup and registers them on the Realtime sess | Env | Default | Purpose | |---|---|---| -| `OPENAI_API_KEY` | — | required | +| `OPENAI_API_KEY` | none | required | | `MODEL` | `gpt-realtime` | Realtime model | | `VOICE` | `marin` | Realtime voice | | `INSTRUCTIONS` | generic | system prompt / persona | | `WS_HOST` / `WS_PORT` | `0.0.0.0` / `8765` | where the device connects | -| `HA_MCP_URL` / `HA_TOKEN` | — | enable HA control (both required) | -| `MUSIC_PLAYER` | — | default Music Assistant speaker | +| `HA_MCP_URL` / `HA_TOKEN` | none | enable HA control (both required) | +| `MUSIC_PLAYER` | none | default Music Assistant speaker | | `VAD_*` | sane defaults | OpenAI server-VAD tuning | | `FOLLOWUP_WINDOW_SECONDS` | `6.0` | after each reply, how long the mic stays open for a follow-up before the broker disconnects the device (wake word re-arms) | | `MAX_TURNS_PER_WAKE` | `8` | user turns allowed per wake, so TV speech can't spiral a session | | `MAX_SESSION_SECONDS` | `3000` | rotate before the 60-min cap | | `IDLE_REFRESH_SECONDS` | `600` | refresh a stale idle session | -Turn hygiene (`FOLLOWUP_WINDOW_SECONDS` / `MAX_TURNS_PER_WAKE`): set either to `0` to disable that bound — both `0` restores the old unbounded behavior, the no-redeploy rollback lever. +Turn hygiene (`FOLLOWUP_WINDOW_SECONDS` / `MAX_TURNS_PER_WAKE`): set either to `0` to disable that bound. Setting both to `0` restores the old unbounded behavior, which is the no-redeploy rollback lever. ## Firmware -The Voice PE runs ESPHome firmware that streams PCM to this broker. See [`firmware/`](firmware/). Flashing replaces the stock firmware; back up first (`esptool read_flash`) — the ESP32-S3 ROM bootloader makes bricking effectively impossible. +The Voice PE runs ESPHome firmware that streams PCM to this broker. See [`firmware/`](firmware/). Flashing replaces the stock firmware; back up first (`esptool read_flash`): the ESP32-S3 ROM bootloader makes bricking effectively impossible. ## Status & roadmap @@ -92,8 +92,8 @@ The Voice PE runs ESPHome firmware that streams PCM to this broker. See [`firmwa - ✅ Session rotation before the 60-min Realtime cap (no dropouts) - ✅ Background-speech gating (`wait_for_user`) - ✅ End-to-end reliability harness (`make check`) -- ⏳ **Smart routing** — one wake word, fast local intents handled on-device, everything else escalated to the LLM (the elegant form of "local + cloud") -- ⏳ **Barge-in** — true open-mic interruption using the Voice PE's hardware AEC (experimental; the acoustic self-trigger loop is the open problem — an echo-residual calibration rig ships in `broker/tools/`) -- ⏳ **Beamforming** — tap the XMOS array's focused channel to reject off-axis room noise (a TV, another speaker) +- ⏳ **Smart routing**: one wake word, fast local intents handled on-device, everything else escalated to the LLM (the elegant form of "local + cloud") +- ⏳ **Barge-in**: true open-mic interruption using the Voice PE's hardware AEC (experimental; the acoustic self-trigger loop is the open problem, an echo-residual calibration rig ships in `broker/tools/`) +- ⏳ **Beamforming**: tap the XMOS array's focused channel to reject off-axis room noise (a TV, another speaker) MIT licensed. Attribution in [NOTICE.md](NOTICE.md). diff --git a/broker/.env.example b/broker/.env.example index ae9f747..b406050 100644 --- a/broker/.env.example +++ b/broker/.env.example @@ -34,7 +34,7 @@ VAD_RELEASE_DELAY_MS=1200 # speaker goes quiet) to follow up before the broker disconnects the device and # the wake word re-arms; MAX_TURNS_PER_WAKE caps user turns per wake so TV # speech can't spiral a session. Set either to 0 to disable (both 0 = old -# unbounded behavior) — the no-redeploy rollback lever. +# unbounded behavior), which is the no-redeploy rollback lever. FOLLOWUP_WINDOW_SECONDS=6.0 MAX_TURNS_PER_WAKE=8 diff --git a/broker/Makefile b/broker/Makefile index 17388ae..420721f 100644 --- a/broker/Makefile +++ b/broker/Makefile @@ -1,10 +1,10 @@ -# voice-pe realtime broker — reliability harness. +# voice-pe realtime broker: reliability harness. # # `make check` drives the broker end-to-end (synthesizes speech, streams it like # the firmware, transcribes the reply, asserts content + latency). Needs # OPENAI_API_KEY in the environment and a RUNNING broker. # -# WS defaults to :8766 (an ISOLATED broker), NOT the live puck's :8765 — the +# WS defaults to :8766 (an ISOLATED broker), NOT the live puck's :8765. The # broker is single-client, so pointing this at :8765 kicks the device. # # make check # against ws://127.0.0.1:8766 diff --git a/broker/realtime_broker/agent.py b/broker/realtime_broker/agent.py index 654f16b..de6df34 100644 --- a/broker/realtime_broker/agent.py +++ b/broker/realtime_broker/agent.py @@ -2,7 +2,7 @@ Configures voice, system prompt, server-side VAD, and the available tools: the Home Assistant MCP tools (if HA control is enabled) plus two custom broker -tools — `get_weather` (live HA weather, which HA's MCP doesn't surface) and +tools, `get_weather` (live HA weather, which HA's MCP doesn't surface) and `end_conversation` (clean "ok, bye" stop). Handlers for the custom tools are registered by the server (they need HA access / the device connection). """ @@ -34,7 +34,7 @@ class VoicePERealtimeService(OpenAIRealtimeLLMService): Upstream's _handle_context treats the FIRST context frame as conversation setup: it replays the context as conversation items and issues a bare response.create. With server_vad the audio commit already created the user - item and auto-created the response, so that double-fires — OpenAI rejects + item and auto-created the response, so that double-fires, OpenAI rejects it (conversation_already_has_active_response) and Pipecat treats any error event as fatal, killing the session's receive loop. Conversation state lives server-side here; the only thing context frames must deliver is new @@ -109,7 +109,7 @@ async def _handle_evt_error(self, evt) -> None: "to you: TV or other media dialogue, or a conversation between other " "people. If you just answered and the next utterance could be a " "follow-up, reaction, or challenge to your answer ('are you sure?', " - "'okay, and what about...'), it IS addressed to you — answer it " + "'okay, and what about...'), it IS addressed to you. Answer it " "instead of calling this. Calling it means stay silent and keep " "listening. Produce no spoken reply when you call it." ), @@ -119,7 +119,7 @@ async def _handle_evt_error(self, evt) -> None: # Appended to the configured persona instructions. This device is far-field and # its mic hears the whole room (TV, other people), so the model must gate on -# whether speech is actually addressed to it — the OpenAI-recommended pattern +# whether speech is actually addressed to it: the OpenAI-recommended pattern # for rejecting non-addressed speech (there is no speaker separation at the API # layer). BACKGROUND_GUIDANCE = ( @@ -131,7 +131,7 @@ async def _handle_evt_error(self, evt) -> None: "right after you answer, the next utterance is usually the same user " "following up. A follow-up question, reaction, or challenge to what you just " "said ('are you sure?', 'okay, and...', 'what about tomorrow?') is addressed " - "to you even when it does not name you — answer it. When torn between " + "to you even when it does not name you. Answer it. When torn between " "answering a plausible follow-up and staying silent, answer: a wrongly " "ignored user must repeat themselves, which is worse than a wrongly " "answered TV line." @@ -146,7 +146,7 @@ def build_audio_input(config: Config, threshold: float | None) -> AudioInput: transcription. threshold=None disables turn detection entirely (serialized as - turn_detection: null), which also discards the server's VAD state — + turn_detection: null), which also discards the server's VAD state, used to drop a speech-in-progress segment after a device disconnect. """ return AudioInput( @@ -168,7 +168,7 @@ def build_audio_input(config: Config, threshold: float | None) -> AudioInput: # off lets the quiet-but-clean tap reach the VAD intact. noise_reduction=None, # Optionally transcribe each user turn so broker logs show what OpenAI - # heard (self-trigger / "janky" diagnosis). Off by default — it bills a + # heard (self-trigger / "janky" diagnosis). Off by default: it bills a # Whisper pass per turn. whisper-1 because gpt-4o-transcribe yielded # zero transcription events on gpt-realtime. transcription=( diff --git a/broker/realtime_broker/config.py b/broker/realtime_broker/config.py index 7c70f62..bce6c56 100644 --- a/broker/realtime_broker/config.py +++ b/broker/realtime_broker/config.py @@ -52,7 +52,7 @@ class Config: # response.done) to take another turn before the broker disconnects the # device and the wake word re-arms; max_turns_per_wake caps committed user # turns per WS connection. Either set to 0 disables that bound (both 0 = - # exactly the pre-hygiene behavior) — the no-redeploy rollback lever. + # exactly the pre-hygiene behavior): the no-redeploy rollback lever. followup_window_seconds: float = 6.0 max_turns_per_wake: int = 8 @@ -64,7 +64,7 @@ class Config: # An idle Realtime session goes stale server-side WITHOUT the socket dying: # a 47-min-old session accepted audio and returned nothing while ws.state # stayed OPEN (2026-06-10). Refresh the session whenever no device has been - # connected for this long — free, and invisible to the user. + # connected for this long: free, and invisible to the user. idle_refresh_seconds: int = 600 # 10 min @property diff --git a/broker/realtime_broker/server.py b/broker/realtime_broker/server.py index ab021a5..04cd24a 100644 --- a/broker/realtime_broker/server.py +++ b/broker/realtime_broker/server.py @@ -61,7 +61,7 @@ class _UserTranscriptLogger(FrameProcessor): Consumes the TranscriptionFrame instead of re-pushing it. If one reaches the user aggregator upstream, the aggregator emulates VAD (a spurious pipeline interruption) and then pushes a context frame that makes the - service double-fire response.create — OpenAI rejects it + service double-fire response.create, OpenAI rejects it (conversation_already_has_active_response) and Pipecat treats any error event as fatal, silently killing the session's receive loop. Nothing upstream of here needs the transcript, so log it and stop it. @@ -85,7 +85,7 @@ class _BotPlaybackGate(FrameProcessor): voice in the mic feed to trip server_vad at any threshold a normal-volume user can also cross (measured: bleed trips 0.6, user is inaudible at 0.7+). So: sensitive threshold while idle, strict while - the bot has the floor. Barge-in still works — it just needs a slightly + the bot has the floor. Barge-in still works, it just needs a slightly raised voice. 2. Barge-in flush: on a real interruption, tell the device to drop its @@ -209,17 +209,17 @@ async def reset_vad(self, drain: float = 0.4) -> None: Clearing the input buffer removes the audio BYTES but not the VAD state machine: when the device vanishes mid-utterance, server_vad still holds speech-started, and once post-reconnect silence gives it - its window it commits whatever is buffered — the tail fragment that - drained from the pipeline after the clear, or nothing at all — and + its window it commits whatever is buffered, the tail fragment that + drained from the pipeline after the clear, or nothing at all, and auto-creates a response. The model greets the ghost turn ("I'm here when you're ready"; soak 2026-07-02, 5/5 then 2/6 with a delayed clear alone). Disabling turn detection makes the server drop the pending segment; then let the stale pipeline tail drain (`drain` - seconds — 0 on connect, when any tail drained long ago), wipe the + seconds, 0 on connect, when any tail drained long ago), wipe the buffer, and re-enable at the current threshold. Single-flight: if a reset is already in flight, joining callers - no-op — the running one clears and re-enables for everyone. In + no-op, the running one clears and re-enables for everyone. In particular a fast reconnect's on-connect reset must NOT preempt the disconnect reset mid-drain, or the clear fires before the stale tail lands and the ghost returns. _set_threshold defers its @@ -252,8 +252,8 @@ class _MicInputGate(FrameProcessor): server-side noise reduction (removed because it scrubbed the quiet NS-tap mic to nothing), that residual is loud enough for server_vad to read as user speech, so the bot answers its own echo in a runaway loop. While the - speaker is playing — tracked by the playback gate's buffer-accurate model, - plus the VAD release margin — replace the incoming mic audio with silence + speaker is playing, tracked by the playback gate's buffer-accurate model, + plus the VAD release margin, replace the incoming mic audio with silence so OpenAI has nothing to trigger on. This makes the assistant turn-based; real barge-in needs echo cancellation, not just an open mic. """ @@ -281,12 +281,12 @@ async def process_frame(self, frame: Frame, direction: FrameDirection) -> None: # silent false wake (TV says the wake word, nobody follows up) would otherwise # stream mic audio to OpenAI until session rotation: the firmware's 10s # auto-stop only arms after the first bot audio chunk, which a silent wake -# never produces. Internal constant on purpose — a safety floor, not a knob. +# never produces. Internal constant on purpose: a safety floor, not a knob. _INITIAL_GRACE_SECONDS = 10.0 class _TurnHygiene(FrameProcessor): - """Bound how long one wake keeps the conversation open (JAY-84). + """Bound how long one wake keeps the conversation open (turn hygiene). TV speech can keep re-tripping server_vad and spiral a single wake into a minutes-long open-mic session. Two structural bounds, both broker-side @@ -295,8 +295,8 @@ class _TurnHygiene(FrameProcessor): zero firmware changes): - Follow-up window: after each reply the user gets - config.followup_window_seconds — measured from when the SPEAKER goes - quiet (the playback gate's clock + VAD release), not response.done — to + config.followup_window_seconds, measured from when the SPEAKER goes + quiet (the playback gate's clock + VAD release), not response.done, to take another turn. Expiry with nothing in flight -> disconnect. A real follow-up starts a fresh window after its reply, so natural conversation is untouched. @@ -306,7 +306,7 @@ class _TurnHygiene(FrameProcessor): spiral even when each TV line lands inside the window. Either knob at 0 disables that bound (both 0 = exactly the pre-hygiene - behavior) — the no-redeploy rollback lever. The OpenAI session stays + behavior): the no-redeploy rollback lever. The OpenAI session stays alive either way; context carries to the next wake as today. Must sit between the service and the assistant context aggregator: the @@ -350,7 +350,7 @@ async def process_frame(self, frame: Frame, direction: FrameDirection) -> None: # from the receive loop, strictly before it. if any(fc.function_name != "wait_for_user" for fc in frame.function_calls): # HA tools / get_weather / play_music trigger a second - # response.create for the verbalization — stay awaiting until + # response.create for the verbalization: stay awaiting until # THAT response ends. wait_for_user has run_llm=False (no # follow-up response), so an ignored TV line clears awaiting # on this response's end; playback_end is already past, so @@ -381,14 +381,14 @@ def on_device_connect(self) -> None: self._tool_followup = False self._gen += 1 # Stamp the socket this connection's state belongs to (pipecat swaps - # transport._websocket before this handler's task runs) — the watcher + # transport._websocket before this handler's task runs): the watcher # re-checks it before sending, so a tick that interleaves between the # swap and this reset can't fire the OLD connection's expiry at the # NEW socket and kill a fresh wake. self._conn_ws = self._get_ws() if self._watch_task is not None: # Always cancel-and-recreate rather than reuse: an old watcher can - # be parked in ws.send on a half-dead kicked socket for seconds — + # be parked in ws.send on a half-dead kicked socket for seconds. # skipping creation here would leave the new connection with NO # watcher once it finishes (window + budget silently off). task, self._watch_task = self._watch_task, None @@ -406,7 +406,7 @@ async def _watch(self) -> None: # ~0.5s tick; single-flight like the gate's _restore_task. Exits (and # clears itself) after signaling the close: the firmware closes the WS # in response, and the next wake's on_device_connect starts a fresh - # watcher. end_conversation needs no special-casing — its disconnect + # watcher. end_conversation needs no special-casing: its disconnect # makes the device drop, and on_device_disconnect cancels us. loop = asyncio.get_running_loop() release = self._config.vad_release_delay_ms / 1000 @@ -421,7 +421,7 @@ async def _watch(self) -> None: if self._awaiting_response and now - self._awaiting_since > 60.0: # Fail-open guard: OpenAI can kill a response without a # response.done reaching the pipeline, and a hung tool's - # verbalization response may never materialize — either would + # verbalization response may never materialize: either would # hold the window open forever. 60s is far above any real # response/tool latency (play_music's HA path ~20s); don't # shorten it below that. @@ -437,7 +437,7 @@ async def _watch(self) -> None: ) + window: # max() folds in the initial grace: before the first committed # turn playback_end is stale (or zero), and a silent false - # wake must still get bounded — grace + window — instead of + # wake must still get bounded (grace + window) instead of # streaming mic audio to OpenAI until rotation. reason = f"follow-up window expired ({self._turns} turns)" elif ( @@ -465,7 +465,7 @@ async def _watch(self) -> None: else: logger.info("turn hygiene: %s; disconnecting device", reason) if self._gen == my_gen: - # Only clear our own handle — a re-wake during the send above + # Only clear our own handle: a re-wake during the send above # already replaced _watch_task with the new watcher's. self._watch_task = None return @@ -557,7 +557,7 @@ def _start_music(config: Config, query: str, speaker: str | None) -> str: None, ) if target is None: - # A speaker was named but didn't match — don't silently play on the + # A speaker was named but didn't match: don't silently play on the # wrong one; tell the user what's available. return f"I couldn't find a speaker called {speaker}. Available: {avail}." else: @@ -643,12 +643,12 @@ async def _wait_for_user(params): # The realtime service pushes user TranscriptionFrames UPSTREAM # (Pipecat >= 0.0.92); the logger must sit between the aggregator # and the service so it can intercept (and consume) them before - # the aggregator reacts to them — see _UserTranscriptLogger. + # the aggregator reacts to them: see _UserTranscriptLogger. _UserTranscriptLogger(), service, # Turn hygiene must observe the service's raw downstream frames; # the assistant aggregator below consumes response-end and - # function-call frames — see _TurnHygiene. + # function-call frames: see _TurnHygiene. hygiene, aggregator.assistant(), gate, @@ -690,8 +690,8 @@ async def _on_connect(_transport, client): device_connected = True logger.info("Device connected: %s", getattr(client, "remote_address", client)) # The device opens a fresh websocket per wake, but the OpenAI session - # is reused for context. Anything left from the previous connection — - # uncommitted buffer audio AND a speech-in-progress VAD segment — + # is reused for context. Anything left from the previous connection + # (uncommitted buffer audio AND a speech-in-progress VAD segment) # would surface as a ghost turn before the real question (stray # 'Bye.', TV test 2026-07-01). Full reset, no drain: any stale # pipeline tail finished draining while no device was connected, and @@ -707,7 +707,7 @@ async def _on_disconnect(_transport, client, *args): # WiFi blip: the dead socket never closed), Pipecat swaps the # transport's websocket to the NEW client before the old handler # exits and fires this event for the OLD one. The device is still - # here — don't flag it disconnected, and above all don't reset VAD + # here: don't flag it disconnected, and above all don't reset VAD # and clear the buffer while the user's real question streams in. # The on-connect reset already dealt with the stale state. current = getattr(transport.input(), "_websocket", None) @@ -719,13 +719,13 @@ async def _on_disconnect(_transport, client, *args): logger.info("Device disconnected") # A disconnect mid-utterance leaves server VAD holding a # speech-in-progress segment (plus mic-stream tail still draining - # through the pipeline). A buffer clear alone can't kill it — see + # through the pipeline). A buffer clear alone can't kill it: see # _BotPlaybackGate.reset_vad, which disables and re-enables turn # detection to drop the segment before it becomes a ghost turn - # (stray 'Bye.' in the TV test 2026-07-01; 'I'm here when you're - # ready' in the soak 2026-07-02). + # (observed on the next wake as a stray 'Bye.' or an unprompted + # 'I'm here when you're ready'). asyncio.create_task(gate.reset_vad()) - # Cancel the hygiene watcher for a REAL disconnect only — the stale + # Cancel the hygiene watcher for a REAL disconnect only: the stale # guard above already returned for a kicked old socket, so it can't # cancel the new connection's window/budget state. await hygiene.on_device_disconnect() @@ -737,7 +737,7 @@ async def _on_disconnect(_transport, client, *args): # Whichever comes first: the session dies on its own, it ages out, or # the OpenAI socket dies underneath it. OpenAI drops idle Realtime # sockets well before our max-age rotation, and Pipecat keeps the - # stale handle and pumps audio into it — the device hears silence + # stale handle and pumps audio into it: the device hears silence # until the next rotation. Poll the socket and rotate the moment it # goes dead (websockets' keepalive flips state within ~40s). # @@ -768,7 +768,7 @@ async def _on_disconnect(_transport, client, *args): break except asyncio.TimeoutError: pass - # Two death modes: the socket itself dies (idle drop — keepalive + # Two death modes: the socket itself dies (idle drop: keepalive # flips the state within ~40s), or Pipecat's receive loop exits on # an OpenAI error event while the socket stays OPEN (brain-dead # session: audio goes in, nothing comes back). @@ -789,7 +789,7 @@ async def _on_disconnect(_transport, client, *args): if not runner_task.done(): await task.cancel() # Await the runner to fully finish so the websocket server releases the - # port before the next session rebinds — otherwise rotation can hit an + # port before the next session rebinds: otherwise rotation can hit an # intermittent "address already in use". try: await asyncio.wait_for(asyncio.shield(runner_task), timeout=10) diff --git a/broker/requirements.txt b/broker/requirements.txt index 71c4947..96efd22 100644 --- a/broker/requirements.txt +++ b/broker/requirements.txt @@ -1,5 +1,5 @@ # Pinned for reproducibility. Pipecat emits the GA Realtime session JSON, so a -# silent minor bump could change the wire shape — bump deliberately. +# silent minor bump could change the wire shape, so bump deliberately. pipecat-ai[mcp,openai,websocket]==0.0.97 python-dotenv==1.2.1 websockets==15.0.1 diff --git a/broker/tools/harness.py b/broker/tools/harness.py index 360e214..1fea79d 100644 --- a/broker/tools/harness.py +++ b/broker/tools/harness.py @@ -1,21 +1,21 @@ """End-to-end reliability harness for the voice-pe realtime broker. -Drives the broker exactly like the Voice PE firmware does — streams PCM16 / -24 kHz / mono speech up the WebSocket in 20 ms frames with trailing silence — +Drives the broker exactly like the Voice PE firmware does, streams PCM16 / +24 kHz / mono speech up the WebSocket in 20 ms frames with trailing silence, then collects the spoken reply, transcribes it, and asserts on content, turn-taking, and latency. No hardware needed; test speech is synthesized with OpenAI TTS and replies are transcribed with Whisper. Targets a RUNNING broker. Defaults to ws://127.0.0.1:8766 on purpose so it does -NOT fight the live puck on :8765 (the broker is single-client — connecting +NOT fight the live puck on :8765 (the broker is single-client, connecting kicks whoever is already on). Spin an isolated broker on 8766 to run this. OPENAI_API_KEY=... python -m broker.tools.harness [ws://host:port] [--soak N] -The legacy SCENARIOS assume the JAY-84 turn hygiene is DISABLED (start the +The legacy SCENARIOS assume the turn-hygiene behavior is DISABLED (start the broker with FOLLOWUP_WINDOW_SECONDS=0): synth/whisper turnaround between turns can approach the 6s follow-up window and would race the close. The -hygiene feature has its own set — `--hygiene` runs HYGIENE_SCENARIOS only, +hygiene feature has its own set, `--hygiene` runs HYGIENE_SCENARIOS only, against an isolated broker at 8766 started with FOLLOWUP_WINDOW_SECONDS=6 MAX_TURNS_PER_WAKE=2. @@ -45,10 +45,10 @@ # ---------------------------------------------------------------------------- -# OpenAI TTS / STT helpers (plain HTTP, no extra deps — matches existing tools) +# OpenAI TTS / STT helpers (plain HTTP, no extra deps: matches existing tools) # ---------------------------------------------------------------------------- def _send(req: urllib.request.Request, attempts: int = 4) -> bytes: - """POST with retry — OpenAI's audio endpoints occasionally blip (429/5xx, + """POST with retry, OpenAI's audio endpoints occasionally blip (429/5xx, even a transient 404). A flaky API call must not fail a scenario.""" delay = 1.0 for n in range(attempts): @@ -162,7 +162,7 @@ async def ask(self, text: str, voice: str = "alloy") -> Reply: return await self._collect() async def ask_pcm(self, pcm: bytes) -> Reply: - """ask() with pre-synthesized PCM — hygiene scenarios must start their + """ask() with pre-synthesized PCM, hygiene scenarios must start their next turn inside the follow-up window, and a synth() round trip alone can eat most of it.""" await self._stream(pcm) @@ -186,7 +186,7 @@ async def session(url: str): # ---------------------------------------------------------------------------- -# Scenarios — each returns (passed: bool, detail: str, latency_ms: float | None) +# Scenarios: each returns (passed: bool, detail: str, latency_ms: float | None) # ---------------------------------------------------------------------------- async def s_capital(url): async with session(url) as c: @@ -235,7 +235,7 @@ async def s_reconnect(url): r1 = await c.ask("What is the capital of France?") if not r1.got_audio: return False, "first session got no audio", r1.first_audio_ms - async with session(url) as c: # fresh connection — the fjfricke #9 bug + async with session(url) as c: # fresh connection, the fjfricke #9 bug r2 = await c.ask("In one short sentence, what is two plus two?") t = r2.transcript().lower() ok = r2.got_audio and any(w in t for w in ("four", "4")) @@ -260,9 +260,9 @@ async def s_background_rejection(url): async def s_challenge_follow_up(url): - # Both real turns the TV test swallowed (2026-07-01) were follow-ups to + # Both real turns swallowed under background TV noise were follow-ups to # the bot's own answer. A bare challenge ("are you sure?") has no topic - # words at all — the hardest addressed-speech case for the background + # words at all: the hardest addressed-speech case for the background # gate. The bot must answer, not wait_for_user it away. async with session(url) as c: r1 = await c.ask("In one short sentence, what is the capital of France?") @@ -276,10 +276,10 @@ async def s_challenge_follow_up(url): async def s_tv_line_after_answer(url): # Counter-metric to the follow-up bias in BACKGROUND_GUIDANCE: a # conversational TV line in a different voice right after the bot - # answers — the exact false-accept from the TV test ("It's good, huh?", - # 2026-07-01 20:49:56). Want silence. Pre-declared tradeoff: swallowing + # answers: the exact false-accept class this gate must reject ("It's + # good, huh?"). Want silence. Pre-declared tradeoff: swallowing # a real follow-up is worse than answering a TV line, so under --soak - # this scenario is allowed to be the flakier of the pair — but it must + # this scenario is allowed to be the flakier of the pair: but it must # not fail while challenge_follow_up also fails. async with session(url) as c: r1 = await c.ask("In one short sentence, what is the capital of France?") @@ -329,14 +329,14 @@ async def s_mid_speech_disconnect(url): # ---------------------------------------------------------------------------- -# Turn-hygiene scenarios (JAY-84) — run with --hygiene against an ISOLATED +# Turn-hygiene scenarios: run with --hygiene against an ISOLATED # broker on 8766 started with FOLLOWUP_WINDOW_SECONDS=6 MAX_TURNS_PER_WAKE=2. # Kept out of SCENARIOS: the legacy set assumes the feature is off (W=0), # because synth/whisper turnaround between turns can approach the 6s window. # ---------------------------------------------------------------------------- async def _watch_disconnect(c: Client, timeout: float) -> float | None: """Stream silence (the device's mic never closes) and wait for the - broker's {"type":"disconnect"} text frame — or a server-side close, which + broker's {"type":"disconnect"} text frame, or a server-side close, which counts too. Mirrors test_bargein.py's interrupt-frame capture. Returns the monotonic arrival time, or None on timeout.""" pump = asyncio.create_task(c._stream(b"\x00\x00" * int(RATE * timeout))) @@ -391,7 +391,7 @@ async def h_followup_window_allows(url): async def h_turn_budget_cap(url): # With MAX_TURNS_PER_WAKE=2: two Q/A turns succeed, then the broker - # disconnects once the 2nd reply finishes playing — no 3rd turn granted. + # disconnects once the 2nd reply finishes playing: no 3rd turn granted. # The budget close (playback end + release) usually beats r2's 2.5s # collect idle, so the disconnect frame tends to land inside r2.texts. q2 = synth("In one short sentence, what is two plus two?") @@ -541,12 +541,12 @@ async def run( t0 = time.monotonic() try: ok, detail, lat = await fn(url) - except Exception as e: # noqa: BLE001 — a crash IS a failed scenario + except Exception as e: # noqa: BLE001 (a crash IS a failed scenario) ok, detail, lat = False, f"EXCEPTION {type(e).__name__}: {e}", None dt = time.monotonic() - t0 if lat is not None: latencies.append(lat) - latstr = f"{lat:.0f}ms 1st-audio" if lat is not None else "—" + latstr = f"{lat:.0f}ms 1st-audio" if lat is not None else "n/a" print(f"[{'PASS' if ok else 'FAIL'}] {label:<26} {latstr:<16} {dt:4.1f}s {detail}") results.append((label, ok, detail, lat)) diff --git a/broker/tools/hook_check.py b/broker/tools/hook_check.py index 7ccf09d..c660ba4 100644 --- a/broker/tools/hook_check.py +++ b/broker/tools/hook_check.py @@ -1,6 +1,6 @@ """PostToolUse guard: run `make check` only when a broker .py file was edited. -Wired in .claude/settings.json. Keeps the loop cheap — a no-op for unrelated +Wired in .claude/settings.json. Keeps the loop cheap, a no-op for unrelated edits, and only the realtime check (which spends a couple OpenAI turns) runs when broker Python actually changes. Exit 2 feeds the failure back to Claude so the loop cannot close on red. @@ -11,7 +11,7 @@ import sys # Repo root, derived from this file's location (/broker/tools/hook_check.py) -# so the hook is portable — no hardcoded personal path. +# so the hook is portable: no hardcoded personal path. REPO = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) data = json.load(sys.stdin) fp = (data.get("tool_input") or {}).get("file_path", "") diff --git a/broker/tools/m2_analyze.py b/broker/tools/m2_analyze.py index 10e3991..947b0c8 100644 --- a/broker/tools/m2_analyze.py +++ b/broker/tools/m2_analyze.py @@ -16,7 +16,7 @@ starts at an arbitrary time before playback), then per-window (320ms, hop 160ms) normalized cross-correlation at the global lag +/- 150ms search range, scored only over windows where the reference is active. Baseline is -the same statistic over reference-silent windows — it shows what NCC noise +the same statistic over reference-silent windows, it shows what NCC noise looks like for this capture. """ @@ -60,7 +60,7 @@ def max_ncc(r: np.ndarray, m: np.ndarray) -> float: """Max |NCC| of window r against every sample offset of segment m. Sample-accurate: NCC collapses within a few samples of the true lag, so - any strided search misses the peak. Vectorized — numerator via one + any strided search misses the peak. Vectorized, numerator via one cross-correlation, per-offset mean/energy via cumulative sums. """ r0 = r - r.mean() @@ -88,7 +88,7 @@ def main() -> None: # shorter one, which would produce a garbage global lag and a false # "residual uncorrelated" verdict. The capture must cover the clip. raise SystemExit( - "capture shorter than the reference — bad take, retake it " + "capture shorter than the reference. Bad take, retake it " f"(capture {len(mic)/RATE:.1f}s < ref {len(ref)/RATE:.1f}s)" ) @@ -109,7 +109,7 @@ def main() -> None: (active if np.abs(r).mean() > ref_gate else silent).append((best, rms)) if not active: - raise SystemExit("no reference-active windows found — alignment failed?") + raise SystemExit("no reference-active windows found, alignment failed?") a_ncc = np.array([x[0] for x in active]) a_rms = np.array([x[1] for x in active]) @@ -125,14 +125,14 @@ def main() -> None: p50 = float(np.median(a_ncc)) if p50 >= 0.6: - print("\nVERDICT: NCC gate viable — echo residual clearly correlated. Build M1.") + print("\nVERDICT: NCC gate viable, echo residual clearly correlated. Build M1.") elif p50 >= 0.3: - print("\nVERDICT: gray zone (0.3-0.6) — per the plan, STOP and redesign " + print("\nVERDICT: gray zone (0.3-0.6): per the plan, STOP and redesign " "(volume cap -> aec_corr_factor DFU -> software AEC).") else: print("\nVERDICT: residual uncorrelated. If mic RMS during playback is " "near the silent baseline, XMOS AEC already buries the echo at " - "this volume — measure at max volume before concluding.") + "this volume, measure at max volume before concluding.") if __name__ == "__main__": diff --git a/broker/tools/m2_capture.py b/broker/tools/m2_capture.py index 1072e43..3618551 100644 --- a/broker/tools/m2_capture.py +++ b/broker/tools/m2_capture.py @@ -8,7 +8,7 @@ Run it on the port the puck targets (the live broker must be stopped first), wake the puck, and play the reference clip through the puck's media_player via m2_play_ref.py. What lands here is -the XMOS ch1 post-AEC residual of that playback — the real signal the +the XMOS ch1 post-AEC residual of that playback, the real signal the NCC gate would see. docker stop voicepe @@ -43,7 +43,7 @@ async def main() -> None: out = Path(args.out_dir) out.mkdir(exist_ok=True) - # Resume numbering after existing captures — a fresh run must never + # Resume numbering after existing captures: a fresh run must never # overwrite takes recorded by a previous one mid-calibration-session. counter = max( (int(p.stem.split("_")[1]) for p in out.glob("capture_*.pcm") @@ -80,7 +80,7 @@ async def handle(ws) -> None: print(f"[m2] device disconnected: {nbytes/2/RATE:.1f}s captured in {pcm_path}") async with websockets.serve(handle, args.host, args.port, max_size=None): - print(f"[m2] capture server on ws://{args.host}:{args.port} — wake the puck when ready") + print(f"[m2] capture server on ws://{args.host}:{args.port}, wake the puck when ready") await asyncio.Future() diff --git a/broker/tools/m2_play_ref.py b/broker/tools/m2_play_ref.py index f876a14..b9ba3f9 100644 --- a/broker/tools/m2_play_ref.py +++ b/broker/tools/m2_play_ref.py @@ -3,7 +3,7 @@ Generates a known TTS clip (once, cached next to the captures), serves it over HTTP from this machine, and asks Home Assistant to play it on the puck's media_player entity. This drives the speaker through the normal HA -media path — the broker never sends a binary frame, so the firmware mic +media path, the broker never sends a binary frame, so the firmware mic gate stays open and m2_capture.py records the XMOS post-AEC residual of exactly this clip. diff --git a/broker/tools/test_bargein.py b/broker/tools/test_bargein.py index 11b1625..9dec691 100644 --- a/broker/tools/test_bargein.py +++ b/broker/tools/test_bargein.py @@ -82,7 +82,7 @@ async def recv(): print(f" <-- TEXT FRAME: {msg[:200]}") state["text_frames"].append(msg) # Only count interrupt frames that arrive AFTER we sent - # the interruptor — an early one is a spurious boundary + # the interruptor: an early one is a spurious boundary # flush (a bug), not a successful barge-in. if "interrupt" in msg and interrupt_sent.is_set(): state["interrupt_frame"] = True diff --git a/broker/tools/test_client.py b/broker/tools/test_client.py index fed65ad..80f70c4 100644 --- a/broker/tools/test_client.py +++ b/broker/tools/test_client.py @@ -93,7 +93,7 @@ async def main() -> None: write_wav("response.wav", bytes(out)) print("wrote response.wav") else: - print("NO AUDIO RETURNED — check broker logs") + print("NO AUDIO RETURNED, check broker logs") if __name__ == "__main__": diff --git a/broker/tools/test_rotation.py b/broker/tools/test_rotation.py index 64289b1..31bccff 100644 --- a/broker/tools/test_rotation.py +++ b/broker/tools/test_rotation.py @@ -1,7 +1,8 @@ """Proves the broker rotates the OpenAI Realtime session SEAMLESSLY mid-call. -OpenAI caps a Realtime session at ~60 min and treats expiry as fatal — the bug -fjfricke/ha-openai-realtime crashes on (#8). Our broker rotates proactively +OpenAI caps a Realtime session at ~60 min and treats expiry as fatal, a failure +mode the original proof-of-concept didn't handle +(fjfricke/ha-openai-realtime#8). Our broker rotates proactively before the cap and rebuilds the session under a still-connected device. This test forces that path: start a broker with a tiny MAX_SESSION_SECONDS, hold ONE device connection open across several rotations, and assert every turn still diff --git a/firmware/CRASH_ANALYSIS.md b/firmware/CRASH_ANALYSIS.md index 4648ac1..b147f6c 100644 --- a/firmware/CRASH_ANALYSIS.md +++ b/firmware/CRASH_ANALYSIS.md @@ -1,4 +1,4 @@ -# Dual-mode firmware crash — root cause + fixes +# Dual-mode firmware crash: root cause + fixes **Symptom:** device works for one voice interaction, then faults (`Fault - Unknown`, core 1, idle-task PC); wake-word engine dead until reboot. @@ -8,7 +8,7 @@ FreeRTOS task. The component does main-loop-only work from that task: 1. **[CRITICAL] Cross-thread speaker + queue access.** `process_received_audio_` (called from `WEBSOCKET_EVENT_DATA`) calls `speaker_->play/start/stop` and - mutates `audio_queue_` on the websocket task, with no lock — while HA Assist + mutates `audio_queue_` on the websocket task, with no lock, while HA Assist drives the SAME speaker and `loop()` touches the same queue on the main loop. `std::queue` isn't thread-safe; concurrent push/pop across cores corrupts the heap → fault later (idle task), wake engine dead. → defer ALL audio/speaker @@ -16,11 +16,11 @@ FreeRTOS task. The component does main-loop-only work from that task: 2. **[HIGH] Use-after-free on `audio_queue_.front()`** held across `pop()`/`push()`. 3. **[MED] `portMAX_DELAY`** on websocket send can block the main loop → task watchdog fault. → bounded timeout. -4. **[MED] `websocket_client_` lifecycle race** — destroyed on main loop while +4. **[MED] `websocket_client_` lifecycle race**: destroyed on main loop while sender threads deref it. → guard with flag/mutex. -5. **[LOW] mic-callback reads shared `state_`/client unlocked** — fold into lock. +5. **[LOW] mic-callback reads shared `state_`/client unlocked**: fold into lock. -**Fix order:** (1)+(2) defer audio to loop() behind a mutex — almost certainly +**Fix order:** (1)+(2) defer audio to loop() behind a mutex, almost certainly resolves the crash; (3) bounded send timeout; (4) lifecycle guard. Audit performed 2026-05-26. diff --git a/firmware/LICENSE.esphome b/firmware/LICENSE.esphome new file mode 100644 index 0000000..215460a --- /dev/null +++ b/firmware/LICENSE.esphome @@ -0,0 +1,709 @@ +# ESPHome License + +Copyright (c) 2019 ESPHome + +The ESPHome License is made up of two base licenses: MIT and the GNU GENERAL PUBLIC LICENSE. +The C++/runtime codebase of the ESPHome project (file extensions .c, .cpp, .h, .hpp, .tcc, .ino) are +published under the GPLv3 license. The python codebase and all other parts of this codebase are +published under the MIT license. + +Both MIT and GPLv3 licenses are attached to this document. + +## MIT License + +Copyright (c) 2019 ESPHome + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +## GPLv3 License + + GNU GENERAL PUBLIC LICENSE + Version 3, 29 June 2007 + + Copyright (C) 2007 Free Software Foundation, Inc. + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. diff --git a/firmware/README.md b/firmware/README.md index 807f610..57480ab 100644 --- a/firmware/README.md +++ b/firmware/README.md @@ -2,7 +2,7 @@ ESPHome firmware for the Home Assistant Voice PE (ESP32-S3) that streams audio to the realtime broker. -## `voice_pe_dual.yaml` — single wake word +## `voice_pe_dual.yaml`: single wake word Built on the **official** [`home-assistant-voice-pe`](https://github.com/esphome/home-assistant-voice-pe) firmware, with one wake word routed to the realtime broker: @@ -10,14 +10,14 @@ Built on the **official** [`home-assistant-voice-pe`](https://github.com/esphome |---|---| | **Hey Mycroft** | the realtime broker (`voice_assistant_websocket`) | -On "Hey Mycroft" the device plays a wake cue, opens the broker WebSocket, and streams PCM from the same mic (`i2s_mics` — echo-cancelled by the XMOS chip); on disconnect it re-arms wake-word detection and plays an end cue. Audio plays back through the media speaker. +On "Hey Mycroft" the device plays a wake cue, opens the broker WebSocket, and streams PCM from the same mic (`i2s_mics`: echo-cancelled by the XMOS chip); on disconnect it re-arms wake-word detection and plays an end cue. Audio plays back through the media speaker. How it's wired (additive edits to the official YAML): - `external_components` adds the local `voice_assistant_websocket` component - a `voice_assistant_websocket:` block points at `i2s_mics` + `media_resampling_speaker` + the broker URL - `on_wake_word_detected` starts the websocket session -> **Dual-mode variant** — an earlier build routed **Hey Jarvis / Okay Nabu** to the stock HA Assist pipeline *and* **Hey Mycroft** to the broker (local + realtime on one device). That's preserved at git tag `dual-2wake-v0` / branch `archive/dual-2wake`. v1 ships single-wake for a cleaner default; the elegant form of the hybrid (one wake word + smart local/cloud routing) is on the roadmap. +> **Dual-mode variant**: an earlier build routed **Hey Jarvis / Okay Nabu** to the stock HA Assist pipeline *and* **Hey Mycroft** to the broker (local + realtime on one device). That's preserved at git tag `dual-2wake-v0` / branch `archive/dual-2wake`. v1 ships single-wake for a cleaner default; the elegant form of the hybrid (one wake word + smart local/cloud routing) is on the roadmap. ## Setup @@ -36,7 +36,7 @@ It backs up the stock firmware first, then flashes. After reboot the device join ## Status -Compiles clean (`firmware.factory.bin`) and runs on-device. Known limitation: the far-field mic captures the whole room, so in a loud room the assistant can pick up a TV or another speaker — the broker mitigates this with the `wait_for_user` gate, and device-side beamforming (tapping the XMOS focused channel) is on the roadmap. +Compiles clean (`firmware.factory.bin`) and runs on-device. Known limitation: the far-field mic captures the whole room, so in a loud room the assistant can pick up a TV or another speaker, the broker mitigates this with the `wait_for_user` gate, and device-side beamforming (tapping the XMOS focused channel) is on the roadmap. ## Credit diff --git a/firmware/components/voice_assistant_websocket/.esphome_component.yml b/firmware/components/voice_assistant_websocket/.esphome_component.yml index 7a0f05a..a22fd96 100644 --- a/firmware/components/voice_assistant_websocket/.esphome_component.yml +++ b/firmware/components/voice_assistant_websocket/.esphome_component.yml @@ -1,7 +1,7 @@ name: Voice Assistant WebSocket description: WebSocket-based voice assistant component for OpenAI Realtime API version: "1.0.0" -author: OpenAI Realtime Voice Agent +author: jayzuccarelli dependencies: - i2s_audio platforms: diff --git a/firmware/components/voice_assistant_websocket/__init__.py b/firmware/components/voice_assistant_websocket/__init__.py index 29b9b4b..79b678a 100644 --- a/firmware/components/voice_assistant_websocket/__init__.py +++ b/firmware/components/voice_assistant_websocket/__init__.py @@ -7,7 +7,7 @@ from esphome.core import CORE from esphome.components.esp32 import add_idf_component -CODEOWNERS = ["@openai-realtime-voice-agent"] +CODEOWNERS = ["@jayzuccarelli"] DEPENDENCIES = ["microphone", "speaker"] voice_assistant_websocket_ns = cg.esphome_ns.namespace("voice_assistant_websocket") diff --git a/firmware/flash.sh b/firmware/flash.sh index 221f171..b4072f8 100755 --- a/firmware/flash.sh +++ b/firmware/flash.sh @@ -16,10 +16,10 @@ PORT="${1:-/dev/ttyACM0}" RUN=(docker run --rm --device="$PORT" -v "$PWD":/config --entrypoint python "$IMG" -m esptool --port "$PORT") [ -e "$PORT" ] || { echo "No device at $PORT (try: lsusb; ls /dev/ttyACM* /dev/ttyUSB*)"; exit 1; } -[ -f "$BIN" ] || { echo "Missing $BIN — compile first."; exit 1; } +[ -f "$BIN" ] || { echo "Missing $BIN, compile first."; exit 1; } echo ">> Verifying it's an ESP32-S3 (not some other serial device)..." -"${RUN[@]}" flash_id | grep -q "ESP32-S3" || { echo "Not an ESP32-S3 on $PORT — aborting."; exit 1; } +"${RUN[@]}" flash_id | grep -q "ESP32-S3" || { echo "Not an ESP32-S3 on $PORT, aborting."; exit 1; } echo ">> [1/2] Backing up stock firmware (16 MB) -> stock-voice-pe-backup.bin" "${RUN[@]}" --baud 460800 read_flash 0 0x1000000 /config/stock-voice-pe-backup.bin diff --git a/firmware/secrets.yaml.example b/firmware/secrets.yaml.example index cdae22b..63128bd 100644 --- a/firmware/secrets.yaml.example +++ b/firmware/secrets.yaml.example @@ -1,5 +1,5 @@ # Copy to secrets.yaml and fill in. secrets.yaml is gitignored. -wifi_ssid: "YourWiFiSSID" # 2.4 GHz only — the ESP32-S3 can't join 5 GHz +wifi_ssid: "YourWiFiSSID" # 2.4 GHz only, the ESP32-S3 can't join 5 GHz wifi_password: "YourWiFiPassword" broker_url: "ws://YOUR_SERVER_IP:8765" # where the broker listens (same LAN as the device) # Used by Home Assistant to talk to the device (adopt it in HA with this key). diff --git a/firmware/voice_pe_dual.yaml b/firmware/voice_pe_dual.yaml index 81d0971..8d71440 100644 --- a/firmware/voice_pe_dual.yaml +++ b/firmware/voice_pe_dual.yaml @@ -1,3 +1,12 @@ +# Derived from esphome/home-assistant-voice-pe, file `home-assistant-voice.yaml`. +# Copyright (c) 2019 ESPHome. Licensed under the ESPHome License, which applies +# MIT terms to non-C++ files such as this one. Full license text, including the +# MIT permission notice, is in `firmware/LICENSE.esphome`. +# Upstream: https://github.com/esphome/home-assistant-voice-pe +# +# Modified here to route the wake word to a server-side realtime broker instead +# of the stock Assist pipeline. See the models block for which wake word. + substitutions: # Phases of the Voice Assistant # The voice assistant is ready to be triggered by a wake word @@ -14,7 +23,7 @@ substitutions: voice_assist_not_ready_phase_id: '10' # The voice assistant encountered an error voice_assist_error_phase_id: '11' - # Change this to true in case you ahve a hidden SSID at home. + # Change this to true in case you have a hidden SSID at home. hidden_ssid: "false" # Substitutions for audio files jack_connected_sound_file: https://github.com/esphome/home-assistant-voice-pe/raw/dev/sounds/jack_connected.flac