Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,6 @@ stock-voice-pe-backup.bin
*.wav
*.pcm

# internal ops runbook (not for public repo)
# personal notes and agent context, kept local
CLAUDE.md
SUNDAY_RUNBOOK.md
11 changes: 0 additions & 11 deletions CLAUDE.md

This file was deleted.

2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Canonical verify loop for voice-pe-realtime.
# `make check` streams synthesized speech into a broker, transcribes the
# spoken reply, and asserts on content no Voice PE device, no being home.
# spoken reply, and asserts on content: no Voice PE device, no being home.
# Defaults to the ISOLATED dev broker on 8766: check.py's own default (8765)
# is the LIVE broker, and hook-fired checks were kicking the puck mid-use
# and billing OpenAI every run. Target live deliberately with
Expand Down
15 changes: 11 additions & 4 deletions NOTICE.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,24 @@
The ESPHome firmware component in
`firmware/components/voice_assistant_websocket/` derives from
**[fjfricke/ha-openai-realtime](https://github.com/fjfricke/ha-openai-realtime)**
(MIT License) the proof-of-concept that first bridged a Home Assistant
(MIT License): the proof-of-concept that first bridged a Home Assistant
Voice PE to the OpenAI Realtime API over WebSocket. That project's MIT license
and copyright are preserved in
`firmware/components/voice_assistant_websocket/LICENSE`.

The device YAML in `firmware/voice_pe_dual.yaml` is derived from
**[esphome/home-assistant-voice-pe](https://github.com/esphome/home-assistant-voice-pe)**
(`home-assistant-voice.yaml`, Copyright (c) 2019 ESPHome), modified to route the
wake word to the broker instead of the stock Assist pipeline. The ESPHome License
applies MIT terms to non-C++ files such as YAML; its full text, including the MIT
permission notice, is preserved in `firmware/LICENSE.esphome`.

The broker is an independent rewrite (clean module structure, SSE-based Home
Assistant MCP control, persistent-session context, reproducible build), but
the original bridging approach came from that project.

Built on:
- [Pipecat](https://github.com/pipecat-ai/pipecat) voice-agent framework (BSD-2-Clause)
- [ESPHome](https://esphome.io) device firmware
- [Home Assistant](https://www.home-assistant.io) home automation + MCP Server
- [Pipecat](https://github.com/pipecat-ai/pipecat): voice-agent framework (BSD-2-Clause)
- [ESPHome](https://esphome.io): device firmware
- [Home Assistant](https://www.home-assistant.io): home automation + MCP Server
- [OpenAI Realtime API](https://platform.openai.com/docs/guides/realtime)
28 changes: 14 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# voice-pe-realtime

Self-hosted, real-time speech-to-speech for the [Home Assistant Voice PE](https://www.home-assistant.io/voice-pe/) talk to your home with ChatGPT-Voice-style latency, and have it actually *do* things.
Self-hosted, real-time speech-to-speech for the [Home Assistant Voice PE](https://www.home-assistant.io/voice-pe/): talk to your home with ChatGPT-Voice-style latency, and have it actually *do* things.

Instead of the turn-based `wake → STT → LLM → TTS` Assist pipeline, the Voice PE streams audio straight to a small server-side **broker** that holds an **OpenAI Realtime** session and controls Home Assistant over **MCP**. One round trip, natural voice, real actions.

Expand All @@ -25,16 +25,16 @@ The canonical pattern: **the agent runs server-side; the device is a thin full-d

## Why

The stock Voice PE pipeline runs STT → LLM → TTS sequentially a latency floor that feels clunky next to ChatGPT Voice. Routing audio through a persistent Realtime session collapses that to a single round trip with a natural voice, while MCP gives the model first-class control of the home.
The stock Voice PE pipeline runs STT → LLM → TTS sequentially, a latency floor that feels clunky next to ChatGPT Voice. Routing audio through a persistent Realtime session collapses that to a single round trip with a natural voice, while MCP gives the model first-class control of the home.

## Reliability

Speech-to-speech on a $59 puck is easy to demo and hard to keep up. This repo treats robustness as the feature:

- **Session rotation** OpenAI caps a Realtime session at ~60 min and treats expiry as fatal. The broker rotates the session *before* the cap (and rebuilds after any death) under a still-connected device, so long-lived pucks never drop. Proven continuous across forced rotations.
- **Idle refresh** a stale idle session (socket open, silently dead) is refreshed proactively.
- **Turn hygiene**a device that vanishes mid-utterance (Wi-Fi blip, session timeout) leaves OpenAI's server VAD holding a speech-in-progress segment that would come back as a ghost turn on the next wake. Clearing the input buffer isn't enough (the bytes go, the VAD state doesn't); the broker disables and re-enables turn detection on disconnect to drop the segment for real. Background speech (a TV, a side conversation) is gated with the OpenAI-recommended `wait_for_user` pattern with an explicit follow-up bias, so "are you sure about that?" right after an answer gets answered instead of ignored.
- **A real test harness** `make check` drives the broker end-to-end exactly like the firmware (streams PCM, transcribes the spoken reply, asserts content + first-audio latency). No hardware needed.
- **Session rotation**: OpenAI caps a Realtime session at ~60 min and treats expiry as fatal. The broker rotates the session *before* the cap (and rebuilds after any death) under a still-connected device, so long-lived pucks never drop. Proven continuous across forced rotations.
- **Idle refresh**: a stale idle session (socket open, silently dead) is refreshed proactively.
- **Turn hygiene**: a device that vanishes mid-utterance (Wi-Fi blip, session timeout) leaves OpenAI's server VAD holding a speech-in-progress segment that would come back as a ghost turn on the next wake. Clearing the input buffer isn't enough (the bytes go, the VAD state doesn't); the broker disables and re-enables turn detection on disconnect to drop the segment for real. Background speech (a TV, a side conversation) is gated with the OpenAI-recommended `wait_for_user` pattern, with an explicit follow-up bias, so "are you sure about that?" right after an answer gets answered instead of ignored.
- **A real test harness**: `make check` drives the broker end-to-end exactly like the firmware (streams PCM, transcribes the spoken reply, asserts content + first-audio latency). No hardware needed.

```bash
cd broker && OPENAI_API_KEY=... make check # 10 scenarios, pass/fail + p50/p95 latency
Expand Down Expand Up @@ -66,24 +66,24 @@ The broker fetches HA's tools at startup and registers them on the Realtime sess

| Env | Default | Purpose |
|---|---|---|
| `OPENAI_API_KEY` | | required |
| `OPENAI_API_KEY` | none | required |
| `MODEL` | `gpt-realtime` | Realtime model |
| `VOICE` | `marin` | Realtime voice |
| `INSTRUCTIONS` | generic | system prompt / persona |
| `WS_HOST` / `WS_PORT` | `0.0.0.0` / `8765` | where the device connects |
| `HA_MCP_URL` / `HA_TOKEN` | | enable HA control (both required) |
| `MUSIC_PLAYER` | | default Music Assistant speaker |
| `HA_MCP_URL` / `HA_TOKEN` | none | enable HA control (both required) |
| `MUSIC_PLAYER` | none | default Music Assistant speaker |
| `VAD_*` | sane defaults | OpenAI server-VAD tuning |
| `FOLLOWUP_WINDOW_SECONDS` | `6.0` | after each reply, how long the mic stays open for a follow-up before the broker disconnects the device (wake word re-arms) |
| `MAX_TURNS_PER_WAKE` | `8` | user turns allowed per wake, so TV speech can't spiral a session |
| `MAX_SESSION_SECONDS` | `3000` | rotate before the 60-min cap |
| `IDLE_REFRESH_SECONDS` | `600` | refresh a stale idle session |

Turn hygiene (`FOLLOWUP_WINDOW_SECONDS` / `MAX_TURNS_PER_WAKE`): set either to `0` to disable that bound both `0` restores the old unbounded behavior, the no-redeploy rollback lever.
Turn hygiene (`FOLLOWUP_WINDOW_SECONDS` / `MAX_TURNS_PER_WAKE`): set either to `0` to disable that bound. Setting both to `0` restores the old unbounded behavior, which is the no-redeploy rollback lever.

## Firmware

The Voice PE runs ESPHome firmware that streams PCM to this broker. See [`firmware/`](firmware/). Flashing replaces the stock firmware; back up first (`esptool read_flash`) the ESP32-S3 ROM bootloader makes bricking effectively impossible.
The Voice PE runs ESPHome firmware that streams PCM to this broker. See [`firmware/`](firmware/). Flashing replaces the stock firmware; back up first (`esptool read_flash`): the ESP32-S3 ROM bootloader makes bricking effectively impossible.

## Status & roadmap

Expand All @@ -92,8 +92,8 @@ The Voice PE runs ESPHome firmware that streams PCM to this broker. See [`firmwa
- ✅ Session rotation before the 60-min Realtime cap (no dropouts)
- ✅ Background-speech gating (`wait_for_user`)
- ✅ End-to-end reliability harness (`make check`)
- ⏳ **Smart routing** one wake word, fast local intents handled on-device, everything else escalated to the LLM (the elegant form of "local + cloud")
- ⏳ **Barge-in**true open-mic interruption using the Voice PE's hardware AEC (experimental; the acoustic self-trigger loop is the open problem an echo-residual calibration rig ships in `broker/tools/`)
- ⏳ **Beamforming** tap the XMOS array's focused channel to reject off-axis room noise (a TV, another speaker)
- ⏳ **Smart routing**: one wake word, fast local intents handled on-device, everything else escalated to the LLM (the elegant form of "local + cloud")
- ⏳ **Barge-in**: true open-mic interruption using the Voice PE's hardware AEC (experimental; the acoustic self-trigger loop is the open problem, an echo-residual calibration rig ships in `broker/tools/`)
- ⏳ **Beamforming**: tap the XMOS array's focused channel to reject off-axis room noise (a TV, another speaker)

MIT licensed. Attribution in [NOTICE.md](NOTICE.md).
2 changes: 1 addition & 1 deletion broker/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ VAD_RELEASE_DELAY_MS=1200
# speaker goes quiet) to follow up before the broker disconnects the device and
# the wake word re-arms; MAX_TURNS_PER_WAKE caps user turns per wake so TV
# speech can't spiral a session. Set either to 0 to disable (both 0 = old
# unbounded behavior) the no-redeploy rollback lever.
# unbounded behavior), which is the no-redeploy rollback lever.
FOLLOWUP_WINDOW_SECONDS=6.0
MAX_TURNS_PER_WAKE=8

Expand Down
4 changes: 2 additions & 2 deletions broker/Makefile
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# voice-pe realtime broker reliability harness.
# voice-pe realtime broker: reliability harness.
#
# `make check` drives the broker end-to-end (synthesizes speech, streams it like
# the firmware, transcribes the reply, asserts content + latency). Needs
# OPENAI_API_KEY in the environment and a RUNNING broker.
#
# WS defaults to :8766 (an ISOLATED broker), NOT the live puck's :8765 — the
# WS defaults to :8766 (an ISOLATED broker), NOT the live puck's :8765. The
# broker is single-client, so pointing this at :8765 kicks the device.
#
# make check # against ws://127.0.0.1:8766
Expand Down
14 changes: 7 additions & 7 deletions broker/realtime_broker/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Configures voice, system prompt, server-side VAD, and the available tools:
the Home Assistant MCP tools (if HA control is enabled) plus two custom broker
tools `get_weather` (live HA weather, which HA's MCP doesn't surface) and
tools, `get_weather` (live HA weather, which HA's MCP doesn't surface) and
`end_conversation` (clean "ok, bye" stop). Handlers for the custom tools are
registered by the server (they need HA access / the device connection).
"""
Expand Down Expand Up @@ -34,7 +34,7 @@ class VoicePERealtimeService(OpenAIRealtimeLLMService):
Upstream's _handle_context treats the FIRST context frame as conversation
setup: it replays the context as conversation items and issues a bare
response.create. With server_vad the audio commit already created the user
item and auto-created the response, so that double-fires OpenAI rejects
item and auto-created the response, so that double-fires, OpenAI rejects
it (conversation_already_has_active_response) and Pipecat treats any error
event as fatal, killing the session's receive loop. Conversation state
lives server-side here; the only thing context frames must deliver is new
Expand Down Expand Up @@ -109,7 +109,7 @@ async def _handle_evt_error(self, evt) -> None:
"to you: TV or other media dialogue, or a conversation between other "
"people. If you just answered and the next utterance could be a "
"follow-up, reaction, or challenge to your answer ('are you sure?', "
"'okay, and what about...'), it IS addressed to you — answer it "
"'okay, and what about...'), it IS addressed to you. Answer it "
"instead of calling this. Calling it means stay silent and keep "
"listening. Produce no spoken reply when you call it."
),
Expand All @@ -119,7 +119,7 @@ async def _handle_evt_error(self, evt) -> None:

# Appended to the configured persona instructions. This device is far-field and
# its mic hears the whole room (TV, other people), so the model must gate on
# whether speech is actually addressed to it the OpenAI-recommended pattern
# whether speech is actually addressed to it: the OpenAI-recommended pattern
# for rejecting non-addressed speech (there is no speaker separation at the API
# layer).
BACKGROUND_GUIDANCE = (
Expand All @@ -131,7 +131,7 @@ async def _handle_evt_error(self, evt) -> None:
"right after you answer, the next utterance is usually the same user "
"following up. A follow-up question, reaction, or challenge to what you just "
"said ('are you sure?', 'okay, and...', 'what about tomorrow?') is addressed "
"to you even when it does not name you — answer it. When torn between "
"to you even when it does not name you. Answer it. When torn between "
"answering a plausible follow-up and staying silent, answer: a wrongly "
"ignored user must repeat themselves, which is worse than a wrongly "
"answered TV line."
Expand All @@ -146,7 +146,7 @@ def build_audio_input(config: Config, threshold: float | None) -> AudioInput:
transcription.

threshold=None disables turn detection entirely (serialized as
turn_detection: null), which also discards the server's VAD state
turn_detection: null), which also discards the server's VAD state,
used to drop a speech-in-progress segment after a device disconnect.
"""
return AudioInput(
Expand All @@ -168,7 +168,7 @@ def build_audio_input(config: Config, threshold: float | None) -> AudioInput:
# off lets the quiet-but-clean tap reach the VAD intact.
noise_reduction=None,
# Optionally transcribe each user turn so broker logs show what OpenAI
# heard (self-trigger / "janky" diagnosis). Off by default it bills a
# heard (self-trigger / "janky" diagnosis). Off by default: it bills a
# Whisper pass per turn. whisper-1 because gpt-4o-transcribe yielded
# zero transcription events on gpt-realtime.
transcription=(
Expand Down
4 changes: 2 additions & 2 deletions broker/realtime_broker/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ class Config:
# response.done) to take another turn before the broker disconnects the
# device and the wake word re-arms; max_turns_per_wake caps committed user
# turns per WS connection. Either set to 0 disables that bound (both 0 =
# exactly the pre-hygiene behavior) the no-redeploy rollback lever.
# exactly the pre-hygiene behavior): the no-redeploy rollback lever.
followup_window_seconds: float = 6.0
max_turns_per_wake: int = 8

Expand All @@ -64,7 +64,7 @@ class Config:
# An idle Realtime session goes stale server-side WITHOUT the socket dying:
# a 47-min-old session accepted audio and returned nothing while ws.state
# stayed OPEN (2026-06-10). Refresh the session whenever no device has been
# connected for this long free, and invisible to the user.
# connected for this long: free, and invisible to the user.
idle_refresh_seconds: int = 600 # 10 min

@property
Expand Down
Loading
Loading