Roadmap: v0.1 - Foundation

Principle

Private speech AI = on-device models + zero network calls + local-only storage

Every component runs locally — GPU inference via MLX (Metal), Neural Engine via CoreML. Audio never leaves the process, no cloud fallback, no telemetry. Your voice stays on your hardware.

---

Where we are

8 protocols, 11 models, 7 pipeline types — ASR, TTS, speech-to-speech, VAD, diarization, speaker embedding, speech enhancement. All on-device via MLX + CoreML.

Architecture

Layer 1: Primitives          Layer 2: Pipelines              Layer 3: Apps
──────────────────          ──────────────────              ─────────────
Qwen3ASR                    StreamingASR (VAD→ASR) ✅        CLI (9 cmds) ✅
ParakeetASR (CoreML)        DiarizationPipeline ✅           HTTP server ✅
Qwen3TTS                    Voice pipeline (→LLM→) ✅        WebSocket streaming ✅
CosyVoiceTTS                Voice cloning (x-vector) ✅      iOS demo app ✅
PersonaPlex                 Streaming diarized ASR ✅        Benchmark dashboard 🔜
KokoroTTS (CoreML)          MeetingTranscriber 🔜
PiperTTS (CoreML) 🔜        Music separation 🔜
SileroVAD / PyannoteVAD
WeSpeaker                   speech-core (C++)
DeepFilterNet3              Cross-platform pipeline engine ✅

                            Layer 2.5: Platform Foundation
                            ──────────────────────────────
                            AudioIO manager 🔜
                            PipelineLLM adapters 🔜
                            Model loading orchestrator 🔜
                            Tool calling bridge 🔜

---

v0.1.0

Done

• CI/CD: release build + metallib + 19 safe tests on every PR (CI/CD: release build + metallib + 19 safe tests on PR #85, CI/CD: build + safe tests on PR #77)
• Fix Homebrew install docs ('speech' command #67, Document ARM Homebrew requirement #84)
• Fix PersonaPlexDemo .app bundle build (PersonaPlexDemo fails to wrap #71, Fix PersonaPlexDemo .app bundle build script #83)
• Model memory management: unload() API + memory footprint (Model memory management: unload API + peak RSS reporting #79)
• Document memory requirements and 8GB compatibility (Document memory requirements and 8GB device compatibility #74)
• Voice cloning: Speaker Encoder (ECAPA-TDNN, x-vector mode) (Voice cloning: port Speech Tokenizer Encoder + Speaker Encoder for zero-shot TTS #87, Add voice cloning via ECAPA-TDNN speaker encoder #90)
• WebSocket streaming: OpenAI Realtime API (/v1/realtime) (Add WebSocket streaming to HTTP API server #73, Add OpenAI Realtime API WebSocket endpoint #91)
• 8-bit quantization across all models (Support for Qwen TTS 8-bit model #12, Add 8-bit quantization support across all models #53)
• Composable voice pipeline: VAD → ASR → LLM → TTS (Add composable voice pipeline: VAD → ASR → LLM → TTS #72)
• Streaming ASR with real-time diarization (Streaming ASR with real-time speaker diarization #80)
• Tool-calling intent engine in speech-core C++ (Experiment: tool-calling intent engine for voice pipeline #76)
• speech-core: cross-platform C++ pipeline engine (repo)
• iOS demo app: TamagotchiDemo — all-CoreML voice chat (iOS demo app with SwiftUI #75, Add iOS SpeechDemo app #102)
• Qwen3Chat 0.6B CoreML module with prompt caching
• Kokoro-82M CoreML TTS — 50 voices, 10 languages, ANE (Add on-device TTS for iOS: Kokoro-82M (CoreML) + Piper (VITS) #93)

In progress

• ASR accuracy benchmarks: WER on LibriSpeech test-clean (Benchmark suite: WER, DER, RTF across all models #78)
• TTS quality and latency benchmarks across M1-M4 (TTS quality & latency benchmarks across Apple Silicon #88)
• Speaker-aware transcription (Demo: speaker-aware speech recognition #55)
• Piper TTS ultra-lightweight voices — ONNX to CoreML, 39 languages (Add on-device TTS for iOS: Kokoro-82M (CoreML) + Piper (VITS) #93)

Platform foundation (new)

The glue between Layer 2 (Pipelines) and Layer 3 (Apps) — what every voice app needs but should not reimplement.

• AudioIO manager — reusable mic input + speaker output with AEC, eliminates ~80 lines of AVAudioEngine boilerplate per app (Add reusable AudioIO manager to eliminate AVAudioEngine boilerplate #119)
• PipelineLLM adapters — built-in adapters for Qwen3Chat (local) and remote APIs (Claude, OpenAI), eliminates ~60-line bridge per app (Add built-in PipelineLLM adapters (local + remote) #120)
• Model loading orchestrator — parallel loading of N models with unified progress callback (Add unified model loading orchestrator with aggregated progress #123)
• Tool calling bridge — expose speech-core tool calling to Swift via PipelineTool closures (Bridge tool calling from speech-core to Swift #124, blocked on speech-core#5)
• STT confidence filtering — real confidence scores from ASR models + configurable threshold to filter garbage transcriptions (Add STT confidence filtering to VoicePipeline #125)
• Concurrency fixes — atomic cancelled flags, tracked Tasks with cancellation, missing continuation error paths (Fix Swift concurrency issues: data races, untracked Tasks, missing continuations #121)
• Qwen3Chat ModelMemoryManageable — only model missing unload / memoryFootprint (Add ModelMemoryManageable conformance to Qwen3Chat #122)
• Normalize init patterns — make Qwen3TTS/ASR public init internal, enforce fromPretrained (Normalize model initialization patterns across all modules #126)

speech-core upstream

Pipeline engine improvements that unlock new capabilities in speech-swift.

• Expose tool calling in C API (speech-core#5)
• Add denoise pre-processing stage — plug DeepFilterNet3 before VAD (speech-core#6)
• Per-stage latency metrics in pipeline events (speech-core#7)
• Conversation history limits — masking, token-aware trimming (speech-core#8)
• Partial transcription events for real-time STT display (speech-core#9)

Models and research

• Qwen3-TTS CoreML backend for Neural Engine (Add Qwen3-TTS CoreML backend (Neural Engine) #106)
• Sortformer diarization via CoreML (Add Sortformer speaker diarization (CoreML) #82)
• Music source separation: Mel-Band RoFormer + Open-Unmix (Add music source separation (Mel-Band RoFormer + Open-Unmix) #94)
• Model compression: mixed-bit palettization, outlier decomposition (Model compression experiments: mixed-bit palettization, outlier decomposition, stateful CoreML #110)

---

Feedback welcome. What would you prioritize differently?