Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,22 +13,22 @@ This file is for any AI coding agent working in this repo (Claude, Codex, Cursor
3. Rewrite or re-record lines in the cloned voice, with inline emotion markers (e.g. `<whisper>`, `<excited>`, `<sad>`).
4. Preview against the video; export muxed output.

Status: the clone → script → synthesize → export pipeline works end to end. macOS runs the MLX engines — `VoxCPM2` by default, with `CosyVoice3`, `Qwen3-TTS`, and `Chatterbox` (multilingual) selectable from the toolbar; Windows/Linux run `VoxCPM2` through speech-core's LiteRT backend.
Status: the clone → script → synthesize → export pipeline works end to end. macOS runs the MLX engines — `CosyVoice3` by default, with `VoxCPM2`, `Qwen3-TTS`, and `Chatterbox` (multilingual) selectable from the toolbar; Windows/Linux run `VoxCPM2` through speech-core's LiteRT backend.

## Stack

**Tauri** (Rust shell + web frontend) wrapping the Soniqo speech engines.

- **Rust process** — Tauri app, owns the window, menu, file pickers, IPC, model lifecycle, file I/O. Talks to a voice-cloning TTS backend through a **sidecar** chosen at compile time per OS:
- **macOS (Apple Silicon)** — `speech-swift` (Swift / MLX) via the `swift-sidecar/` binary. `VoxCPM2` is the default cloning engine; `CosyVoice3`, `Qwen3-TTS`, and the multilingual `Chatterbox` are also selectable (all MLX, macOS-only).
- **macOS (Apple Silicon)** — `speech-swift` (Swift / MLX) via the `swift-sidecar/` binary. `CosyVoice3` is the default cloning engine; `VoxCPM2`, `Qwen3-TTS`, and the multilingual `Chatterbox` are also selectable (all MLX, macOS-only).
- **Windows / Linux (x86_64)** — `speech-core` (C++) via the `core-sidecar/` binary, cloning + cloned-voice TTS with the `VoxCPM2` **LiteRT** model through the C ABI in `include/speech_core/voxcpm2_c.h`.
- **v1+** — `speech-core`'s broader C ABI (`speech_core_c.h`) for STT (Parakeet), VAD (Silero), noise suppression (DeepFilterNet3), audio utilities.
- **Web frontend** — **React + Vite**, rendered into the OS WebView (WKWebView on macOS, WebView2 on Windows, WebKitGTK on Linux). Owns the video timeline, voice-clone manager, script editor with emotion markers, and waveform views. Talks to Rust via Tauri `invoke()` commands and events.
- **Bridge mechanism** — a stateful **sidecar binary** bundled with the app. Tauri spawns it; Rust talks to it over stdin/stdout using an NDJSON protocol (one JSON object per line each way). The sidecar loads the model once and keeps it resident across calls, so per-line synthesis after warmup is fast. The protocol (`ping` / `init_model` / per-engine `synthesize_*`) is implemented by both sidecars — the Swift sidecar handles every macOS engine (`synthesize_voxcpm2` / `_cosyvoice` / `_chatterbox` / `_icl`), the C++ sidecar handles `synthesize_voxcpm2` — so `SidecarManager` (`src-tauri/src/lib.rs`) only varies the binary path per OS. `init_model` carries the selected engine. Code: `swift-sidecar/` (Swift package, macOS) and `core-sidecar/` (CMake C++, Windows/Linux).

**Target platforms.**

- **v0: macOS (Apple Silicon)** — MLX engines via `swift-sidecar`: `VoxCPM2` (default, cloning + emotional markers) plus selectable `CosyVoice3`, `Qwen3-TTS`, and the multilingual `Chatterbox`. These engines are macOS-only — Windows/Linux don't list them.
- **v0: macOS (Apple Silicon)** — MLX engines via `swift-sidecar`: `CosyVoice3` (default, cloning + emotional markers) plus selectable `VoxCPM2`, `Qwen3-TTS`, and the multilingual `Chatterbox`. These engines are macOS-only — Windows/Linux don't list them.
- **v0: Windows / Linux (x86_64)** — `VoxCPM2` via speech-core's LiteRT backend (`core-sidecar`). Same clone + emotion-marker story without MLX. ASR-graded retry isn't wired here yet (see Notes); the first successful take is accepted.

Why Tauri (vs Electron): smaller binaries, native shell, easier C++ FFI from Rust, desktop-first distribution. Matches the "deploy-anywhere" positioning.
Expand All @@ -38,7 +38,7 @@ Why Tauri (vs Electron): smaller binaries, native shell, easier C++ FFI from Rus
## Sibling repos under `~/repos/`

- **speech-core** — C++ engine. **v0 dependency on Windows/Linux**: the `core-sidecar` links its `speech_core_models_litert` static lib + the `libLiteRt` runtime and drives `VoxCPM2` via `include/speech_core/voxcpm2_c.h`. Also the v1+ source of truth for VAD / STT / non-cloned TTS / enhancement. Build it with `-DSPEECH_CORE_WITH_LITERT=ON -DLITERT_DIR=...` first (see its `AGENTS.md` for the C ABI and CMake targets).
- **speech-swift** — speech models runtime for Apple Silicon (MLX / CoreML). The macOS voice-cloning backend: `VoxCPM2TTS` (default), `CosyVoiceTTS`, `Qwen3TTS` (ICL clone API in `Qwen3TTS+ICL.swift`), and `ChatterboxTTS` (multilingual, `Sources/ChatterboxTTS/`).
- **speech-swift** — speech models runtime for Apple Silicon (MLX / CoreML). The macOS voice-cloning backend: `CosyVoiceTTS` (default), `VoxCPM2TTS`, `Qwen3TTS` (ICL clone API in `Qwen3TTS+ICL.swift`), and `ChatterboxTTS` (multilingual, `Sources/ChatterboxTTS/`).
- **speech-models** — model artifacts on Hugging Face (`aufklarer/`). Studio bundles or downloads from here on first use.

## Build
Expand Down Expand Up @@ -114,7 +114,7 @@ pnpm tauri build # add --no-bundle to skip msi/nsis install
- Per-OS Tauri bundle settings live in `tauri.{macos,windows,linux}.conf.json` (merged over `tauri.conf.json`): `externalBin` selects the sidecar, `resources` ships its runtime (`mlx.metallib` on macOS, `libLiteRt.dll`/`.so` on Windows/Linux). `tauri-build` verifies `externalBin` on **every** cargo build, so stage `src-tauri/binaries/<name>-<triple>` first (it's gitignored) or even `cargo test --lib` fails.
- **macOS sidecar (`swift-sidecar`)**: the build doesn't emit `mlx.metallib` next to the binary on its own — copy it once from the speech-swift build that does (`~/repos/speech-swift/.build/arm64-apple-macosx/debug/mlx.metallib` → `swift-sidecar/.build/arm64-apple-macosx/debug/mlx.metallib`) or you'll get `MLX error: Failed to load the default metallib`. The Rust `colocate_metallib` helper (macOS-only) handles the bundled `.app` layout.
- **Windows/Linux sidecar (`core-sidecar`)**: `cfg`'d off macOS. On `init_model` it loads the bundle from `SONIQO_VOXCPM2_BUNDLE_DIR` if set, else calls `sc_voxcpm2_create_from_pretrained` to download+cache it (`SONIQO_VOXCPM2_MODEL_ID`, default `soniqo/VoxCPM2-LiteRT`; cache via `SONIQO_MODEL_CACHE_DIR`/`SPEECH_CORE_CACHE_DIR`). The CMake colocates `libLiteRt` next to the binary; libcurl is linked statically (no extra DLL). `cfgValue` from the synth ladder has no LiteRT knob and is ignored (the ladder still varies `seed`).
- **TTS model**: macOS MLX path defaults to `aufklarer/VoxCPM2-MLX-int8`. Windows/Linux use the `VoxCPM2-LiteRT` bundle — downloaded on first run (resumable; see speech-core's `SPEECH_CORE_WITH_HF_DOWNLOAD`) or supplied via `SONIQO_VOXCPM2_BUNDLE_DIR`.
- **TTS model**: macOS defaults to the CosyVoice3 MLX engine; VoxCPM2 remains selectable via `aufklarer/VoxCPM2-MLX-int8`. Windows/Linux use the `VoxCPM2-LiteRT` bundle — downloaded on first run (resumable; see speech-core's `SPEECH_CORE_WITH_HF_DOWNLOAD`) or supplied via `SONIQO_VOXCPM2_BUNDLE_DIR`.
- Generated clip audio is cached under `dirs::cache_dir()/audio.soniqo.studio/clips/` (`~/Library/Caches/...` on macOS, `%LOCALAPPDATA%\...` on Windows, `$XDG_CACHE_HOME`/`~/.cache` on Linux). The Rust and sidecar sides compute this independently and must stay in sync.
- **Follow-ups**: (1) ASR-graded retry is macOS-only (`GRADING_AVAILABLE` in `lib.rs`); Windows/Linux accept the first successful take — wiring Parakeet-via-sidecar grading is TODO. (2) The Windows/Linux CI lanes (`build.yml` `build-windows` / `build-linux`) build the sidecar, run `ctest`, `pnpm tauri build`, and attach installers to the Release on tag — but they check out speech-core at the moving `ref: main` (not a pinned SHA) and run no model inference, so an x86_64 LiteRT runtime regression can still slip past CI. First-run model download is handled by speech-core, so installers don't embed the bundle.

Expand Down
28 changes: 14 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,16 +30,16 @@ The clone is local. The synth is local. No audio leaves your machine.
- **Tauri 2** shell (Rust + the OS-native WebView) so the shipped app is a small native binary, not a Chromium fork.
- **React + Vite** frontend for the timeline, voice library, and script editor.
- **A warm sidecar process** holds the speech engine resident so per-line synthesis is fast after the first warm-up. Tauri spawns it once and talks NDJSON over stdin/stdout. On macOS this is the **Swift sidecar** (`swift-sidecar/`, MLX); on Windows/Linux the **C++ sidecar** (`core-sidecar/`, LiteRT).
- **VoxCPM2** is the default engine on every platform — via [`speech-swift`](https://github.com/soniqo/speech-swift) (MLX) on macOS and [`speech-core`](https://github.com/soniqo/speech-core) (LiteRT) on Windows/Linux. On macOS you can switch engines from the toolbar: **CosyVoice3**, **Qwen3-TTS**, **Chatterbox** (multilingual cloning across 23 languages), **OmniVoice** (600+ language cloning), **Indic-Mio** (Hindi/Indic emotion tags), and **Fish Audio S2 Pro** (experimental clone + bracket markers). Those MLX engines are macOS-only; Windows/Linux runs VoxCPM2.
- **CosyVoice 3** is the default engine on macOS via [`speech-swift`](https://github.com/soniqo/speech-swift) (MLX). Windows/Linux default to **VoxCPM2** via [`speech-core`](https://github.com/soniqo/speech-core) (LiteRT). On macOS you can switch engines from the toolbar: **VoxCPM2**, **Qwen3-TTS**, **Chatterbox** (multilingual cloning across 23 languages), **OmniVoice** (600+ language cloning), **Indic-Mio** (Hindi/Indic emotion tags), and **Fish Audio S2 Pro** (experimental clone + bracket markers). Those MLX engines are macOS-only; Windows/Linux runs VoxCPM2.

## Engines

Switch engine from the toolbar dropdown (macOS only — Windows/Linux always use VoxCPM2, so the dropdown doesn't appear).

| Engine | Platforms | Backend | Voice cloning | Emotion markers | Languages |
|---|---|---|:---:|---|:---:|
| **VoxCPM2** · default | macOS · Windows · Linux | MLX / LiteRT | ✅ | style instructions | 30 |
| **CosyVoice 3** | macOS only | MLX | ✅ | style instructions | 9 |
| **CosyVoice 3** · macOS default | macOS only | MLX | ✅ | style instructions | 9 |
| **VoxCPM2** · Windows/Linux default | macOS · Windows · Linux | MLX / LiteRT | ✅ | style instructions | 30 |
| **Qwen3-TTS** | macOS only | MLX | ✅ (ICL) | — | 10 |
| **Chatterbox** | macOS only | MLX | ✅ | intensity only¹ | 23 |
| **OmniVoice** | macOS only | MLX | ✅ | restricted instruct² | 600+ |
Expand Down Expand Up @@ -95,15 +95,15 @@ Grab the latest build from the [**releases page**](https://github.com/soniqo/spe

Every platform **downloads its speech model on first run** and caches it, so the installers stay small:

- **macOS** — `.dmg` (~46 MB); drag into `/Applications`. First run pulls ~2.75 GB of MLX weights into `~/Library/Caches/qwen3-speech/`.
- **macOS** — `.dmg` (~46 MB); drag into `/Applications`. First run pulls the CosyVoice 3 MLX weights into `~/Library/Caches/qwen3-speech/`; selecting VoxCPM2 later pulls its ~2.75 GB MLX weights.
- **Windows** — `.msi` or the NSIS `-setup.exe`. First run pulls the ~8.8 GB VoxCPM2-LiteRT bundle into `%LOCALAPPDATA%\speech-core`.
- **Linux** — `.deb` or `.AppImage`. First run pulls the same bundle into `~/.cache/speech-core`.

The Windows/Linux LiteRT bundle is fp16 and needs **~10 GiB of free RAM** to load — an 8 GB machine may fall short.

The **macOS build is signed and notarized** (from v0.0.5 on) — it opens like any other app, no Gatekeeper hoops. The Windows installers are still unsigned: SmartScreen needs *More info → Run anyway*.

### Manual model download (macOS)
### Manual VoxCPM2 model download (macOS)

If the in-app download keeps failing on a flaky or slow network (`Download stalled for …: no progress` / `Failed to download …`), fetch the model yourself and place it where the app looks. Two pieces: the model weights and a small set of tokenizer files.

Expand Down Expand Up @@ -142,7 +142,7 @@ cd swift-sidecar && swift build # builds the Swift sidecar
cd .. && pnpm tauri dev # launches the app, hot-reloads the UI
```

Same ~2.75 GB model download on first synth (into `~/Library/Caches/qwen3-speech/`see [Manual model download](#manual-model-download-macos) if your network keeps dropping it).
The selected MLX engine downloads on first synth (into `~/Library/Caches/qwen3-speech/`). For VoxCPM2 specifically, see [Manual VoxCPM2 model download](#manual-voxcpm2-model-download-macos) if your network keeps dropping it.

### Dev loop — Windows / Linux

Expand All @@ -160,20 +160,20 @@ pnpm tauri dev

Measured on an Apple Silicon Mac (M-series, unified memory). The **resident** column is the real process footprint (Activity Monitor's "Memory" — `vmmap` physical footprint), which is the figure to check against your RAM. **MLX active/peak** is MLX's own accounting (peak is over a multi-line session). Note: plain `ps rss` under-reports by ~3× on Apple Silicon — Metal unified-memory buffers don't count as RSS, so use the resident figures below.

The default **VoxCPM2** engine:
The selectable **VoxCPM2** MLX engine:

| Variant | Disk | MLX active | MLX peak | Resident (real) | Default |
|---|---|---|---|---|---|
| `aufklarer/VoxCPM2-MLX-int8` | 2.75 GB | 3.1 GB | 5.4 GB | **~4–5 GB** | ✅ |
| `aufklarer/VoxCPM2-MLX-bf16` | 4.6 GB | 9.1 GB | 11.4 GB | ~12 GB | |
| Variant | Disk | MLX active | MLX peak | Resident (real) |
|---|---|---|---|---|
| `aufklarer/VoxCPM2-MLX-int8` | 2.75 GB | 3.1 GB | 5.4 GB | **~4–5 GB** |
| `aufklarer/VoxCPM2-MLX-bf16` | 4.6 GB | 9.1 GB | 11.4 GB | ~12 GB |

The other macOS engines load separately when you select them — only one is resident at a time (switching unloads the previous): **Chatterbox** ~4 GB resident (1.3 GB on disk), **CosyVoice 3** lighter, **Qwen3-TTS** (1.7B bf16) heavier. OmniVoice is downloaded and loaded separately when selected.
The macOS engines load separately when selected — only one is resident at a time (switching unloads the previous): **Chatterbox** ~4 GB resident (1.3 GB on disk), **CosyVoice 3** lighter than VoxCPM2, **Qwen3-TTS** (1.7B bf16) heavier. OmniVoice is downloaded and loaded separately when selected.

The MLX buffer cache is capped at 1 GB (`SONIQO_MLX_CACHE_MB` to override) — without that cap, peak grows to tens of GB on long sessions as varying-shape buffers accumulate. Override the default model with `SONIQO_VOXCPM2_MODEL_ID=aufklarer/VoxCPM2-MLX-bf16` if you want the higher-fidelity weights.
The MLX buffer cache is capped at 1 GB (`SONIQO_MLX_CACHE_MB` to override) — without that cap, peak grows to tens of GB on long sessions as varying-shape buffers accumulate. Override the VoxCPM2 model with `SONIQO_VOXCPM2_MODEL_ID=aufklarer/VoxCPM2-MLX-bf16` if you want the higher-fidelity weights.

### Try the demo

Hit **Load demo** in the top bar. It bootstraps a Scene 04 storyboard with two cloned voices (Anna and Marek) and four lines of dialogue — one with each emotion marker — then synthesizes everything via VoxCPM2.
Hit **Load demo** in the top bar. It bootstraps a Scene 04 storyboard with two cloned voices (Anna and Marek) and four lines of dialogue — one with each emotion marker — then synthesizes everything through the currently selected engine (CosyVoice 3 by default on macOS).

### Packaging your own .app / .dmg

Expand Down
Loading
Loading