diff --git a/AGENTS.md b/AGENTS.md index e4cd02c..11f0a0d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,14 +13,14 @@ This file is for any AI coding agent working in this repo (Claude, Codex, Cursor 3. Rewrite or re-record lines in the cloned voice, with inline emotion markers (e.g. ``, ``, ``). 4. Preview against the video; export muxed output. -Status: the clone → script → synthesize → export pipeline works end to end. macOS runs the MLX engines — `VoxCPM2` by default, with `CosyVoice3`, `Qwen3-TTS`, and `Chatterbox` (multilingual) selectable from the toolbar; Windows/Linux run `VoxCPM2` through speech-core's LiteRT backend. +Status: the clone → script → synthesize → export pipeline works end to end. macOS runs the MLX engines — `CosyVoice3` by default, with `VoxCPM2`, `Qwen3-TTS`, and `Chatterbox` (multilingual) selectable from the toolbar; Windows/Linux run `VoxCPM2` through speech-core's LiteRT backend. ## Stack **Tauri** (Rust shell + web frontend) wrapping the Soniqo speech engines. - **Rust process** — Tauri app, owns the window, menu, file pickers, IPC, model lifecycle, file I/O. Talks to a voice-cloning TTS backend through a **sidecar** chosen at compile time per OS: - - **macOS (Apple Silicon)** — `speech-swift` (Swift / MLX) via the `swift-sidecar/` binary. `VoxCPM2` is the default cloning engine; `CosyVoice3`, `Qwen3-TTS`, and the multilingual `Chatterbox` are also selectable (all MLX, macOS-only). + - **macOS (Apple Silicon)** — `speech-swift` (Swift / MLX) via the `swift-sidecar/` binary. `CosyVoice3` is the default cloning engine; `VoxCPM2`, `Qwen3-TTS`, and the multilingual `Chatterbox` are also selectable (all MLX, macOS-only). - **Windows / Linux (x86_64)** — `speech-core` (C++) via the `core-sidecar/` binary, cloning + cloned-voice TTS with the `VoxCPM2` **LiteRT** model through the C ABI in `include/speech_core/voxcpm2_c.h`. - **v1+** — `speech-core`'s broader C ABI (`speech_core_c.h`) for STT (Parakeet), VAD (Silero), noise suppression (DeepFilterNet3), audio utilities. - **Web frontend** — **React + Vite**, rendered into the OS WebView (WKWebView on macOS, WebView2 on Windows, WebKitGTK on Linux). Owns the video timeline, voice-clone manager, script editor with emotion markers, and waveform views. Talks to Rust via Tauri `invoke()` commands and events. @@ -28,7 +28,7 @@ Status: the clone → script → synthesize → export pipeline works end to end **Target platforms.** -- **v0: macOS (Apple Silicon)** — MLX engines via `swift-sidecar`: `VoxCPM2` (default, cloning + emotional markers) plus selectable `CosyVoice3`, `Qwen3-TTS`, and the multilingual `Chatterbox`. These engines are macOS-only — Windows/Linux don't list them. +- **v0: macOS (Apple Silicon)** — MLX engines via `swift-sidecar`: `CosyVoice3` (default, cloning + emotional markers) plus selectable `VoxCPM2`, `Qwen3-TTS`, and the multilingual `Chatterbox`. These engines are macOS-only — Windows/Linux don't list them. - **v0: Windows / Linux (x86_64)** — `VoxCPM2` via speech-core's LiteRT backend (`core-sidecar`). Same clone + emotion-marker story without MLX. ASR-graded retry isn't wired here yet (see Notes); the first successful take is accepted. Why Tauri (vs Electron): smaller binaries, native shell, easier C++ FFI from Rust, desktop-first distribution. Matches the "deploy-anywhere" positioning. @@ -38,7 +38,7 @@ Why Tauri (vs Electron): smaller binaries, native shell, easier C++ FFI from Rus ## Sibling repos under `~/repos/` - **speech-core** — C++ engine. **v0 dependency on Windows/Linux**: the `core-sidecar` links its `speech_core_models_litert` static lib + the `libLiteRt` runtime and drives `VoxCPM2` via `include/speech_core/voxcpm2_c.h`. Also the v1+ source of truth for VAD / STT / non-cloned TTS / enhancement. Build it with `-DSPEECH_CORE_WITH_LITERT=ON -DLITERT_DIR=...` first (see its `AGENTS.md` for the C ABI and CMake targets). -- **speech-swift** — speech models runtime for Apple Silicon (MLX / CoreML). The macOS voice-cloning backend: `VoxCPM2TTS` (default), `CosyVoiceTTS`, `Qwen3TTS` (ICL clone API in `Qwen3TTS+ICL.swift`), and `ChatterboxTTS` (multilingual, `Sources/ChatterboxTTS/`). +- **speech-swift** — speech models runtime for Apple Silicon (MLX / CoreML). The macOS voice-cloning backend: `CosyVoiceTTS` (default), `VoxCPM2TTS`, `Qwen3TTS` (ICL clone API in `Qwen3TTS+ICL.swift`), and `ChatterboxTTS` (multilingual, `Sources/ChatterboxTTS/`). - **speech-models** — model artifacts on Hugging Face (`aufklarer/`). Studio bundles or downloads from here on first use. ## Build @@ -114,7 +114,7 @@ pnpm tauri build # add --no-bundle to skip msi/nsis install - Per-OS Tauri bundle settings live in `tauri.{macos,windows,linux}.conf.json` (merged over `tauri.conf.json`): `externalBin` selects the sidecar, `resources` ships its runtime (`mlx.metallib` on macOS, `libLiteRt.dll`/`.so` on Windows/Linux). `tauri-build` verifies `externalBin` on **every** cargo build, so stage `src-tauri/binaries/-` first (it's gitignored) or even `cargo test --lib` fails. - **macOS sidecar (`swift-sidecar`)**: the build doesn't emit `mlx.metallib` next to the binary on its own — copy it once from the speech-swift build that does (`~/repos/speech-swift/.build/arm64-apple-macosx/debug/mlx.metallib` → `swift-sidecar/.build/arm64-apple-macosx/debug/mlx.metallib`) or you'll get `MLX error: Failed to load the default metallib`. The Rust `colocate_metallib` helper (macOS-only) handles the bundled `.app` layout. - **Windows/Linux sidecar (`core-sidecar`)**: `cfg`'d off macOS. On `init_model` it loads the bundle from `SONIQO_VOXCPM2_BUNDLE_DIR` if set, else calls `sc_voxcpm2_create_from_pretrained` to download+cache it (`SONIQO_VOXCPM2_MODEL_ID`, default `soniqo/VoxCPM2-LiteRT`; cache via `SONIQO_MODEL_CACHE_DIR`/`SPEECH_CORE_CACHE_DIR`). The CMake colocates `libLiteRt` next to the binary; libcurl is linked statically (no extra DLL). `cfgValue` from the synth ladder has no LiteRT knob and is ignored (the ladder still varies `seed`). -- **TTS model**: macOS MLX path defaults to `aufklarer/VoxCPM2-MLX-int8`. Windows/Linux use the `VoxCPM2-LiteRT` bundle — downloaded on first run (resumable; see speech-core's `SPEECH_CORE_WITH_HF_DOWNLOAD`) or supplied via `SONIQO_VOXCPM2_BUNDLE_DIR`. +- **TTS model**: macOS defaults to the CosyVoice3 MLX engine; VoxCPM2 remains selectable via `aufklarer/VoxCPM2-MLX-int8`. Windows/Linux use the `VoxCPM2-LiteRT` bundle — downloaded on first run (resumable; see speech-core's `SPEECH_CORE_WITH_HF_DOWNLOAD`) or supplied via `SONIQO_VOXCPM2_BUNDLE_DIR`. - Generated clip audio is cached under `dirs::cache_dir()/audio.soniqo.studio/clips/` (`~/Library/Caches/...` on macOS, `%LOCALAPPDATA%\...` on Windows, `$XDG_CACHE_HOME`/`~/.cache` on Linux). The Rust and sidecar sides compute this independently and must stay in sync. - **Follow-ups**: (1) ASR-graded retry is macOS-only (`GRADING_AVAILABLE` in `lib.rs`); Windows/Linux accept the first successful take — wiring Parakeet-via-sidecar grading is TODO. (2) The Windows/Linux CI lanes (`build.yml` `build-windows` / `build-linux`) build the sidecar, run `ctest`, `pnpm tauri build`, and attach installers to the Release on tag — but they check out speech-core at the moving `ref: main` (not a pinned SHA) and run no model inference, so an x86_64 LiteRT runtime regression can still slip past CI. First-run model download is handled by speech-core, so installers don't embed the bundle. diff --git a/README.md b/README.md index 320679a..b22dfaa 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ The clone is local. The synth is local. No audio leaves your machine. - **Tauri 2** shell (Rust + the OS-native WebView) so the shipped app is a small native binary, not a Chromium fork. - **React + Vite** frontend for the timeline, voice library, and script editor. - **A warm sidecar process** holds the speech engine resident so per-line synthesis is fast after the first warm-up. Tauri spawns it once and talks NDJSON over stdin/stdout. On macOS this is the **Swift sidecar** (`swift-sidecar/`, MLX); on Windows/Linux the **C++ sidecar** (`core-sidecar/`, LiteRT). -- **VoxCPM2** is the default engine on every platform — via [`speech-swift`](https://github.com/soniqo/speech-swift) (MLX) on macOS and [`speech-core`](https://github.com/soniqo/speech-core) (LiteRT) on Windows/Linux. On macOS you can switch engines from the toolbar: **CosyVoice3**, **Qwen3-TTS**, **Chatterbox** (multilingual cloning across 23 languages), **OmniVoice** (600+ language cloning), **Indic-Mio** (Hindi/Indic emotion tags), and **Fish Audio S2 Pro** (experimental clone + bracket markers). Those MLX engines are macOS-only; Windows/Linux runs VoxCPM2. +- **CosyVoice 3** is the default engine on macOS via [`speech-swift`](https://github.com/soniqo/speech-swift) (MLX). Windows/Linux default to **VoxCPM2** via [`speech-core`](https://github.com/soniqo/speech-core) (LiteRT). On macOS you can switch engines from the toolbar: **VoxCPM2**, **Qwen3-TTS**, **Chatterbox** (multilingual cloning across 23 languages), **OmniVoice** (600+ language cloning), **Indic-Mio** (Hindi/Indic emotion tags), and **Fish Audio S2 Pro** (experimental clone + bracket markers). Those MLX engines are macOS-only; Windows/Linux runs VoxCPM2. ## Engines @@ -38,8 +38,8 @@ Switch engine from the toolbar dropdown (macOS only — Windows/Linux always use | Engine | Platforms | Backend | Voice cloning | Emotion markers | Languages | |---|---|---|:---:|---|:---:| -| **VoxCPM2** · default | macOS · Windows · Linux | MLX / LiteRT | ✅ | style instructions | 30 | -| **CosyVoice 3** | macOS only | MLX | ✅ | style instructions | 9 | +| **CosyVoice 3** · macOS default | macOS only | MLX | ✅ | style instructions | 9 | +| **VoxCPM2** · Windows/Linux default | macOS · Windows · Linux | MLX / LiteRT | ✅ | style instructions | 30 | | **Qwen3-TTS** | macOS only | MLX | ✅ (ICL) | — | 10 | | **Chatterbox** | macOS only | MLX | ✅ | intensity only¹ | 23 | | **OmniVoice** | macOS only | MLX | ✅ | restricted instruct² | 600+ | @@ -95,7 +95,7 @@ Grab the latest build from the [**releases page**](https://github.com/soniqo/spe Every platform **downloads its speech model on first run** and caches it, so the installers stay small: -- **macOS** — `.dmg` (~46 MB); drag into `/Applications`. First run pulls ~2.75 GB of MLX weights into `~/Library/Caches/qwen3-speech/`. +- **macOS** — `.dmg` (~46 MB); drag into `/Applications`. First run pulls the CosyVoice 3 MLX weights into `~/Library/Caches/qwen3-speech/`; selecting VoxCPM2 later pulls its ~2.75 GB MLX weights. - **Windows** — `.msi` or the NSIS `-setup.exe`. First run pulls the ~8.8 GB VoxCPM2-LiteRT bundle into `%LOCALAPPDATA%\speech-core`. - **Linux** — `.deb` or `.AppImage`. First run pulls the same bundle into `~/.cache/speech-core`. @@ -103,7 +103,7 @@ The Windows/Linux LiteRT bundle is fp16 and needs **~10 GiB of free RAM** to loa The **macOS build is signed and notarized** (from v0.0.5 on) — it opens like any other app, no Gatekeeper hoops. The Windows installers are still unsigned: SmartScreen needs *More info → Run anyway*. -### Manual model download (macOS) +### Manual VoxCPM2 model download (macOS) If the in-app download keeps failing on a flaky or slow network (`Download stalled for …: no progress` / `Failed to download …`), fetch the model yourself and place it where the app looks. Two pieces: the model weights and a small set of tokenizer files. @@ -142,7 +142,7 @@ cd swift-sidecar && swift build # builds the Swift sidecar cd .. && pnpm tauri dev # launches the app, hot-reloads the UI ``` -Same ~2.75 GB model download on first synth (into `~/Library/Caches/qwen3-speech/` — see [Manual model download](#manual-model-download-macos) if your network keeps dropping it). +The selected MLX engine downloads on first synth (into `~/Library/Caches/qwen3-speech/`). For VoxCPM2 specifically, see [Manual VoxCPM2 model download](#manual-voxcpm2-model-download-macos) if your network keeps dropping it. ### Dev loop — Windows / Linux @@ -160,20 +160,20 @@ pnpm tauri dev Measured on an Apple Silicon Mac (M-series, unified memory). The **resident** column is the real process footprint (Activity Monitor's "Memory" — `vmmap` physical footprint), which is the figure to check against your RAM. **MLX active/peak** is MLX's own accounting (peak is over a multi-line session). Note: plain `ps rss` under-reports by ~3× on Apple Silicon — Metal unified-memory buffers don't count as RSS, so use the resident figures below. -The default **VoxCPM2** engine: +The selectable **VoxCPM2** MLX engine: -| Variant | Disk | MLX active | MLX peak | Resident (real) | Default | -|---|---|---|---|---|---| -| `aufklarer/VoxCPM2-MLX-int8` | 2.75 GB | 3.1 GB | 5.4 GB | **~4–5 GB** | ✅ | -| `aufklarer/VoxCPM2-MLX-bf16` | 4.6 GB | 9.1 GB | 11.4 GB | ~12 GB | | +| Variant | Disk | MLX active | MLX peak | Resident (real) | +|---|---|---|---|---| +| `aufklarer/VoxCPM2-MLX-int8` | 2.75 GB | 3.1 GB | 5.4 GB | **~4–5 GB** | +| `aufklarer/VoxCPM2-MLX-bf16` | 4.6 GB | 9.1 GB | 11.4 GB | ~12 GB | -The other macOS engines load separately when you select them — only one is resident at a time (switching unloads the previous): **Chatterbox** ~4 GB resident (1.3 GB on disk), **CosyVoice 3** lighter, **Qwen3-TTS** (1.7B bf16) heavier. OmniVoice is downloaded and loaded separately when selected. +The macOS engines load separately when selected — only one is resident at a time (switching unloads the previous): **Chatterbox** ~4 GB resident (1.3 GB on disk), **CosyVoice 3** lighter than VoxCPM2, **Qwen3-TTS** (1.7B bf16) heavier. OmniVoice is downloaded and loaded separately when selected. -The MLX buffer cache is capped at 1 GB (`SONIQO_MLX_CACHE_MB` to override) — without that cap, peak grows to tens of GB on long sessions as varying-shape buffers accumulate. Override the default model with `SONIQO_VOXCPM2_MODEL_ID=aufklarer/VoxCPM2-MLX-bf16` if you want the higher-fidelity weights. +The MLX buffer cache is capped at 1 GB (`SONIQO_MLX_CACHE_MB` to override) — without that cap, peak grows to tens of GB on long sessions as varying-shape buffers accumulate. Override the VoxCPM2 model with `SONIQO_VOXCPM2_MODEL_ID=aufklarer/VoxCPM2-MLX-bf16` if you want the higher-fidelity weights. ### Try the demo -Hit **Load demo** in the top bar. It bootstraps a Scene 04 storyboard with two cloned voices (Anna and Marek) and four lines of dialogue — one with each emotion marker — then synthesizes everything via VoxCPM2. +Hit **Load demo** in the top bar. It bootstraps a Scene 04 storyboard with two cloned voices (Anna and Marek) and four lines of dialogue — one with each emotion marker — then synthesizes everything through the currently selected engine (CosyVoice 3 by default on macOS). ### Packaging your own .app / .dmg diff --git a/README.zh-CN.md b/README.zh-CN.md index caea193..57fa28c 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -30,7 +30,7 @@ - **Tauri 2** 外壳（Rust + 操作系统原生 WebView），因此发布的应用是一个小巧的原生二进制，而非 Chromium 分支。 - **React + Vite** 前端，负责时间轴、音色库与脚本编辑器。 - **常驻的 sidecar 进程**让语音引擎保持加载状态，因此首次预热后逐行合成很快。Tauri 启动它一次，并通过 stdin/stdout 以 NDJSON 通信。在 macOS 上是 **Swift sidecar**（`swift-sidecar/`，MLX）；在 Windows/Linux 上是 **C++ sidecar**（`core-sidecar/`，LiteRT）。 -- **VoxCPM2** 是所有平台上的默认引擎——在 macOS 上经由 [`speech-swift`](https://github.com/soniqo/speech-swift)（MLX），在 Windows/Linux 上经由 [`speech-core`](https://github.com/soniqo/speech-core)（LiteRT）。在 macOS 上可从工具栏切换引擎：**CosyVoice3**、**Qwen3-TTS**、**Chatterbox**（支持 23 种语言的多语言克隆）、**OmniVoice**（600+ 语言克隆）、**Indic-Mio**（Hindi/Indic 情感标签）以及 **Fish Audio S2 Pro**（实验性克隆 + 方括号标记）。这些 MLX 引擎仅限 macOS；Windows/Linux 仅运行 VoxCPM2。 +- **CosyVoice 3** 是 macOS 上的默认引擎，经由 [`speech-swift`](https://github.com/soniqo/speech-swift)（MLX）运行。Windows/Linux 默认使用 **VoxCPM2**，经由 [`speech-core`](https://github.com/soniqo/speech-core)（LiteRT）运行。在 macOS 上可从工具栏切换引擎：**VoxCPM2**、**Qwen3-TTS**、**Chatterbox**（支持 23 种语言的多语言克隆）、**OmniVoice**（600+ 语言克隆）、**Indic-Mio**（Hindi/Indic 情感标签）以及 **Fish Audio S2 Pro**（实验性克隆 + 方括号标记）。这些 MLX 引擎仅限 macOS；Windows/Linux 仅运行 VoxCPM2。 ## 引擎 @@ -38,8 +38,8 @@ | 引擎 | 平台 | 后端 | 语音克隆 | 情感标记 | 语言数 | |---|---|---|:---:|---|:---:| -| **VoxCPM2** · 默认 | macOS · Windows · Linux | MLX / LiteRT | ✅ | 风格指令 | 30 | -| **CosyVoice 3** | 仅 macOS | MLX | ✅ | 风格指令 | 9 | +| **CosyVoice 3** · macOS 默认 | 仅 macOS | MLX | ✅ | 风格指令 | 9 | +| **VoxCPM2** · Windows/Linux 默认 | macOS · Windows · Linux | MLX / LiteRT | ✅ | 风格指令 | 30 | | **Qwen3-TTS** | 仅 macOS | MLX | ✅（ICL） | — | 10 | | **Chatterbox** | 仅 macOS | MLX | ✅ | 仅强度¹ | 23 | | **OmniVoice** | 仅 macOS | MLX | ✅ | 受限指令² | 600+ | @@ -95,7 +95,7 @@ MLX 引擎（CosyVoice 3、Qwen3-TTS、Chatterbox、OmniVoice、Indic-Mio、Fish 每个平台都会在**首次运行时下载语音模型**并缓存，因此安装包本身很小： -- **macOS** — `.dmg`（约 46 MB）；拖入 `/Applications`。首次运行会将约 2.75 GB 的 MLX 权重下载到 `~/Library/Caches/qwen3-speech/`。 +- **macOS** — `.dmg`（约 46 MB）；拖入 `/Applications`。首次运行会将 CosyVoice 3 的 MLX 权重下载到 `~/Library/Caches/qwen3-speech/`；之后选择 VoxCPM2 时会再下载约 2.75 GB 的 VoxCPM2 MLX 权重。 - **Windows** — `.msi` 或 NSIS `-setup.exe`。首次运行会将约 8.8 GB 的 VoxCPM2-LiteRT 模型包下载到 `%LOCALAPPDATA%\speech-core`。 - **Linux** — `.deb` 或 `.AppImage`。首次运行会将同样的模型包下载到 `~/.cache/speech-core`。 @@ -103,7 +103,7 @@ Windows/Linux 的 LiteRT 模型包为 fp16 格式，加载时约需 **10 GiB 空 **macOS 构建已签名并经过公证**（自 v0.0.5 起）——像普通应用一样直接打开，无需绕过 Gatekeeper。Windows 安装包仍未签名：SmartScreen 需要点击 *More info → Run anyway*。 -### 手动下载模型（macOS） +### 手动下载 VoxCPM2 模型（macOS） 如果应用内下载在不稳定或缓慢的网络上反复失败（`Download stalled for …: no progress` / `Failed to download …`），可以自行下载模型并放到应用读取的目录。共两部分：模型权重和一小组分词器文件。 @@ -142,7 +142,7 @@ cd swift-sidecar && swift build # 构建 Swift sidecar cd .. && pnpm tauri dev # 启动应用，热重载 UI ``` -首次合成时同样会下载约 2.75 GB 的模型（位于 `~/Library/Caches/qwen3-speech/`——如果网络不稳定，请参阅[手动下载模型](#手动下载模型macos)）。 +首次合成时会下载当前选择的 MLX 引擎模型（位于 `~/Library/Caches/qwen3-speech/`）。如果 VoxCPM2 下载不稳定，请参阅[手动下载 VoxCPM2 模型](#手动下载-voxcpm2-模型macos)。 ### 开发循环 — Windows / Linux @@ -160,20 +160,20 @@ pnpm tauri dev 在一台 Apple Silicon Mac（M 系列，统一内存）上测得。**驻留（真实）** 列是实际进程占用（活动监视器的“内存”——`vmmap` 物理足迹），这是你应与 RAM 对照的数字。**MLX 活跃/峰值** 是 MLX 自身的统计（峰值为多行会话期间）。注意：在 Apple Silicon 上 `ps rss` 会少报约 3 倍——Metal 统一内存缓冲不计入 RSS，请以下方驻留数字为准。 -默认的 **VoxCPM2** 引擎： +可选的 **VoxCPM2** MLX 引擎： -| 变体 | 磁盘 | MLX 活跃 | MLX 峰值 | 驻留（真实） | 默认 | -|---|---|---|---|---|---| -| `aufklarer/VoxCPM2-MLX-int8` | 2.75 GB | 3.1 GB | 5.4 GB | **约 4–5 GB** | ✅ | -| `aufklarer/VoxCPM2-MLX-bf16` | 4.6 GB | 9.1 GB | 11.4 GB | 约 12 GB | | +| 变体 | 磁盘 | MLX 活跃 | MLX 峰值 | 驻留（真实） | +|---|---|---|---|---| +| `aufklarer/VoxCPM2-MLX-int8` | 2.75 GB | 3.1 GB | 5.4 GB | **约 4–5 GB** | +| `aufklarer/VoxCPM2-MLX-bf16` | 4.6 GB | 9.1 GB | 11.4 GB | 约 12 GB | -其他 macOS 引擎在被选中时单独加载——同一时刻只有一个驻留（切换会卸载上一个）：**Chatterbox** 驻留约 4 GB（磁盘 1.3 GB），**CosyVoice 3** 更轻，**Qwen3-TTS**（1.7B bf16）更重。OmniVoice 会在被选中时单独下载并加载。 +macOS 引擎在被选中时单独加载——同一时刻只有一个驻留（切换会卸载上一个）：**Chatterbox** 驻留约 4 GB（磁盘 1.3 GB），**CosyVoice 3** 比 VoxCPM2 更轻，**Qwen3-TTS**（1.7B bf16）更重。OmniVoice 会在被选中时单独下载并加载。 -MLX 缓冲缓存上限为 1 GB（可用 `SONIQO_MLX_CACHE_MB` 覆盖）——若没有该上限，长会话中峰值会随着不同形状的缓冲累积增长到数十 GB。如需更高保真度的权重，用 `SONIQO_VOXCPM2_MODEL_ID=aufklarer/VoxCPM2-MLX-bf16` 覆盖默认模型。 +MLX 缓冲缓存上限为 1 GB（可用 `SONIQO_MLX_CACHE_MB` 覆盖）——若没有该上限，长会话中峰值会随着不同形状的缓冲累积增长到数十 GB。如需更高保真度的 VoxCPM2 权重，用 `SONIQO_VOXCPM2_MODEL_ID=aufklarer/VoxCPM2-MLX-bf16` 覆盖 VoxCPM2 模型。 ### 试用演示 -点击顶栏的 **Load demo（加载演示）**。它会引导出一个 Scene 04 分镜，包含两个克隆音色（Anna 与 Marek）和四行对白——每行带一个情感标记——然后通过 VoxCPM2 合成全部内容。 +点击顶栏的 **Load demo（加载演示）**。它会引导出一个 Scene 04 分镜，包含两个克隆音色（Anna 与 Marek）和四行对白——每行带一个情感标记——然后通过当前选择的引擎合成全部内容（macOS 默认是 CosyVoice 3）。 ### 打包你自己的 .app / .dmg diff --git a/src/state/projectStore.test.ts b/src/state/projectStore.test.ts index 43d4fe2..784e55b 100644 --- a/src/state/projectStore.test.ts +++ b/src/state/projectStore.test.ts @@ -28,6 +28,12 @@ beforeEach(() => { }); }); +describe("projectStore model defaults", () => { + it("starts on CosyVoice so macOS loads it by default", () => { + expect(useProjectStore.getState().model.engine).toBe("cosyvoice"); + }); +}); + describe("projectStore.addTrack / removeTrack", () => { it("adds a track to the project", () => { const t = speakerTrack(); diff --git a/src/state/projectStore.ts b/src/state/projectStore.ts index 857d50e..9a597c9 100644 --- a/src/state/projectStore.ts +++ b/src/state/projectStore.ts @@ -136,7 +136,7 @@ export const useProjectStore = create((set) => ({ selection: { kind: "none" }, transport: { playing: false, positionSec: 0, zoomPxPerSec: 100 }, model: { - engine: "voxcpm2", + engine: "cosyvoice", language: "en", engines: [], status: "idle", diff --git a/swift-sidecar/Package.swift b/swift-sidecar/Package.swift index ab93168..bb96956 100644 --- a/swift-sidecar/Package.swift +++ b/swift-sidecar/Package.swift @@ -12,10 +12,9 @@ let package = Package( name: "soniqo-tts-sidecar", dependencies: [ // Products consumed from speech-swift: - // VoxCPM2TTS — primary engine (simple cloning: refAudio + optional - // instruct for emotional markers, no transcript needed). - // CosyVoiceTTS — legacy fallback (kept for one release behind - // SONIQO_TTS_ENGINE=cosyvoice while voxcpm2 stabilises). + // CosyVoiceTTS — default macOS engine (zero-shot clone with transcript). + // VoxCPM2TTS — simple cloning: refAudio + optional instruct for + // emotional markers, no transcript needed. // Qwen3TTS — older ICL path, kept behind SONIQO_TTS_ENGINE=qwen3. // ChatterboxTTS — multilingual zero-shot cloning (per-call language). // IndicMioTTS — Hindi/Indic emotion-marker synthesis. diff --git a/swift-sidecar/Sources/soniqo-tts-sidecar/main.swift b/swift-sidecar/Sources/soniqo-tts-sidecar/main.swift index 1f40b29..b50ccdb 100644 --- a/swift-sidecar/Sources/soniqo-tts-sidecar/main.swift +++ b/swift-sidecar/Sources/soniqo-tts-sidecar/main.swift @@ -503,8 +503,8 @@ final class FishAudioHolder: @unchecked Sendable { let fishAudioHolder = FishAudioHolder() -// Engine selector. VoxCPM2 is the default. SONIQO_TTS_ENGINE remains a -// process-level compatibility default, but Studio chooses the engine per +// Engine selector. CosyVoice is the default. SONIQO_TTS_ENGINE remains a +// process-level compatibility override, but Studio chooses the engine per // request so a user can switch without restarting the app. enum TTSEngine: String { case voxcpm2, cosyvoice, qwen3, chatterbox, omnivoice @@ -515,7 +515,7 @@ enum TTSEngine: String { let defaultEngine: TTSEngine = { if let raw = ProcessInfo.processInfo.environment["SONIQO_TTS_ENGINE"]?.lowercased(), let e = TTSEngine(rawValue: raw) { return e } - return .voxcpm2 + return .cosyvoice }() func requestedEngine(for request: Request) -> TTSEngine? {