From 7f54e59b2e392548262a07f14d4f3817b2105f1b Mon Sep 17 00:00:00 2001 From: Robert Date: Fri, 3 Jul 2026 13:25:47 -0700 Subject: [PATCH 1/8] docs: design audio.cpp TTS provider integration --- ...-07-03-audio-cpp-tts-integration-design.md | 253 ++++++++++++++++++ ....cpp-TTS-provider-and-setup-integration.md | 69 +++++ 2 files changed, 322 insertions(+) create mode 100644 Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md create mode 100644 backlog/tasks/task-12124 - Design-audio.cpp-TTS-provider-and-setup-integration.md diff --git a/Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md b/Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md new file mode 100644 index 0000000000..c40616b89a --- /dev/null +++ b/Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md @@ -0,0 +1,253 @@ +# audio.cpp TTS Provider And Setup Integration Design + +## Status + +Accepted for specification after brainstorming on 2026-07-03. + +## Related Task + +- `TASK-12124` + +## Purpose + +Integrate [`0xShug0/audio.cpp`](https://github.com/0xShug0/audio.cpp) as an optional tldw_server audio backend. The first implementation should follow Approach A: add a production-shaped `audio_cpp` TTS provider and setup path first, then move toward the broader Audio Studio runtime platform after the TTS path proves reliable. + +The first slice must not bypass the existing audio API surface. `POST /api/v1/audio/speech` should continue to route through `TTSServiceV2`, the TTS adapter registry, existing auth, quotas, fallback behavior, history, generated-file storage, metrics, and endpoint error mapping. + +## Source Facts And Constraints + +The upstream `audiocpp_server` exposes: + +- `GET /health` +- `GET /v1/models` +- `POST /v1/audio/speech` +- `POST /v1/audio/transcriptions` +- `POST /v1/tasks/run` + +The upstream server speech route accepts OpenAI-style JSON and returns `audio/wav` by default. It accepts `response_format: "json"` for base64 WAV in JSON. Its examples use `voice_ref` as a server-local path and request options such as `max_tokens` and `seed`. + +The server README describes `audiocpp_server` as CUDA-only for the current HTTP adapter. The broader project README documents CPU, CUDA, Vulkan, and Metal build paths, but the first managed server target should treat CUDA as the verified server backend and make other server backends explicit future verification work. + +The server registers configured model ids on startup when `lazy_load` is true, but loaded models and task sessions remain resident after first use until the server exits. tldw must surface that memory-residency behavior in setup docs and health/status messaging. + +`audio.cpp` is Apache-2.0 licensed. tldw_server is GPLv2 per project notes. The first design treats `audio.cpp` as an optional external executable/service installed by explicit user action. Vendoring source, statically linking, or bundling prebuilt binaries requires separate legal/package review. + +## Architecture + +Add `audio_cpp` as a first-class TTS provider, not endpoint-specific glue. + +Request path: + +```text +POST /api/v1/audio/speech + -> TTSServiceV2 + -> TTSAdapterFactory / TTSAdapterRegistry + -> AudioCppTTSAdapter + -> AudioCppClient + -> audiocpp_server /v1/audio/speech +``` + +The provider has two runtime modes behind the same adapter: + +- External server mode: `providers.audio_cpp.base_url` points at an already running `audiocpp_server`. +- Managed sidecar mode: `providers.audio_cpp.managed: true` and `server_binary_path` let tldw start `audiocpp_server --config `, wait on `/health`, reuse the same HTTP client path, and stop the process on shutdown. + +Core units: + +- `AudioCppTTSAdapter`: implements the tldw TTS adapter contract, capabilities, request validation, response conversion, metadata, and provider error mapping. +- `AudioCppClient`: small HTTP client for `/health`, `/v1/models`, and `/v1/audio/speech`, using existing retry, timeout, and HTTP client patterns where practical. +- `AudioCppSidecarSupervisor`: optional process owner for managed mode, modeled after existing sidecar patterns but scoped to `audio_cpp`. +- `AudioCppServerConfig`: renderer/validator for the upstream server JSON file. +- `install_tts_audio_cpp.py`: setup/helper script with pure functions for layout and config patching plus explicit opt-in clone/build/model-manager actions. + +Do not add generic Audio Studio `/v1/tasks/run` abstractions in the first implementation. The TTS provider should leave room for those by reusing `AudioCppClient` and the sidecar supervisor. + +## Configuration + +Add `audio_cpp` to `tldw_Server_API/Config_Files/tts_providers_config.yaml`, disabled by default. + +```yaml +providers: + audio_cpp: + enabled: false + managed: false + base_url: "http://127.0.0.1:8080" + server_binary_path: null + server_config_path: "models/audio_cpp/server.json" + models_root: "models/audio_cpp" + shared_scratch_dir: "models/audio_cpp/runtime/scratch" + backend: "cuda" # setup/build hint; not necessarily emitted into server.json + device: 0 + threads: 1 + lazy_load: true + model: "pocket-tts" + family: "pocket_tts" + model_path: "models/audio_cpp/pocket-tts" + task: "tts" + mode: "offline" + load_options: + language: "english" + session_options: + language: "english" + timeout: 300 + sample_rate: 24000 + retain_request_artifacts: false + request_option_allowlist: + - max_tokens + - seed + voices: + alba: + voice_id: "alba" +``` + +For managed mode, tldw renders an upstream-compatible server config with explicit model entries. Provider settings such as `backend` are setup/build hints unless upstream server config documents a matching field. The first implementation should support one configured TTS model entry and leave multi-model config expansion for a later pass unless it is trivial inside the config renderer. + +For external server mode, `shared_scratch_dir` is required only for requests that need server-local files, such as reference audio. If the external server cannot read the configured scratch directory, `voice_reference` requests must fail with a clear provider validation error. Basic text-to-speech can work with only `base_url`. + +## Setup And Installer Scope + +The setup story should support three paths: + +1. Existing server: user provides `base_url`; tldw verifies `/health` and `/v1/models`. +2. Managed sidecar: user provides or installer creates `server_binary_path`; tldw starts the server, waits for `/health`, and shuts it down with the app. +3. Full setup/admin flow: helper clones/builds `audio.cpp`, patches `tts_providers_config.yaml`, and guides model installation. + +Model installation must be explicit. Upstream ships `tools/model_manager.py` for package listing, package info, and model installation. Some packages are gated, some need Hugging Face tokens, and some require source files or conversion. tldw should wrap or document that tool instead of reimplementing model download logic in the first pass. + +No model download should happen silently from normal server startup or a speech request. + +Installer behavior: + +- Use pure helper functions for repo-root resolution, runtime layout, config rendering, and YAML patching. +- Offer CLI flags for clone/build/config update/model package installation. +- Keep network operations explicit and user/admin initiated. +- Do not put secrets or tokens in generated config files. +- Keep platform support truthful: CUDA server build is the first managed target; CPU/Vulkan/Metal server support is future verification unless upstream server docs change. + +## TTS Request Behavior + +The adapter translates `TTSRequest` into `audiocpp_server` speech JSON. + +Supported first-pass fields: + +- `model`: configured audio.cpp model id by default, or request model when it resolves to `audio_cpp`. +- `input`: request text. +- `voice_reference`: staged WAV file path passed as `voice_ref` when the runtime can read the staged file. +- `voice`: mapped to upstream voice ids only through configured `providers.audio_cpp.voices`; generic tldw voices are not passed blindly. +- `response_format`: request `wav` or upstream JSON/base64 WAV, then use tldw's existing conversion path for `mp3`, `opus`, `flac`, `aac`, and `pcm`. +- `speed`: pass only if the configured model advertises or the config explicitly maps it to an allowed request option; otherwise ignore and record metadata. +- `extra_params`: pass only allowlisted scalar request options, initially `max_tokens` and `seed` unless configuration expands the allowlist. + +Reference-audio handling: + +- Managed sidecar mode can materialize reference bytes under `shared_scratch_dir` because tldw controls both the path and the server. +- External server mode must verify or require a shared scratch directory readable by the external server before accepting `voice_reference`. +- Reference audio should be validated and converted to a server-readable WAV shape before `voice_ref` is passed. +- Temp names must be generated by tldw, not derived from user filenames. +- Files are deleted after the request unless `retain_request_artifacts` is enabled for diagnostics. + +Streaming semantics: + +- Upstream warns that framework-wide streaming inference is not generally supported and models should be treated as offline-only. +- tldw should advertise `supports_streaming: true` only in the compatibility sense required by `/audio/speech` streaming responses, while metadata should state `incremental_streaming: false`. +- For `stream=true`, the adapter generates full audio and returns it as a single chunk through the existing streaming response path. +- Do not claim token/audio incremental streaming until a specific upstream server route and model family are verified. + +## Response Handling + +Native response: + +- Prefer `audio/wav` bytes from `/v1/audio/speech` when the requested tldw output can be served as WAV or converted locally. +- Use `response_format: "json"` only when it materially simplifies metadata or base64 decoding. + +Output conversion: + +- If tldw request format differs from upstream WAV, convert using existing `AudioConverter` behavior in `TTSServiceV2` or adapter-local conversion consistent with existing providers. +- The adapter response should include provider metadata: `provider=audio_cpp`, upstream `model`, `base_url` redacted to origin only when useful, `managed`, `incremental_streaming=false`, `voice_reference_mode`, and ignored/unsupported request options. + +Voice catalog: + +- Expose configured voices from `providers.audio_cpp.voices`. +- Do not try to infer model-owned voice ids from `/v1/models` unless upstream adds a stable field for it. + +## Error Handling + +Map failures into existing TTS exception types: + +- Unreachable server: `TTSNetworkError` or `TTSProviderUnavailableError`. +- `/health` failure or managed startup timeout: `TTSProviderInitializationError`. +- Missing model in `/v1/models`: `TTSModelNotFoundError` or provider configuration error depending on whether the model was request-selected or configured. +- Unsupported format or unsupported reference-audio mode: `TTSValidationError`. +- Upstream 4xx/5xx: `TTSProviderError` with sanitized details. +- Conversion failure: `TTSGenerationError`. + +User-facing errors must not include raw request text, secrets, full local model paths, arbitrary stderr, or generated scratch paths. Logs may include sanitized diagnostics and structured error categories. + +## Security And Privacy + +Primary risks: + +- Server-local file path exposure through `voice_ref` and future task routes. +- Arbitrary option passthrough that could reach model/session/server internals. +- Installer clone/build/download operations with network and toolchain side effects. +- Resident model memory pressure after lazy first use. +- License and distribution boundary confusion. + +Controls: + +- Keep managed config, scratch, and logs under tldw-controlled roots. +- Validate paths with resolved absolute path checks before writing or passing them to the sidecar. +- In external mode, require explicit shared-scratch configuration for file-backed requests. +- Use an allowlist for request options. +- Do not accept arbitrary server command args, environment variables, config JSON, or file paths from normal speech requests. +- Do not auto-download models during inference. +- Treat `audio.cpp` as an optional external component; no vendoring or bundled binaries in the first implementation. + +## Testing + +Default CI should not require a real `audio.cpp` binary or model. + +Required tests: + +- Unit tests for `AudioCppClient` request construction, health/model parsing, response decoding, and error mapping with mocked HTTP. +- Adapter tests for request translation, configured voice mapping, unsupported generic voice behavior, reference-audio staging/cleanup, stream-compatible single-chunk response, response metadata, and conversion handoff. +- Sidecar supervisor tests with fake process/startup probes. +- Server config renderer tests for model entries, lazy load, CUDA backend fields, and path validation. +- Installer helper tests for runtime layout and `tts_providers_config.yaml` patching. +- Endpoint-level test that `audio_cpp` can be selected through model/provider routing without bypassing TTSServiceV2, using mocks. + +Optional real-runtime checks: + +- Smoke helper for an installed `audiocpp_server` and configured model. +- Manual verification for managed CUDA server startup, `/health`, `/v1/models`, basic TTS, reference-audio TTS, and shutdown cleanup. + +Bandit should run on touched backend/helper paths before implementation completion. For this design-only task, record a non-code skip. + +## Documentation + +Update or add docs for: + +- TTS provider setup guide entry for `audio_cpp`. +- First-time CPU/GPU audio setup pages, with CUDA server caveat and external-server option. +- Admin/setup installer card or equivalent setup UI guidance, if implementation reaches UI. +- Audio API provider catalog docs showing `incremental_streaming=false`. +- Troubleshooting for server unreachable, model missing, shared scratch unreadable, model memory residency, gated model packages, and build prerequisites. + +## Approach C Follow-up Path + +After Approach A works, move toward Approach C in staged follow-ups: + +1. Reuse `AudioCppClient` and `AudioCppSidecarSupervisor` for `/v1/tasks/run`. +2. Add Audio Studio provider adapters for VAD, diarization, source separation, voice conversion, music generation, and pipelines. +3. Add STT provider support around `/v1/audio/transcriptions`. +4. Promote upstream model package discovery and model-manager integration into setup/admin UI. +5. Add broader capability envelopes so WebUI routes can show which `audio.cpp` tasks are installed, configured, and ready. + +The key boundary is that TTS must prove the runtime management, path safety, error mapping, and setup story before the generic task platform is introduced. + +## Open Questions For Implementation Planning + +- Which initial model package should the installer present first: `pocket_tts`, `qwen3_tts_0_6b_base`, or a small non-gated model if upstream supports one well enough? +- Should managed mode support multiple configured TTS models in the first implementation, or one model entry plus later expansion? +- Should the health/status endpoint expose resident-model memory warnings for `audio_cpp` specifically, or rely on provider metadata first? +- Should `audio_cpp` model aliases be added to `TTSAdapterFactory.MODEL_PROVIDER_MAP`, or should the provider be selected only through explicit provider hints and configured default provider? diff --git a/backlog/tasks/task-12124 - Design-audio.cpp-TTS-provider-and-setup-integration.md b/backlog/tasks/task-12124 - Design-audio.cpp-TTS-provider-and-setup-integration.md new file mode 100644 index 0000000000..2f81dbb7ac --- /dev/null +++ b/backlog/tasks/task-12124 - Design-audio.cpp-TTS-provider-and-setup-integration.md @@ -0,0 +1,69 @@ +--- +id: TASK-12124 +title: Design audio.cpp TTS provider and setup integration +status: Done +labels: +- audio +- tts +- design +- setup +references: +- https://github.com/0xShug0/audio.cpp +- https://raw.githubusercontent.com/0xShug0/audio.cpp/release-0.1/app/server/README.md +- https://raw.githubusercontent.com/0xShug0/audio.cpp/release-0.1/README.md +- https://raw.githubusercontent.com/0xShug0/audio.cpp/release-0.1/LICENSE +documentation: +- Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md +modified_files: +- Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md +- backlog/tasks/task-12124 - Design-audio.cpp-TTS-provider-and-setup-integration.md +--- + +## Description + + +Write the accepted design for integrating `0xShug0/audio.cpp` as a tldw_server audio backend using Approach A first: a first-class `audio_cpp` TTS provider with external-server and optional managed-sidecar modes, setup/admin installer guidance, reference-audio constraints, and a staged path toward broader Audio Studio support after TTS proves out. + + +## Acceptance Criteria + +- [x] #1 Design documents `audio_cpp` as a TTS provider that uses existing TTS service, adapter registry, fallback, quota, history, and storage behavior. +- [x] #2 Design covers external-server and managed-sidecar runtime modes, setup/admin installer flow, and explicit model installation boundaries. +- [x] #3 Design records reviewed constraints around CUDA-first server support, server-local paths, single-chunk streaming compatibility, configured voice mappings, and licensing/bundling boundaries. +- [x] #4 Design defines testing, security, error mapping, and the follow-up path from Approach A to broader Audio Studio/Approach C support. + + +## Implementation Plan + + +Write `Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md` from the approved brainstorming sections and review corrections. Keep implementation out of scope until the user reviews the written spec and approves transition to planning. + + +## Implementation Notes + + + +- Created `Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md` from the approved brainstorming sections. +- Kept the first slice scoped to Approach A: a first-class `audio_cpp` TTS provider and setup story, with Approach C deferred until TTS proves the runtime and path-safety model. +- Recorded review corrections for upstream server support, server-local reference-audio paths, single-chunk streaming compatibility, configured voice mappings, model-manager wrapping, and optional external-component licensing boundaries. +- Verification is documentation-only: no backend code paths were changed, so Bandit is not applicable for this task. + + + +## Final Summary + + + +Design spec written for integrating `0xShug0/audio.cpp` as an optional tldw_server TTS backend. The spec covers architecture, configuration, setup/installer scope, TTS request and response behavior, security, testing, documentation, and the staged follow-up path from Approach A to broader Audio Studio support. + + + +## Definition of Done + +- [x] #1 Acceptance criteria completed +- [x] #2 Tests or verification recorded +- [x] #3 Documentation updated when relevant +- [x] #4 Bandit run for touched code when applicable or document non-code/environment skip +- [x] #5 Final summary added +- [x] #6 Known skips or blockers documented + From 3f847e56b850b1f270a8dcfacf65debccb55e35c Mon Sep 17 00:00:00 2001 From: Robert Date: Fri, 3 Jul 2026 13:49:13 -0700 Subject: [PATCH 2/8] docs: tighten audio.cpp TTS integration spec --- ...-07-03-audio-cpp-tts-integration-design.md | 109 ++++++++++++++---- ....cpp-TTS-provider-and-setup-integration.md | 3 +- 2 files changed, 86 insertions(+), 26 deletions(-) diff --git a/Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md b/Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md index c40616b89a..38323816eb 100644 --- a/Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md +++ b/Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md @@ -62,6 +62,24 @@ Core units: Do not add generic Audio Studio `/v1/tasks/run` abstractions in the first implementation. The TTS provider should leave room for those by reusing `AudioCppClient` and the sidecar supervisor. +## Provider Registration And Routing + +The implementation should register `audio_cpp` explicitly in the existing TTS provider registry: + +- Add `TTSProvider.AUDIO_CPP = "audio_cpp"`. +- Add default adapter mapping for `AudioCppTTSAdapter`. +- Add provider aliases for `audio_cpp`, `audio-cpp`, and `audiocpp`. +- Add `audio_cpp` to provider priority only as disabled-by-default documentation/config; it should not affect default selection unless enabled. +- Add `audio_cpp` to format preferences after the adapter's conversion behavior is verified. + +Model routing should avoid stealing existing aliases. Today, generic `pocket-tts` routes to the existing PocketTTS provider. The first `audio_cpp` implementation should prefer explicit provider selection or namespaced model aliases such as: + +- `audio_cpp:pocket-tts` +- `audio-cpp/pocket-tts` +- `audiocpp/pocket-tts` + +Do not remap bare `pocket-tts` to `audio_cpp` unless a later migration intentionally changes the existing PocketTTS behavior and covers that compatibility break in tests and release notes. + ## Configuration Add `audio_cpp` to `tldw_Server_API/Config_Files/tts_providers_config.yaml`, disabled by default. @@ -70,40 +88,64 @@ Add `audio_cpp` to `tldw_Server_API/Config_Files/tts_providers_config.yaml`, dis providers: audio_cpp: enabled: false - managed: false - base_url: "http://127.0.0.1:8080" - server_binary_path: null - server_config_path: "models/audio_cpp/server.json" - models_root: "models/audio_cpp" - shared_scratch_dir: "models/audio_cpp/runtime/scratch" backend: "cuda" # setup/build hint; not necessarily emitted into server.json - device: 0 - threads: 1 - lazy_load: true - model: "pocket-tts" - family: "pocket_tts" + base_url: "http://127.0.0.1:8080" + model: "audio-cpp/pocket-tts" model_path: "models/audio_cpp/pocket-tts" - task: "tts" - mode: "offline" - load_options: - language: "english" - session_options: - language: "english" + binary_path: null + device: "cuda" timeout: 300 sample_rate: 24000 - retain_request_artifacts: false - request_option_allowlist: - - max_tokens - - seed - voices: - alba: - voice_id: "alba" + max_concurrent_generations: 1 + auto_download: false + extra_params: + managed: false + allow_remote_base_url: false + server: + host: "127.0.0.1" + port: 8080 + autoselect_port: true + port_probe_max: 10 + startup_timeout_seconds: 30 + healthcheck_interval_seconds: 0.25 + startup_backoff_seconds: 5 + idle_shutdown_seconds: 900 + terminate_timeout_seconds: 10 + server_config_path: "models/audio_cpp/server.json" + models_root: "models/audio_cpp" + shared_scratch_dir: "models/audio_cpp/runtime/scratch" + lazy_load: true + device: 0 + threads: 1 + model: + id: "pocket-tts" + family: "pocket_tts" + path: "models/audio_cpp/pocket-tts" + task: "tts" + mode: "offline" + load_options: + language: "english" + session_options: + language: "english" + retain_request_artifacts: false + external_voice_reference_mode: "disabled" # disabled | shared_path + request_option_allowlist: + - max_tokens + - seed + voices: + alba: + upstream_value: "alba" + request_field: null # set only after upstream server support is verified ``` +The current `ProviderConfig` schema preserves known provider fields plus `extra_params`. The first implementation should either extend that schema deliberately or keep audio.cpp-specific runtime fields under `extra_params` as shown above. Do not place new top-level provider keys in YAML unless the config schema is updated in the same change, because otherwise those fields may be dropped during Pydantic parsing or config serialization. + For managed mode, tldw renders an upstream-compatible server config with explicit model entries. Provider settings such as `backend` are setup/build hints unless upstream server config documents a matching field. The first implementation should support one configured TTS model entry and leave multi-model config expansion for a later pass unless it is trivial inside the config renderer. For external server mode, `shared_scratch_dir` is required only for requests that need server-local files, such as reference audio. If the external server cannot read the configured scratch directory, `voice_reference` requests must fail with a clear provider validation error. Basic text-to-speech can work with only `base_url`. +External reference-audio support must be opt-in. Because upstream currently documents request-time audio paths as server-local and does not document a cheap file-read probe endpoint, tldw cannot fully verify external server readability during normal initialization. The default `external_voice_reference_mode` should be `disabled`; `shared_path` should be treated as an admin assertion that the external server can read the configured path. + ## Setup And Installer Scope The setup story should support three paths: @@ -124,6 +166,15 @@ Installer behavior: - Do not put secrets or tokens in generated config files. - Keep platform support truthful: CUDA server build is the first managed target; CPU/Vulkan/Metal server support is future verification unless upstream server docs change. +Managed sidecar behavior: + +- Bind managed sidecars to loopback hosts only. +- Autoselect a free port by default and derive `base_url` from the selected host and port. +- Wait for `/health` with configurable timeout and polling interval. +- Back off after startup failure to avoid tight restart loops. +- Support idle shutdown because upstream keeps loaded models and sessions resident until process exit. +- Keep stdout/stderr handling sanitized and avoid surfacing arbitrary process output in user-facing errors. + ## TTS Request Behavior The adapter translates `TTSRequest` into `audiocpp_server` speech JSON. @@ -146,6 +197,8 @@ Reference-audio handling: - Temp names must be generated by tldw, not derived from user filenames. - Files are deleted after the request unless `retain_request_artifacts` is enabled for diagnostics. +Configured voice mappings need a verified upstream request field before they are sent. The current server README documents `voice_ref` for `/v1/audio/speech`, but it does not document a built-in voice field for that route. Until implementation verifies the server source or a real runtime for fields such as `voice`, `voice_id`, or another request key, configured voices should be exposed as catalog metadata only or fail with a clear validation error when requested without a reference audio path. + Streaming semantics: - Upstream warns that framework-wide streaming inference is not generally supported and models should be treated as offline-only. @@ -164,6 +217,7 @@ Output conversion: - If tldw request format differs from upstream WAV, convert using existing `AudioConverter` behavior in `TTSServiceV2` or adapter-local conversion consistent with existing providers. - The adapter response should include provider metadata: `provider=audio_cpp`, upstream `model`, `base_url` redacted to origin only when useful, `managed`, `incremental_streaming=false`, `voice_reference_mode`, and ignored/unsupported request options. +- Advertise only formats the adapter can actually return or convert. The first pass should not advertise `ogg`, `webm`, or `ulaw` just because the public request schema accepts them, unless conversion for those formats is verified and tested. Voice catalog: @@ -198,6 +252,7 @@ Controls: - Keep managed config, scratch, and logs under tldw-controlled roots. - Validate paths with resolved absolute path checks before writing or passing them to the sidecar. - In external mode, require explicit shared-scratch configuration for file-backed requests. +- Restrict `base_url` to loopback origins by default. Allow non-loopback origins only through an admin-controlled `allow_remote_base_url` setting. - Use an allowlist for request options. - Do not accept arbitrary server command args, environment variables, config JSON, or file paths from normal speech requests. - Do not auto-download models during inference. @@ -215,6 +270,10 @@ Required tests: - Server config renderer tests for model entries, lazy load, CUDA backend fields, and path validation. - Installer helper tests for runtime layout and `tts_providers_config.yaml` patching. - Endpoint-level test that `audio_cpp` can be selected through model/provider routing without bypassing TTSServiceV2, using mocks. +- Registry tests for `TTSProvider.AUDIO_CPP`, provider aliases, default adapter mapping, and namespaced model aliases that do not change existing `pocket-tts` routing. +- Config tests proving audio.cpp-specific fields survive load/serialize either through explicit `ProviderConfig` fields or through `extra_params`. +- Security tests for loopback-only default `base_url`, remote opt-in, path containment, and disabled external reference-audio mode. +- Format tests proving unsupported public schema formats are rejected unless conversion is implemented. Optional real-runtime checks: @@ -250,4 +309,4 @@ The key boundary is that TTS must prove the runtime management, path safety, err - Which initial model package should the installer present first: `pocket_tts`, `qwen3_tts_0_6b_base`, or a small non-gated model if upstream supports one well enough? - Should managed mode support multiple configured TTS models in the first implementation, or one model entry plus later expansion? - Should the health/status endpoint expose resident-model memory warnings for `audio_cpp` specifically, or rely on provider metadata first? -- Should `audio_cpp` model aliases be added to `TTSAdapterFactory.MODEL_PROVIDER_MAP`, or should the provider be selected only through explicit provider hints and configured default provider? +- Which upstream server request field, if any, should configured built-in voices use for `/v1/audio/speech`? diff --git a/backlog/tasks/task-12124 - Design-audio.cpp-TTS-provider-and-setup-integration.md b/backlog/tasks/task-12124 - Design-audio.cpp-TTS-provider-and-setup-integration.md index 2f81dbb7ac..305576fc5f 100644 --- a/backlog/tasks/task-12124 - Design-audio.cpp-TTS-provider-and-setup-integration.md +++ b/backlog/tasks/task-12124 - Design-audio.cpp-TTS-provider-and-setup-integration.md @@ -46,6 +46,7 @@ Write `Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md` fr - Created `Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md` from the approved brainstorming sections. - Kept the first slice scoped to Approach A: a first-class `audio_cpp` TTS provider and setup story, with Approach C deferred until TTS proves the runtime and path-safety model. - Recorded review corrections for upstream server support, server-local reference-audio paths, single-chunk streaming compatibility, configured voice mappings, model-manager wrapping, and optional external-component licensing boundaries. +- Amended the spec after review to tighten current-codebase integration risks: config-schema preservation, provider enum/routing, namespaced model aliases, loopback-only default `base_url`, external reference-audio opt-in, sidecar lifecycle controls, verified voice request fields, and format-advertising tests. - Verification is documentation-only: no backend code paths were changed, so Bandit is not applicable for this task. @@ -54,7 +55,7 @@ Write `Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md` fr -Design spec written for integrating `0xShug0/audio.cpp` as an optional tldw_server TTS backend. The spec covers architecture, configuration, setup/installer scope, TTS request and response behavior, security, testing, documentation, and the staged follow-up path from Approach A to broader Audio Studio support. +Design spec written and amended for integrating `0xShug0/audio.cpp` as an optional tldw_server TTS backend. The spec covers architecture, provider registration, configuration, setup/installer scope, TTS request and response behavior, security, testing, documentation, and the staged follow-up path from Approach A to broader Audio Studio support. From 30cb4f1c0cc496e86d8ea98e43342f014f0c9be0 Mon Sep 17 00:00:00 2001 From: Robert Date: Fri, 3 Jul 2026 14:28:49 -0700 Subject: [PATCH 3/8] docs: plan audio.cpp TTS provider implementation --- ...io-cpp-tts-provider-implementation-plan.md | 574 ++++++++++++++++++ ....cpp-TTS-provider-and-setup-integration.md | 68 +++ 2 files changed, 642 insertions(+) create mode 100644 Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md create mode 100644 backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md diff --git a/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md new file mode 100644 index 0000000000..181be4234e --- /dev/null +++ b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md @@ -0,0 +1,574 @@ +# audio.cpp TTS Provider Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add `audio_cpp` as a disabled-by-default TTS provider backed by an external or managed `audiocpp_server`, with tested registry routing, configuration, HTTP client behavior, adapter behavior, sidecar lifecycle, installer scaffolding, documentation, and safe default behavior. + +**Architecture:** Keep the existing `/api/v1/audio/speech` path intact. Requests flow through `TTSServiceV2`, the TTS adapter registry, fallback, history, storage, quota, and metrics before reaching a new `AudioCppTTSAdapter`. The adapter uses a small `AudioCppClient` for `/health`, `/v1/models`, and `/v1/audio/speech`. Optional managed mode starts a loopback `audiocpp_server` sidecar, renders upstream server config from provider `extra_params`, waits for health, and reuses the same client path. The first slice does not expose generic `/v1/tasks/run` orchestration. + +**Tech Stack:** Python 3, FastAPI TTS stack, Pydantic config models, Loguru, httpx, asyncio subprocess management, existing audio conversion utilities, pytest, Ruff, Bandit + +--- + +## Current Workspace Note + +The current checkout is the primary workspace, not a linked worktree, and it contains unrelated untracked files: + +- `tldw_Server_API/Config_Files/templates/watchlists/cti_osint_report_markdown.md` +- `tldw_Server_API/Config_Files/templates/watchlists/news_briefing_markdown.md` +- `tldw_Server_API/Databases/system_logs.jsonl` + +Before source edits begin, choose one: + +- Preferred: create or switch to an isolated worktree/branch for this implementation. +- Acceptable: continue in place while staging only files listed in this plan. + +Record that decision in `TASK-12125` before editing source files. + +## File Map + +- Create: `tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py` + Responsibility: HTTP client for health, model listing, speech requests, response decoding, timeout handling, sanitized upstream errors, and injectable test transport. +- Create: `tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py` + Responsibility: parse provider `extra_params`, validate loopback/remote policy, validate scratch/model paths, render upstream server JSON, and normalize allowlisted request options. +- Create: `tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py` + Responsibility: manage optional loopback `audiocpp_server` startup, autoselect port, health wait, backoff, idle shutdown, and sanitized process output. +- Create: `tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py` + Responsibility: implement the TTS adapter contract, capabilities, request translation, reference-audio staging, format metadata, one-shot streaming compatibility, and provider error mapping. +- Create: `Helper_Scripts/install_tts_audio_cpp.py` + Responsibility: provide explicit admin helper functions for runtime layout, config patching, optional clone/build commands, and optional model-manager command construction. +- Modify: `tldw_Server_API/app/core/TTS/adapter_registry.py` + Responsibility: add `TTSProvider.AUDIO_CPP`, aliases, namespaced model aliases, default adapter mapping, and provider metadata without changing bare `pocket-tts` routing. +- Modify: `tldw_Server_API/Config_Files/tts_providers_config.yaml` + Responsibility: add disabled `audio_cpp` provider config and format preferences using only schema-preserved fields plus `extra_params`. +- Modify: `Docs/STT-TTS/TTS-SETUP-GUIDE.md` + Responsibility: document external server setup, managed sidecar setup, model-manager use, CUDA-first managed support, memory residency, and security boundaries. +- Create: `tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py` + Responsibility: prove provider aliases, namespaced model routing, default adapter registration, and non-regression for bare `pocket-tts`. +- Create: `tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py` + Responsibility: prove YAML loading preserves `audio_cpp` settings under `extra_params`, disabled defaults, and format preferences. +- Create: `tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py` + Responsibility: prove base URL policy, remote opt-in, scratch/model path containment, option allowlist filtering, and server JSON rendering. +- Create: `tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py` + Responsibility: prove health/model/speech HTTP behavior with mocked transports and sanitized failures. +- Create: `tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py` + Responsibility: prove capabilities, text-only synthesis, reference-audio modes, ignored options metadata, conversion handoff, and one-shot streaming compatibility. +- Create: `tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py` + Responsibility: prove loopback-only command construction, port selection, health polling, startup failure backoff, and shutdown behavior with fakes. +- Create: `tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py` + Responsibility: prove installer pure functions patch config and construct explicit commands without network or compiler dependencies. +- Create: `tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py` + Responsibility: prove `TTSServiceV2` can select `audio_cpp` by provider/model hint and produce a speech response through a mocked adapter/client path. + +## Stage 1: Registry And Config Scaffold + +**Goal:** Register `audio_cpp` as a disabled provider with exact routing semantics and schema-safe configuration. + +**Success Criteria:** `audio_cpp`, `audio-cpp`, and `audiocpp` resolve to `TTSProvider.AUDIO_CPP`; `audio_cpp:pocket-tts`, `audio-cpp/pocket-tts`, and `audiocpp/pocket-tts` route to `audio_cpp`; bare `pocket-tts` still routes to `pocket_tts`; config loads with all audio.cpp runtime settings preserved under `extra_params`; provider remains disabled. + +**Tests:** `test_audio_cpp_registry.py`, `test_audio_cpp_tts_config.py` + +**Status:** Not Started + +- [ ] **Step 1: Write failing registry tests** + +Add tests like: + +```python +def test_audio_cpp_provider_aliases_resolve(): + assert TTSAdapterRegistry.resolve_provider("audio_cpp") == TTSProvider.AUDIO_CPP + assert TTSAdapterRegistry.resolve_provider("audio-cpp") == TTSProvider.AUDIO_CPP + assert TTSAdapterRegistry.resolve_provider("audiocpp") == TTSProvider.AUDIO_CPP + + +def test_audio_cpp_model_aliases_do_not_steal_pocket_tts(): + assert MODEL_PROVIDER_MAP["audio_cpp:pocket-tts"] == TTSProvider.AUDIO_CPP + assert MODEL_PROVIDER_MAP["audio-cpp/pocket-tts"] == TTSProvider.AUDIO_CPP + assert MODEL_PROVIDER_MAP["audiocpp/pocket-tts"] == TTSProvider.AUDIO_CPP + assert MODEL_PROVIDER_MAP["pocket-tts"] == TTSProvider.POCKET_TTS +``` + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py +``` + +Expected result before implementation: failure because `TTSProvider.AUDIO_CPP` is not present. + +- [ ] **Step 2: Write failing config tests** + +Assert that `tts_providers_config.yaml` loads with: + +- `providers.audio_cpp.enabled` as `false` +- `providers.audio_cpp.extra_params.managed` as `false` +- `providers.audio_cpp.extra_params.allow_remote_base_url` as `false` +- `providers.audio_cpp.extra_params.external_voice_reference_mode` as `disabled` +- `providers.audio_cpp.extra_params.request_option_allowlist` containing `max_tokens` and `seed` +- no unsupported `ogg`, `webm`, or `ulaw` advertised for `audio_cpp` + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py +``` + +Expected result before implementation: failure because `audio_cpp` config is absent. + +- [ ] **Step 3: Implement registry and YAML scaffold** + +Modify: + +- `tldw_Server_API/app/core/TTS/adapter_registry.py` +- `tldw_Server_API/Config_Files/tts_providers_config.yaml` + +Implementation requirements: + +- Add `TTSProvider.AUDIO_CPP = "audio_cpp"`. +- Add aliases for `audio_cpp`, `audio-cpp`, and `audiocpp`. +- Add default adapter mapping to `tldw_Server_API.app.core.TTS.adapters.audio_cpp_adapter.AudioCppTTSAdapter`. +- Add namespaced model aliases only. +- Keep `pocket-tts` mapped to `TTSProvider.POCKET_TTS`. +- Add a disabled YAML provider block that keeps audio.cpp-specific fields inside `extra_params`. +- Add format preferences only for formats the first pass can return or convert: `wav`, `mp3`, `opus`, `flac`, `aac`, and `pcm`. + +- [ ] **Step 4: Re-run focused tests** + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py +``` + +Expected result after implementation: both test files pass. + +## Stage 2: HTTP Client And Config Validation + +**Goal:** Add reusable, testable support code for speaking to `audiocpp_server` and rendering managed server config. + +**Success Criteria:** The client can call `/health`, `/v1/models`, and `/v1/audio/speech`; JSON/base64 and WAV responses decode deterministically; upstream failures map to sanitized TTS exceptions; base URLs are loopback-only by default; remote URLs require explicit admin opt-in; file paths stay under configured roots; option passthrough is allowlisted. + +**Tests:** `test_audio_cpp_client.py`, `test_audio_cpp_config.py` + +**Status:** Not Started + +- [ ] **Step 1: Write failing client tests** + +Use `httpx.MockTransport` or the project equivalent to cover: + +- `health()` returns a healthy result for HTTP 200. +- `list_models()` returns model ids from `/v1/models`. +- `speech()` returns WAV bytes when upstream returns `audio/wav`. +- `speech()` decodes base64 audio when upstream returns JSON. +- Upstream 4xx and 5xx do not expose raw request text, full local paths, or response bodies longer than the sanitized limit. + +Example shape: + +```python +transport = httpx.MockTransport(handler) +async with httpx.AsyncClient(transport=transport, base_url="http://127.0.0.1:8080") as http_client: + client = AudioCppClient(base_url="http://127.0.0.1:8080", http_client=http_client) + response = await client.speech({"model": "pocket-tts", "input": "hello"}) +assert response.audio_bytes.startswith(b"RIFF") +``` + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py +``` + +Expected result before implementation: import failure for `audio_cpp_client`. + +- [ ] **Step 2: Write failing config validation tests** + +Cover: + +- `http://127.0.0.1:` and `http://localhost:` are accepted by default. +- `http://example.com:8080` is rejected unless `allow_remote_base_url` is true. +- managed sidecar hosts other than `127.0.0.1` or `localhost` are rejected. +- `shared_scratch_dir` and `model.path` resolve under configured roots. +- temp reference names are generated by tldw and are not derived from user filenames. +- non-scalar `extra_params` values are not passed upstream. +- only `max_tokens` and `seed` pass through with the default allowlist. +- server JSON includes a single configured TTS model entry with `id`, `family`, `path`, `task`, `mode`, `load_options`, and `session_options`. + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py +``` + +Expected result before implementation: import failure for `audio_cpp_config`. + +- [ ] **Step 3: Implement client and config modules** + +Create: + +- `tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py` +- `tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py` + +Implementation requirements: + +- Prefer injected `httpx.AsyncClient` for tests. +- Use the existing TTS exception hierarchy for provider, network, model, validation, and generation errors. +- Redact request text, full paths, secrets, and arbitrary upstream body details from raised messages. +- Keep config parsing independent from the adapter so sidecar and installer code can reuse it. +- Treat provider `backend` as a setup hint unless upstream config documents a matching server field. + +- [ ] **Step 4: Re-run focused tests** + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py +``` + +Expected result after implementation: both test files pass. + +## Stage 3: TTS Adapter Behavior + +**Goal:** Implement `AudioCppTTSAdapter` so public speech requests work through the existing TTS service contract. + +**Success Criteria:** The adapter reports accurate capabilities, validates unsupported request shapes early, translates `TTSRequest` into upstream speech JSON, handles reference audio according to managed/external mode, returns full audio bytes so `TTSServiceV2` can perform conversion, and supports `stream=true` as one full chunk with `incremental_streaming=false` metadata. + +**Tests:** `test_audio_cpp_adapter.py`, `test_audio_cpp_tts_service.py` + +**Status:** Not Started + +- [ ] **Step 1: Write failing adapter tests** + +Cover: + +- capabilities include `supports_streaming=True` for API compatibility and metadata says incremental streaming is false. +- supported formats exclude `ogg`, `webm`, and `ulaw`. +- text-only request posts `model`, `input`, and allowed options. +- `request.stream=True` still returns `audio_data` so service-level conversion can run before the endpoint streams one chunk. +- `voice_reference` is rejected in external mode when `external_voice_reference_mode` is `disabled`. +- `voice_reference` in managed mode is staged as WAV under `shared_scratch_dir` and passed as `voice_ref`. +- configured voice mappings with `request_field: null` are catalog metadata only and fail clearly when requested without reference audio. +- ignored fields such as unsupported `speed` are recorded in metadata instead of passed blindly. + +Example assertion: + +```python +response = await adapter.generate(TTSRequest(text="hello", model="audio_cpp:pocket-tts", stream=True)) +assert response.audio_data == wav_bytes +assert response.audio_stream is None +assert response.metadata["incremental_streaming"] is False +``` + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py +``` + +Expected result before implementation: import failure for `audio_cpp_adapter`. + +- [ ] **Step 2: Write failing service routing test** + +Use the TTS factory/service test pattern to prove: + +- explicit provider `audio_cpp` selects `AudioCppTTSAdapter`. +- namespaced model `audio-cpp/pocket-tts` selects `audio_cpp`. +- disabled provider does not become the default provider without explicit selection. +- public service call receives audio bytes and metadata from the mocked adapter path. + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py +``` + +Expected result before implementation: routing or adapter import failure. + +- [ ] **Step 3: Implement adapter** + +Create `tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py`. + +Implementation requirements: + +- Use `AudioCppConfig` and `AudioCppClient`. +- For `stream=True`, generate full upstream audio and return `audio_data`, not `audio_stream`, in the first pass so existing conversion code can run. +- Include metadata keys: `provider`, `model`, `managed`, `incremental_streaming`, `voice_reference_mode`, `ignored_options`, and `upstream_response_format`. +- Pass only allowlisted scalar options. +- Pass `voice_ref` only when a managed sidecar can read the scratch path or external `shared_path` mode is explicitly configured. +- Do not pass generic `voice`, `voice_id`, or unverified voice fields. +- Clean request scratch files after generation unless `retain_request_artifacts` is true. + +- [ ] **Step 4: Re-run adapter and service tests** + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py ` + tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py +``` + +Expected result after implementation: both test files pass. + +## Stage 4: Managed Sidecar, Installer, And Docs + +**Goal:** Add optional managed sidecar support plus explicit admin setup helpers and user-facing setup guidance. + +**Success Criteria:** Managed mode starts only loopback sidecars, selects or validates a port, renders server config, waits for `/health`, backs off after startup failure, supports idle shutdown, and never exposes arbitrary command args or stderr in user-facing errors. The installer helper can patch config and print explicit clone/build/model commands without doing network work in tests. Docs describe external server and managed sidecar setup truthfully. + +**Tests:** `test_audio_cpp_sidecar_supervisor.py`, `test_audio_cpp_installer.py` + +**Status:** Not Started + +- [ ] **Step 1: Write failing sidecar tests** + +Cover: + +- non-loopback managed host raises `TTSValidationError`. +- autoselect port chooses an unused loopback port and updates the derived base URL. +- command construction uses only configured binary path and generated server config path. +- startup health polling succeeds with a fake process plus fake client. +- startup timeout terminates the fake process and stores a backoff deadline. +- idle shutdown stops an idle process after the configured interval. +- sanitized errors omit raw stderr and full config paths. + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py +``` + +Expected result before implementation: import failure for `audio_cpp_sidecar_supervisor`. + +- [ ] **Step 2: Write failing installer tests** + +Cover pure functions only: + +- runtime layout resolves under repo-local `models/audio_cpp`. +- generated YAML patch sets `providers.audio_cpp.enabled` only when `--enable-provider` is supplied. +- `base_url`, `model_path`, `binary_path`, and `extra_params.server` are written without secrets. +- clone, build, and model-manager commands are constructed explicitly but not executed in unit tests. +- generated provider config keeps runtime-specific settings under `extra_params`. + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py +``` + +Expected result before implementation: missing installer module or helper failures. + +- [ ] **Step 3: Implement sidecar supervisor and installer helper** + +Create: + +- `tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py` +- `Helper_Scripts/install_tts_audio_cpp.py` + +Modify adapter initialization so managed mode can call `AudioCppSidecarSupervisor.ensure_started()` before speech generation. + +Implementation requirements: + +- No arbitrary extra command args from normal speech requests. +- No inherited secret environment values unless existing project process helpers already sanitize them. +- Generated server config lives under configured audio.cpp runtime root. +- Sidecar stdout/stderr may be logged at debug with truncation, but user-facing exceptions stay sanitized. +- Installer network and build operations require explicit CLI flags. + +- [ ] **Step 4: Update setup documentation** + +Modify `Docs/STT-TTS/TTS-SETUP-GUIDE.md` with: + +- external `audiocpp_server` mode. +- managed sidecar mode. +- CUDA-first managed support statement. +- model-manager package install guidance. +- no silent model download during normal startup or inference. +- memory residency note for lazy-loaded models and sessions. +- loopback, remote-base-url opt-in, and reference-audio shared-path warnings. +- license boundary: optional external component, no vendored binaries in this implementation. + +- [ ] **Step 5: Re-run focused tests** + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py +``` + +Expected result after implementation: both test files pass. + +## Stage 5: Final Verification And Task Closeout + +**Goal:** Verify the complete implementation slice, run security checks, update tracking, and prepare the next Approach C decision. + +**Success Criteria:** All focused audio.cpp tests pass; adjacent TTS routing/config tests pass; Ruff passes for touched Python files; Bandit reports no new findings in touched implementation files; Backlog task records verification and remaining limitations; commits are scoped to this task. + +**Tests:** All files created in this plan plus adjacent registry, config, and service tests. + +**Status:** Not Started + +- [ ] **Step 1: Run focused test suite** + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py ` + tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py +``` + +Expected result: all focused tests pass. + +- [ ] **Step 2: Run adjacent regression tests** + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m pytest -q ` + tldw_Server_API/tests/TTS_NEW/unit/test_fish_s2_registry.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_pocket_tts_cpp_adapter.py ` + tldw_Server_API/tests/TTS_NEW/unit/test_pocket_tts_cpp_installer.py +``` + +Expected result: adjacent provider routing and installer tests pass. If any named file is absent in this checkout, record the absent path in `TASK-12125` and run the closest existing adjacent test discovered with `rg --files`. + +- [ ] **Step 3: Run Ruff on touched Python files** + +Run after source files exist: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m ruff check ` + tldw_Server_API/app/core/TTS/adapter_registry.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py ` + Helper_Scripts/install_tts_audio_cpp.py ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py ` + tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py +``` + +Expected result: Ruff exits 0. + +- [ ] **Step 4: Run Bandit on touched implementation files** + +Run: + +```powershell +. .\.venv\Scripts\Activate.ps1 +python -m bandit -r ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py ` + Helper_Scripts/install_tts_audio_cpp.py ` + -f json -o "$env:TEMP\bandit_audio_cpp_tts.json" +``` + +Expected result: no new high or medium findings in touched code. Fix new findings before closeout. + +- [ ] **Step 5: Update task tracking and self-review** + +Update `TASK-12125` with: + +- implementation decision for worktree versus in-place edits. +- touched files. +- test outputs. +- Ruff result. +- Bandit result path and summary. +- known limitations, including CUDA-first managed server support and no generic `/v1/tasks/run` API in this slice. +- final summary. + +Run: + +```powershell +git diff --check -- ` + tldw_Server_API/app/core/TTS/adapter_registry.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py ` + tldw_Server_API/Config_Files/tts_providers_config.yaml ` + Helper_Scripts/install_tts_audio_cpp.py ` + Docs/STT-TTS/TTS-SETUP-GUIDE.md ` + backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md ` + docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md +``` + +Expected result: no whitespace errors. + +- [ ] **Step 6: Commit scoped slices** + +Prefer small commits by stage. Stage explicit paths only, and do not include unrelated untracked files. + +Example final commit shape: + +```powershell +git add ` + tldw_Server_API/app/core/TTS/adapter_registry.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py ` + tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py ` + tldw_Server_API/Config_Files/tts_providers_config.yaml ` + Helper_Scripts/install_tts_audio_cpp.py ` + Docs/STT-TTS/TTS-SETUP-GUIDE.md ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py ` + tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py ` + tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py ` + tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py ` + backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md +git commit -m "feat(tts): add audio_cpp provider integration" +``` + +Expected result: commit succeeds with only task-scoped files. + +## Non-Goals For This Slice + +- Do not expose generic Audio Studio `/v1/tasks/run` orchestration. +- Do not auto-download models during server startup or inference. +- Do not vendor `audio.cpp` source or binaries. +- Do not remap bare `pocket-tts` to `audio_cpp`. +- Do not advertise `ogg`, `webm`, or `ulaw` until conversion is verified and tested. +- Do not pass arbitrary server command args, environment variables, or unverified voice fields from normal speech requests. + +## Handoff + +When executing this plan, use the test-first order in each stage. After each stage, update the stage status, record command output in `TASK-12125`, and commit only scoped files. If a test path named above differs in the current checkout, discover the nearest existing test with `rg --files tldw_Server_API/tests | rg "tts|TTS"` and record the substitution in the task notes. diff --git a/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md new file mode 100644 index 0000000000..161ce53c04 --- /dev/null +++ b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md @@ -0,0 +1,68 @@ +--- +id: TASK-12125 +title: Implement audio.cpp TTS provider and setup integration +status: In Progress +labels: +- audio +- tts +- implementation +- setup +references: +- Docs/superpowers/specs/2026-07-03-audio-cpp-tts-integration-design.md +- https://github.com/0xShug0/audio.cpp +- https://raw.githubusercontent.com/0xShug0/audio.cpp/release-0.1/app/server/README.md +documentation: +- docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md +modified_files: +- backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md +- docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md +--- + +## Description + + +Implement the accepted Approach A design for `0xShug0/audio.cpp`: a disabled-by-default `audio_cpp` TTS provider that routes through the existing tldw_server TTS service, can call an external `audiocpp_server`, can optionally manage a loopback sidecar, and includes setup/admin documentation plus tests for registry, configuration, request safety, adapter behavior, and verification. + + +## Acceptance Criteria + +- [ ] #1 `audio_cpp` is registered as a first-class TTS provider with explicit aliases, namespaced model aliases, disabled-by-default config, and no regression to existing `pocket_tts` routing. +- [ ] #2 The adapter/client can synthesize through `audiocpp_server` using the existing `/api/v1/audio/speech` flow, with tested request translation, WAV response handling, one-shot streaming compatibility, format conversion handoff, and sanitized error mapping. +- [ ] #3 Reference-audio and option passthrough behavior is safe by default: loopback-only base URLs unless explicitly allowed, external reference audio disabled unless configured, server-local scratch paths constrained, and only allowlisted scalar options sent upstream. +- [ ] #4 Managed sidecar support can render upstream server config, choose a loopback port, wait for health, avoid tight restart loops, and shut down cleanly without exposing arbitrary command args or process output. +- [ ] #5 Installer/setup helpers and documentation cover explicit clone/build/config/model steps without silent network downloads during normal server startup or inference. +- [ ] #6 Focused pytest, Ruff, and Bandit verification are recorded for the touched implementation scope. + + +## Implementation Plan + + +Follow `docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md`. Use test-driven slices and update this task with touched files, verification output, and any scoped deviations before finalization. + + +## Implementation Notes + + + +- Created the implementation task and linked plan after the accepted design task `TASK-12124`. +- Current workspace note: active branch is `codex/parakeet-onnx-wav-fallback`, with unrelated untracked runtime/template files present. Source edits should proceed in an isolated worktree or after an explicit in-place decision. + + + +## Final Summary + + + +Pending implementation. + + + +## Definition of Done + +- [ ] #1 Acceptance criteria completed +- [ ] #2 Tests or verification recorded +- [ ] #3 Documentation updated when relevant +- [ ] #4 Bandit run for touched code when applicable or documented as non-code/environment skip +- [ ] #5 Final summary added +- [ ] #6 Known skips or blockers documented + From 499c894023b934110e1b6bb8351b0ed81c141e02 Mon Sep 17 00:00:00 2001 From: Robert Date: Fri, 3 Jul 2026 14:50:23 -0700 Subject: [PATCH 4/8] feat(tts): register audio_cpp provider scaffold --- ...io-cpp-tts-provider-implementation-plan.md | 10 ++-- ....cpp-TTS-provider-and-setup-integration.md | 16 +++++- .../Config_Files/tts_providers_config.yaml | 54 ++++++++++++++++++ .../app/core/TTS/adapter_registry.py | 12 +++- .../tests/TTS_NEW/fixtures/empty_config.txt | 1 + .../TTS_NEW/unit/test_audio_cpp_registry.py | 32 +++++++++++ .../TTS_NEW/unit/test_audio_cpp_tts_config.py | 56 +++++++++++++++++++ 7 files changed, 173 insertions(+), 8 deletions(-) create mode 100644 tldw_Server_API/tests/TTS_NEW/fixtures/empty_config.txt create mode 100644 tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py create mode 100644 tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py diff --git a/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md index 181be4234e..5a24e9fbe7 100644 --- a/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md +++ b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md @@ -68,9 +68,9 @@ Record that decision in `TASK-12125` before editing source files. **Tests:** `test_audio_cpp_registry.py`, `test_audio_cpp_tts_config.py` -**Status:** Not Started +**Status:** Complete -- [ ] **Step 1: Write failing registry tests** +- [x] **Step 1: Write failing registry tests** Add tests like: @@ -97,7 +97,7 @@ python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.p Expected result before implementation: failure because `TTSProvider.AUDIO_CPP` is not present. -- [ ] **Step 2: Write failing config tests** +- [x] **Step 2: Write failing config tests** Assert that `tts_providers_config.yaml` loads with: @@ -117,7 +117,7 @@ python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config Expected result before implementation: failure because `audio_cpp` config is absent. -- [ ] **Step 3: Implement registry and YAML scaffold** +- [x] **Step 3: Implement registry and YAML scaffold** Modify: @@ -134,7 +134,7 @@ Implementation requirements: - Add a disabled YAML provider block that keeps audio.cpp-specific fields inside `extra_params`. - Add format preferences only for formats the first pass can return or convert: `wav`, `mp3`, `opus`, `flac`, `aac`, and `pcm`. -- [ ] **Step 4: Re-run focused tests** +- [x] **Step 4: Re-run focused tests** Run: diff --git a/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md index 161ce53c04..505bb13460 100644 --- a/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md +++ b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md @@ -14,8 +14,13 @@ references: documentation: - docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md modified_files: +- Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md - backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md -- docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md +- tldw_Server_API/Config_Files/tts_providers_config.yaml +- tldw_Server_API/app/core/TTS/adapter_registry.py +- tldw_Server_API/tests/TTS_NEW/fixtures/empty_config.txt +- tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py +- tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py --- ## Description @@ -26,7 +31,7 @@ Implement the accepted Approach A design for `0xShug0/audio.cpp`: a disabled-by- ## Acceptance Criteria -- [ ] #1 `audio_cpp` is registered as a first-class TTS provider with explicit aliases, namespaced model aliases, disabled-by-default config, and no regression to existing `pocket_tts` routing. +- [x] #1 `audio_cpp` is registered as a first-class TTS provider with explicit aliases, namespaced model aliases, disabled-by-default config, and no regression to existing `pocket_tts` routing. - [ ] #2 The adapter/client can synthesize through `audiocpp_server` using the existing `/api/v1/audio/speech` flow, with tested request translation, WAV response handling, one-shot streaming compatibility, format conversion handoff, and sanitized error mapping. - [ ] #3 Reference-audio and option passthrough behavior is safe by default: loopback-only base URLs unless explicitly allowed, external reference audio disabled unless configured, server-local scratch paths constrained, and only allowlisted scalar options sent upstream. - [ ] #4 Managed sidecar support can render upstream server config, choose a loopback port, wait for health, avoid tight restart loops, and shut down cleanly without exposing arbitrary command args or process output. @@ -46,6 +51,13 @@ Follow `docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation- - Created the implementation task and linked plan after the accepted design task `TASK-12124`. - Current workspace note: active branch is `codex/parakeet-onnx-wav-fallback`, with unrelated untracked runtime/template files present. Source edits should proceed in an isolated worktree or after an explicit in-place decision. +- Implementation decision: using isolated git worktree `C:\Users\GDesktop-1\Working\tldw\.worktrees\audio-cpp-tts-provider` on branch `codex/audio-cpp-tts-provider`. The parent checkout remains on `dev` with unrelated untracked files left untouched. +- Baseline attempt with `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_fish_s2_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_pocket_tts_cpp_registry.py` stopped during collection because the shared venv was missing declared dependency `pytest-asyncio>=1.3.0`. +- Installed declared dependency `pytest-asyncio==1.4.0` into the shared project venv from cache, then reran the same baseline: 5 passed, 6 warnings in 83.26s. +- Stage 1 red test: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py` failed as expected with 5 failures for missing `TTSProvider.AUDIO_CPP`, missing `providers.audio_cpp`, and missing `format_preferences.audio_cpp`. +- Stage 1 implementation added `TTSProvider.AUDIO_CPP`, provider/model aliases, lazy default adapter mapping, disabled `audio_cpp` YAML config, and format preferences limited to `wav`, `mp3`, `opus`, `flac`, `aac`, and `pcm`. +- Stage 1 green test: the same audio.cpp test command passed with 5 passed, 5 warnings in 35.09s. +- Stage 1 adjacent regression: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_fish_s2_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_pocket_tts_cpp_registry.py` passed with 5 passed, 6 warnings in 40.86s. diff --git a/tldw_Server_API/Config_Files/tts_providers_config.yaml b/tldw_Server_API/Config_Files/tts_providers_config.yaml index 479b6edbca..c9c044fdbe 100644 --- a/tldw_Server_API/Config_Files/tts_providers_config.yaml +++ b/tldw_Server_API/Config_Files/tts_providers_config.yaml @@ -5,6 +5,7 @@ provider_priority: - kitten_tts # Local ONNX voices with first-use model download - pocket_tts_cpp # Local PocketTTS.cpp voice cloning + - audio_cpp # External or managed audio.cpp server (disabled by default) - openai # Reliable, high quality - kokoro # Local, fast - pocket_tts # Local ONNX voice cloning @@ -93,6 +94,58 @@ providers: cache_max_bytes_per_user: 1073741824 persist_direct_voice_references: false + # audio.cpp Configuration (external or managed audiocpp_server) + audio_cpp: + enabled: false # Disabled by default; enable after installing/running audio.cpp + backend: "cuda" + base_url: "http://127.0.0.1:8080" + model: "audio-cpp/pocket-tts" + model_path: "models/audio_cpp/pocket-tts" + binary_path: null + device: "cuda" + timeout: 300 + sample_rate: 24000 + max_concurrent_generations: 1 + auto_download: false + extra_params: + managed: false + allow_remote_base_url: false + retain_request_artifacts: false + external_voice_reference_mode: "disabled" # disabled | shared_path + request_option_allowlist: + - max_tokens + - seed + server: + host: "127.0.0.1" + port: 8080 + autoselect_port: true + port_probe_max: 10 + startup_timeout_seconds: 30 + healthcheck_interval_seconds: 0.25 + startup_backoff_seconds: 5 + idle_shutdown_seconds: 900 + terminate_timeout_seconds: 10 + server_config_path: "models/audio_cpp/server.json" + models_root: "models/audio_cpp" + shared_scratch_dir: "models/audio_cpp/runtime/scratch" + lazy_load: true + device: 0 + threads: 1 + model: + id: "pocket-tts" + family: "pocket_tts" + path: "models/audio_cpp/pocket-tts" + task: "tts" + mode: "offline" + load_options: + language: "english" + session_options: + language: "english" + voices: + alba: + upstream_value: "alba" + request_field: null + # KittenTTS ONNX Configuration (local; English single-speaker voices) kitten_tts: enabled: true @@ -427,6 +480,7 @@ format_preferences: index_tts: ["mp3", "wav"] pocket_tts: ["wav", "mp3", "opus", "flac", "pcm", "aac"] pocket_tts_cpp: ["wav", "mp3", "opus", "flac", "pcm", "aac"] + audio_cpp: ["wav", "mp3", "opus", "flac", "aac", "pcm"] lux_tts: ["wav", "mp3", "opus", "flac", "pcm", "aac"] qwen3_tts: ["wav", "mp3", "opus", "flac", "pcm"] diff --git a/tldw_Server_API/app/core/TTS/adapter_registry.py b/tldw_Server_API/app/core/TTS/adapter_registry.py index 09649bbf60..eb8326d6fa 100644 --- a/tldw_Server_API/app/core/TTS/adapter_registry.py +++ b/tldw_Server_API/app/core/TTS/adapter_registry.py @@ -14,6 +14,8 @@ from tldw_Server_API.app.core.Infrastructure.provider_registry import ( ProviderRegistryBase, ProviderRegistryConfig, +) +from tldw_Server_API.app.core.Infrastructure.provider_registry import ( ProviderStatus as RegistryProviderStatus, ) from tldw_Server_API.app.core.Utils.pydantic_compat import model_dump_compat @@ -85,6 +87,7 @@ class TTSProvider(Enum): SUPERTONIC2 = "supertonic2" POCKET_TTS = "pocket_tts" POCKET_TTS_CPP = "pocket_tts_cpp" + AUDIO_CPP = "audio_cpp" ECHO_TTS = "echo_tts" QWEN3_TTS = "qwen3_tts" OMNIVOICE = "omnivoice" @@ -301,6 +304,7 @@ class TTSAdapterRegistry: TTSProvider.SUPERTONIC2: "tldw_Server_API.app.core.TTS.adapters.supertonic2_adapter.Supertonic2OnnxAdapter", TTSProvider.POCKET_TTS: "tldw_Server_API.app.core.TTS.adapters.pocket_tts_adapter.PocketTTSOnnxAdapter", TTSProvider.POCKET_TTS_CPP: "tldw_Server_API.app.core.TTS.adapters.pocket_tts_cpp_adapter.PocketTTSCppAdapter", + TTSProvider.AUDIO_CPP: "tldw_Server_API.app.core.TTS.adapters.audio_cpp_adapter.AudioCppTTSAdapter", TTSProvider.ECHO_TTS: "tldw_Server_API.app.core.TTS.adapters.echo_tts_adapter.EchoTTSAdapter", TTSProvider.QWEN3_TTS: "tldw_Server_API.app.core.TTS.adapters.qwen3_tts_adapter.Qwen3TTSAdapter", TTSProvider.OMNIVOICE: "tldw_Server_API.app.core.TTS.adapters.omnivoice_adapter.OmniVoiceAdapter", @@ -1173,7 +1177,7 @@ class TTSAdapterFactory: "dia-1.6b": TTSProvider.DIA, # Chatterbox models - **{alias: TTSProvider.CHATTERBOX for alias in CHATTERBOX_MODEL_PROVIDER_ALIASES}, + **dict.fromkeys(CHATTERBOX_MODEL_PROVIDER_ALIASES, TTSProvider.CHATTERBOX), # VibeVoice models "vibevoice": TTSProvider.VIBEVOICE, @@ -1228,6 +1232,12 @@ class TTSAdapterFactory: "pocket_tts_cpp": TTSProvider.POCKET_TTS_CPP, "pocket-tts-cpp": TTSProvider.POCKET_TTS_CPP, + # audio.cpp models use explicit namespaced aliases only. Bare + # "pocket-tts" remains mapped to the existing PocketTTS provider. + "audio_cpp:pocket-tts": TTSProvider.AUDIO_CPP, + "audio-cpp/pocket-tts": TTSProvider.AUDIO_CPP, + "audiocpp/pocket-tts": TTSProvider.AUDIO_CPP, + # Echo-TTS models "echo-tts": TTSProvider.ECHO_TTS, "echo_tts": TTSProvider.ECHO_TTS, diff --git a/tldw_Server_API/tests/TTS_NEW/fixtures/empty_config.txt b/tldw_Server_API/tests/TTS_NEW/fixtures/empty_config.txt new file mode 100644 index 0000000000..5dc7eaf285 --- /dev/null +++ b/tldw_Server_API/tests/TTS_NEW/fixtures/empty_config.txt @@ -0,0 +1 @@ +[TTS-Settings] diff --git a/tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py b/tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py new file mode 100644 index 0000000000..f03e1efbe8 --- /dev/null +++ b/tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py @@ -0,0 +1,32 @@ +import pytest + +from tldw_Server_API.app.core.TTS.adapter_registry import ( + TTSAdapterFactory, + TTSAdapterRegistry, + TTSProvider, +) + + +@pytest.mark.unit +def test_audio_cpp_provider_aliases_resolve(): + assert TTSAdapterRegistry.resolve_provider("audio_cpp") == TTSProvider.AUDIO_CPP + assert TTSAdapterRegistry.resolve_provider("audio-cpp") == TTSProvider.AUDIO_CPP + assert TTSAdapterRegistry.resolve_provider("audiocpp") == TTSProvider.AUDIO_CPP + + +@pytest.mark.unit +def test_audio_cpp_model_aliases_are_namespaced_and_do_not_steal_pocket_tts(): + factory = TTSAdapterFactory({}) + + assert factory.get_provider_for_model("audio_cpp:pocket-tts") == TTSProvider.AUDIO_CPP + assert factory.get_provider_for_model("audio-cpp/pocket-tts") == TTSProvider.AUDIO_CPP + assert factory.get_provider_for_model("audiocpp/pocket-tts") == TTSProvider.AUDIO_CPP + assert factory.get_provider_for_model("pocket-tts") == TTSProvider.POCKET_TTS + + +@pytest.mark.unit +def test_audio_cpp_default_adapter_is_registered_lazily(): + assert ( + TTSAdapterRegistry.DEFAULT_ADAPTERS[TTSProvider.AUDIO_CPP] + == "tldw_Server_API.app.core.TTS.adapters.audio_cpp_adapter.AudioCppTTSAdapter" + ) diff --git a/tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py b/tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py new file mode 100644 index 0000000000..da69bab3ec --- /dev/null +++ b/tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py @@ -0,0 +1,56 @@ +from pathlib import Path + +import pytest + +from tldw_Server_API.app.core.TTS.tts_config import TTSConfigManager + +EMPTY_CONFIG_TXT_PATH = Path("tldw_Server_API/tests/TTS_NEW/fixtures/empty_config.txt") + + +@pytest.mark.unit +def test_audio_cpp_yaml_config_is_disabled_and_preserves_runtime_settings(): + manager = TTSConfigManager( + yaml_path=Path("tldw_Server_API/Config_Files/tts_providers_config.yaml"), + config_txt_path=EMPTY_CONFIG_TXT_PATH, + ) + + provider_config = manager.get_provider_config("audio_cpp") + + assert provider_config is not None + assert provider_config.enabled is False + assert provider_config.backend == "cuda" + assert provider_config.base_url == "http://127.0.0.1:8080" + assert provider_config.model == "audio-cpp/pocket-tts" + assert provider_config.model_path == "models/audio_cpp/pocket-tts" + assert provider_config.binary_path is None + assert provider_config.device == "cuda" + assert provider_config.timeout == 300 + assert provider_config.sample_rate == 24000 + assert provider_config.max_concurrent_generations == 1 + assert provider_config.auto_download is False + + extra = provider_config.extra_params + assert extra["managed"] is False + assert extra["allow_remote_base_url"] is False + assert extra["external_voice_reference_mode"] == "disabled" + assert extra["retain_request_artifacts"] is False + assert extra["request_option_allowlist"] == ["max_tokens", "seed"] + assert extra["server"]["host"] == "127.0.0.1" + assert extra["server"]["autoselect_port"] is True + assert extra["server"]["model"]["id"] == "pocket-tts" + assert extra["voices"]["alba"]["request_field"] is None + + +@pytest.mark.unit +def test_audio_cpp_format_preferences_are_limited_to_verified_conversion_targets(): + manager = TTSConfigManager( + yaml_path=Path("tldw_Server_API/Config_Files/tts_providers_config.yaml"), + config_txt_path=EMPTY_CONFIG_TXT_PATH, + ) + + formats = manager.get_config().format_preferences["audio_cpp"] + + assert formats == ["wav", "mp3", "opus", "flac", "aac", "pcm"] + assert "ogg" not in formats + assert "webm" not in formats + assert "ulaw" not in formats From e8ffa09cb0cebed6772c38c1955f12ca687ea858 Mon Sep 17 00:00:00 2001 From: Robert Date: Fri, 3 Jul 2026 15:01:35 -0700 Subject: [PATCH 5/8] feat(tts): add audio_cpp client config helpers --- ...io-cpp-tts-provider-implementation-plan.md | 10 +- ....cpp-TTS-provider-and-setup-integration.md | 10 + .../app/core/TTS/adapters/audio_cpp_client.py | 214 +++++++++++++++++ .../app/core/TTS/adapters/audio_cpp_config.py | 216 ++++++++++++++++++ .../unit/adapters/test_audio_cpp_client.py | 103 +++++++++ .../unit/adapters/test_audio_cpp_config.py | 136 +++++++++++ 6 files changed, 684 insertions(+), 5 deletions(-) create mode 100644 tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py create mode 100644 tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py create mode 100644 tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py create mode 100644 tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py diff --git a/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md index 5a24e9fbe7..8a9f756e9c 100644 --- a/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md +++ b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md @@ -155,9 +155,9 @@ Expected result after implementation: both test files pass. **Tests:** `test_audio_cpp_client.py`, `test_audio_cpp_config.py` -**Status:** Not Started +**Status:** Complete -- [ ] **Step 1: Write failing client tests** +- [x] **Step 1: Write failing client tests** Use `httpx.MockTransport` or the project equivalent to cover: @@ -186,7 +186,7 @@ python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_c Expected result before implementation: import failure for `audio_cpp_client`. -- [ ] **Step 2: Write failing config validation tests** +- [x] **Step 2: Write failing config validation tests** Cover: @@ -208,7 +208,7 @@ python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_c Expected result before implementation: import failure for `audio_cpp_config`. -- [ ] **Step 3: Implement client and config modules** +- [x] **Step 3: Implement client and config modules** Create: @@ -223,7 +223,7 @@ Implementation requirements: - Keep config parsing independent from the adapter so sidecar and installer code can reuse it. - Treat provider `backend` as a setup hint unless upstream config documents a matching server field. -- [ ] **Step 4: Re-run focused tests** +- [x] **Step 4: Re-run focused tests** Run: diff --git a/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md index 505bb13460..243e63e69d 100644 --- a/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md +++ b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md @@ -18,7 +18,11 @@ modified_files: - backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md - tldw_Server_API/Config_Files/tts_providers_config.yaml - tldw_Server_API/app/core/TTS/adapter_registry.py +- tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py +- tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py - tldw_Server_API/tests/TTS_NEW/fixtures/empty_config.txt +- tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py +- tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py - tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py - tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py --- @@ -58,6 +62,12 @@ Follow `docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation- - Stage 1 implementation added `TTSProvider.AUDIO_CPP`, provider/model aliases, lazy default adapter mapping, disabled `audio_cpp` YAML config, and format preferences limited to `wav`, `mp3`, `opus`, `flac`, `aac`, and `pcm`. - Stage 1 green test: the same audio.cpp test command passed with 5 passed, 5 warnings in 35.09s. - Stage 1 adjacent regression: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_fish_s2_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_pocket_tts_cpp_registry.py` passed with 5 passed, 6 warnings in 40.86s. +- Stage 2 red test: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py` failed during collection with missing `audio_cpp_client` and `audio_cpp_config` modules. +- Stage 2 implementation added `AudioCppClient`, `AudioCppSpeechResult`, `AudioCppConfig`, base URL and managed-host validation, option allowlist filtering, server-config rendering, path containment checks, and generated scratch reference names. +- Stage 2 green test: the same client/config command passed with 11 passed, 10 warnings in 52.02s, then after Ruff fixes passed again with 11 passed, 10 warnings in 68.63s. +- Stage 2 combined focused test: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py` passed with 16 passed, 10 warnings in 99.73s. +- Stage 2 adjacent regression: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_fish_s2_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_pocket_tts_cpp_registry.py` passed with 5 passed, 6 warnings in 64.47s. +- Stage 2 Ruff check passed for `audio_cpp_client.py`, `audio_cpp_config.py`, `test_audio_cpp_client.py`, and `test_audio_cpp_config.py`; `git diff --check` also passed for the Stage 2 touched files. diff --git a/tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py b/tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py new file mode 100644 index 0000000000..52db508898 --- /dev/null +++ b/tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py @@ -0,0 +1,214 @@ +"""HTTP client for audiocpp_server.""" + +from __future__ import annotations + +import base64 +import re +from dataclasses import dataclass, field +from typing import Any + +import httpx + +from ..tts_exceptions import ( + TTSGenerationError, + TTSNetworkError, + TTSProviderError, + TTSTimeoutError, +) +from .audio_cpp_config import PROVIDER_KEY, validate_base_url + +_WINDOWS_PATH_RE = re.compile(r"\b[A-Za-z]:\\[^\r\n\t]+") +_POSIX_PATH_RE = re.compile(r"(? str: + sanitized = str(text or "").strip() + for term in redaction_terms or []: + if term: + sanitized = sanitized.replace(str(term), "[redacted-input]") + sanitized = _WINDOWS_PATH_RE.sub("[redacted-path]", sanitized) + sanitized = _POSIX_PATH_RE.sub("[redacted-path]", sanitized) + sanitized = _SECRET_RE.sub(lambda match: f"{match.group(1)}=[redacted-secret]", sanitized) + sanitized = re.sub(r"\s+", " ", sanitized).strip() + if len(sanitized) > 300: + return f"{sanitized[:300]}..." + return sanitized + + +class AudioCppClient: + """Small async client for audiocpp_server routes used by the TTS adapter.""" + + def __init__( + self, + *, + base_url: str, + http_client: httpx.AsyncClient | None = None, + timeout: float = 300.0, + allow_remote_base_url: bool = False, + ) -> None: + self.base_url = validate_base_url( + base_url, + allow_remote_base_url=allow_remote_base_url, + ) + self.timeout = timeout + self._http_client = http_client or httpx.AsyncClient(timeout=timeout, trust_env=False) + self._owns_client = http_client is None + + async def close(self) -> None: + if self._owns_client: + await self._http_client.aclose() + + def _url(self, path: str) -> str: + return f"{self.base_url}{path}" + + def _content_type(self, response: httpx.Response) -> str: + return str(response.headers.get("content-type") or "").split(";", 1)[0].strip().lower() + + async def _request(self, method: str, path: str, **kwargs: Any) -> httpx.Response: + try: + response = await self._http_client.request( + method, + self._url(path), + timeout=self.timeout, + **kwargs, + ) + except httpx.TimeoutException as exc: + raise TTSTimeoutError( + "Timeout waiting for audio.cpp server", + provider=PROVIDER_KEY, + error_code="TIMEOUT", + ) from exc + except httpx.RequestError as exc: + raise TTSNetworkError( + "Network error communicating with audio.cpp server", + provider=PROVIDER_KEY, + error_code="NETWORK_ERROR", + details={"error_type": type(exc).__name__}, + ) from exc + return response + + def _raise_for_status(self, response: httpx.Response, *, payload: dict[str, Any] | None = None) -> None: + if response.status_code < 400: + return + request_text = "" + if isinstance(payload, dict): + request_text = str(payload.get("input") or "") + raise TTSProviderError( + f"audio.cpp server returned HTTP {response.status_code}", + provider=PROVIDER_KEY, + error_code=f"HTTP_{response.status_code}", + details={ + "status_code": response.status_code, + "response_text": _sanitize_error_text( + response.text, + redaction_terms=[request_text], + ), + }, + ) + + async def health(self) -> dict[str, Any]: + response = await self._request("GET", "/health") + self._raise_for_status(response) + if not response.content: + return {"status": "ok"} + try: + return dict(response.json()) + except ValueError: + return {"status": response.text.strip() or "ok"} + + async def list_models(self) -> list[str]: + response = await self._request("GET", "/v1/models") + self._raise_for_status(response) + try: + payload = response.json() + except ValueError as exc: + raise TTSProviderError( + "audio.cpp /v1/models returned invalid JSON", + provider=PROVIDER_KEY, + error_code="INVALID_MODELS_RESPONSE", + ) from exc + + models = payload.get("data") if isinstance(payload, dict) else None + if models is None and isinstance(payload, dict): + models = payload.get("models") + if not isinstance(models, list): + return [] + + model_ids: list[str] = [] + for model in models: + if isinstance(model, dict): + model_id = str(model.get("id") or "").strip() + else: + model_id = str(model or "").strip() + if model_id: + model_ids.append(model_id) + return model_ids + + async def speech(self, payload: dict[str, Any]) -> AudioCppSpeechResult: + response = await self._request("POST", "/v1/audio/speech", json=payload) + self._raise_for_status(response, payload=payload) + content_type = self._content_type(response) or "application/octet-stream" + if content_type != "application/json": + return AudioCppSpeechResult( + audio_bytes=response.content, + content_type=content_type, + metadata={"upstream_response_format": content_type}, + ) + return self._decode_json_speech_response(response) + + def _decode_json_speech_response(self, response: httpx.Response) -> AudioCppSpeechResult: + try: + payload = response.json() + except ValueError as exc: + raise TTSGenerationError( + "audio.cpp speech JSON response was invalid", + provider=PROVIDER_KEY, + error_code="INVALID_SPEECH_RESPONSE", + ) from exc + if not isinstance(payload, dict): + raise TTSGenerationError( + "audio.cpp speech JSON response must be an object", + provider=PROVIDER_KEY, + error_code="INVALID_SPEECH_RESPONSE", + ) + + encoded_audio = None + for key in ("audio", "audio_base64", "audio_data", "data"): + value = payload.get(key) + if isinstance(value, str) and value.strip(): + encoded_audio = value + break + if not encoded_audio: + raise TTSGenerationError( + "audio.cpp speech JSON response did not include audio", + provider=PROVIDER_KEY, + error_code="MISSING_AUDIO", + ) + try: + audio_bytes = base64.b64decode(encoded_audio, validate=True) + except ValueError as exc: + raise TTSGenerationError( + "audio.cpp speech JSON audio was not valid base64", + provider=PROVIDER_KEY, + error_code="INVALID_AUDIO_BASE64", + ) from exc + return AudioCppSpeechResult( + audio_bytes=audio_bytes, + content_type="application/json", + metadata={ + "upstream_response_format": "application/json", + "json_format": payload.get("format") or payload.get("response_format"), + }, + ) diff --git a/tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py b/tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py new file mode 100644 index 0000000000..64c640fc2a --- /dev/null +++ b/tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py @@ -0,0 +1,216 @@ +"""Configuration helpers for the audio.cpp TTS provider.""" + +from __future__ import annotations + +import ipaddress +import uuid +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any +from urllib.parse import urlparse + +from ..tts_exceptions import TTSValidationError + +PROVIDER_KEY = "audio_cpp" +_SCALAR_TYPES = (str, int, float, bool) + + +def _as_bool(value: Any, default: bool = False) -> bool: + if value is None: + return default + if isinstance(value, bool): + return value + if isinstance(value, str): + normalized = value.strip().lower() + if normalized in {"1", "true", "yes", "on"}: + return True + if normalized in {"0", "false", "no", "off"}: + return False + return bool(value) + + +def _is_loopback_host(host: str | None) -> bool: + normalized = str(host or "").strip().lower() + if not normalized: + return False + if normalized == "localhost": + return True + try: + return ipaddress.ip_address(normalized).is_loopback + except ValueError: + return False + + +def validate_base_url(base_url: str | None, *, allow_remote_base_url: bool = False) -> str: + """Normalize and validate an audiocpp_server base URL.""" + normalized = str(base_url or "").strip().rstrip("/") + parsed = urlparse(normalized) + if parsed.scheme not in {"http", "https"} or not parsed.netloc: + raise TTSValidationError( + "audio.cpp base_url must be an absolute http(s) URL", + provider=PROVIDER_KEY, + error_code="invalid_base_url", + ) + if not allow_remote_base_url and not _is_loopback_host(parsed.hostname): + raise TTSValidationError( + "audio.cpp base_url must be loopback unless allow_remote_base_url is enabled", + provider=PROVIDER_KEY, + error_code="remote_base_url_disabled", + ) + return normalized + + +def validate_managed_host(host: str | None) -> str: + """Normalize managed sidecar bind host and reject non-loopback addresses.""" + normalized = str(host or "127.0.0.1").strip().lower() + if normalized == "localhost": + return "127.0.0.1" + try: + parsed = ipaddress.ip_address(normalized) + except ValueError as exc: + raise TTSValidationError( + "audio.cpp managed sidecar host must be a loopback address", + provider=PROVIDER_KEY, + error_code="invalid_managed_host", + ) from exc + if not parsed.is_loopback: + raise TTSValidationError( + "audio.cpp managed sidecar host must be a loopback address", + provider=PROVIDER_KEY, + error_code="invalid_managed_host", + ) + return "127.0.0.1" if parsed.version == 4 else "::1" + + +def filter_request_options( + options: dict[str, Any] | None, + *, + allowlist: tuple[str, ...] | list[str] | set[str], +) -> tuple[dict[str, Any], dict[str, str]]: + """Return allowlisted scalar options and reasons for ignored options.""" + allowed = {str(item) for item in allowlist} + filtered: dict[str, Any] = {} + ignored: dict[str, str] = {} + for key, value in (options or {}).items(): + normalized_key = str(key) + if normalized_key not in allowed: + ignored[normalized_key] = "not_allowlisted" + continue + if value is None: + ignored[normalized_key] = "none_value" + continue + if not isinstance(value, _SCALAR_TYPES): + ignored[normalized_key] = "non_scalar" + continue + filtered[normalized_key] = value + return filtered, ignored + + +@dataclass(frozen=True) +class AudioCppConfig: + """Parsed audio.cpp provider configuration.""" + + base_url: str + model: str + model_path: str | None + timeout: int + managed: bool + allow_remote_base_url: bool + external_voice_reference_mode: str + retain_request_artifacts: bool + request_option_allowlist: tuple[str, ...] + server: dict[str, Any] = field(default_factory=dict) + repo_root: Path = field(default_factory=Path.cwd) + + @classmethod + def from_provider_config( + cls, + config: dict[str, Any], + *, + repo_root: Path | None = None, + ) -> AudioCppConfig: + extra_params = dict(config.get("extra_params") or {}) + allow_remote = _as_bool(extra_params.get("allow_remote_base_url"), default=False) + mode = str(extra_params.get("external_voice_reference_mode") or "disabled").strip().lower() + if mode not in {"disabled", "shared_path"}: + raise TTSValidationError( + "audio.cpp external_voice_reference_mode must be disabled or shared_path", + provider=PROVIDER_KEY, + error_code="invalid_voice_reference_mode", + ) + allowlist = extra_params.get("request_option_allowlist") or ("max_tokens", "seed") + return cls( + base_url=validate_base_url( + config.get("base_url") or "http://127.0.0.1:8080", + allow_remote_base_url=allow_remote, + ), + model=str(config.get("model") or "audio-cpp/pocket-tts"), + model_path=config.get("model_path"), + timeout=int(config.get("timeout") or 300), + managed=_as_bool(extra_params.get("managed"), default=False), + allow_remote_base_url=allow_remote, + external_voice_reference_mode=mode, + retain_request_artifacts=_as_bool(extra_params.get("retain_request_artifacts"), default=False), + request_option_allowlist=tuple(str(item) for item in allowlist), + server=dict(extra_params.get("server") or {}), + repo_root=Path(repo_root or Path.cwd()).resolve(strict=False), + ) + + @property + def models_root(self) -> Path: + return self._resolve_repo_path(self.server.get("models_root") or "models/audio_cpp") + + @property + def shared_scratch_dir(self) -> Path: + configured = self.server.get("shared_scratch_dir") or "models/audio_cpp/runtime/scratch" + scratch_dir = self._resolve_repo_path(configured) + self._ensure_within(scratch_dir, self.models_root, "shared_scratch_dir") + return scratch_dir + + def _resolve_repo_path(self, value: Any) -> Path: + path = Path(str(value)).expanduser() + if not path.is_absolute(): + path = self.repo_root / path + return path.resolve(strict=False) + + def _ensure_within(self, path: Path, root: Path, field_name: str) -> None: + try: + path.relative_to(root) + except ValueError as exc: + raise TTSValidationError( + f"audio.cpp {field_name} must resolve under {root.name}", + provider=PROVIDER_KEY, + error_code="path_outside_audio_cpp_root", + ) from exc + + def render_server_config(self) -> dict[str, Any]: + """Render a single-model upstream server config dictionary.""" + host = validate_managed_host(self.server.get("host")) + model_config = dict(self.server.get("model") or {}) + configured_model_path = model_config.get("path") or self.model_path + model_path = self._resolve_repo_path(configured_model_path) + self._ensure_within(model_path, self.models_root, "model.path") + + model_entry = { + "id": str(model_config.get("id") or "pocket-tts"), + "family": str(model_config.get("family") or "pocket_tts"), + "path": str(model_path), + "task": str(model_config.get("task") or "tts"), + "mode": str(model_config.get("mode") or "offline"), + "load_options": dict(model_config.get("load_options") or {}), + "session_options": dict(model_config.get("session_options") or {}), + } + return { + "host": host, + "port": int(self.server.get("port") or 8080), + "lazy_load": _as_bool(self.server.get("lazy_load"), default=True), + "device": self.server.get("device", 0), + "threads": int(self.server.get("threads") or 1), + "models_root": str(self.models_root), + "shared_scratch_dir": str(self.shared_scratch_dir), + "models": [model_entry], + } + + def build_reference_scratch_path(self, original_filename: str | None = None) -> Path: + """Build a server-readable WAV scratch path without using user filenames.""" + return self.shared_scratch_dir / f"voice_ref_{uuid.uuid4().hex}.wav" diff --git a/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py b/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py new file mode 100644 index 0000000000..3a5d005934 --- /dev/null +++ b/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py @@ -0,0 +1,103 @@ +import base64 +import json + +import httpx +import pytest + +from tldw_Server_API.app.core.TTS.adapters.audio_cpp_client import AudioCppClient +from tldw_Server_API.app.core.TTS.tts_exceptions import TTSNetworkError, TTSProviderError + +WAV_BYTES = b"RIFF\x24\x00\x00\x00WAVEfmt " + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_audio_cpp_client_health_and_models_use_expected_routes(): + seen: list[tuple[str, str]] = [] + + def handler(request: httpx.Request) -> httpx.Response: + seen.append((request.method, request.url.path)) + if request.url.path == "/health": + return httpx.Response(200, json={"status": "ok"}) + if request.url.path == "/v1/models": + return httpx.Response(200, json={"data": [{"id": "pocket-tts"}]}) + return httpx.Response(404) + + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client: + client = AudioCppClient(base_url="http://127.0.0.1:8080", http_client=http_client) + + assert await client.health() == {"status": "ok"} + assert await client.list_models() == ["pocket-tts"] + + assert seen == [("GET", "/health"), ("GET", "/v1/models")] + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_audio_cpp_client_speech_returns_wav_bytes(): + captured_payload: dict[str, object] = {} + + def handler(request: httpx.Request) -> httpx.Response: + captured_payload.update(json.loads(request.content.decode("utf-8"))) + return httpx.Response(200, content=WAV_BYTES, headers={"content-type": "audio/wav"}) + + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client: + client = AudioCppClient(base_url="http://127.0.0.1:8080/", http_client=http_client) + result = await client.speech({"model": "pocket-tts", "input": "hello"}) + + assert result.audio_bytes == WAV_BYTES + assert result.content_type == "audio/wav" + assert captured_payload == {"model": "pocket-tts", "input": "hello"} + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_audio_cpp_client_speech_decodes_json_base64_audio(): + encoded = base64.b64encode(WAV_BYTES).decode("ascii") + + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, json={"audio": encoded, "format": "wav"}) + + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client: + client = AudioCppClient(base_url="http://127.0.0.1:8080", http_client=http_client) + result = await client.speech({"model": "pocket-tts", "input": "hello"}) + + assert result.audio_bytes == WAV_BYTES + assert result.content_type == "application/json" + assert result.metadata["json_format"] == "wav" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_audio_cpp_client_sanitizes_upstream_http_errors(): + request_text = "sensitive request text" + leaked_path = r"C:\Users\GDesktop-1\Working\tldw\models\audio_cpp\secret.wav" + long_body = f"{request_text} failed at {leaked_path} " + ("x" * 600) + + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(500, text=long_body) + + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client: + client = AudioCppClient(base_url="http://127.0.0.1:8080", http_client=http_client) + with pytest.raises(TTSProviderError) as exc_info: + await client.speech({"model": "pocket-tts", "input": request_text}) + + error = exc_info.value + assert request_text not in str(error) + assert leaked_path not in str(error) + assert error.details["status_code"] == 500 + assert request_text not in error.details["response_text"] + assert leaked_path not in error.details["response_text"] + assert len(error.details["response_text"]) <= 320 + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_audio_cpp_client_maps_transport_errors_to_tts_network_error(): + def handler(request: httpx.Request) -> httpx.Response: + raise httpx.ConnectError("connection refused", request=request) + + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as http_client: + client = AudioCppClient(base_url="http://127.0.0.1:8080", http_client=http_client) + with pytest.raises(TTSNetworkError): + await client.health() diff --git a/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py b/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py new file mode 100644 index 0000000000..c553fb2ee8 --- /dev/null +++ b/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py @@ -0,0 +1,136 @@ +from pathlib import Path + +import pytest + +from tldw_Server_API.app.core.TTS.adapters.audio_cpp_config import ( + AudioCppConfig, + filter_request_options, + validate_base_url, + validate_managed_host, +) +from tldw_Server_API.app.core.TTS.tts_exceptions import TTSValidationError + + +def _provider_config(**overrides): + config = { + "base_url": "http://127.0.0.1:8080", + "model": "audio-cpp/pocket-tts", + "model_path": "models/audio_cpp/pocket-tts", + "timeout": 300, + "extra_params": { + "managed": False, + "allow_remote_base_url": False, + "external_voice_reference_mode": "disabled", + "request_option_allowlist": ["max_tokens", "seed"], + "server": { + "host": "127.0.0.1", + "port": 8080, + "autoselect_port": True, + "models_root": "models/audio_cpp", + "shared_scratch_dir": "models/audio_cpp/runtime/scratch", + "lazy_load": True, + "device": 0, + "threads": 1, + "model": { + "id": "pocket-tts", + "family": "pocket_tts", + "path": "models/audio_cpp/pocket-tts", + "task": "tts", + "mode": "offline", + "load_options": {"language": "english"}, + "session_options": {"language": "english"}, + }, + }, + }, + } + config.update(overrides) + return config + + +@pytest.mark.unit +def test_validate_base_url_requires_loopback_by_default(): + assert validate_base_url("http://127.0.0.1:8080") == "http://127.0.0.1:8080" + assert validate_base_url("http://localhost:8080/") == "http://localhost:8080" + + with pytest.raises(TTSValidationError): + validate_base_url("http://example.com:8080") + + assert ( + validate_base_url("http://example.com:8080", allow_remote_base_url=True) + == "http://example.com:8080" + ) + + +@pytest.mark.unit +def test_validate_managed_host_allows_loopback_only(): + assert validate_managed_host("127.0.0.1") == "127.0.0.1" + assert validate_managed_host("localhost") == "127.0.0.1" + + with pytest.raises(TTSValidationError): + validate_managed_host("0.0.0.0") + + +@pytest.mark.unit +def test_filter_request_options_passes_only_allowlisted_scalars(): + filtered, ignored = filter_request_options( + { + "max_tokens": 128, + "seed": 42, + "temperature": 0.7, + "nested": {"bad": True}, + "items": [1, 2], + "empty": None, + }, + allowlist=("max_tokens", "seed", "nested", "items", "empty"), + ) + + assert filtered == {"max_tokens": 128, "seed": 42} + assert ignored == { + "temperature": "not_allowlisted", + "nested": "non_scalar", + "items": "non_scalar", + "empty": "none_value", + } + + +@pytest.mark.unit +def test_audio_cpp_config_renders_single_model_server_config(): + cfg = AudioCppConfig.from_provider_config(_provider_config(), repo_root=Path.cwd()) + + server_config = cfg.render_server_config() + + assert server_config["host"] == "127.0.0.1" + assert server_config["port"] == 8080 + assert server_config["lazy_load"] is True + assert server_config["device"] == 0 + assert server_config["threads"] == 1 + assert len(server_config["models"]) == 1 + model = server_config["models"][0] + assert model["id"] == "pocket-tts" + assert model["family"] == "pocket_tts" + assert Path(model["path"]).is_absolute() + assert model["task"] == "tts" + assert model["mode"] == "offline" + assert model["load_options"] == {"language": "english"} + assert model["session_options"] == {"language": "english"} + + +@pytest.mark.unit +def test_audio_cpp_config_rejects_model_paths_outside_models_root(): + config = _provider_config() + config["extra_params"]["server"]["model"]["path"] = "../outside-model" + + with pytest.raises(TTSValidationError): + AudioCppConfig.from_provider_config(config, repo_root=Path.cwd()).render_server_config() + + +@pytest.mark.unit +def test_audio_cpp_config_builds_reference_paths_without_user_filename(): + cfg = AudioCppConfig.from_provider_config(_provider_config(), repo_root=Path.cwd()) + + path = cfg.build_reference_scratch_path("my private voice.wav") + + assert path.parent == cfg.shared_scratch_dir + assert path.suffix == ".wav" + assert "my private voice" not in path.name + assert path.name.startswith("voice_ref_") From ea846f36b9b435aa914dc1c170d608775b3edae9 Mon Sep 17 00:00:00 2001 From: Robert Date: Fri, 3 Jul 2026 15:24:47 -0700 Subject: [PATCH 6/8] feat(tts): add audio_cpp adapter behavior --- ...io-cpp-tts-provider-implementation-plan.md | 10 +- ....cpp-TTS-provider-and-setup-integration.md | 14 +- .../core/TTS/adapters/audio_cpp_adapter.py | 422 ++++++++++++++++++ .../integration/test_audio_cpp_tts_service.py | 172 +++++++ .../unit/adapters/test_audio_cpp_adapter.py | 228 ++++++++++ 5 files changed, 839 insertions(+), 7 deletions(-) create mode 100644 tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py create mode 100644 tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py create mode 100644 tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py diff --git a/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md index 8a9f756e9c..98f9a59eca 100644 --- a/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md +++ b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md @@ -244,9 +244,9 @@ Expected result after implementation: both test files pass. **Tests:** `test_audio_cpp_adapter.py`, `test_audio_cpp_tts_service.py` -**Status:** Not Started +**Status:** Complete -- [ ] **Step 1: Write failing adapter tests** +- [x] **Step 1: Write failing adapter tests** Cover: @@ -277,7 +277,7 @@ python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_a Expected result before implementation: import failure for `audio_cpp_adapter`. -- [ ] **Step 2: Write failing service routing test** +- [x] **Step 2: Write failing service routing test** Use the TTS factory/service test pattern to prove: @@ -295,7 +295,7 @@ python -m pytest -q tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts Expected result before implementation: routing or adapter import failure. -- [ ] **Step 3: Implement adapter** +- [x] **Step 3: Implement adapter** Create `tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py`. @@ -309,7 +309,7 @@ Implementation requirements: - Do not pass generic `voice`, `voice_id`, or unverified voice fields. - Clean request scratch files after generation unless `retain_request_artifacts` is true. -- [ ] **Step 4: Re-run adapter and service tests** +- [x] **Step 4: Re-run adapter and service tests** Run: diff --git a/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md index 243e63e69d..c36e5290e6 100644 --- a/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md +++ b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md @@ -20,7 +20,10 @@ modified_files: - tldw_Server_API/app/core/TTS/adapter_registry.py - tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py - tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py +- tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py - tldw_Server_API/tests/TTS_NEW/fixtures/empty_config.txt +- tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py +- tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py - tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py - tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py - tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py @@ -36,8 +39,8 @@ Implement the accepted Approach A design for `0xShug0/audio.cpp`: a disabled-by- ## Acceptance Criteria - [x] #1 `audio_cpp` is registered as a first-class TTS provider with explicit aliases, namespaced model aliases, disabled-by-default config, and no regression to existing `pocket_tts` routing. -- [ ] #2 The adapter/client can synthesize through `audiocpp_server` using the existing `/api/v1/audio/speech` flow, with tested request translation, WAV response handling, one-shot streaming compatibility, format conversion handoff, and sanitized error mapping. -- [ ] #3 Reference-audio and option passthrough behavior is safe by default: loopback-only base URLs unless explicitly allowed, external reference audio disabled unless configured, server-local scratch paths constrained, and only allowlisted scalar options sent upstream. +- [x] #2 The adapter/client can synthesize through `audiocpp_server` using the existing `/api/v1/audio/speech` flow, with tested request translation, WAV response handling, one-shot streaming compatibility, format conversion handoff, and sanitized error mapping. +- [x] #3 Reference-audio and option passthrough behavior is safe by default: loopback-only base URLs unless explicitly allowed, external reference audio disabled unless configured, server-local scratch paths constrained, and only allowlisted scalar options sent upstream. - [ ] #4 Managed sidecar support can render upstream server config, choose a loopback port, wait for health, avoid tight restart loops, and shut down cleanly without exposing arbitrary command args or process output. - [ ] #5 Installer/setup helpers and documentation cover explicit clone/build/config/model steps without silent network downloads during normal server startup or inference. - [ ] #6 Focused pytest, Ruff, and Bandit verification are recorded for the touched implementation scope. @@ -68,6 +71,13 @@ Follow `docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation- - Stage 2 combined focused test: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py` passed with 16 passed, 10 warnings in 99.73s. - Stage 2 adjacent regression: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_fish_s2_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_pocket_tts_cpp_registry.py` passed with 5 passed, 6 warnings in 64.47s. - Stage 2 Ruff check passed for `audio_cpp_client.py`, `audio_cpp_config.py`, `test_audio_cpp_client.py`, and `test_audio_cpp_config.py`; `git diff --check` also passed for the Stage 2 touched files. +- Stage 3 red test: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py` failed during collection with missing `tldw_Server_API.app.core.TTS.adapters.audio_cpp_adapter`, as expected before implementation. +- Stage 3 implementation added `AudioCppTTSAdapter`, tested capabilities, allowlisted request translation, full-byte one-shot streaming compatibility, managed/shared reference-audio staging, catalog-only voice behavior, and service routing through `TTSServiceV2`. +- Stage 3 green test: the same adapter/service command passed with 9 passed, 7 warnings in 80.59s, then after Ruff fixes passed again with 9 passed, 7 warnings in 79.52s. +- Stage 3 combined focused test: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py` passed with 25 passed, 12 warnings in 117.02s. +- Stage 3 adjacent regression: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_fish_s2_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_pocket_tts_cpp_registry.py` passed with 5 passed, 6 warnings in 64.60s. +- Stage 3 Ruff check passed for `audio_cpp_adapter.py`, `test_audio_cpp_adapter.py`, and `test_audio_cpp_tts_service.py`. +- Stage 3 post-review refactor moved reference-audio file writes off the event loop; final focused rerun passed with 9 passed, 7 warnings in 76.92s. diff --git a/tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py b/tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py new file mode 100644 index 0000000000..14b00dae6e --- /dev/null +++ b/tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py @@ -0,0 +1,422 @@ +"""audio.cpp TTS provider adapter.""" + +from __future__ import annotations + +import asyncio +import base64 +import inspect +import math +from contextlib import suppress +from pathlib import Path +from typing import Any + +from loguru import logger + +from ..tts_exceptions import ( + TTSGenerationError, + TTSModelNotFoundError, + TTSProviderInitializationError, + TTSProviderNotConfiguredError, + TTSValidationError, +) +from .audio_cpp_client import AudioCppClient, AudioCppSpeechResult +from .audio_cpp_config import PROVIDER_KEY as AUDIO_CPP_PROVIDER_KEY +from .audio_cpp_config import AudioCppConfig, filter_request_options +from .base import ( + AudioFormat, + ProviderStatus, + TTSAdapter, + TTSCapabilities, + TTSRequest, + TTSResponse, + VoiceInfo, +) + +_SUPPORTED_LANGUAGES = {"en"} +_SUPPORTED_FORMATS = { + AudioFormat.WAV, + AudioFormat.MP3, + AudioFormat.OPUS, + AudioFormat.FLAC, + AudioFormat.AAC, + AudioFormat.PCM, +} +_NAMESPACED_MODEL_PREFIXES = ( + "audio_cpp:", + "audio-cpp:", + "audiocpp:", + "audio_cpp/", + "audio-cpp/", + "audiocpp/", +) +_CONTENT_TYPE_FORMATS = { + "audio/wav": AudioFormat.WAV, + "audio/x-wav": AudioFormat.WAV, + "audio/mpeg": AudioFormat.MP3, + "audio/mp3": AudioFormat.MP3, + "audio/opus": AudioFormat.OPUS, + "audio/flac": AudioFormat.FLAC, + "audio/aac": AudioFormat.AAC, + "audio/l16": AudioFormat.PCM, + "application/octet-stream": AudioFormat.WAV, +} + + +class AudioCppTTSAdapter(TTSAdapter): + """Registry-facing adapter for an audiocpp_server speech endpoint.""" + + PROVIDER_KEY = AUDIO_CPP_PROVIDER_KEY + SUPPORTED_FORMATS = _SUPPORTED_FORMATS + + def __init__(self, config: dict[str, Any] | None = None): + super().__init__(config) + cfg = config or {} + self._audio_cpp_config = AudioCppConfig.from_provider_config(cfg, repo_root=Path.cwd()) + self.sample_rate = int(cfg.get("sample_rate") or 24000) + self.max_text_length = int(cfg.get("max_text_length") or 5000) + self._client: Any | None = cfg.get("client") or cfg.get("_client") + self._owns_client = self._client is None + self._available_models: list[str] = [] + self._voices = self._parse_voice_catalog(cfg) + + async def ensure_initialized(self) -> bool: + """Propagate initialization errors so registry logs keep the root cause.""" + if self._initialized: + return True + + async with self._init_lock: + if self._initialized: + return True + + self._status = ProviderStatus.INITIALIZING + try: + success = await self.initialize() + if success: + self._capabilities = await self.get_capabilities() + self._status = ProviderStatus.AVAILABLE + self._initialized = True + else: + self._status = ProviderStatus.ERROR + return success + except Exception: + self._status = ProviderStatus.ERROR + raise + + async def initialize(self) -> bool: + if self._client is None: + self._client = AudioCppClient( + base_url=self._audio_cpp_config.base_url, + timeout=float(self._audio_cpp_config.timeout), + allow_remote_base_url=self._audio_cpp_config.allow_remote_base_url, + ) + self._owns_client = True + + health = await self._client.health() + if isinstance(health, dict): + status = str(health.get("status") or health.get("state") or "ok").strip().lower() + if status in {"error", "failed", "unhealthy"}: + raise TTSProviderInitializationError( + "audio.cpp server health check failed", + provider=self.PROVIDER_KEY, + details={"status": status}, + ) + + self._available_models = await self._client.list_models() + self._validate_configured_model_available() + logger.info("audio.cpp TTS adapter initialized model={}", self._configured_upstream_model()) + return True + + async def get_capabilities(self) -> TTSCapabilities: + return TTSCapabilities( + provider_name="audio.cpp", + supported_languages=_SUPPORTED_LANGUAGES, + supported_voices=list(self._voices.values()), + supported_formats=self.SUPPORTED_FORMATS, + max_text_length=self.max_text_length, + supports_streaming=True, + supports_voice_cloning=True, + supports_emotion_control=False, + supports_speech_rate=False, + supports_pitch_control=False, + supports_volume_control=False, + supports_ssml=False, + supports_phonemes=False, + supports_multi_speaker=False, + supports_background_audio=False, + latency_ms=250, + sample_rate=self.sample_rate, + default_format=AudioFormat.WAV, + metadata={ + "incremental_streaming": False, + "native_streaming": False, + "managed": self._audio_cpp_config.managed, + "voice_reference_mode": self._voice_reference_mode_metadata(), + }, + ) + + async def generate(self, request: TTSRequest) -> TTSResponse: + if not await self.ensure_initialized(): + raise TTSProviderNotConfiguredError( + "audio.cpp adapter not initialized", + provider=self.PROVIDER_KEY, + ) + if self._client is None: + raise TTSProviderInitializationError( + "audio.cpp client is not available", + provider=self.PROVIDER_KEY, + ) + + is_valid, error = await self.validate_request(request) + if not is_valid: + raise TTSValidationError( + error or "audio.cpp request is invalid", + provider=self.PROVIDER_KEY, + ) + + payload, ignored_options, voice_used, staged_reference = await self._build_payload(request) + try: + result = await self._client.speech(payload) + return self._build_response( + request=request, + result=result, + ignored_options=ignored_options, + voice_used=voice_used, + ) + finally: + self._cleanup_reference(staged_reference) + + async def _cleanup_resources(self) -> None: + client = self._client + if client is None or not self._owns_client: + return + close = getattr(client, "close", None) + if not callable(close): + return + maybe_close = close() + if inspect.isawaitable(maybe_close): + await maybe_close + + def _parse_voice_catalog(self, config: dict[str, Any]) -> dict[str, VoiceInfo]: + extras = config.get("extra_params") if isinstance(config.get("extra_params"), dict) else {} + voices = extras.get("voices") if isinstance(extras, dict) else None + if not isinstance(voices, dict): + return {} + + catalog: dict[str, VoiceInfo] = {} + for voice_id, raw_mapping in voices.items(): + normalized_id = str(voice_id).strip() + if not normalized_id or not isinstance(raw_mapping, dict): + continue + catalog[normalized_id] = VoiceInfo( + id=normalized_id, + name=str(raw_mapping.get("name") or normalized_id), + language=str(raw_mapping.get("language") or "en"), + gender=raw_mapping.get("gender"), + description=raw_mapping.get("description"), + ) + return catalog + + def _voice_mapping(self, voice_id: str | None) -> dict[str, Any] | None: + if not voice_id: + return None + extras = self.config.get("extra_params") if isinstance(self.config.get("extra_params"), dict) else {} + voices = extras.get("voices") if isinstance(extras, dict) else None + if not isinstance(voices, dict): + return None + mapping = voices.get(str(voice_id)) + return dict(mapping) if isinstance(mapping, dict) else None + + def _configured_upstream_model(self) -> str: + server_model = self._audio_cpp_config.server.get("model") + if isinstance(server_model, dict): + model_id = str(server_model.get("id") or "").strip() + if model_id: + return model_id + return self._strip_model_namespace(self._audio_cpp_config.model) + + def _resolve_upstream_model(self, request: TTSRequest) -> str: + return self._strip_model_namespace(request.model or self._audio_cpp_config.model) + + @staticmethod + def _strip_model_namespace(model: str | None) -> str: + value = str(model or "").strip() + lowered = value.lower() + for prefix in _NAMESPACED_MODEL_PREFIXES: + if lowered.startswith(prefix): + return value[len(prefix):].strip() or "pocket-tts" + return value or "pocket-tts" + + def _validate_configured_model_available(self) -> None: + if not self._available_models: + return + configured_model = self._configured_upstream_model().lower() + known_models = { + self._strip_model_namespace(model).lower() + for model in self._available_models + if str(model).strip() + } + if configured_model not in known_models: + raise TTSModelNotFoundError( + "audio.cpp configured model is not listed by the server", + provider=self.PROVIDER_KEY, + details={"model": configured_model}, + ) + + async def _build_payload( + self, + request: TTSRequest, + ) -> tuple[dict[str, Any], dict[str, str], str | None, Path | None]: + ignored_options: dict[str, str] = {} + payload: dict[str, Any] = { + "model": self._resolve_upstream_model(request), + "input": self.preprocess_text(request.text), + } + + extras = request.extra_params if isinstance(request.extra_params, dict) else {} + filtered_options, ignored_extras = filter_request_options( + extras, + allowlist=self._audio_cpp_config.request_option_allowlist, + ) + payload.update(filtered_options) + ignored_options.update(ignored_extras) + + if request.speed is not None and not math.isclose(float(request.speed), 1.0, abs_tol=1e-12): + ignored_options["speed"] = "unsupported" + + voice_used = request.voice + self._apply_voice_mapping(request, payload) + staged_reference = await self._stage_reference_audio(request.voice_reference) + if staged_reference is not None: + payload["voice_ref"] = str(staged_reference) + voice_used = request.voice or "reference_audio" + + return payload, ignored_options, voice_used, staged_reference + + def _apply_voice_mapping(self, request: TTSRequest, payload: dict[str, Any]) -> None: + mapping = self._voice_mapping(request.voice) + if mapping is None: + return + + request_field = mapping.get("request_field") + if request_field is None: + if not request.voice_reference: + raise TTSValidationError( + f"audio.cpp voice '{request.voice}' is catalog metadata only; provide reference audio", + provider=self.PROVIDER_KEY, + details={"voice": request.voice}, + ) + return + + field_name = str(request_field).strip() + if not field_name or field_name in {"voice", "voice_id"}: + raise TTSValidationError( + "audio.cpp voice mapping uses an unsupported request field", + provider=self.PROVIDER_KEY, + details={"voice": request.voice, "request_field": field_name}, + ) + payload[field_name] = mapping.get("upstream_value") or request.voice + + async def _stage_reference_audio(self, voice_reference: Any) -> Path | None: + if voice_reference is None: + return None + if ( + not self._audio_cpp_config.managed + and self._audio_cpp_config.external_voice_reference_mode != "shared_path" + ): + raise TTSValidationError( + "audio.cpp voice_reference requires managed mode or external shared_path mode", + provider=self.PROVIDER_KEY, + details={"external_voice_reference_mode": self._audio_cpp_config.external_voice_reference_mode}, + ) + + audio_bytes = self._extract_voice_reference_bytes(voice_reference) + path = self._audio_cpp_config.build_reference_scratch_path() + await asyncio.to_thread(self._write_reference_audio_sync, path, audio_bytes) + return path + + @staticmethod + def _write_reference_audio_sync(path: Path, audio_bytes: bytes) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(audio_bytes) + + def _extract_voice_reference_bytes(self, voice_reference: Any) -> bytes: + if isinstance(voice_reference, bytes): + return voice_reference + if isinstance(voice_reference, bytearray): + return bytes(voice_reference) + if isinstance(voice_reference, str): + try: + return base64.b64decode(voice_reference, validate=True) + except ValueError as exc: + raise TTSValidationError( + "audio.cpp voice_reference must be bytes or base64 audio", + provider=self.PROVIDER_KEY, + details={"type": "str"}, + ) from exc + raise TTSValidationError( + "audio.cpp voice_reference must be bytes or base64 audio", + provider=self.PROVIDER_KEY, + details={"type": type(voice_reference).__name__}, + ) + + def _cleanup_reference(self, staged_reference: Path | None) -> None: + if staged_reference is None or self._audio_cpp_config.retain_request_artifacts: + return + with suppress(OSError, ValueError): + staged_reference.unlink(missing_ok=True) + + def _build_response( + self, + *, + request: TTSRequest, + result: AudioCppSpeechResult, + ignored_options: dict[str, str], + voice_used: str | None, + ) -> TTSResponse: + if not result.audio_bytes: + raise TTSGenerationError( + "audio.cpp server returned empty audio", + provider=self.PROVIDER_KEY, + error_code="EMPTY_AUDIO", + ) + + metadata = dict(result.metadata or {}) + content_type = result.content_type or metadata.get("upstream_response_format") or "audio/wav" + metadata.update( + { + "provider": self.PROVIDER_KEY, + "model": request.model or self._audio_cpp_config.model, + "managed": self._audio_cpp_config.managed, + "incremental_streaming": False, + "voice_reference_mode": self._voice_reference_mode_metadata(), + "ignored_options": ignored_options, + "upstream_response_format": content_type, + } + ) + + return TTSResponse( + audio_data=result.audio_bytes, + audio_stream=None, + format=self._resolve_response_format(content_type, metadata), + sample_rate=self.sample_rate, + channels=1, + voice_used=voice_used, + provider=self.PROVIDER_KEY, + model=request.model or self._audio_cpp_config.model, + metadata=metadata, + ) + + def _resolve_response_format(self, content_type: str, metadata: dict[str, Any]) -> AudioFormat: + normalized = str(content_type or "").split(";", 1)[0].strip().lower() + if normalized == "application/json": + json_format = str(metadata.get("json_format") or "").strip().lower() + if json_format: + with suppress(ValueError): + return AudioFormat(json_format) + return AudioFormat.WAV + return _CONTENT_TYPE_FORMATS.get(normalized, AudioFormat.WAV) + + def _voice_reference_mode_metadata(self) -> str: + if self._audio_cpp_config.managed: + return "managed" + return self._audio_cpp_config.external_voice_reference_mode diff --git a/tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py b/tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py new file mode 100644 index 0000000000..9795cd78fb --- /dev/null +++ b/tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py @@ -0,0 +1,172 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest + +from tldw_Server_API.app.api.v1.schemas.audio_schemas import OpenAISpeechRequest +from tldw_Server_API.app.core.TTS import adapter_registry as adapter_registry_module +from tldw_Server_API.app.core.TTS import tts_service_v2 as tts_service_v2_module +from tldw_Server_API.app.core.TTS.adapter_registry import TTSAdapterFactory, TTSProvider +from tldw_Server_API.app.core.TTS.adapters.audio_cpp_adapter import AudioCppTTSAdapter +from tldw_Server_API.app.core.TTS.adapters.audio_cpp_client import AudioCppSpeechResult +from tldw_Server_API.app.core.TTS.tts_service_v2 import TTSServiceV2 + +WAV_BYTES = b"RIFF$\x00\x00\x00WAVEfmt " + + +class _FakeAudioCppClient: + def __init__(self) -> None: + self.payloads: list[dict[str, object]] = [] + + async def health(self) -> dict[str, str]: + return {"status": "ok"} + + async def list_models(self) -> list[str]: + return ["pocket-tts"] + + async def speech(self, payload: dict[str, object]) -> AudioCppSpeechResult: + self.payloads.append(dict(payload)) + return AudioCppSpeechResult( + audio_bytes=WAV_BYTES, + content_type="audio/wav", + metadata={"upstream_latency_ms": 5}, + ) + + async def close(self) -> None: + return None + + +class _FakeMemoryMonitor: + def is_memory_critical(self) -> bool: + return False + + +class _FakeResourceManager: + def __init__(self) -> None: + self.memory_monitor = _FakeMemoryMonitor() + self.touched: list[tuple[str, object]] = [] + + def touch_model(self, provider: str, model: object) -> None: + self.touched.append((provider, model)) + + +async def _fake_resource_manager() -> _FakeResourceManager: + return _FakeResourceManager() + + +def _audio_cpp_provider_config(client: _FakeAudioCppClient, *, enabled: bool = True) -> dict[str, object]: + return { + "enabled": enabled, + "base_url": "http://127.0.0.1:8080", + "model": "audio-cpp/pocket-tts", + "model_path": "models/audio_cpp/pocket-tts", + "sample_rate": 24000, + "timeout": 300, + "client": client, + "extra_params": { + "managed": False, + "allow_remote_base_url": False, + "external_voice_reference_mode": "disabled", + "request_option_allowlist": ["max_tokens", "seed"], + "server": { + "host": "127.0.0.1", + "port": 8080, + "models_root": "models/audio_cpp", + "shared_scratch_dir": str(Path("models/audio_cpp/runtime/test_scratch")), + "model": { + "id": "pocket-tts", + "family": "pocket_tts", + "path": "models/audio_cpp/pocket-tts", + "task": "tts", + "mode": "offline", + }, + }, + }, + } + + +def _factory_config(client: _FakeAudioCppClient, *, enabled: bool = True) -> dict[str, object]: + return { + "provider_priority": ["pocket_tts", "audio_cpp"], + "performance": { + "token_estimation_enabled": False, + "max_concurrent_generations": 1, + }, + "providers": { + "audio_cpp": _audio_cpp_provider_config(client, enabled=enabled), + "pocket_tts": {"enabled": False}, + }, + } + + +@pytest.fixture(autouse=True) +def _stub_resource_manager(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + adapter_registry_module, + "get_resource_manager", + _fake_resource_manager, + raising=True, + ) + monkeypatch.setattr( + tts_service_v2_module, + "get_resource_manager", + _fake_resource_manager, + raising=True, + ) + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_factory_routes_audio_cpp_explicit_provider_and_namespaced_model(): + client = _FakeAudioCppClient() + factory = TTSAdapterFactory(_factory_config(client)) + + explicit_adapter = await factory.registry.get_adapter("audio_cpp") + model_adapter = await factory.get_adapter_by_model("audio-cpp/pocket-tts") + + assert isinstance(explicit_adapter, AudioCppTTSAdapter) + assert model_adapter is explicit_adapter + assert factory.get_provider_for_model("audio-cpp/pocket-tts") == TTSProvider.AUDIO_CPP + assert factory.get_provider_for_model("pocket-tts") == TTSProvider.POCKET_TTS + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_disabled_audio_cpp_provider_does_not_initialize_by_default(): + client = _FakeAudioCppClient() + factory = TTSAdapterFactory(_factory_config(client, enabled=False)) + + assert await factory.registry.get_adapter("audio_cpp") is None + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_generate_speech_uses_audio_cpp_adapter_for_namespaced_model(): + client = _FakeAudioCppClient() + factory = TTSAdapterFactory(_factory_config(client)) + service = TTSServiceV2(factory=factory) + request = OpenAISpeechRequest( + model="audio-cpp/pocket-tts", + input="hello from service", + voice="af_heart", + response_format="wav", + stream=True, + extra_params={"seed": 9}, + ) + + try: + chunks = [chunk async for chunk in service.generate_speech(request, fallback=False)] + finally: + await service.shutdown() + + assert chunks == [WAV_BYTES] + assert client.payloads[-1] == { + "model": "pocket-tts", + "input": "hello from service", + "seed": 9, + } + metadata = request._tts_metadata + assert metadata["provider"] == "audio_cpp" + assert metadata["model"] == "audio-cpp/pocket-tts" + assert metadata["incremental_streaming"] is False diff --git a/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py b/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py new file mode 100644 index 0000000000..3463fb7d98 --- /dev/null +++ b/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py @@ -0,0 +1,228 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest + +from tldw_Server_API.app.core.TTS.adapters.audio_cpp_adapter import AudioCppTTSAdapter +from tldw_Server_API.app.core.TTS.adapters.audio_cpp_client import AudioCppSpeechResult +from tldw_Server_API.app.core.TTS.adapters.base import AudioFormat, TTSRequest +from tldw_Server_API.app.core.TTS.tts_exceptions import TTSValidationError + +WAV_BYTES = b"RIFF$\x00\x00\x00WAVEfmt " + + +class _FakeAudioCppClient: + def __init__(self, *, audio_bytes: bytes = WAV_BYTES, models: list[str] | None = None) -> None: + self.audio_bytes = audio_bytes + self.models = models or ["pocket-tts"] + self.payloads: list[dict[str, object]] = [] + self.closed = False + + async def health(self) -> dict[str, str]: + return {"status": "ok"} + + async def list_models(self) -> list[str]: + return self.models + + async def speech(self, payload: dict[str, object]) -> AudioCppSpeechResult: + self.payloads.append(dict(payload)) + return AudioCppSpeechResult( + audio_bytes=self.audio_bytes, + content_type="audio/wav", + metadata={"upstream_latency_ms": 12}, + ) + + async def close(self) -> None: + self.closed = True + + +def _provider_config( + *, + client: _FakeAudioCppClient, + managed: bool = False, + external_voice_reference_mode: str = "disabled", + retain_request_artifacts: bool = False, + scratch_dir: Path | None = None, +) -> dict[str, object]: + scratch = scratch_dir or Path("models/audio_cpp/runtime/test_scratch") + return { + "enabled": True, + "base_url": "http://127.0.0.1:8080", + "model": "audio-cpp/pocket-tts", + "model_path": "models/audio_cpp/pocket-tts", + "sample_rate": 24000, + "timeout": 300, + "client": client, + "extra_params": { + "managed": managed, + "allow_remote_base_url": False, + "external_voice_reference_mode": external_voice_reference_mode, + "retain_request_artifacts": retain_request_artifacts, + "request_option_allowlist": ["max_tokens", "seed"], + "server": { + "host": "127.0.0.1", + "port": 8080, + "models_root": "models/audio_cpp", + "shared_scratch_dir": str(scratch), + "model": { + "id": "pocket-tts", + "family": "pocket_tts", + "path": "models/audio_cpp/pocket-tts", + "task": "tts", + "mode": "offline", + }, + }, + "voices": { + "alba": { + "name": "Alba", + "language": "en", + "upstream_value": "alba", + "request_field": None, + } + }, + }, + } + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_capabilities_advertise_one_shot_streaming_and_verified_formats(): + client = _FakeAudioCppClient() + adapter = AudioCppTTSAdapter(_provider_config(client=client)) + + assert await adapter.ensure_initialized() is True + + capabilities = adapter.capabilities + assert capabilities is not None + assert capabilities.supports_streaming is True + assert capabilities.metadata["incremental_streaming"] is False + assert {AudioFormat.OGG, AudioFormat.WEBM, AudioFormat.ULAW}.isdisjoint( + capabilities.supported_formats + ) + assert [voice.id for voice in capabilities.supported_voices] == ["alba"] + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_text_only_request_posts_model_input_and_allowlisted_options(): + client = _FakeAudioCppClient() + adapter = AudioCppTTSAdapter(_provider_config(client=client)) + await adapter.ensure_initialized() + + response = await adapter.generate( + TTSRequest( + text=" hello audio.cpp ", + model="audio_cpp:pocket-tts", + format=AudioFormat.WAV, + stream=False, + speed=1.25, + extra_params={ + "max_tokens": 128, + "seed": 42, + "temperature": 0.7, + "nested": {"ignored": True}, + }, + ) + ) + + assert response.audio_data == WAV_BYTES + assert response.format == AudioFormat.WAV + assert client.payloads[-1] == { + "model": "pocket-tts", + "input": "hello audio.cpp", + "max_tokens": 128, + "seed": 42, + } + assert response.metadata["ignored_options"] == { + "temperature": "not_allowlisted", + "nested": "not_allowlisted", + "speed": "unsupported", + } + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_stream_request_returns_full_audio_bytes_for_service_conversion(): + client = _FakeAudioCppClient() + adapter = AudioCppTTSAdapter(_provider_config(client=client)) + await adapter.ensure_initialized() + + response = await adapter.generate( + TTSRequest( + text="hello", + model="audio-cpp/pocket-tts", + format=AudioFormat.MP3, + stream=True, + ) + ) + + assert response.audio_data == WAV_BYTES + assert response.audio_stream is None + assert response.format == AudioFormat.WAV + assert response.metadata["incremental_streaming"] is False + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_external_mode_rejects_voice_reference_when_disabled(): + client = _FakeAudioCppClient() + adapter = AudioCppTTSAdapter(_provider_config(client=client)) + await adapter.ensure_initialized() + + with pytest.raises(TTSValidationError, match="voice_reference"): + await adapter.generate( + TTSRequest( + text="hello", + model="audio_cpp:pocket-tts", + format=AudioFormat.WAV, + stream=False, + voice_reference=b"RIFF reference audio", + ) + ) + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_managed_mode_stages_reference_audio_under_shared_scratch_dir(): + client = _FakeAudioCppClient() + scratch_dir = Path("models/audio_cpp/runtime/test_scratch") + adapter = AudioCppTTSAdapter( + _provider_config(client=client, managed=True, scratch_dir=scratch_dir) + ) + await adapter.ensure_initialized() + + response = await adapter.generate( + TTSRequest( + text="hello", + model="audio_cpp:pocket-tts", + format=AudioFormat.WAV, + stream=False, + voice_reference=b"RIFF reference audio", + ) + ) + + staged_path = Path(str(client.payloads[-1]["voice_ref"])) + assert response.audio_data == WAV_BYTES + assert staged_path.parent == (Path.cwd() / scratch_dir).resolve(strict=False) + assert staged_path.suffix == ".wav" + assert not staged_path.exists() + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_catalog_only_voice_mapping_requires_reference_audio(): + client = _FakeAudioCppClient() + adapter = AudioCppTTSAdapter(_provider_config(client=client)) + await adapter.ensure_initialized() + + with pytest.raises(TTSValidationError, match="reference audio"): + await adapter.generate( + TTSRequest( + text="hello", + voice="alba", + model="audio_cpp:pocket-tts", + format=AudioFormat.WAV, + stream=False, + ) + ) From d5eddbffd5049fdc626ed3dc546766beebfb6df8 Mon Sep 17 00:00:00 2001 From: Robert Date: Fri, 3 Jul 2026 15:53:28 -0700 Subject: [PATCH 7/8] feat(tts): add audio_cpp sidecar setup --- Docs/STT-TTS/TTS-SETUP-GUIDE.md | 102 +++++ ...io-cpp-tts-provider-implementation-plan.md | 12 +- Helper_Scripts/install_tts_audio_cpp.py | 350 ++++++++++++++++++ ....cpp-TTS-provider-and-setup-integration.md | 17 +- .../core/TTS/adapters/audio_cpp_adapter.py | 35 +- .../adapters/audio_cpp_sidecar_supervisor.py | 329 ++++++++++++++++ .../test_audio_cpp_sidecar_supervisor.py | 215 +++++++++++ .../TTS_NEW/unit/test_audio_cpp_installer.py | 156 ++++++++ 8 files changed, 1198 insertions(+), 18 deletions(-) create mode 100644 Helper_Scripts/install_tts_audio_cpp.py create mode 100644 tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py create mode 100644 tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py create mode 100644 tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py diff --git a/Docs/STT-TTS/TTS-SETUP-GUIDE.md b/Docs/STT-TTS/TTS-SETUP-GUIDE.md index 8b8cc9239d..47805ca574 100644 --- a/Docs/STT-TTS/TTS-SETUP-GUIDE.md +++ b/Docs/STT-TTS/TTS-SETUP-GUIDE.md @@ -175,6 +175,9 @@ python Helper_Scripts/TTS_Installers/install_tts_vibevoice.py --variant 1.5B python Helper_Scripts/TTS_Installers/install_tts_omnivoice_sidecar.py \ --model-path models/omnivoice_sidecar/models/OmniVoice +# audio.cpp sidecar config helper (explicit clone/build/model flags) +python Helper_Scripts/install_tts_audio_cpp.py --patch-config + # NeuTTS (deps; optional prefetch) python Helper_Scripts/TTS_Installers/install_tts_neutts.py --prefetch @@ -244,6 +247,105 @@ Example request: } ``` +### audio.cpp Setup + +`audio_cpp` is an optional TTS provider backed by the external +[`0xShug0/audio.cpp`](https://github.com/0xShug0/audio.cpp) executable or HTTP +server. It is disabled by default and does not vendor audio.cpp source or +prebuilt binaries into tldw_server. + +The first supported path is CUDA-first for the managed HTTP server. Upstream also +documents other build backends for parts of audio.cpp, but this tldw integration +treats non-CUDA managed server builds as future verification work unless you run +and validate the external server yourself. + +#### External Server Mode + +Run `audiocpp_server` yourself and point tldw at its loopback URL: + +```yaml +providers: + audio_cpp: + enabled: true + base_url: "http://127.0.0.1:8080" + model: "audio-cpp/pocket-tts" + auto_download: false + extra_params: + managed: false + allow_remote_base_url: false + external_voice_reference_mode: "disabled" +``` + +The adapter checks `/health` and `/v1/models` during initialization. By default, +`base_url` must be loopback. Set `allow_remote_base_url: true` only when an +admin intentionally exposes a trusted remote audio.cpp server. + +Reference-audio cloning in external mode is disabled by default because upstream +expects `voice_ref` to be a path readable by the audio.cpp server process. To use +it with a separate server, set `external_voice_reference_mode: "shared_path"` and +configure `shared_scratch_dir` to a directory that both tldw and the server can +read. + +#### Managed Sidecar Mode + +Managed mode lets tldw start a loopback sidecar with: + +```text +audiocpp_server --config +``` + +Patch the provider config without enabling it: + +```bash +python Helper_Scripts/install_tts_audio_cpp.py --patch-config +``` + +Enable it in the generated config or run: + +```bash +python Helper_Scripts/install_tts_audio_cpp.py --patch-config --enable-provider +``` + +The helper builds repo-local paths under `models/audio_cpp`, sets +`extra_params.managed: true`, and writes runtime-specific settings under +`extra_params.server`. It does not clone, build, or download models unless you +pass explicit admin flags such as `--clone`, `--configure`, `--build`, or +`--install-model`. + +The generated sidecar config stays under `models/audio_cpp`, binds to +`127.0.0.1`, autoselects a free port by default, waits for `/health`, backs off +after startup failure, and can stop after an idle interval. Normal speech +requests cannot inject extra command arguments or environment variables. + +#### Build And Model Package Commands + +The helper exposes explicit commands for operators who want a single entry point: + +```bash +python Helper_Scripts/install_tts_audio_cpp.py --clone +python Helper_Scripts/install_tts_audio_cpp.py --configure --build +python Helper_Scripts/install_tts_audio_cpp.py --install-model --package-id pocket-tts +``` + +Model installation is always explicit. audio.cpp's upstream +`tools/model_manager.py` handles package installation, including any gated +packages or token requirements. Do not put Hugging Face tokens or API keys in +`tts_providers_config.yaml`. + +No model download happens during normal tldw startup or a `/audio/speech` +request. If the configured model files are missing, initialization or generation +fails closed instead of fetching assets silently. + +Runtime note: audio.cpp can register lazy-loaded model ids at server startup, but +models and task sessions may remain resident after first use until the sidecar +process exits. Use `idle_shutdown_seconds` to release that memory in managed +mode, or restart an external server when you need to unload resident models. + +License and packaging note: audio.cpp is Apache-2.0 while tldw_server is GPLv2 +per project metadata. This implementation treats audio.cpp as an optional +external component installed by user/admin action. Vendoring, static linking, or +shipping prebuilt audio.cpp binaries needs separate legal and packaging review. + ### Model Auto-Download Controls Local providers (Kokoro, Higgs, Dia, Chatterbox, VibeVoice) can auto-download models the first time you use them. You can control this behavior globally or per provider. diff --git a/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md index 98f9a59eca..549d4daeca 100644 --- a/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md +++ b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md @@ -330,9 +330,9 @@ Expected result after implementation: both test files pass. **Tests:** `test_audio_cpp_sidecar_supervisor.py`, `test_audio_cpp_installer.py` -**Status:** Not Started +**Status:** Complete -- [ ] **Step 1: Write failing sidecar tests** +- [x] **Step 1: Write failing sidecar tests** Cover: @@ -353,7 +353,7 @@ python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_s Expected result before implementation: import failure for `audio_cpp_sidecar_supervisor`. -- [ ] **Step 2: Write failing installer tests** +- [x] **Step 2: Write failing installer tests** Cover pure functions only: @@ -372,7 +372,7 @@ python -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer. Expected result before implementation: missing installer module or helper failures. -- [ ] **Step 3: Implement sidecar supervisor and installer helper** +- [x] **Step 3: Implement sidecar supervisor and installer helper** Create: @@ -389,7 +389,7 @@ Implementation requirements: - Sidecar stdout/stderr may be logged at debug with truncation, but user-facing exceptions stay sanitized. - Installer network and build operations require explicit CLI flags. -- [ ] **Step 4: Update setup documentation** +- [x] **Step 4: Update setup documentation** Modify `Docs/STT-TTS/TTS-SETUP-GUIDE.md` with: @@ -402,7 +402,7 @@ Modify `Docs/STT-TTS/TTS-SETUP-GUIDE.md` with: - loopback, remote-base-url opt-in, and reference-audio shared-path warnings. - license boundary: optional external component, no vendored binaries in this implementation. -- [ ] **Step 5: Re-run focused tests** +- [x] **Step 5: Re-run focused tests** Run: diff --git a/Helper_Scripts/install_tts_audio_cpp.py b/Helper_Scripts/install_tts_audio_cpp.py new file mode 100644 index 0000000000..3972d7644d --- /dev/null +++ b/Helper_Scripts/install_tts_audio_cpp.py @@ -0,0 +1,350 @@ +#!/usr/bin/env python3 +"""Install/config helper for the optional audio.cpp TTS runtime.""" + +from __future__ import annotations + +import argparse +import subprocess # nosec B404 - installer CLI intentionally runs explicit argv lists +import sys +from collections.abc import Sequence +from dataclasses import dataclass +from pathlib import Path + +from loguru import logger + +DEFAULT_REPO_URL = "https://github.com/0xShug0/audio.cpp" +DEFAULT_RUNTIME_BASE = Path("models") / "audio_cpp" +DEFAULT_CONFIG_PATH = Path("tldw_Server_API") / "Config_Files" / "tts_providers_config.yaml" +DEFAULT_SOURCE_DIR = Path("external") / "audio.cpp" +PROVIDER_NAME = "audio_cpp" + + +@dataclass(frozen=True) +class AudioCppRuntimeLayout: + """Resolved repo-local paths for an audio.cpp sidecar runtime.""" + + provider_name: str + runtime_base: Path + binary_path: Path + model_path: Path + server_config_path: Path + shared_scratch_dir: Path + source_dir: Path + build_dir: Path + + +def default_binary_name(platform_name: str | None = None) -> str: + platform_key = (platform_name or sys.platform).lower() + if platform_key.startswith("win"): + return "audiocpp_server.exe" + return "audiocpp_server" + + +def resolve_repo_root(start: Path | None = None) -> Path: + probe = (start or Path(__file__)).resolve() + candidates = (probe,) + tuple(probe.parents) + for candidate in candidates: + if (candidate / "pyproject.toml").exists() and (candidate / "tldw_Server_API").is_dir(): + return candidate + raise FileNotFoundError(f"Unable to resolve repository root from {probe}") + + +def build_runtime_layout( + runtime_base: Path, + repo_root: Path | None = None, + *, + platform_name: str | None = None, +) -> AudioCppRuntimeLayout: + root = repo_root if repo_root is not None else resolve_repo_root() + base_candidate = runtime_base.expanduser() + base = base_candidate if base_candidate.is_absolute() else root / base_candidate + return AudioCppRuntimeLayout( + provider_name=PROVIDER_NAME, + runtime_base=base, + binary_path=root / "bin" / default_binary_name(platform_name), + model_path=base / "pocket-tts", + server_config_path=base / "server.json", + shared_scratch_dir=base / "runtime" / "scratch", + source_dir=root / DEFAULT_SOURCE_DIR, + build_dir=base / "_build", + ) + + +def _path_for_config(path: Path, repo_root: Path | None) -> str: + if not path.is_absolute(): + return path.as_posix() + if repo_root is not None: + try: + return path.relative_to(repo_root).as_posix() + except ValueError: + pass + return path.as_posix() + + +def _find_provider_block(lines: list[str], provider_name: str) -> tuple[int | None, int | None, int | None]: + in_providers = False + providers_indent: int | None = None + for idx, line in enumerate(lines): + stripped = line.strip() + if not stripped or stripped.startswith("#"): + continue + indent = len(line) - len(line.lstrip(" ")) + if not in_providers: + if stripped == "providers:": + in_providers = True + providers_indent = indent + continue + if providers_indent is not None and indent <= providers_indent: + in_providers = False + continue + if stripped.startswith(f"{provider_name}:"): + block_start = idx + block_indent = indent + block_end = idx + 1 + while block_end < len(lines): + next_line = lines[block_end] + next_stripped = next_line.strip() + if not next_stripped or next_stripped.startswith("#"): + block_end += 1 + continue + next_indent = len(next_line) - len(next_line.lstrip(" ")) + if next_indent <= block_indent: + break + block_end += 1 + return block_start, block_end, block_indent + return None, None, None + + +def _find_providers_insert_at(lines: list[str]) -> tuple[int, int]: + for idx, line in enumerate(lines): + if line.strip() == "providers:": + providers_indent = len(line) - len(line.lstrip(" ")) + insert_at = len(lines) + for probe in range(idx + 1, len(lines)): + stripped = lines[probe].strip() + if not stripped or stripped.startswith("#"): + continue + indent = len(lines[probe]) - len(lines[probe].lstrip(" ")) + if indent <= providers_indent: + insert_at = probe + break + return insert_at, providers_indent + 2 + lines.extend(["", "providers:"]) + return len(lines), 2 + + +def _render_provider_block( + *, + layout: AudioCppRuntimeLayout, + repo_root: Path | None, + enable_provider: bool, + base_url: str, + provider_indent: int, +) -> list[str]: + provider_prefix = " " * provider_indent + key_prefix = provider_prefix + " " + nested_prefix = key_prefix + " " + server_prefix = nested_prefix + " " + model_prefix = server_prefix + " " + model_nested_prefix = model_prefix + " " + + binary_path = _path_for_config(layout.binary_path, repo_root) + model_path = _path_for_config(layout.model_path, repo_root) + server_config_path = _path_for_config(layout.server_config_path, repo_root) + models_root = _path_for_config(layout.runtime_base, repo_root) + shared_scratch_dir = _path_for_config(layout.shared_scratch_dir, repo_root) + return [ + f"{provider_prefix}{PROVIDER_NAME}:", + f"{key_prefix}enabled: {'true' if enable_provider else 'false'}", + f'{key_prefix}backend: "cuda"', + f'{key_prefix}base_url: "{base_url}"', + f'{key_prefix}model: "audio-cpp/pocket-tts"', + f'{key_prefix}model_path: "{model_path}"', + f'{key_prefix}binary_path: "{binary_path}"', + f"{key_prefix}device: cuda", + f"{key_prefix}timeout: 300", + f"{key_prefix}sample_rate: 24000", + f"{key_prefix}max_concurrent_generations: 1", + f"{key_prefix}auto_download: false", + f"{key_prefix}extra_params:", + f"{nested_prefix}managed: true", + f"{nested_prefix}allow_remote_base_url: false", + f'{nested_prefix}external_voice_reference_mode: "disabled"', + f"{nested_prefix}retain_request_artifacts: false", + f"{nested_prefix}server:", + f'{server_prefix}host: "127.0.0.1"', + f"{server_prefix}port: 8080", + f"{server_prefix}autoselect_port: true", + f"{server_prefix}port_probe_max: 10", + f"{server_prefix}startup_timeout_seconds: 30", + f"{server_prefix}healthcheck_interval_seconds: 0.25", + f"{server_prefix}startup_backoff_seconds: 5", + f"{server_prefix}idle_shutdown_seconds: 900", + f"{server_prefix}terminate_timeout_seconds: 10", + f'{server_prefix}server_config_path: "{server_config_path}"', + f'{server_prefix}models_root: "{models_root}"', + f'{server_prefix}shared_scratch_dir: "{shared_scratch_dir}"', + f"{server_prefix}lazy_load: true", + f"{server_prefix}device: 0", + f"{server_prefix}threads: 1", + f"{server_prefix}model:", + f'{model_prefix}id: "pocket-tts"', + f'{model_prefix}family: "pocket_tts"', + f'{model_prefix}path: "{model_path}"', + f'{model_prefix}task: "tts"', + f'{model_prefix}mode: "offline"', + f"{model_prefix}load_options:", + f'{model_nested_prefix}language: "english"', + f"{model_prefix}session_options:", + f'{model_nested_prefix}language: "english"', + f"{nested_prefix}request_option_allowlist:", + f'{server_prefix}- "max_tokens"', + f'{server_prefix}- "seed"', + ] + + +def patch_tts_config( + *, + config_path: Path, + layout: AudioCppRuntimeLayout, + repo_root: Path | None = None, + enable_provider: bool = False, + base_url: str = "http://127.0.0.1:8080", +) -> bool: + """Patch only the audio_cpp provider block.""" + if not config_path.exists(): + logger.warning("Config file not found at {}; skipping update.", config_path) + return False + + lines = config_path.read_text(encoding="utf-8").splitlines() + block_start, block_end, block_indent = _find_provider_block(lines, PROVIDER_NAME) + if block_indent is None: + insert_at, provider_indent = _find_providers_insert_at(lines) + else: + insert_at = block_start if block_start is not None else len(lines) + provider_indent = block_indent + + block_lines = _render_provider_block( + layout=layout, + repo_root=repo_root, + enable_provider=enable_provider, + base_url=base_url, + provider_indent=provider_indent, + ) + if block_start is not None and block_end is not None: + lines[block_start:block_end] = block_lines + else: + lines[insert_at:insert_at] = block_lines + + config_path.write_text("\n".join(lines) + "\n", encoding="utf-8") + logger.info("Updated audio.cpp provider configuration at {}", config_path) + return True + + +def build_clone_command(repo_url: str, source_dir: Path) -> list[str]: + return ["git", "clone", "--depth", "1", repo_url, str(source_dir)] + + +def build_cmake_configure_command( + *, + source_dir: Path, + build_dir: Path, + install_dir: Path, + backend: str = "cuda", +) -> list[str]: + return [ + "cmake", + "-S", + str(source_dir), + "-B", + str(build_dir), + f"-DCMAKE_INSTALL_PREFIX={install_dir}", + f"-DAUDIOCPP_BACKEND={backend}", + ] + + +def build_cmake_build_command(build_dir: Path) -> list[str]: + return ["cmake", "--build", str(build_dir), "--config", "Release"] + + +def build_model_manager_command( + *, + source_dir: Path, + package_id: str, + models_root: Path, + python_executable: str = sys.executable, +) -> list[str]: + return [ + python_executable, + str(source_dir / "tools" / "model_manager.py"), + "install", + package_id, + "--models-root", + str(models_root), + ] + + +def _run_checked(command: Sequence[str]) -> None: + subprocess.run([str(part) for part in command], check=True) # nosec B603 + + +def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Install/configure the optional audio.cpp TTS runtime") + parser.add_argument("--repo-url", default=DEFAULT_REPO_URL) + parser.add_argument("--runtime-base", default=str(DEFAULT_RUNTIME_BASE)) + parser.add_argument("--source-dir") + parser.add_argument("--build-dir") + parser.add_argument("--config-path", default=str(DEFAULT_CONFIG_PATH)) + parser.add_argument("--backend", default="cuda") + parser.add_argument("--package-id", default="pocket-tts") + parser.add_argument("--enable-provider", action="store_true") + parser.add_argument("--clone", action="store_true", help="Clone audio.cpp") + parser.add_argument("--configure", action="store_true", help="Run cmake configure") + parser.add_argument("--build", action="store_true", help="Run cmake build") + parser.add_argument("--install-model", action="store_true", help="Run upstream model_manager.py install") + parser.add_argument("--patch-config", action="store_true", help="Patch tts_providers_config.yaml") + return parser.parse_args(argv) + + +def main(argv: Sequence[str] | None = None) -> int: + args = parse_args(argv) + repo_root = resolve_repo_root() + layout = build_runtime_layout(Path(args.runtime_base), repo_root=repo_root) + source_dir = Path(args.source_dir).expanduser() if args.source_dir else layout.source_dir + build_dir = Path(args.build_dir).expanduser() if args.build_dir else layout.build_dir + + if args.clone: + _run_checked(build_clone_command(args.repo_url, source_dir)) + if args.configure: + _run_checked( + build_cmake_configure_command( + source_dir=source_dir, + build_dir=build_dir, + install_dir=layout.runtime_base, + backend=args.backend, + ) + ) + if args.build: + _run_checked(build_cmake_build_command(build_dir)) + if args.install_model: + _run_checked( + build_model_manager_command( + source_dir=source_dir, + package_id=args.package_id, + models_root=layout.runtime_base, + ) + ) + if args.patch_config: + patch_tts_config( + config_path=repo_root / Path(args.config_path), + layout=layout, + repo_root=repo_root, + enable_provider=args.enable_provider, + ) + + logger.info("audio.cpp runtime layout: {}", layout.runtime_base) + return 0 + + +if __name__ == "__main__": # pragma: no cover - CLI entrypoint + raise SystemExit(main()) diff --git a/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md index c36e5290e6..c3cd0b52ce 100644 --- a/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md +++ b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md @@ -14,18 +14,23 @@ references: documentation: - docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md modified_files: +- Docs/STT-TTS/TTS-SETUP-GUIDE.md - Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md - backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md +- Helper_Scripts/install_tts_audio_cpp.py - tldw_Server_API/Config_Files/tts_providers_config.yaml - tldw_Server_API/app/core/TTS/adapter_registry.py +- tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py - tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py - tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py -- tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py +- tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py - tldw_Server_API/tests/TTS_NEW/fixtures/empty_config.txt - tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py - tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py - tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py - tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py +- tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py +- tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py - tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py - tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py --- @@ -41,8 +46,8 @@ Implement the accepted Approach A design for `0xShug0/audio.cpp`: a disabled-by- - [x] #1 `audio_cpp` is registered as a first-class TTS provider with explicit aliases, namespaced model aliases, disabled-by-default config, and no regression to existing `pocket_tts` routing. - [x] #2 The adapter/client can synthesize through `audiocpp_server` using the existing `/api/v1/audio/speech` flow, with tested request translation, WAV response handling, one-shot streaming compatibility, format conversion handoff, and sanitized error mapping. - [x] #3 Reference-audio and option passthrough behavior is safe by default: loopback-only base URLs unless explicitly allowed, external reference audio disabled unless configured, server-local scratch paths constrained, and only allowlisted scalar options sent upstream. -- [ ] #4 Managed sidecar support can render upstream server config, choose a loopback port, wait for health, avoid tight restart loops, and shut down cleanly without exposing arbitrary command args or process output. -- [ ] #5 Installer/setup helpers and documentation cover explicit clone/build/config/model steps without silent network downloads during normal server startup or inference. +- [x] #4 Managed sidecar support can render upstream server config, choose a loopback port, wait for health, avoid tight restart loops, and shut down cleanly without exposing arbitrary command args or process output. +- [x] #5 Installer/setup helpers and documentation cover explicit clone/build/config/model steps without silent network downloads during normal server startup or inference. - [ ] #6 Focused pytest, Ruff, and Bandit verification are recorded for the touched implementation scope. @@ -78,6 +83,12 @@ Follow `docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation- - Stage 3 adjacent regression: `..\..\.venv\Scripts\python.exe -m pytest -q tldw_Server_API/tests/TTS_NEW/unit/test_fish_s2_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_pocket_tts_cpp_registry.py` passed with 5 passed, 6 warnings in 64.60s. - Stage 3 Ruff check passed for `audio_cpp_adapter.py`, `test_audio_cpp_adapter.py`, and `test_audio_cpp_tts_service.py`. - Stage 3 post-review refactor moved reference-audio file writes off the event loop; final focused rerun passed with 9 passed, 7 warnings in 76.92s. +- Stage 4 red test: `..\..\.venv\Scripts\python.exe -m pytest -q --tb=short tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py` failed as expected with 9 missing-module/import failures for `audio_cpp_sidecar_supervisor` and `Helper_Scripts.install_tts_audio_cpp`. +- Stage 4 implementation added `AudioCppSidecarSupervisor`, managed-mode adapter startup wiring, `Helper_Scripts/install_tts_audio_cpp.py`, sidecar/installer tests, and `Docs/STT-TTS/TTS-SETUP-GUIDE.md` setup guidance. +- Stage 4 green test: the same sidecar/installer command passed with 9 passed, 6 warnings in 71.75s. +- Stage 4 self-review found that omitting an explicit subprocess environment would inherit parent secrets. Added a failing sidecar test that set `HF_TOKEN` and `OPENAI_API_KEY` and expected them not to be passed to the child process; it failed with missing `env`, then passed after adding an allowlisted sidecar env. +- Stage 4 Ruff check passed for `audio_cpp_sidecar_supervisor.py`, `audio_cpp_adapter.py`, `install_tts_audio_cpp.py`, `test_audio_cpp_sidecar_supervisor.py`, and `test_audio_cpp_installer.py` after env hardening. +- Stage 4 adapter regression: `..\..\.venv\Scripts\python.exe -m pytest -q --tb=short tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py` passed with 18 passed, 9 warnings in 85.71s. diff --git a/tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py b/tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py index 14b00dae6e..683573e653 100644 --- a/tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py +++ b/tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py @@ -22,6 +22,7 @@ from .audio_cpp_client import AudioCppClient, AudioCppSpeechResult from .audio_cpp_config import PROVIDER_KEY as AUDIO_CPP_PROVIDER_KEY from .audio_cpp_config import AudioCppConfig, filter_request_options +from .audio_cpp_sidecar_supervisor import AudioCppSidecarSupervisor from .base import ( AudioFormat, ProviderStatus, @@ -76,6 +77,8 @@ def __init__(self, config: dict[str, Any] | None = None): self.max_text_length = int(cfg.get("max_text_length") or 5000) self._client: Any | None = cfg.get("client") or cfg.get("_client") self._owns_client = self._client is None + self._sidecar_supervisor: Any | None = cfg.get("sidecar_supervisor") or cfg.get("_sidecar_supervisor") + self._owns_sidecar_supervisor = self._sidecar_supervisor is None self._available_models: list[str] = [] self._voices = self._parse_voice_catalog(cfg) @@ -104,8 +107,17 @@ async def ensure_initialized(self) -> bool: async def initialize(self) -> bool: if self._client is None: + base_url = self._audio_cpp_config.base_url + if self._audio_cpp_config.managed: + if self._sidecar_supervisor is None: + self._sidecar_supervisor = AudioCppSidecarSupervisor( + self.config, + repo_root=self._audio_cpp_config.repo_root, + ) + self._owns_sidecar_supervisor = True + base_url = await self._sidecar_supervisor.ensure_started() self._client = AudioCppClient( - base_url=self._audio_cpp_config.base_url, + base_url=base_url, timeout=float(self._audio_cpp_config.timeout), allow_remote_base_url=self._audio_cpp_config.allow_remote_base_url, ) @@ -187,14 +199,19 @@ async def generate(self, request: TTSRequest) -> TTSResponse: async def _cleanup_resources(self) -> None: client = self._client - if client is None or not self._owns_client: - return - close = getattr(client, "close", None) - if not callable(close): - return - maybe_close = close() - if inspect.isawaitable(maybe_close): - await maybe_close + if client is not None and self._owns_client: + close = getattr(client, "close", None) + if callable(close): + maybe_close = close() + if inspect.isawaitable(maybe_close): + await maybe_close + supervisor = self._sidecar_supervisor + if supervisor is not None and self._owns_sidecar_supervisor: + shutdown = getattr(supervisor, "shutdown", None) + if callable(shutdown): + maybe_shutdown = shutdown() + if inspect.isawaitable(maybe_shutdown): + await maybe_shutdown def _parse_voice_catalog(self, config: dict[str, Any]) -> dict[str, VoiceInfo]: extras = config.get("extra_params") if isinstance(config.get("extra_params"), dict) else {} diff --git a/tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py b/tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py new file mode 100644 index 0000000000..3ee0a215a3 --- /dev/null +++ b/tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py @@ -0,0 +1,329 @@ +"""Managed sidecar supervisor for audiocpp_server.""" + +from __future__ import annotations + +import asyncio +import contextlib +import json +import os +import socket +import time +from pathlib import Path +from typing import Any + +from loguru import logger + +from ..tts_exceptions import TTSError, TTSProviderInitializationError +from .audio_cpp_client import AudioCppClient +from .audio_cpp_config import PROVIDER_KEY, AudioCppConfig, validate_managed_host + +_SUBPROCESS_ENV_ALLOWLIST = { + "COMSPEC", + "CUDA_HOME", + "CUDA_PATH", + "CUDA_VISIBLE_DEVICES", + "DYLD_LIBRARY_PATH", + "HIP_PATH", + "LD_LIBRARY_PATH", + "NVIDIA_DRIVER_CAPABILITIES", + "NVIDIA_VISIBLE_DEVICES", + "PATH", + "PATHEXT", + "ROCM_PATH", + "SystemRoot", + "TEMP", + "TMP", + "WINDIR", +} + + +def is_port_free(host: str, port: int) -> bool: + """Return True when a TCP bind probe succeeds for host:port.""" + family = socket.AF_INET6 if ":" in host else socket.AF_INET + with socket.socket(family, socket.SOCK_STREAM) as sock: + sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + try: + sock.bind((host, int(port))) + except OSError: + return False + return True + + +class AudioCppSidecarSupervisor: + """Own the lifecycle of an optional loopback audiocpp_server process.""" + + def __init__(self, provider_config: dict[str, Any], repo_root: Path | None = None) -> None: + self._provider_config = dict(provider_config or {}) + self._repo_root = Path(repo_root or Path.cwd()).resolve(strict=False) + self._audio_cpp_config = AudioCppConfig.from_provider_config( + self._provider_config, + repo_root=self._repo_root, + ) + self._server = dict(self._audio_cpp_config.server or {}) + self._host = validate_managed_host(self._server.get("host")) + self._start_port = int(self._server.get("port") or 8080) + self._autoselect_port = self._as_bool(self._server.get("autoselect_port"), default=True) + self._port_probe_max = max(0, int(self._server.get("port_probe_max") or 10)) + self._startup_timeout_seconds = float(self._server.get("startup_timeout_seconds") or 30.0) + self._healthcheck_interval_seconds = float(self._server.get("healthcheck_interval_seconds") or 0.25) + self._startup_backoff_seconds = float(self._server.get("startup_backoff_seconds") or 5.0) + self._idle_shutdown_seconds = float(self._server.get("idle_shutdown_seconds") or 900.0) + self._terminate_timeout_seconds = float(self._server.get("terminate_timeout_seconds") or 10.0) + self._binary_path = self._resolve_repo_path(self._provider_config.get("binary_path")) + self.server_config_path = self._resolve_server_config_path() + self._process: asyncio.subprocess.Process | None = None + self._client: Any | None = None + self._port: int | None = None + self._base_url: str | None = None + self._last_failure_at: float | None = None + self._last_activity_at: float | None = None + self._lock = asyncio.Lock() + + @property + def port(self) -> int | None: + return self._port + + @property + def base_url(self) -> str | None: + return self._base_url + + @property + def last_failure_at(self) -> float | None: + return self._last_failure_at + + async def ensure_started(self) -> str: + """Start the sidecar when needed and return its loopback base URL.""" + async with self._lock: + if self._is_process_running() and self._base_url: + if await self._shutdown_if_idle_locked(): + logger.debug("Restarting audio.cpp sidecar after idle shutdown") + else: + self._last_activity_at = time.time() + return self._base_url + + if self._last_failure_at is not None: + elapsed = time.time() - self._last_failure_at + if elapsed < self._startup_backoff_seconds: + raise TTSProviderInitializationError( + "audio.cpp sidecar startup is backing off after a recent failure", + provider=PROVIDER_KEY, + error_code="SIDECAR_BACKOFF", + ) + + selected_port = self._select_port() + self._write_server_config(selected_port) + base_url = self._build_base_url(self._host, selected_port) + try: + self._port = selected_port + self._base_url = base_url + self._process = await self._spawn_sidecar() + self._client = AudioCppClient( + base_url=base_url, + timeout=max(self._healthcheck_interval_seconds, 0.01), + allow_remote_base_url=False, + ) + await self._wait_for_ready() + except Exception as exc: + self._record_failure() + await self._stop_process_locked() + await self._close_client() + raise TTSProviderInitializationError( + "audio.cpp sidecar did not reach /health", + provider=PROVIDER_KEY, + error_code="SIDECAR_STARTUP_FAILED", + ) from exc + + self._last_activity_at = time.time() + return base_url + + async def shutdown_if_idle(self) -> bool: + async with self._lock: + return await self._shutdown_if_idle_locked() + + async def shutdown(self) -> None: + async with self._lock: + await self._stop_process_locked() + await self._close_client() + + @staticmethod + def _as_bool(value: Any, *, default: bool = False) -> bool: + if value is None: + return default + if isinstance(value, bool): + return value + if isinstance(value, str): + normalized = value.strip().lower() + if normalized in {"1", "true", "yes", "on"}: + return True + if normalized in {"0", "false", "no", "off"}: + return False + return bool(value) + + def _resolve_repo_path(self, value: Any) -> Path: + if value is None or str(value).strip() == "": + raise TTSProviderInitializationError( + "audio.cpp managed mode requires binary_path", + provider=PROVIDER_KEY, + error_code="MISSING_BINARY_PATH", + ) + path = Path(str(value)).expanduser() + if not path.is_absolute(): + path = self._repo_root / path + return path.resolve(strict=False) + + def _resolve_server_config_path(self) -> Path: + configured = self._server.get("server_config_path") or "models/audio_cpp/server.json" + path = Path(str(configured)).expanduser() + if not path.is_absolute(): + path = self._repo_root / path + resolved = path.resolve(strict=False) + models_root = self._audio_cpp_config.models_root + try: + resolved.relative_to(models_root) + except ValueError as exc: + raise TTSProviderInitializationError( + "audio.cpp server_config_path must stay under the audio.cpp runtime root", + provider=PROVIDER_KEY, + error_code="SERVER_CONFIG_PATH_OUTSIDE_ROOT", + ) from exc + return resolved + + def _select_port(self) -> int: + if not self._autoselect_port: + if not is_port_free(self._host, self._start_port): + raise TTSProviderInitializationError( + "audio.cpp sidecar configured port is unavailable", + provider=PROVIDER_KEY, + error_code="SIDECAR_PORT_UNAVAILABLE", + ) + return self._start_port + + for offset in range(self._port_probe_max + 1): + candidate = self._start_port + offset + if is_port_free(self._host, candidate): + return candidate + + raise TTSProviderInitializationError( + "audio.cpp sidecar could not find a free loopback port", + provider=PROVIDER_KEY, + error_code="SIDECAR_PORT_UNAVAILABLE", + ) + + def _write_server_config(self, port: int) -> None: + config = self._audio_cpp_config.render_server_config() + config["host"] = self._host + config["port"] = int(port) + self.server_config_path.parent.mkdir(parents=True, exist_ok=True) + self.server_config_path.write_text( + json.dumps(config, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + async def _spawn_sidecar(self) -> asyncio.subprocess.Process: + logger.debug("Starting audio.cpp sidecar on {}:{}", self._host, self._port) + return await asyncio.create_subprocess_exec( + str(self._binary_path), + "--config", + str(self.server_config_path), + cwd=str(self._repo_root), + env=self._build_subprocess_env(), + ) + + def _build_subprocess_env(self) -> dict[str, str]: + env: dict[str, str] = {} + allowed_upper = {key.upper() for key in _SUBPROCESS_ENV_ALLOWLIST} + for key, value in os.environ.items(): + if key.upper() in allowed_upper and value: + env[key] = value + return env + + async def _wait_for_ready(self) -> None: + if self._client is None: + raise RuntimeError("audio.cpp sidecar health client is not initialized") + deadline = asyncio.get_running_loop().time() + self._startup_timeout_seconds + + while asyncio.get_running_loop().time() < deadline: + if self._process is not None and self._process.returncode is not None: + raise RuntimeError("audio.cpp sidecar exited during startup") + health = await self._probe_health() + if health: + return + await asyncio.sleep(self._healthcheck_interval_seconds) + + raise RuntimeError("audio.cpp sidecar did not reach /health") + + async def _probe_health(self) -> bool: + try: + payload = await self._client.health() + except (TTSError, RuntimeError) as exc: + logger.debug("audio.cpp sidecar health probe failed: {}", type(exc).__name__) + return False + if not isinstance(payload, dict): + return False + status = str(payload.get("status") or payload.get("state") or "ok").strip().lower() + return status not in {"error", "failed", "unhealthy"} + + async def _shutdown_if_idle_locked(self) -> bool: + if self._idle_shutdown_seconds <= 0 or self._last_activity_at is None: + return False + if (time.time() - self._last_activity_at) < self._idle_shutdown_seconds: + return False + if self._process is None or self._process.returncode is not None: + self._clear_process_state() + return False + await self._stop_process_locked() + return True + + def _is_process_running(self) -> bool: + return self._process is not None and self._process.returncode is None + + def _record_failure(self) -> None: + self._last_failure_at = time.time() + + async def _close_client(self) -> None: + client = self._client + self._client = None + if client is None: + return + close = getattr(client, "close", None) + if callable(close): + maybe_close = close() + if hasattr(maybe_close, "__await__"): + await maybe_close + + async def _stop_process_locked(self) -> None: + process = self._process + if process is None: + self._clear_process_state() + return + if process.returncode is not None: + self._clear_process_state(process) + return + with contextlib.suppress(ProcessLookupError): + process.terminate() + try: + await asyncio.wait_for(process.wait(), timeout=self._terminate_timeout_seconds) + except asyncio.TimeoutError: + with contextlib.suppress(ProcessLookupError): + process.kill() + with contextlib.suppress(Exception): + await process.wait() + finally: + self._clear_process_state(process) + + def _clear_process_state(self, process: asyncio.subprocess.Process | None = None) -> None: + if process is not None and self._process is not process: + return + self._process = None + self._base_url = None + self._port = None + self._last_activity_at = None + + @staticmethod + def _build_base_url(host: str, port: int) -> str: + if ":" in host and not host.startswith("["): + return f"http://[{host}]:{port}" + return f"http://{host}:{port}" + + +__all__ = ["AudioCppSidecarSupervisor", "is_port_free"] diff --git a/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py b/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py new file mode 100644 index 0000000000..912bef7ad0 --- /dev/null +++ b/tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py @@ -0,0 +1,215 @@ +from __future__ import annotations + +import json +import shutil +from pathlib import Path + +import pytest + +from tldw_Server_API.app.core.TTS.tts_exceptions import ( + TTSProviderInitializationError, + TTSValidationError, +) + + +class _FakeProcess: + def __init__(self, *, stderr_text: str = "") -> None: + self.returncode = None + self.terminate_called = False + self.kill_called = False + self.stderr_text = stderr_text + + async def wait(self) -> int: + self.returncode = 0 + return 0 + + def terminate(self) -> None: + self.terminate_called = True + self.returncode = 0 + + def kill(self) -> None: + self.kill_called = True + self.returncode = -9 + + +class _ReadyClient: + def __init__(self, *, base_url: str, **_kwargs) -> None: + self.base_url = base_url + self.health_calls = 0 + + async def health(self) -> dict[str, str]: + self.health_calls += 1 + return {"status": "ok"} + + async def close(self) -> None: + return None + + +class _NeverReadyClient: + def __init__(self, *, base_url: str, **_kwargs) -> None: + self.base_url = base_url + + async def health(self) -> dict[str, str]: + raise RuntimeError( + "raw stderr: token=secret C:/Users/GDesktop-1/Working/tldw/models/audio_cpp/server.json" + ) + + async def close(self) -> None: + return None + + +def _workspace_test_dir(name: str) -> Path: + root = Path.cwd() / "models" / "audio_cpp" / "test_artifacts" / name + if root.exists(): + shutil.rmtree(root) + root.mkdir(parents=True, exist_ok=True) + return root + + +def _provider_config(test_root: Path, *, host: str = "127.0.0.1") -> dict[str, object]: + return { + "base_url": "http://127.0.0.1:8080", + "model": "audio-cpp/pocket-tts", + "model_path": "models/audio_cpp/pocket-tts", + "binary_path": str(test_root / "bin" / "audiocpp_server"), + "timeout": 300, + "extra_params": { + "managed": True, + "allow_remote_base_url": False, + "server": { + "host": host, + "port": 8080, + "autoselect_port": True, + "port_probe_max": 3, + "startup_timeout_seconds": 0.05, + "healthcheck_interval_seconds": 0.01, + "startup_backoff_seconds": 5, + "idle_shutdown_seconds": 1, + "terminate_timeout_seconds": 0.1, + "server_config_path": "models/audio_cpp/server.json", + "models_root": "models/audio_cpp", + "shared_scratch_dir": "models/audio_cpp/runtime/scratch", + "lazy_load": True, + "device": 0, + "threads": 1, + "model": { + "id": "pocket-tts", + "family": "pocket_tts", + "path": "models/audio_cpp/pocket-tts", + "task": "tts", + "mode": "offline", + }, + }, + }, + } + + +@pytest.mark.unit +def test_sidecar_rejects_non_loopback_host(): + from tldw_Server_API.app.core.TTS.adapters.audio_cpp_sidecar_supervisor import ( + AudioCppSidecarSupervisor, + ) + + test_root = _workspace_test_dir("sidecar_rejects_non_loopback") + + with pytest.raises(TTSValidationError, match="loopback"): + AudioCppSidecarSupervisor( + _provider_config(test_root, host="0.0.0.0"), + repo_root=test_root, + ) + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_sidecar_autoselects_port_renders_config_and_uses_fixed_command(monkeypatch): + from tldw_Server_API.app.core.TTS.adapters import audio_cpp_sidecar_supervisor as supervisor_module + from tldw_Server_API.app.core.TTS.adapters.audio_cpp_sidecar_supervisor import AudioCppSidecarSupervisor + + probed_ports: list[int] = [] + spawned: list[tuple[tuple[object, ...], dict[str, object]]] = [] + test_root = _workspace_test_dir("sidecar_autoselect") + + def _fake_is_port_free(host: str, port: int) -> bool: + assert host == "127.0.0.1" + probed_ports.append(port) + return port == 8082 + + async def _fake_spawn(*args, **kwargs): + spawned.append((args, kwargs)) + return _FakeProcess() + + monkeypatch.setattr(supervisor_module, "is_port_free", _fake_is_port_free, raising=True) + monkeypatch.setattr(supervisor_module.asyncio, "create_subprocess_exec", _fake_spawn, raising=True) + monkeypatch.setattr(supervisor_module, "AudioCppClient", _ReadyClient, raising=True) + monkeypatch.setenv("HF_TOKEN", "secret-token") + monkeypatch.setenv("OPENAI_API_KEY", "secret-key") + + supervisor = AudioCppSidecarSupervisor(_provider_config(test_root), repo_root=test_root) + + base_url = await supervisor.ensure_started() + + assert base_url == "http://127.0.0.1:8082" + assert supervisor.port == 8082 + assert probed_ports == [8080, 8081, 8082] + assert len(spawned) == 1 + + command, kwargs = spawned[0] + assert command == (str(test_root / "bin" / "audiocpp_server"), "--config", str(supervisor.server_config_path)) + assert kwargs["cwd"] == str(test_root) + assert "HF_TOKEN" not in kwargs["env"] + assert "OPENAI_API_KEY" not in kwargs["env"] + + server_config = json.loads(supervisor.server_config_path.read_text(encoding="utf-8")) + assert server_config["host"] == "127.0.0.1" + assert server_config["port"] == 8082 + assert server_config["models"][0]["id"] == "pocket-tts" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_startup_timeout_terminates_process_records_backoff_and_sanitizes_error(monkeypatch): + from tldw_Server_API.app.core.TTS.adapters import audio_cpp_sidecar_supervisor as supervisor_module + from tldw_Server_API.app.core.TTS.adapters.audio_cpp_sidecar_supervisor import AudioCppSidecarSupervisor + + process = _FakeProcess(stderr_text="token=secret full local path") + test_root = _workspace_test_dir("sidecar_timeout") + + async def _fake_spawn(*args, **kwargs): # noqa: ARG001 + return process + + monkeypatch.setattr(supervisor_module, "is_port_free", lambda host, port: True, raising=True) + monkeypatch.setattr(supervisor_module.asyncio, "create_subprocess_exec", _fake_spawn, raising=True) + monkeypatch.setattr(supervisor_module, "AudioCppClient", _NeverReadyClient, raising=True) + + supervisor = AudioCppSidecarSupervisor(_provider_config(test_root), repo_root=test_root) + + with pytest.raises(TTSProviderInitializationError) as excinfo: + await supervisor.ensure_started() + + assert process.terminate_called is True + assert supervisor.last_failure_at is not None + message = str(excinfo.value) + assert "token=secret" not in message + assert str(supervisor.server_config_path) not in message + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_shutdown_if_idle_stops_live_process_once(): + from tldw_Server_API.app.core.TTS.adapters.audio_cpp_sidecar_supervisor import ( + AudioCppSidecarSupervisor, + ) + + test_root = _workspace_test_dir("sidecar_idle_shutdown") + supervisor = AudioCppSidecarSupervisor(_provider_config(test_root), repo_root=test_root) + process = _FakeProcess() + supervisor._process = process + supervisor._base_url = "http://127.0.0.1:8080" + supervisor._last_activity_at = 0 + + first = await supervisor.shutdown_if_idle() + second = await supervisor.shutdown_if_idle() + + assert first is True + assert second is False + assert process.terminate_called is True diff --git a/tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py b/tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py new file mode 100644 index 0000000000..fa58061ab7 --- /dev/null +++ b/tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py @@ -0,0 +1,156 @@ +from __future__ import annotations + +import shutil +from pathlib import Path + +import pytest + + +def _workspace_test_dir(name: str) -> Path: + root = Path.cwd() / "models" / "audio_cpp" / "test_artifacts" / name + if root.exists(): + shutil.rmtree(root) + root.mkdir(parents=True, exist_ok=True) + return root + + +@pytest.mark.unit +def test_audio_cpp_installer_builds_repo_local_runtime_layout(): + from Helper_Scripts.install_tts_audio_cpp import build_runtime_layout + + repo_root = Path(__file__).resolve().parents[4] + + layout = build_runtime_layout(Path("models") / "audio_cpp", repo_root=repo_root, platform_name="linux") + + assert layout.provider_name == "audio_cpp" + assert layout.runtime_base.relative_to(repo_root).as_posix() == "models/audio_cpp" + assert layout.binary_path.relative_to(repo_root).as_posix() == "bin/audiocpp_server" + assert layout.model_path.relative_to(repo_root).as_posix() == "models/audio_cpp/pocket-tts" + assert layout.server_config_path.relative_to(repo_root).as_posix() == "models/audio_cpp/server.json" + assert layout.shared_scratch_dir.relative_to(repo_root).as_posix() == "models/audio_cpp/runtime/scratch" + + +@pytest.mark.unit +def test_audio_cpp_installer_patches_config_without_enabling_provider_by_default(): + from Helper_Scripts.install_tts_audio_cpp import build_runtime_layout, patch_tts_config + + test_root = _workspace_test_dir("installer_patch_disabled") + config_path = test_root / "tts_providers_config.yaml" + config_path.write_text( + """ +providers: + audio_cpp: + enabled: false + base_url: "http://127.0.0.1:8080" + extra_params: + managed: false +""".strip() + + "\n", + encoding="utf-8", + ) + repo_root = test_root / "repo" + layout = build_runtime_layout(Path("models") / "audio_cpp", repo_root=repo_root, platform_name="linux") + + changed = patch_tts_config( + config_path=config_path, + layout=layout, + repo_root=repo_root, + enable_provider=False, + base_url="http://127.0.0.1:9010", + ) + + content = config_path.read_text(encoding="utf-8") + assert changed is True + assert "audio_cpp:\n enabled: false" in content + assert 'base_url: "http://127.0.0.1:9010"' in content + assert 'binary_path: "bin/audiocpp_server"' in content + assert 'model_path: "models/audio_cpp/pocket-tts"' in content + assert 'server_config_path: "models/audio_cpp/server.json"' in content + assert "HF_TOKEN" not in content + assert "api_key" not in content + + +@pytest.mark.unit +def test_audio_cpp_installer_can_enable_provider_when_requested(): + from Helper_Scripts.install_tts_audio_cpp import build_runtime_layout, patch_tts_config + + test_root = _workspace_test_dir("installer_patch_enabled") + config_path = test_root / "tts_providers_config.yaml" + config_path.write_text("providers:\n", encoding="utf-8") + repo_root = test_root / "repo" + layout = build_runtime_layout(Path("models") / "audio_cpp", repo_root=repo_root, platform_name="linux") + + patch_tts_config( + config_path=config_path, + layout=layout, + repo_root=repo_root, + enable_provider=True, + base_url="http://127.0.0.1:8080", + ) + + content = config_path.read_text(encoding="utf-8") + assert "audio_cpp:\n enabled: true" in content + assert "extra_params:\n managed: true" in content + + +@pytest.mark.unit +def test_audio_cpp_installer_builds_explicit_clone_build_and_model_manager_commands(): + from Helper_Scripts.install_tts_audio_cpp import ( + build_clone_command, + build_cmake_build_command, + build_cmake_configure_command, + build_model_manager_command, + ) + + test_root = _workspace_test_dir("installer_commands") + source_dir = test_root / "external" / "audio.cpp" + build_dir = test_root / "build" + install_dir = test_root / "install" + + assert build_clone_command("https://github.com/0xShug0/audio.cpp", source_dir) == [ + "git", + "clone", + "--depth", + "1", + "https://github.com/0xShug0/audio.cpp", + str(source_dir), + ] + assert build_cmake_configure_command( + source_dir=source_dir, + build_dir=build_dir, + install_dir=install_dir, + backend="cuda", + ) == [ + "cmake", + "-S", + str(source_dir), + "-B", + str(build_dir), + f"-DCMAKE_INSTALL_PREFIX={install_dir}", + "-DAUDIOCPP_BACKEND=cuda", + ] + assert build_cmake_build_command(build_dir) == ["cmake", "--build", str(build_dir), "--config", "Release"] + assert build_model_manager_command( + source_dir=source_dir, + package_id="pocket-tts", + models_root=install_dir / "models", + python_executable="python", + ) == [ + "python", + str(source_dir / "tools" / "model_manager.py"), + "install", + "pocket-tts", + "--models-root", + str(install_dir / "models"), + ] + + +@pytest.mark.unit +def test_audio_cpp_installer_builds_windows_binary_layout(): + from Helper_Scripts.install_tts_audio_cpp import build_runtime_layout + + repo_root = Path(__file__).resolve().parents[4] + + layout = build_runtime_layout(Path("models") / "audio_cpp", repo_root=repo_root, platform_name="win32") + + assert layout.binary_path.relative_to(repo_root).as_posix() == "bin/audiocpp_server.exe" From c88b5d954923637d8b64b40162bf1035438bfbdf Mon Sep 17 00:00:00 2001 From: Robert Date: Fri, 3 Jul 2026 16:01:21 -0700 Subject: [PATCH 8/8] chore(tts): close audio_cpp tts task --- ...io-cpp-tts-provider-implementation-plan.md | 14 +++++------ ....cpp-TTS-provider-and-setup-integration.md | 24 ++++++++++++------- 2 files changed, 22 insertions(+), 16 deletions(-) diff --git a/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md index 549d4daeca..13dbbc620a 100644 --- a/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md +++ b/Docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation-plan.md @@ -423,9 +423,9 @@ Expected result after implementation: both test files pass. **Tests:** All files created in this plan plus adjacent registry, config, and service tests. -**Status:** Not Started +**Status:** Complete -- [ ] **Step 1: Run focused test suite** +- [x] **Step 1: Run focused test suite** Run: @@ -444,7 +444,7 @@ python -m pytest -q ` Expected result: all focused tests pass. -- [ ] **Step 2: Run adjacent regression tests** +- [x] **Step 2: Run adjacent regression tests** Run: @@ -458,7 +458,7 @@ python -m pytest -q ` Expected result: adjacent provider routing and installer tests pass. If any named file is absent in this checkout, record the absent path in `TASK-12125` and run the closest existing adjacent test discovered with `rg --files`. -- [ ] **Step 3: Run Ruff on touched Python files** +- [x] **Step 3: Run Ruff on touched Python files** Run after source files exist: @@ -483,7 +483,7 @@ python -m ruff check ` Expected result: Ruff exits 0. -- [ ] **Step 4: Run Bandit on touched implementation files** +- [x] **Step 4: Run Bandit on touched implementation files** Run: @@ -500,7 +500,7 @@ python -m bandit -r ` Expected result: no new high or medium findings in touched code. Fix new findings before closeout. -- [ ] **Step 5: Update task tracking and self-review** +- [x] **Step 5: Update task tracking and self-review** Update `TASK-12125` with: @@ -530,7 +530,7 @@ git diff --check -- ` Expected result: no whitespace errors. -- [ ] **Step 6: Commit scoped slices** +- [x] **Step 6: Commit scoped slices** Prefer small commits by stage. Stage explicit paths only, and do not include unrelated untracked files. diff --git a/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md index c3cd0b52ce..42969cdc3e 100644 --- a/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md +++ b/backlog/tasks/task-12125 - Implement-audio.cpp-TTS-provider-and-setup-integration.md @@ -1,7 +1,7 @@ --- id: TASK-12125 title: Implement audio.cpp TTS provider and setup integration -status: In Progress +status: Done labels: - audio - tts @@ -48,7 +48,7 @@ Implement the accepted Approach A design for `0xShug0/audio.cpp`: a disabled-by- - [x] #3 Reference-audio and option passthrough behavior is safe by default: loopback-only base URLs unless explicitly allowed, external reference audio disabled unless configured, server-local scratch paths constrained, and only allowlisted scalar options sent upstream. - [x] #4 Managed sidecar support can render upstream server config, choose a loopback port, wait for health, avoid tight restart loops, and shut down cleanly without exposing arbitrary command args or process output. - [x] #5 Installer/setup helpers and documentation cover explicit clone/build/config/model steps without silent network downloads during normal server startup or inference. -- [ ] #6 Focused pytest, Ruff, and Bandit verification are recorded for the touched implementation scope. +- [x] #6 Focused pytest, Ruff, and Bandit verification are recorded for the touched implementation scope. ## Implementation Plan @@ -89,6 +89,12 @@ Follow `docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation- - Stage 4 self-review found that omitting an explicit subprocess environment would inherit parent secrets. Added a failing sidecar test that set `HF_TOKEN` and `OPENAI_API_KEY` and expected them not to be passed to the child process; it failed with missing `env`, then passed after adding an allowlisted sidecar env. - Stage 4 Ruff check passed for `audio_cpp_sidecar_supervisor.py`, `audio_cpp_adapter.py`, `install_tts_audio_cpp.py`, `test_audio_cpp_sidecar_supervisor.py`, and `test_audio_cpp_installer.py` after env hardening. - Stage 4 adapter regression: `..\..\.venv\Scripts\python.exe -m pytest -q --tb=short tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py` passed with 18 passed, 9 warnings in 85.71s. +- Stage 5 focused audio.cpp suite: `..\..\.venv\Scripts\python.exe -m pytest -q --tb=short tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_tts_config.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_config.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_client.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_adapter.py tldw_Server_API/tests/TTS_NEW/unit/adapters/test_audio_cpp_sidecar_supervisor.py tldw_Server_API/tests/TTS_NEW/unit/test_audio_cpp_installer.py tldw_Server_API/tests/TTS_NEW/integration/test_audio_cpp_tts_service.py` passed with 34 passed, 14 warnings in 124.60s. +- Stage 5 adjacent regression note: the plan-named `tldw_Server_API/tests/TTS_NEW/unit/adapters/test_pocket_tts_cpp_adapter.py` is absent in this checkout. Substituted `tldw_Server_API/tests/TTS_NEW/unit/test_pocket_tts_cpp_registry.py` plus `test_pocket_tts_cpp_installer.py`. +- Stage 5 adjacent regression: `..\..\.venv\Scripts\python.exe -m pytest -q --tb=short --basetemp .pytest_tmp_adjacent tldw_Server_API/tests/TTS_NEW/unit/test_fish_s2_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_pocket_tts_cpp_registry.py tldw_Server_API/tests/TTS_NEW/unit/test_pocket_tts_cpp_installer.py` passed with 14 passed, 6 warnings in 44.55s. The first substitute run without `--basetemp` failed only because pytest tried to create temp paths under `C:\Users\GDesktop-1\AppData\Local\Temp\pytest-of-GDesktop-1`, outside the writable sandbox. +- Stage 5 Ruff check passed for all touched Python files in the implementation plan. +- Stage 5 Bandit check passed: `..\..\.venv\Scripts\python.exe -m bandit -r tldw_Server_API/app/core/TTS/adapters/audio_cpp_client.py tldw_Server_API/app/core/TTS/adapters/audio_cpp_config.py tldw_Server_API/app/core/TTS/adapters/audio_cpp_sidecar_supervisor.py tldw_Server_API/app/core/TTS/adapters/audio_cpp_adapter.py Helper_Scripts/install_tts_audio_cpp.py -f json -o models/audio_cpp/test_artifacts/bandit_audio_cpp_tts.json`; report summary had 0 results, 0 high, 0 medium, 0 low findings. +- Known limitations: no live `audiocpp_server` smoke test was run in this environment; managed mode remains documented as CUDA-first; generic `/v1/tasks/run`, STT, VAD, diarization, and other Approach C task surfaces are not included in this TTS slice. @@ -96,16 +102,16 @@ Follow `docs/superpowers/plans/2026-07-03-audio-cpp-tts-provider-implementation- -Pending implementation. +Approach A is implemented for TTS: `audio_cpp` is registered as a disabled-by-default provider, routes through the existing `TTSServiceV2` flow, includes client/config/adapter/managed-sidecar support, has explicit setup helpers and docs, and keeps request options, paths, reference audio, process args, and subprocess environment constrained by default. Verification passed for focused audio.cpp tests, adjacent provider regressions, Ruff, and Bandit. Follow-up Approach C work should reuse `AudioCppClient` and `AudioCppSidecarSupervisor` for broader `/v1/tasks/run`-style audio processing surfaces after a real runtime smoke test is available. ## Definition of Done -- [ ] #1 Acceptance criteria completed -- [ ] #2 Tests or verification recorded -- [ ] #3 Documentation updated when relevant -- [ ] #4 Bandit run for touched code when applicable or documented as non-code/environment skip -- [ ] #5 Final summary added -- [ ] #6 Known skips or blockers documented +- [x] #1 Acceptance criteria completed +- [x] #2 Tests or verification recorded +- [x] #3 Documentation updated when relevant +- [x] #4 Bandit run for touched code when applicable or documented as non-code/environment skip +- [x] #5 Final summary added +- [x] #6 Known skips or blockers documented