Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions docs/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -801,9 +801,18 @@ tts.synthesize("Bonjour tout le monde.", "fr", [](const float* samples, size_t l
generation — the structural fix is an L=128 graph bucket (see follow-ups). A piece too
short to split
that still overflows is trimmed to the window with a log line, as before.
- **Latent buckets.** A bundle may ship extra fixed-L exports of the two L-dependent graphs
next to the base ones — `vector_estimator_L128.tflite` + `vocoder_L128.tflite` (≈ 9 s) — produced
by `speech-models/stmodels/export_litert.py --latent-frames 64 128`. They are discovered at
construction (`latent_buckets()`) and loaded on first use, so memory only grows when a long
piece actually needs one. Each piece runs on the smallest bucket whose window holds its
predicted duration (`choose_latent_bucket()`): short sentences keep the cheap L=64 graph, a
sentence the base window cannot hold is generated in one pass on L=128 — one coherent
prosodic contour instead of a split — and a split only remains for text longer than the
largest bucket. The planner's window is the largest bucket plus the 10% stretch.
- `set_speed()` divides the predicted duration (default 1.05); `set_seed()` fixes the latent
noise for reproducible output; `SUPERTONIC_LATENT_FRAMES` overrides the window only for
experiments against a re-exported graph.
noise for reproducible output; `SUPERTONIC_LATENT_FRAMES` overrides the base window only for
experiments against a re-exported base graph.
- Tests: `tests/test_supertonic_tokenizer.cpp` (built with `SPEECH_CORE_WITH_LITERT=ON`, no
bundle needed) pins the chunking, capacity, continuation-terminator, and window-fitting
behaviour, including the French paragraph from #140.
Expand Down
50 changes: 47 additions & 3 deletions include/speech_core/models/litert_supertonic_tts.h
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
#include "speech_core/models/litert_engine.h"
#include "speech_core/models/supertonic_tokenizer.h"

#include <array>
#include <atomic>
#include <cstdint>
#include <memory>
Expand Down Expand Up @@ -37,6 +38,13 @@ namespace speech_core {
/// tokenized with a trailing "," rather than a sentence-final ".", and the seam between the two
/// is trimmed to a comma-length pause.
///
/// **Latent buckets.** Because L is fixed per graph, a bundle may ship extra exports of the two
/// L-dependent graphs next to the base ones — `vector_estimator_L128.tflite` + `vocoder_L128.tflite`
/// (≈ 9 s), and so on. They are discovered at construction and loaded lazily; each piece runs on
/// the smallest bucket whose window holds its predicted duration, so a long sentence is generated
/// in one pass (one coherent prosodic contour) and a split only remains for text longer than the
/// largest bucket. Short pieces keep using the cheap base graph.
///
/// Validated end-to-end against the ONNX reference at 66–82 dB mag-STFT SNR (en/de/ko); see the
/// Runner repo's `speech-models/stmodels/controlled_ab.py`. Voice is a precomputed style pair
/// (`voice_styles/<id>.json`); on-device voice cloning is out of scope (the style-extractor isn't
Expand Down Expand Up @@ -83,6 +91,14 @@ class LiteRTSupertonicTts : public TTSInterface {
/// Available voice ids loaded from `voice_styles_dir`.
std::vector<std::string> voices() const;

/// Latent-window buckets in the bundle, in frames of 3072 samples, ascending; [0] is the base
/// graph pair (L=64 for the published bundle), the rest are optional `*_L{N}.tflite` siblings.
std::vector<int> latent_buckets() const;

/// Index into `buckets` (ascending frames) of the smallest window that holds `frames`, or the
/// last one when none does (the caller then tempo-fits or truncates). Pure; unit-tested.
static size_t choose_latent_bucket(const std::vector<int>& buckets, int frames);

private:
struct VoiceStyle {
std::vector<float> style_ttl; // [1,50,256] → 12800
Expand All @@ -98,16 +114,44 @@ class LiteRTSupertonicTts : public TTSInterface {
};
Prepared prepare_chunk(const std::string& text, const std::string& language, bool continuation);

// Physical input slot per signature role. The converter permutes a graph's input slots away
// from the signature order (and differently per toolchain version), so slots are resolved from
// the tensor names (`serving_default_args_N`) at load time; see input_slots() in the .cpp.
// duration_predictor roles: 0 text_ids, 1 style_dp, 2 text_mask
// text_encoder roles: 0 text_ids, 1 style_ttl, 2 text_mask
// vector_estimator roles: 0 noisy, 1 text_emb, 2 style_ttl, 3 latent_mask, 4 text_mask,
// 5 current_step, 6 total_step
using Slots3 = std::array<int, 3>;
using Slots7 = std::array<int, 7>;

// One fixed-L export of the two L-dependent graphs. Handles are raw (freed in destroy_graphs()).
struct LatentBucket {
int frames = 0; // L
std::string vector_path, vocoder_path;
LiteRtModel vector_model = nullptr; LiteRtCompiledModel vector_compiled = nullptr;
LiteRtModel vocoder_model = nullptr; LiteRtCompiledModel vocoder_compiled = nullptr;
Slots7 vector_slots{};
bool failed = false; // load attempted and failed; skipped from then on
bool loaded() const { return vector_compiled && vocoder_compiled; }
};

// text_encoder → vector_estimator × N → vocoder on a prepared piece → trimmed 44.1 kHz PCM.
// piece_index decorrelates the latent noise between the pieces of one utterance.
std::vector<float> synth_prepared(const Prepared& prepared, size_t piece_index);
std::vector<float> synth_prepared(const Prepared& prepared, size_t piece_index,
const LatentBucket& bucket);
// Smallest loadable bucket holding `frames` (loads it on first use; falls back on failure).
const LatentBucket& bucket_for(int frames);
void load_bucket(LatentBucket& bucket);
void discover_buckets(const std::string& vector_estimator_path, const std::string& vocoder_path);
const VoiceStyle& current_voice() const;
void destroy_graphs() noexcept; // idempotent; used by the dtor and ctor-failure cleanup

LiteRtModel duration_model_ = nullptr; LiteRtCompiledModel duration_compiled_ = nullptr;
LiteRtModel encoder_model_ = nullptr; LiteRtCompiledModel encoder_compiled_ = nullptr;
LiteRtModel vector_model_ = nullptr; LiteRtCompiledModel vector_compiled_ = nullptr;
LiteRtModel vocoder_model_ = nullptr; LiteRtCompiledModel vocoder_compiled_ = nullptr;
Slots3 duration_slots_{};
Slots3 encoder_slots_{};
std::vector<LatentBucket> buckets_; // ascending frames; [0] = base, loaded in the ctor
bool hw_accel_ = false;

std::unique_ptr<SupertonicTokenizer> tokenizer_;
std::unordered_map<std::string, VoiceStyle> voices_;
Expand Down
Loading
Loading