Fine-Tune Other languages

[anlcakmak19]

Hi,
I’m Anıl from the Turkish TTS community. I’ve been fine-tuning several TTS models such as Dia, Orpheus, and KaniTTS. While these models work reasonably well, many of them struggle with live performance, especially under parallel requests and real-time inference speed.

From the outside, your work looks significantly faster (almost x100 in speed), which is why I’m very interested in your approach.

I currently have ~250 hours of Turkish speech data. The data was scraped from YouTube, segmented, and transcribed. Audio durations range from 3 seconds to ~20 seconds.

I have a few questions:

Do you think ~250 hours of Turkish data is sufficient to teach a new language on top of an existing base model?

Are you planning to share the training / fine-tuning code?

Do you have a roadmap for releasing a multilingual model?

Thanks in advance for your work and for sharing your insights.
Looking forward to your response.

[ysharma3501]

@anlcakmak19 It should definitely learn Turkish within that time, although I am not sure it will be SOTA in voice cloning performance. I just released finetuning code, you can try training the model.

Kaggle notebook: https://www.kaggle.com/code/yatharthsharma888/miratts-training

Colab notebook: https://colab.research.google.com/drive/1IprDyaMKaZrIvykMfNrxWFeuvj-DQPII?usp=sharing

[anlcakmak19]

@ysharma3501,
I couldn’t find any speaker (or speaker_id) column in the training code.
Does the current pipeline support multi-speaker training?

For example, the dataset
https://huggingface.co/datasets/WpythonW/elevenlabs_multilingual_v2-technical-speech

appears to be a single-speaker dataset. I’m wondering how multi-speaker scenarios are expected to be handled.

Could you clarify whether multi-speaker training is supported, and if so, how the speaker information should be provided?

[Alukarho]

I was trying to train it to speak Spanish and I made the following discoveries: I used 65 hours of a dataset with 3,500 different speakers to provide good nourishment for the model for cloning. Although its cloning is not as good as that of other models, its greatest strength is real-time performance, which works really well for me. However, I encountered the following issues: although it pronounces most of the words in the dataset perfectly, it has trouble constructing new, unseen words, which makes the model inconsistent for now. I should add that I am not very professional in this field. What I like about the model is that for unseen words I can train LoRAs in 10 minutes that adjust those words, but as you can see, there are too many of them. The model lacks something that would allow it to properly construct unseen words. Using this same dataset, I trained other models such as F5 TTS and Index TTS, and they speak perfectly, perhaps because they are different architectures. Once again, congratulations on the great work. I am very excited that with this model I can finally build a local assistant for my own home. Thank you very much.

[tamerkanak]

Hi, I’m Anıl from the Turkish TTS community. I’ve been fine-tuning several TTS models such as Dia, Orpheus, and KaniTTS. While these models work reasonably well, many of them struggle with live performance, especially under parallel requests and real-time inference speed.

From the outside, your work looks significantly faster (almost x100 in speed), which is why I’m very interested in your approach.

I currently have ~250 hours of Turkish speech data. The data was scraped from YouTube, segmented, and transcribed. Audio durations range from 3 seconds to ~20 seconds.

I have a few questions:

Do you think ~250 hours of Turkish data is sufficient to teach a new language on top of an existing base model?

Are you planning to share the training / fine-tuning code?

Do you have a roadmap for releasing a multilingual model?

Thanks in advance for your work and for sharing your insights. Looking forward to your response.

Which TTS models have you trained for Turkish and obtained very good results with? Also, have you trained MiraTTS and gotten results from it?

[Satish-101]

@Alukarho I’m training an Indic language, mainly Punjabi, and my MiraTTS model hasn’t learned to produce intelligible speech, it’s generating gibberish/noise. Could you share your approach or pointers on how you replaced the original tokenizer with a phoneme-based one?