2024
2023
F5-TTS (A Fairytaler that Fakes Fluent
and Faithful Speech with Flow Matching)
improves upon
E2 TTS (Embarrassingly easy, fully non-autoregressive zero shot TTS)
E2-TTS is a streamlined and highly effective approach to zero-shot text-to-speech synthesis, achieving state-of-the-art results through non-autoregressive techniques and flow matching, without the need for complex alignment models or external duration models.
F5-TTS builds on the success of E2-TTS by introducing several key improvements: Faster Training and Inference: Due to architectural changes like the use of ConvNeXt for text refinement and the elimination of rigid alignment methods. Improved Naturalness and Alignment: Through the introduction of Sway Sampling, which enhances the model’s ability to generate fluent, faithful speech from text. Better Zero-Shot Performance: F5-TTS excels in generating speech with high speaker similarity and naturalness, even in unseen scenarios, surpassing E2-TTS.
A source audio clip for voice cloning is limited to 15 seconds. Outputs are limted to 30 seconds... sort of. The generations are as long as you want, but the audio file is stitched together from multiple 30 second clips that were generated.
Inference time "Sway Sampling" improves the model’s performance by focusing more on early flow steps during speech generation, capturing the rough structure of the target speech early in the process. This leads to better alignment between the text and generated speech, especially in the initial stages of inference. (Listen to me saying this)
https://github.com/SWivid/F5-TTS
One you set it up, there is a gradio reference app shipped with the code. Make sure you check out the 'multi-style' tab where you can upload multiple source audio files for multiple 'emotions' you can ascribe to the audio being generated.