New Breakthrough in Text-to-Speech: Meta Releases Machine Learning Model Voicebox

2023-06-19

Last week, Meta Platforms' artificial intelligence research division launched a machine learning model called Voicebox that can generate speech from text. It can perform many untrained tasks, including editing, denoising, and style conversion, which other text-to-speech models cannot do. The model was trained using a special method developed by Meta researchers. Due to concerns about potential misuse, Meta did not publicly release Voicebox, but its initial results are promising and have the potential to power numerous applications in the future. Voicebox is a generative model that can synthesize speech in six languages: English, French, Spanish, German, Polish, and Portuguese. Unlike large language models that attempt to learn statistical patterns of words and text sequences, Voicebox learns the mapping between audio samples and their transcriptions. As a result, this model can handle many downstream tasks with minimal or no fine-tuning. "Our goal is to build a unified model that can perform various text-guided speech generation tasks through context learning," Meta researchers described the technical details of Voicebox in their paper. The model was trained using Meta's "flow matching" technique, which is more efficient and generalizable than diffusion-based learning methods used by other generative models. This technique allows Voicebox to learn from various speech data without the need for meticulous annotation. Researchers were able to train Voicebox using 50,000 hours of speech and transcriptions from audiobooks without manual labeling. This model can be applied to many untrained tasks with little or no fine-tuning. For example, it can generate speech for new text using a two-second audio sample. This capability can be used for voice generation for non-speaking individuals or customizing the voices of game characters and virtual assistants. Voicebox can also perform style conversion in different ways. For instance, by providing two audio and text samples, the model can use the first audio sample as a style reference and modify the second sample to match the sound and intonation of the reference sample. Interestingly, the model can do the same across different languages, which can help people communicate naturally and authentically even if they don't speak the same language. The model can also perform various editing tasks. For example, if there is background noise while recording your voice, you can provide the audio and transcription to Voicebox, and it will mask the parts with background noise. The model uses the transcription to generate the missing parts without background noise. The same technique can be used for editing speech. If you make a mistake while speaking, you can mask that part of the audio sample and pass it along with the edited transcription to Voicebox. The model will generate the missing parts of the new text in a way that matches the surrounding sound and intonation. One interesting application of Voicebox is speech sampling. The model can generate various speech samples from a single text sequence. This capability can be used to generate synthetic data for training other speech processing models. "Our results show that the speech recognition model trained on synthetic speech generated by Voicebox performs almost as well as the model trained on real speech, with an error rate reduction of 1%, compared to the 45% to 70% error rate of synthetic speech generated by previous text-to-speech models," Meta wrote. Voicebox also has limitations. Since it was trained on audiobook data, it does not perform well on informal conversational speech that includes non-linguistic sounds. It also does not have full control over different attributes of generated speech, such as voice style, intonation, emotion, and acoustic conditions. Meta's research team is exploring future technologies to overcome these limitations. The model has not been released to the public due to concerns about the potential misuse of AI-generated content. For example, there have been recent cases where criminals attempted to impersonate someone using AI-generated voices to deceive individuals. Advanced speech synthesis systems like Voicebox could be used for similar purposes or other malicious activities, such as creating fake evidence or manipulating real audio. "As with other powerful new AI innovations, we recognize the potential for abuse and unintended harm that this technology brings," Meta wrote on its AI blog. Due to these concerns, Meta did not release the model but provided technical details about its architecture and training process in the research paper. The paper also includes details about a classifier model that can detect speech and audio generated by Voicebox to mitigate the risks associated with using the model. Source: https://venturebeat.com/ai/meta-announces-voicebox-a-generative-model-for-multiple-voice-synthesis-tasks/

RECENT AI NEWS

RECENT AI TOOLS