Last week, Meta Platforms' artificial intelligence research division launched a machine learning model called Voicebox that can generate speech from text. It can perform many untrained tasks, including editing, denoising, and style conversion, which other text-to-speech models cannot do.
The model was trained using a special method developed by Meta researchers. Due to concerns about potential misuse, Meta did not publicly release Voicebox, but its initial results are promising and have the potential to power numerous applications in the future.
Voicebox is a generative model that can synthesize speech in six languages: English, French, Spanish, German, Polish, and Portuguese. Unlike large language models that attempt to learn statistical patterns of words and text sequences, Voicebox learns the mapping between audio samples and their transcriptions.
As a result, this model can handle many downstream tasks with minimal or no fine-tuning. "Our goal is to build a unified model that can perform various text-guided speech generation tasks through context learning," Meta researchers described the technical details of Voicebox in their paper.
The model was trained using Meta's "flow matching" technique, which is more efficient and generalizable than diffusion-based learning methods used by other generative models. This technique allows Voicebox to learn from various speech data without the need for meticulous annotation. Researchers were able to train Voicebox using 50,000 hours of speech and transcriptions from audiobooks without manual labeling.
This model can be applied to many untrained tasks with little or no fine-tuning. For example, it can generate speech for new text using a two-second audio sample. This capability can be used for voice generation for non-speaking individuals or customizing the voices of game characters and virtual assistants.
Voicebox can also perform style conversion in different ways. For instance, by providing two audio and text samples, the model can use the first audio sample as a style reference and modify the second sample to match the sound and intonation of the reference sample. Interestingly, the model can do the same across different languages, which can help people communicate naturally and authentically even if they don't speak the same language.
The model can also perform various editing tasks. For example, if there is background noise while recording your voice, you can provide the audio and transcription to Voicebox, and it will mask the parts with background noise. The model uses the transcription to generate the missing parts without background noise.
The same technique can be used for editing speech. If you make a mistake while speaking, you can mask that part of the audio sample and pass it along with the edited transcription to Voicebox. The model will generate the missing parts of the new text in a way that matches the surrounding sound and intonation.
One interesting application of Voicebox is speech sampling. The model can generate various speech samples from a single text sequence. This capability can be used to generate synthetic data for training other speech processing models. "Our results show that the speech recognition model trained on synthetic speech generated by Voicebox performs almost as well as the model trained on real speech, with an error rate reduction of 1%, compared to the 45% to 70% error rate of synthetic speech generated by previous text-to-speech models," Meta wrote.
Voicebox also has limitations. Since it was trained on audiobook data, it does not perform well on informal conversational speech that includes non-linguistic sounds. It also does not have full control over different attributes of generated speech, such as voice style, intonation, emotion, and acoustic conditions. Meta's research team is exploring future technologies to overcome these limitations.
The model has not been released to the public due to concerns about the potential misuse of AI-generated content. For example, there have been recent cases where criminals attempted to impersonate someone using AI-generated voices to deceive individuals. Advanced speech synthesis systems like Voicebox could be used for similar purposes or other malicious activities, such as creating fake evidence or manipulating real audio.
"As with other powerful new AI innovations, we recognize the potential for abuse and unintended harm that this technology brings," Meta wrote on its AI blog. Due to these concerns, Meta did not release the model but provided technical details about its architecture and training process in the research paper. The paper also includes details about a classifier model that can detect speech and audio generated by Voicebox to mitigate the risks associated with using the model.
Source: https://venturebeat.com/ai/meta-announces-voicebox-a-generative-model-for-multiple-voice-synthesis-tasks/
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.