Meta released a new speech-to-text model SeamlessM4T

2023-08-23

Meta has announced the launch of a multilingual base model called SeamlessM4T, which can understand and generate translations for nearly 100 languages in real time, whether in speech or text.

SeamlessM4T is a multimodal technology that has been publicly released to help researchers build and deploy universal applications for speech-to-speech, speech-to-text, text-to-speech, and text-to-text translation. Alongside SeamlessM4T, Meta has also released SeamlessAlign, a multimodal translation dataset consisting of 265,000 hours of aligned speech and text.

This product launch marks a significant development in linguistics and AI applications, as it is a single system performing multiple tasks in the domains of speech and text. Previously, approaches in this field mostly involved using different systems for different tasks, such as a system specifically designed for speech-to-speech translation.

What can SeamlessM4T do?

Meta explains that SeamlessM4T can implicitly recognize the source language without the need for a separate language identification model. It can detect speech and text in nearly 100 languages and generate text in nearly 100 languages and speech in 36 languages. Interestingly, it can also identify sentences that mix multiple languages and provide translations in a single target language (e.g., translating a sentence mixing Telugu and Hindi into English speech).

In the BLASER 2.0 test, the model performed exceptionally well in noise and speaker variation, showing an average improvement of 37% and 48% compared to the current state-of-the-art models for speech-to-text tasks.

In a blog post, Meta states, "SeamlessM4T outperforms the previous state-of-the-art competitors. We have also significantly improved performance on low-resource languages (occupying less digital space) and demonstrated strong performance on high-resource languages like English."

Once developed, this will lead to a large-scale universal translation system, enabling people speaking different languages to communicate more effectively.

It is worth noting that Google is also making efforts in this direction and has announced the Universal Speech Model (USM), which performs automatic speech recognition (ASR) for widely used and resource-scarce languages.

How does it work?

To achieve this model, Meta mined billions of sentences from public sources and 4 million hours of speech data, aligning them to create the SeamlessAlign dataset. Overall, the company claims to have aligned over 443,000 hours of speech and text and created approximately 29,000 hours of speech-to-speech alignment. Using this data, the company trained a multitask UnitY model to produce the desired multimodal results.

Meta explains, "The multitask UnitY model consists of three main sequential components. The tasks of the text and speech encoders are to recognize input in nearly 100 languages. The text decoder converts the meaning into text in nearly 100 languages, and then the text-to-unit model decodes it into discrete acoustic units for 36 speech languages... Finally, the decoded discrete units are transformed into speech using the multilingual HiFi-GAN unit vocoder."

Not yet perfect

However, it is important to note that SeamlessM4T is still far from perfect. Evaluations have found toxicity (although 63% lower than state-of-the-art models) and gender bias issues in the model.

According to the white paper detailing the technology, when translating from neutral terms, SeamlessM4T tends to favor male forms excessively (with an average preference of around 10%) and lacks robustness when gender differences are around 3%.

Meta states, "We detected toxicity in the input and output in our demonstration results. If toxicity is only detected in the output, it means that toxicity has been added. In such cases, we provide warnings and do not display the output... Regarding bias, we have started evaluating gender bias in large-scale languages. We can now quantify gender bias in dozens of speech translation directions by extending our previously designed multilingual HolisticBias dataset to speech."

The company emphasizes that this is an ongoing effort, and they will continue researching and taking action in these areas to further improve the robustness and safety of the SeamlessM4T model.

RECENT AI NEWS

RECENT AI TOOLS