ElevenLabs Launches AI Dubbing: An Application for Easily Generating Multilingual Videos and Audios

ElevenLabs, a voice cloning and synthesis startup founded by former Google and Palantir employees, recently announced the launch of AI Dubbing, a professional product that can translate any speech, including long-form content, into over 20 different languages. This service is available to all platform users and offers a new way to dub audio and video content, disrupting an industry that has long relied on manual labor. It also breaks down language barriers and provides assistance to small content creators who may not have the resources to hire human translators. "We have collaborated with hundreds of content creators to test and iterate this feature, making it easier for their content to be accepted by a wider audience," said Mati Staniszewski, CEO and co-founder of ElevenLabs. "We see tremendous potential for independent creators, from video and podcast producers to film and television production companies." ElevenLabs claims that this feature can provide high-quality translated speech within minutes (depending on the length of the content), while preserving the original voice of the speaker, including their emotions and intonations. However, in this era of artificial intelligence, where almost every company is seeking to leverage language models to improve efficiency, ElevenLabs is not the only company exploring speech-to-speech translation. AI Dubbing: How Does It Work? Although AI-based translation involves multiple layers of work, from noise reduction to speech translation, the end-users do not need to go through these steps. They simply need to select the AI Dubbing tool on ElevenLabs, create a new project, choose the source and target languages, and upload the content file. Once the content is uploaded, the tool automatically detects the number of speakers and starts working, displaying a progress bar on the screen. It functions like any other conversion tool on the internet. After completion, the file can be downloaded and used. In the background, the tool utilizes ElevenLabs' proprietary methods to remove background noise, differentiate between music and noise, and the actual dialogue of the speakers. It identifies when the speakers are talking, keeps their voices clear, transcribes their words into their original language using a speech-to-text model, translates and adjusts the text to match the length, and generates the desired speech in the target language while preserving the original voice characteristics of the speakers. Finally, the translated speech is resynchronized with the music and background noise that were initially removed from the file, preparing it for dubbed output. ElevenLabs claims that this work is the culmination of their research in voice cloning, text and audio processing, and multilingual speech synthesis. To generate the final speech from the translated text, the company utilizes its latest Multilingual v2 model. It currently supports over 20 languages, including Hindi, Portuguese, Spanish, Japanese, Ukrainian, Polish, and Arabic, providing users with a wide range of choices to globalize their content. Prior to this end-to-end interface, ElevenLabs offered separate voice cloning and text-to-speech synthesis tools. This means that if someone wanted to translate their audio content, such as a podcast, into another language, they had to first create a clone of their voice on the platform, transcribe and translate the audio separately. Then, using the translated text file and the cloned voice, they could generate audio from the text-to-speech model. Needless to say, this only worked for speech without significant background music or noise. Staniszewski confirmed that the new dubbing feature will be available to all users on the platform, but there will be some character limitations, similar to text-to-speech generation. He stated that approximately one minute of AI Dubbing is usually equivalent to 3,000 characters. AI-Based Speech on the Horizon While ElevenLabs has garnered attention for its continuous development, it is not the only company exploring AI-based speech. A few weeks ago, Microsoft-backed OpenAI made ChatGPT multimodal, enabling it to respond to voice prompts like Alexa. Here, too, audio is converted using speech-to-text and text-to-speech models, but this technology is not available to everyone. OpenAI stated that it is working with select partners to prevent the misuse of these capabilities. One of these partners is Spotify, which is helping its podcasters translate their content into different languages while preserving their own voices. According to Staniszewski, what sets ElevenLabs' AI Dubbing tool apart is its ability to translate videos or audios of any length, with any number of speakers, while preserving their voices and emotions in up to 20 languages, and providing the highest quality results. Other players are also active in the field of AI-based speech and speech synthesis, including MURF.AI, Play.ht, and WellSaid Labs. Recently, Meta also launched SeamlessM4T, an open-source multilingual base model that can understand nearly 100 languages from speech or text and generate real-time translations, whether input, output, or both. According to Market US data, the global market for such tools was valued at $1.2 billion in 2022 and is projected to reach nearly $5 billion by 2032, with a compound annual growth rate slightly above 15.40%.