Innovative Technology from Typecast Empowers Generative Artificial Intelligence to Convey Human Emotions

Language is the foundation of human communication, and so is emotion. Expressing happiness, sadness, anger, frustration, or other emotions helps convey information and connect us.

Although generative artificial intelligence has achieved great success in many other fields, it struggles to capture these subtle differences and handle the complexity of human emotions.

Typecast, a startup company that creates synthetic voices and videos using artificial intelligence, claims to be revolutionizing this field with its new Cross-Speaker Emotion Transfer technology.

This technology allows users to apply emotions recorded from other people's voices to their own voices while maintaining their unique style, enabling faster and more efficient content creation. This technology is now available through Typecast's My Voice Maker feature.

"Artificial intelligence actors have not fully captured the range of human emotions, which is their biggest limiting factor," said Taesu Kim, CEO and co-founder of Neosapience and Typecast, based in Seoul, South Korea.

With Typecast's Cross-Speaker Emotion Transfer, "anyone can use AI actors with real emotional depth based on samples of their own voices."

Understanding Emotions

Although emotions can typically be categorized into seven categories: happiness, sadness, anger, fear, surprise, and disgust, Kim pointed out that this is not enough to express the various emotions in generated speech.

He noted that speaking is not just a one-to-one mapping between given text and output speech.

"Humans can say the same sentence in countless ways. We can also express different emotions within the same sentence (or even the same word)," he said in an interview.

For example, recording the sentence "How could you do this to me?" with the emotion prompt "with a sad voice, as if disappointed" would be completely different from recording it with the emotion prompt "angry, as if scolding."

Similarly, the emotion prompt "she feels sad because her father passed away, but with a smile on her face" is complex and not easily defined by a single category.

"Humans can speak with different emotions, which leads to rich and diverse conversations," Kim and other researchers wrote in their paper about their new technology.

Limitations of Emotion-to-Speech Text

Text-to-speech technology has made significant progress in a short period, driven by mainstream and emerging players such as ChatGPT, LaMDA, LLama, Bard, Claude, and others.

Emotion-to-speech text has also made considerable progress, but it requires a large amount of labeled data, which is not easily obtainable, explained Kim. Capturing the subtle differences in emotions through voice recordings has always been a time-consuming and challenging task.

Furthermore, "it is very difficult to continuously record multiple sentences while maintaining the emotions," Kim and his colleagues wrote.

He explained that in traditional emotion speech synthesis, all training data must have emotion labels. These methods often require additional emotion encodings or reference audios.

But this poses a fundamental challenge because there must be available data for every emotion and every speaker. Additionally, existing methods tend to have issues with label errors due to the difficulty of extracting intensity.

Transferring emotions across speakers becomes even more challenging when assigning an unseen emotion to a speaker. So far, this technology has performed poorly because generating emotional speech from a neutral speaker instead of the original speaker is unnatural. Additionally, controlling emotion intensity is often difficult.

"Even with an emotion speech dataset available, there are still limitations in controlling emotion intensity," Kim and his fellow researchers wrote.

Utilizing Deep Neural Networks and Unsupervised Learning

To address this issue, researchers first input emotion labels into a generative deep neural network, which Kim called a world-first. While successful, this approach was not sufficient to express complex emotions and speaking styles.

Then, researchers developed an unsupervised learning algorithm that can discern speaking styles and emotions from a large database. Kim stated that during the training process, the entire model was not trained using any emotion labels.

This provided representative numbers for a given speech. Although not interpretable by humans, these representations can be used in text-to-speech algorithms to express the emotions present in the database.

Researchers further trained a perceptual neural network to convert natural language emotion descriptions into representations.

"With this technology, users do not need to record hundreds or thousands of different speaking styles/emotions, as it can learn from a large database containing various emotional voices," Kim said.

Adapting Sound Features from Short Segments

Researchers achieved "transferable and controllable emotion speech synthesis" by leveraging latent representations and separating speakers from styles through domain adversarial training and cycle consistency loss.

This technology learns emotional patterns, intonations, and tones by analyzing and understanding a large number of human-recorded voices through audiobooks, videos, and other media.

Kim explained that this approach can transfer emotions to speakers with a neutral reading style using only a few labeled samples, and emotion intensity can be controlled through simple and intuitive scalar values.

He said this helps achieve emotion transfer in a natural way without changing identity. Users only need to record short segments of their own voices and then apply various emotions and intensities, and the AI can adapt to specific sound features.

Users can choose different types of emotional speech recorded by others and apply that style to their own voices while retaining their unique vocal identity. Even speaking in a normal tone, users can express happiness, sadness, anger, or other emotions by recording just five minutes of their own voice.

Typecast's technology has been adopted by Samsung Securities, a subsidiary of Samsung Group, LG Electronics in South Korea, and other companies. Since its establishment in 2017, the startup has raised $268 million in funding. The company is currently working on applying its core speech synthesis technology to facial expressions, according to Kim.

Control in Generative AI is Crucial

Kim pointed out that the media landscape is changing rapidly.

In the past, text-based blogs were the most popular format for corporate media. But now, short videos dominate, and companies and individuals must produce more audio and video content more frequently.

"To convey company messages, high-quality and expressive voices are crucial," Kim said.

Fast and cost-effective content creation is essential, he added, as manual work by human actors is inefficient.

"Control in generative AI is crucial for content creation," Kim said. "We believe these technologies can help ordinary people and companies unleash their creativity and improve productivity."