More Gemini Features Coming to Google TV
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreReal-time 2D animation is a novel and powerful form of communication that allows performers to control cartoon characters in real time, while interacting and improvising with other actors or audience members. For example, Disney's "Star Wars: Forces of Destiny" and "My Little Pony" engage in real-time chat sessions with fans through YouTube or Facebook Live.
Creating realistic real-time 2D animation requires the use of interactive systems that can automatically convert human performances into animation in real time. The key to these systems is achieving good lip synchronization, which essentially means that the mouth of the animated character moves appropriately when speaking, mimicking the movements observed in the performer's mouth.
Good lip synchronization can make real-time 2D animation more convincing, allowing animated characters to more realistically portray their functions. Conversely, improper lip synchronization often disrupts the experience of characters as live performers or participants in conversations.In a recent paper published on arXiv, two researchers introduced a deep-based learning interactive system developed by Adobe and the University of Washington that can automatically generate lip synchronization for layered 2D animated characters. Their system uses Long Short-Term Memory (LSTM) models, a type of Recurrent Neural Network (RNN) architecture commonly used for tasks involving classification, data processing, and prediction.
The two researchers, Wilmot Li and Deepali Aneja, explained, "Since speech is almost every real-time animation's main component, we believe that the most critical problem to address in this field is real-time lip synchronization, which requires converting the actor's speech into corresponding mouth movements (i.e., visual phoneme sequences) in the animated character. In this work, we focus on creating high-quality lip synchronization for real-time 2D animation."
The system developed by Li and Aneja uses a simple LSTM model to convert streaming audio input into corresponding visual phoneme sequences at a rate of 24 frames per second, with a delay of less than 200 milliseconds. In other words, their system allows the lips of animated characters to move in a manner similar to that of human users speaking in real time, with a delay of less than 200 milliseconds between speech and lip movement.Li and Aneja said, "In this work, we make two contributions: identifying appropriate feature representations and network configurations to achieve state-of-the-art results in real-time 2D lip synchronization, and designing a novel augmentation method to collect training data for the model. For handcrafted lip synchronization, professional animators make stylistic decisions regarding specific choices of visual phonemes, as well as the timing and quantity of transitions. Therefore, training a 'universal' model is insufficient for most applications."
Furthermore, obtaining labeled lip synchronization data for training deep learning models can be expensive and time-consuming. Professional animators can spend five to seven hours manually creating visual phoneme sequences for every minute of speech. Recognizing these limitations, Li and Aneja developed a method that can generate training data more quickly and efficiently.
In order to train their LSTM model more effectively, Li and Aneja introduced a new technique of using audio time warping to augment the manually written training data. This data augmentation process achieves good lip synchronization even when training their model on smaller labeled datasets.
To evaluate the effectiveness of their interactive system in generating real-time lip synchronization, the researchers asked human viewers to assess the quality of animations produced using their model compared to animations created using commercial 2D animation tools. They found that most viewers preferred the lip synchronization generated by their method compared to other techniques.
Interestingly, the researchers found that their LSTM model can achieve different lip synchronization styles based on the trained data, while also being widely applicable across a range of speakers. The model achieved impressive results, impressing the researchers, and Adobe decided to integrate it into the Adobe Character Animator software, set to be released in the fall of 2018.
The researchers believe that there are more opportunities to apply modern machine learning techniques to improve the workflow of 2D animation. Li and Aneja stated, "So far, one challenge has been the lack of training data, which is expensive. However, as we have demonstrated in this work, there may be ways to maximize the use of handcrafted animation data by utilizing structured data and automatic editing algorithms, such as dynamic time warping. "
Although the data augmentation strategies proposed by the researchers can greatly reduce the training data requirements for models designed to generate real-time lip synchronization, sufficient manual animation content for training new models still requires a significant amount of work and effort. However, according to Li and Aneja, it may not be necessary to retrain the entire model for each new lip synchronization style.
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreAlphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Read moreAnthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Read moreApple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Read moreGoogle launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
Read moreYouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Read moreGoogle grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Read moreMistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.
Read moreCustom subliminal audio maker

AI Social Media Manager

AI Search Visibility

Workout and Calorie Tracker
Stock research platform

AI interview assistant

Scientific figure generator

AI courses with real code