AI that can read lip movements in videos

2019-12-05

In 2016, researchers from Google and the University of Oxford detailed a system that can annotate video clips with an accuracy of 46.8%, surpassing the accuracy of professional lip readers by 12.4%. However, even the most advanced systems struggle to overcome the ambiguity of lip movements, making them unable to outperform audio-based speech recognition. To develop more efficient language recognition systems, researchers from Alibaba, Zhejiang University, and Stevens Institute of Technology have designed a method called LIBS. Similar to other solutions, LIBS can assist individuals with hearing impairments in tracking videos without subtitles. It is estimated that there are 466 million people worldwide with disabling hearing loss, accounting for approximately 5% of the global population. According to the World Health Organization, this number may increase to over 900 million by 2050. On the other hand, LIBS also benefits non-native language speakers, allowing them to independently watch their favorite movies or TV shows on foreign websites. LIBS extracts useful audio information from multi-scale (including sequence-level, context-level, and frame-level) speaker videos. It aligns this data with video data by identifying the correspondence between the two (due to different sampling rates and occasional blanks at the beginning or end), and refines the extracted features using filtering techniques to further enhance accuracy. The speech recognizer and lip reader components of LIBS primarily rely on an attention-based sequence-to-sequence architecture, which is a machine translation approach for mapping sequence inputs. The researchers trained the model using content mainly sourced from 45,000 spoken sentences from the BBC and CMLR, the largest Mandarin lip-reading corpus in China, which includes over 100,000 natural sentences (including over 3,000 Chinese characters and 20,000 phrases) from Chinese online TV websites. During the training process, the team noticed that the model struggled to produce "reasonable" results on the LRS2 dataset for sentences shorter than 14 characters, as the decoder found it challenging to extract relevant information from such short sentences. On the other hand, if the model was pre-trained on sentences with a maximum length of 16 words, the decoder could leverage context-level knowledge to improve the quality of sentence endings in the LRS2 dataset. Regarding these unexpected scenarios, the researchers wrote in a paper describing their work, "The former may be due to LIBS reducing attention to irrelevant frames, while the latter is likely to be frame-level knowledge extraction, further enhancing the discriminability of video frame features and concentrating attention."

RECENT AI NEWS

RECENT AI TOOLS