Google's New App: Real-time Transcription of Speech

2019-12-22

Over the past 20 years, Google has provided the public with a wealth of information, from text, photos, and videos to maps and other content. However, there is a lot of information in the world that is conveyed through speech. Even though we use recording devices to capture important information from conversations, interviews, speeches, and other content, it is still difficult to parse, recognize, and extract the relevant information from hours of recordings.


Therefore, Google has created Recorder, a new audio recording application that leverages the latest developments in machine learning to transcribe conversations, detect and identify recorded audio types (from broad categories like music or speech to specific sounds like applause, laughter, and whistling), and index the recordings so that users can quickly find and extract the desired segments. All of these features run entirely on the device without the need for an internet connection.



Transcription


The application uses an automatic speech recognition model to transcribe speech, accurately transcribing long recordings (several hours) and indexing the session by mapping words to timestamps calculated by the speech recognition model. This allows users to click on a word in the transcription and start playing from that point in the recording, or search for a word and jump to the exact point in the recording where it was said.




Sound Categorization


Presenting recordings is useful and allows people to search for specific words, but sometimes it is more useful to visually search different parts of the recorded content based on specific time points or sounds. To achieve this, Recorder also represents audio visually as a colored waveform, where each color is associated with a different sound category. This is accomplished by combining research on using CNNs to classify audio sounds (e.g., recognizing dog barks or instrument playing) and audio event detection on previously released datasets to classify distinct sound events in individual audio frames.


Of course, in most cases, many sounds can occur simultaneously. To visualize audio in a very clear way, we decided to color each waveform bar with a color that represents the most prominent sound within a given time period (in our example, a 50ms bar). The colored waveform allows users to understand what types of content were captured in a specific recording and makes it easier to browse through a growing audio library. This provides users with a visual representation of the recording and enables them to search for audio events within the recording.


The program also supports a sliding window feature, which processes partially overlapping 960ms audio frames with a 50ms interval and outputs an S-shaped score vector representing the probability of each supported audio category in that frame. To combine the maximum system accuracy and report the correct sound categorization, developers applied linearization processing to the S-shaped scores and incorporated a thresholding mechanism. This process of analyzing the content of a 960ms window with a smaller 50ms offset allows for more accurate determination of start and end times in a less error-prone manner than analyzing individual 960ms large window slices separately.


Since the model analyzes each audio frame independently, there may be rapid fluctuations between audio categories. This can be addressed by applying adaptive-sized median filtering to the latest model audio class outputs, providing a smooth continuous output. This process runs continuously in real-time, requiring it to meet very strict power consumption limits.



Creating Labels


After recording is complete, Recorder suggests three labels that the application deems represent the most important content, allowing users to quickly compose meaningful titles.


In order to establish these labels immediately upon the end of recording, Recorder analyzes the transcribed recording. First, Recorder calculates the frequency of word occurrences and their grammatical roles in sentences. Terms identified as entities are represented in uppercase letters. Then, we use a part-of-speech tagger (a model that tags each word in a sentence based on its grammatical role) on the device to detect common nouns and proper nouns that the user seems more likely to remember. The Recorder uses a prior score table for term extraction that supports unigrams and bigrams. To generate the scores, we trained an enhanced decision tree using session data and utilized text features such as document word frequency and specificity. Finally, meaningless words and profanity are filtered out, and the top labels are outputted.Recorder incorporates some of our recent work in machine learning research conducted on the device into useful features and runs models on the device to ensure user privacy. The positive feedback loop between machine learning research and user needs reveals exciting opportunities to make our software even more useful. We are excited about future research that will make everyone's ideas and conversations more accessible and searchable.

RECENT AI NEWS

RECENT AI TOOLS