The AI Behind Amazon Alexa's Whisper Mode

2020-01-18

In October 2018, a few months after a brief disclosure, Amazon introduced Whisper Mode for select third-party and first-party Alexa devices. It will expand the feature to all regions in November 2019, so that all Alexa-supported smart appliances now respond to whispers with whispering.


Initially, Amazon was not very forthcoming with the technical details, but "Whisper Mode" uses neural networks (mathematical functional layers loosely modeled after human brain neurons) to differentiate between normal and whispered words. However, in a research paper published in January 2020, it provided a detailed account of the research that led to the expansion.Application scientist Marius Cotescu from Amazon's Voice Research team explained that the main challenge was to convert normal speech into whispered speech while maintaining naturalness and speaker identity. He and his colleagues explored several different conversion techniques, including handcrafted digital signal processing (DSP) based on acoustic analysis of whispered speech, but ultimately settled on two machine learning approaches due to their robustness (ease of generalization to unfamiliar speakers) and performance (outperforming handcrafted signal processors).


Both approaches are based on Gaussian Mixture Models (GMM) and Deep Neural Networks (DNN), involving training algorithms to map the acoustic features of regular speech to the acoustic features of whispered speech. GMM attempts to identify value ranges corresponding to the distribution of input values for each output feature, while DNN (a dense algorithm of simple processing nodes) adjusts its internal settings through the network to predict the process of relevant outputs with specific inputs.


The researchers' system passes the acoustic feature representation to a vocoder, which then converts it into a continuous signal. While the experimental version relied on an open-source vocoder called WORLD, the deployed version of Whisper Mode for customers utilizes a neural vocoder, which further enhances the quality of whispered speech.


The team used two datasets to train their speech conversion system: one created by five professional voice actors from Australia, Canada, Germany, India, and the United States, and another from a popular benchmark in the field. (Both corpora include paired speech from many speakers, one speaking at a normal volume and the other whispering.) To evaluate their system, they compared the output with natural speech recordings and speech recordings input through the vocoder.


In the first set of experiments, the team trained the speech conversion system based on data from a single speaker and tested it on data from the same speaker. They found that while the original recordings sounded the most natural, the synthesized whispers from the model sounded more natural than "speech coding" human speech.


The state-of-the-art text-to-speech conversion models can generate segments that sound almost human-like upon initial listening. In fact, they support neural speech provided by Google Assistant, as well as news anchor voices recently introduced in Alexa and Amazon Polly services, and the celebrity Alexa voice skill featuring Samuel L. Jackson that became available in December last year.

RECENT AI NEWS

RECENT AI TOOLS