Google launches AudioPaLM: Where Text Meets Voice

2023-06-27

Google has made a splash in the field of generative artificial intelligence with the launch of AudioPaLM, a new multimodal language model that combines the capabilities of Google's large-scale language model PaLM-2, released at the 2023 Google I/O conference, and last year's generative audio model AudioLM. AudioPaLM establishes a comprehensive multimodal framework for processing and generating textual and spoken language. AudioPaLM has a wide range of applications, including speech recognition and speech-to-speech translation. Leveraging the expertise of AudioLM, AudioPaLM inherits the ability to capture non-linguistic cues such as speaker identity and intonation, while also integrating the language knowledge embedded in text-based language models like PaLM-2. Additionally, AudioPaLM showcases unique features of audio language models, such as the ability to convert speech from one language to another based on concise spoken prompts. Built on a large-scale Transformer model, AudioPaLM expands the existing vocabulary of text-based language models by adding specialized audio tokens. With a basic task description, a single model consisting only of a decoder can be trained to handle various combinations of speech and text tasks, including speech recognition, text-to-speech synthesis, and speech-to-speech translation. This approach integrates traditionally separate models into a unified architecture and training process. AudioPaLM achieves outstanding performance in speech translation benchmarks and competitive results in speech recognition tasks. It also demonstrates the ability to convert speech to text without prior training on previously unseen language pairs. In addition to speech generation, AudioPaLM can generate transcriptions that include the original language or directly translate the transcriptions, as well as generate speech text for the original speech. AudioPaLM achieves the best results in speech translation benchmarks and demonstrates competitive performance in speech recognition tasks. The model also retains secondary linguistic information, such as speaker identity and intonation, which is often lost in traditional speech-to-text translation systems. The system is expected to outperform existing solutions in terms of speech quality based on automatic and manual evaluations. The researchers state in the paper, "There are further research opportunities in audio tokenization aimed at identifying desirable audio token attributes, developing measurement techniques, and optimizing accordingly. Additionally, more comprehensive benchmarks and metrics are needed for generative audio tasks to drive research progress, as current benchmarks mainly focus on speech recognition and translation." Tech giant Google's battle in the music generation field has just begun. This is not Google's first foray into audio generation. Earlier this year, it released MusicLM, a high-fidelity music generation model that creates music from text descriptions built on AudioLM. It uses a hierarchical sequence-to-sequence approach to generate stable music at 24 kHz. It also introduced MusicCaps, a curated dataset of 5.5k music text pairs aimed at evaluating text-to-music generation. Google's competitors in this field are not lagging behind either. Microsoft recently launched Pengi, an audio language model that leverages transfer learning from text generation tasks to audio tasks. By integrating audio and text inputs, Pengi can generate free-form text output without the need for additional fine-tuning. In addition, Meta, led by Mark Zuckerberg, has introduced MusicGen, which utilizes the powerful capabilities of the Transformer architecture to compose music based on text prompts and combines the generated music with existing melodies. Similar to language models, MusicGen can predict the next section of a music piece, resulting in coherent and structured compositions. It efficiently processes tokens in parallel using Meta's EnCodec audio tokenizer. The model is trained on a dataset of 20,000 hours of licensed music, ensuring a variety of music styles and compositions. Meta has also released Voicebox, a multilingual generative AI model that can perform various speech generation tasks through context learning, even for tasks it has not been explicitly trained on. However, OpenAI, supported by Microsoft and currently considered a leader in the field of generative AI, seems to have lost its way in this music generation competition, as the creators of ChatGPT have not made any announcements in this field recently. Source: https://analyticsindiamag.com/google-unveils-audiopalm-where-text-meets-voice/

RECENT AI NEWS

RECENT AI TOOLS