Microsoft launches Pengi, an audio language model for open-ended tasks

2023-05-29

Pengi, a groundbreaking audio language model developed by Microsoft researchers, leverages transfer learning to transform all audio tasks into text generation tasks. By integrating audio and text inputs, Pengi generates free-form text outputs without the need for additional fine-tuning. Extensive evaluations involving 22 downstream tasks demonstrate Pengi's state-of-the-art performance, highlighting significant advancements in general audio understanding through the integration of language and audio models. The audio language model utilizes transfer learning by treating all audio-related tasks as text generation tasks. It operates by taking recordings and their corresponding texts as inputs and generating free-form text outputs. Pengi's unified architecture allows for the handling of both open-ended and closed tasks without the need for additional fine-tuning or task-specific extensions. During training, Pengi is exposed to a large dataset consisting of audio-text pairs. This dataset encompasses various audio recordings, including human speech, music, and various sounds, along with their respective text transcriptions. The audio recordings are processed through an audio encoder, which converts them into continuous embedded sequences. Simultaneously, the text transcriptions are processed through a text encoder, transforming them into corresponding continuous embedded sequences. These two embedded sequences are integrated into a prefix to prompt the pre-trained frozen language model. The language model then generates tokens in an autoregressive manner based on the audio and text inputs. To evaluate Pengi's capabilities, 22 downstream tasks, including audio captioning, audio question answering, and audio event detection, were assessed. Pengi demonstrated state-of-the-art performance across these tasks, confirming its effectiveness as a powerful audio language model applicable to a wide range of tasks. Pengi's functionalities include generating subtitles for recordings, answering questions related to recordings, detecting events in recordings, translating recordings into text, summarizing recordings, and generating creative text formats such as poetry, code, scripts, music compositions, emails, and letters. Although still under development, Pengi has the potential to revolutionize audio interactions. With Pengi, natural conversations with devices become feasible, enabling unprecedented audio-related functionalities that were previously unattainable. Source: https://analyticsindiamag.com/microsoft-launches-pengi-an-audio-language-model-for-open-ended-tasks/

RECENT AI NEWS

RECENT AI TOOLS