ByteDance and Tsinghua University Jointly Develop SALMONN: An Artificial Intelligence Capable of Understanding All Audio

Researchers from Tsinghua University and ByteDance have developed a new artificial intelligence system called SALMONN that enables machines to understand and reason with audio inputs such as speech, sounds, and music. In a research paper published on arXiv, scientists describe SALMONN as a "large language model (LLM) capable of handling speech, audio events, and music inputs." The system combines two specialized AI models—one for speech processing and one for general audio processing—into a single LLM that can generate text responses to audio prompts. "SALMONN is not just limited to speech or audio event inputs; it can perceive and understand various audio inputs, thus acquiring new capabilities such as multilingual speech recognition and translation and audio-visual co-reasoning," the paper states. "This can be seen as giving the LLM 'ears' and cognitive auditory abilities." An AI model that can listen and understand The researchers demonstrated SALMONN's capabilities on a range of audio inputs, including speech, gunshots, duck calls, and music. When presented with each sound clip, the system generated appropriate descriptive text responses, demonstrating an understanding of the audio content. "Text prompts are used to guide SALMONN in answering open-ended questions about general audio inputs, with the answers being in the LLM text responses," the paper explains. According to the researchers, this cognitive audio question-answering technique represents a significant breakthrough from traditional AI speech and audio systems, which are limited to basic transcription. "Compared to traditional speech and audio processing tasks such as speech recognition and audio captioning, SALMONN achieves cognition-driven audio perception by leveraging the LLM's general knowledge and cognitive capabilities, greatly enhancing the model's versatility and the richness of tasks," the paper notes. The researchers suggest that SALMONN also possesses cross-modal capabilities, such as following verbal instructions without the need for explicit speech-to-text translation training. "SALMONN uses only text-command-based training data, and listening to verbal commands is also a new cross-modal capability," they write. While the current capabilities are promising, the researchers acknowledge limitations in the model's depth of reasoning. However, they remain optimistic about its future potential, stating that SALMONN "takes a step towards artificial general intelligence with auditory capabilities." The potential impact of SALMONN on enterprise data analytics For technology decision-makers, this development may herald a new era of voice-activated data analysis and business intelligence. SALMONN's ability to understand and interpret various audio inputs could fundamentally change the way businesses interact with data, eliminating the need for traditional text-based inputs and opening up new possibilities for voice-activated analysis and data-driven decision-making. Additionally, the team has released a web-based demo that allows users to experience SALMONN's capabilities firsthand. The model is also available on Hugging Face, a leading platform for hosting and sharing machine learning models. In a rapidly evolving world of artificial intelligence, SALMONN offers an intriguing glimpse into the future of machine learning and cognitive computing. It highlights ByteDance and Tsinghua University's commitment to advancing artificial intelligence. As we move towards a world where AI can "see" through computer vision and "hear" through cognitive audio processing, the implications are profound for both businesses and consumers.