From Sound to Vision: Introducing AudioToken, a Technology for Audio to Image Synthesis

2023-06-25

Neural generative models have revolutionized the way we consume digital content, bringing innovation to various aspects. They possess the ability to generate high-quality images, ensure coherence in text spans, and generate speech and audio. Among different approaches, diffusion-based generative models have recently gained widespread attention and demonstrated remarkable results in various tasks. During the diffusion process, the model learns to map a predefined noise distribution to the target data distribution. At each step, the model predicts noise and generates signals from the target distribution. Diffusion models can handle different forms of data representation, such as raw input and latent representation. State-of-the-art models like Stable Diffusion, DALLE, and Midjourney have been developed for text-to-image synthesis tasks. While there has been increasing interest in X-to-Y generation in recent years, audio-to-image models have not been thoroughly explored. The reason for using audio signals instead of textual prompts is the interconnection between images and audio in a video context. In contrast, although text-based generative models can produce impressive images, there is no inherent connection between text descriptions and images, which means that text descriptions are often manually added. Audio signals also have the ability to represent complex scenes and objects, such as different variations of the same instrument (e.g., classical guitar, acoustic guitar, electric guitar) or the same object (e.g., classical guitar recorded in a studio and during a live performance). Manual annotation of this detailed information for different objects is labor-intensive, posing challenges to scalability. Previous research has proposed several methods for generating audio from image inputs, mainly using generative adversarial networks (GANs) to generate images based on audio recordings. However, their work differs significantly from the proposed method. Some methods focus only on generating MNIST digits and do not extend their approach to include general audio sounds. Others do generate images from general audio, but the resulting images are of low quality. To overcome the limitations of these studies, Audio proposes a deep learning model for audio-to-image generation. The overview of this method is shown in the following figure. This method involves utilizing a pre-trained text-to-image generative model and a pre-trained audio representation model to learn the mapping adaptation layer between their outputs and inputs. Drawing inspiration from recent work on text reversal, a dedicated AudioToken is introduced to map audio representations to an embedding vector. This vector is then forwarded as a continuous representation to the network, reflecting a new word embedding. The audio embedder utilizes a pre-trained audio classification network to capture the representation of audio. Typically, the last layer of a discriminative network is used for classification purposes, but it often overlooks important audio details that are irrelevant to the discriminative task. To address this issue, this method combines earlier layers with the last hidden layer to obtain a temporal embedding of the audio signal. Below are some sample results generated by the proposed model. (Source: https://www.marktechpost.com/2023/06/22/from-sound-to-sight-meet-audiotoken-for-audio-to-image-synthesis/)

RECENT AI NEWS

RECENT AI TOOLS