"Stability AI releases model for generating stable audio named Stable Audio"

Harmonai, the audio research lab of Stability AI, has released Stable Audio, a text-controlled audio generation diffusion model. Stable Audio has been trained on 19,500 hours of audio data and can generate 44.1kHz quality audio in real-time using an NVIDIA A100 GPU.

Similar to Stability AI's image generation model, Stable Diffusion, Stable Audio takes user-provided text prompts as input to describe the desired output. Like Stable Diffusion, a U-Net-based diffusion model is at the core of this system. In addition to text prompts, users can also specify the desired output duration in seconds. The model can generate single instrument, full orchestra audio, or more ambient sound effects like crowd noise. According to Stability AI:

"Stable Audio represents the cutting-edge audio generation research achievements of Harmonai, the audio research lab under Stability AI. We will continuously improve the model architecture, dataset, and training process to enhance output quality, controllability, inference speed, and output duration."

Recent advancements in generative AI for text and images have also propelled the development of music generation models. OpenAI's MuseNet, based on GPT-2, can generate a range of MIDI notes that can be converted into sound using MIDI synthesizer software. This year, Google's MusicLM and Meta's MusicGen models operate similarly to autoregressive language models, but they output "audio tokens" instead of text tokens. In 2022, several diffusion-based music generation models emerged, including Harmonai's early projects Dance Diffusion and Riffusion, which use a fine-tuned version of Stable Diffusion to generate spectrogram images that are then converted into sound using classical digital signal processing techniques.

Stable Audio utilizes a pre-trained model called CLAP to map the user's text prompts into an embedding space shared with music features, similar to how OpenAI's CLIP is used in Stable Diffusion. These feature vectors, along with the embedding of the desired output length and a noise vector, are fed into a denoising U-Net model with 970M parameters, based on a system called Moûsai. This outputs the latent space representation of the generated sound, which is then transformed into audio using a variational autoencoder (VAE) called Descript Audio Codec.

Emad Mostaque, CEO of Stability AI, wrote:

"This is the first commercially licensed music model and platform, and the team's work is remarkable. It is still in the experimental stage, but we expect rapid progress so that users can create any audio they imagine and integrate their own data and more content."

Although Stable Audio is currently not open-source, Harmonai states that they will release "open-source models based on Stable Audio" as well as code for training custom models. Harmonai's Github account includes a branch of the Moûsai repository. The Stable Audio website allows users to register for a free version with a maximum of 20 generations per month, but it cannot be used for commercial purposes.