Introducing AudioCraft: Meta AI's New Generation Audio Model Series

2023-08-15

Audio is rapidly becoming one of the new frontiers in artificial intelligence generation. In the pursuit of generating high-fidelity audio, Meta AI faces the challenge of modeling complex signals and patterns at different scales. Among various types of audio, music is particularly daunting as it encompasses both local and global patterns, ranging from individual notes to complex musical structures with multiple instruments. While traditional methods rely on symbolic representations such as MIDI or piano rolls, they fall short in capturing the subtle differences in musical expression and stylistic richness. Meta AI has recently introduced AudioCraft, a series of generative AI models for high-quality audio generation.

The AudioCraft model series is a notable achievement that demonstrates the ability to generate high-quality audio with persistent consistency, while providing a user-friendly interface. Compared to previous work in the field, AudioCraft simplifies the overall design of audio generation models, offering a comprehensive toolkit to explore the state-of-the-art models developed by Meta over the years. Additionally, it empowers users to push boundaries and develop their own custom models.

AudioCraft is based on three core models: MusicGen, AudioGen, and EnCodec. AudioCraft showcases its versatility in generating different forms of audio. MusicGen takes meta-owned and specially licensed music training to transform text-based user input into impressive musical compositions. On the other hand, AudioGen, trained on publicly available sound effects, adeptly generates audio from text prompts related to environmental sounds and sound effects, such as dog barks, car honks, or footsteps on wooden floors.

1) Audio Generation: EnCodec

The AudioCraft model faces the complex challenge of generating audio from raw audio signals, as it involves modeling very long sequences. A typical music track lasts several minutes, sampled at 44.1 kHz (the standard for music recording), and contains millions of time steps. In contrast, text-based generation models like Llama and Llama 2 handle text processed as subwords, with each sample representing only a few thousand time steps.

To address this complexity, Meta AI adopts EnCodec, a neural audio codec that learns discrete audio tokens from the raw signal, effectively creating a new fixed "vocabulary" for music samples. With this approach, they can train autoregressive language models on these discrete audio tokens to generate new tokens, sounds, and music. By subsequently using EnCodec's decoder to convert these tokens back into the audio domain, they achieve the desired audio generation.

The AudioCraft model leverages EnCodec, a neural audio codec that learns discrete audio tokens from the raw waveform. The codec performs parallel streams mapping audio signals to one or several discrete tokens. A single autoregressive language model is then employed to recursively model the audio tokens from EnCodec. These generated tokens are then fed back into the codec, converting them back into the audio domain and producing the final output waveform. This flexibility allows for the use of various conditioning models to control the generation process, such as using a pre-trained text encoder for text-to-audio applications.

EnCodec is a lossy neural codec specifically designed to compress various types of audio and reconstruct the original signal with high fidelity. Its architecture includes an autoencoder with a residual vector quantization bottleneck, generating parallel streams of audio tokens with a fixed vocabulary. These different streams capture different levels of information from the audio waveform, ultimately achieving high-fidelity audio reconstruction across all streams.

2) Text-to-Audio: AudioGen

AudioGen delves into the challenging task of generating audio samples conditioned on descriptive text prompts, extending this approach to include both conditioned and unconditioned audio continuations. In the context of generating complex soundscapes, such as "a dog barking while someone plays a trumpet on a busy street," the model faces the complex challenge of generating three different types of acoustic content with distinct background/foreground elements, durations, and relative positions on the timeline. This combination is highly unlikely to occur in the training data and requires deep generalization, acoustic fidelity, production, and mastering capabilities.

AudioGen introduces an autoregressive text-guided audio generation model, which consists of two main stages. In the initial stage, the original audio is encoded into a discrete token sequence using a neural audio compression model designed for end-to-end training to reconstruct the input audio from the compressed representation. To enhance audio generation fidelity, a set of discriminators applies perceptual losses during this process. This compression technique ensures both high-fidelity audio generation and compactness of the representation.

The second stage employs an autoregressive Transformer-decoder language model to operate on the discrete audio tokens obtained from the first stage. Additionally, the language model is conditioned on text input. To represent the text, a separate text encoder model, T5, pretrained on a large corpus of text, is used. This pretrained text encoder significantly aids in generalizing text concepts that are lacking in the current text-audio dataset. This feature becomes particularly important when dealing with diverse and descriptive limited text annotations. By incorporating such a versatile text encoder, AudioGen effectively addresses the challenge of generating rich and diverse audio samples based on text prompts.

3) Text-to-Music: MusicGen

MusicGen is an AudioCraft model dedicated to the challenging task of text-to-music generation, aiming to generate music compositions based on textual descriptions. This process presents unique difficulties as it involves modeling long-range sequences. Unlike speech, music needs to utilize the entire spectrum, resulting in higher sampling rates for music recordings, typically at 44.1 kHz or 48 kHz, compared to 16 kHz for speech. Furthermore, the complexity of music arises from the presence of harmonies and melodies from multiple instruments, forming intricate structures.

MusicGen is a simple yet controllable music generation model where Meta AI proposes a comprehensive framework for modeling multiple parallel streams of acoustic tokens, offering significant advancements over previous research. To enhance the controllability of generating music, the model incorporates unsupervised melody conditioning, facilitating the production of music that adheres to given harmonic and melodic structures.

MusicGen employs a Transformer-based autoregressive decoder, conditioned on either text or melody representation. The language model operates on quantized units obtained from the EnCodec audio tokenizer, ensuring high-fidelity reconstruction from low frame-rate discrete representations. Each stream in this setup contains discrete tokens from various learned codebooks. To address this complexity, previous approaches proposed different modeling strategies. In this work, Meta AI introduces a novel and versatile modeling framework that can adapt to various codebook interleaving patterns, exploring several different variants. By leveraging these patterns, the model can exploit the internal structure of quantized audio tokens, enhancing the music generation process.

RECENT AI NEWS

RECENT AI TOOLS