Meta releases open-source AI audio tool AudioCraft

2023-08-03

Meta's three AI model suites can create sound effects and music based on descriptions.

On Wednesday, Meta announced that it is open-sourcing AudioCraft, a generative AI tool for generating music and audio based on text prompts. With these tools, content creators can input simple text descriptions to generate complex audio scenes, compose melodies, and even simulate entire virtual orchestras.

AudioCraft consists of three core components: AudioGen, a tool for generating various audio effects and soundscapes; MusicGen, which can create music compositions and melodies based on descriptions; and EnCodec, a neural network-based audio compression codec.

In particular, Meta states that EnCodec, which we first reported on in November last year, has recently been improved to achieve "higher quality music generation while reducing hardware failures." Additionally, AudioGen can create audio effects such as dog barks, car horns, or footsteps on wooden floors. MusicGen can compose songs of various genres based on descriptions like "pop dance, catchy melodies, tropical percussion, and cheerful rhythms, perfect for the beach."

Meta provides several audio samples on its website for evaluation. These results seem to align with their state-of-the-art claims, but it can be said that their quality is not sufficient to replace professionally produced commercial audio effects or music.

Meta points out that while generative AI models around text and static images have received much attention (and are relatively easy for people to experiment with online), the development of generative audio tools has lagged behind. "There is currently some related research, but they are very complex and not very open, making it difficult for people to experiment easily," they wrote. However, they hope that the release of AudioCraft under the MIT license will provide accessible tools for audio and music experimentation, contributing to a wider community.

Meta states, "These models are available for research purposes to further advance people's understanding of the technology. We are pleased to enable researchers and practitioners to train their own models with their own datasets, helping to drive technological progress in this field."

Meta is not the first company to attempt using AI to generate audio and music. In some notable attempts, OpenAI introduced Jukebox in 2020, Google launched MusicLM in January this year, and in December last year, an independent research team created a text-to-music generation platform called Riffusion using Stable Diffusion as the foundation.

These generative audio projects have not garnered as much attention as image synthesis models, but that does not mean the development process is any less complex, as Meta points out on its website:
Generating any type of high-fidelity audio requires modeling complex signals and patterns at different scales. Music can be said to be the most challenging type of audio generation, as it consists of local and global patterns, from a set of notes to the overall musical structure with multiple instruments. Coherent music is generated by processing symbolic representations using MIDI or piano rolls, but these methods do not fully understand the subtle differences in expression and stylistic elements in music. Recent advances utilize self-supervised audio representation learning and some hierarchical or cascaded models to generate music by inputting raw audio into complex systems to capture long-range structures in the signals and generate high-quality audio. But we know there is more to be done in this field.

In the controversy surrounding the use of undisclosed and potentially unethical training materials to create image synthesis models (such as Stable Diffusion, DALL-E, and Midjourney), it is worth noting that Meta states MusicGen is trained on "20,000 hours of music owned by Meta or music specifically licensed for this purpose." This seems to be a step towards a more ethical direction and may please some critics of generative AI.

Interestingly, we will see how open-source developers choose to integrate these Meta audio models into their work. This could lead to the creation of interesting and user-friendly generative audio tools. Currently, for those familiar with coding, the model weights and code for these three AudioCraft tools can be found on GitHub.



Source: https://arstechnica.com/information-technology/2023/08/open-source-audiocraft-can-make-dogs-bark-and-symphonies-soar-from-text-using-ai/

RECENT AI NEWS

RECENT AI TOOLS