More Gemini Features Coming to Google TV
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreAudio is rapidly becoming one of the new frontiers in artificial intelligence generation. In the pursuit of generating high-fidelity audio, Meta AI faces the challenge of modeling complex signals and patterns at different scales. Among various types of audio, music is particularly daunting as it encompasses both local and global patterns, ranging from individual notes to complex musical structures with multiple instruments. While traditional methods rely on symbolic representations such as MIDI or piano rolls, they fall short in capturing the subtle differences in musical expression and stylistic richness. Meta AI has recently introduced AudioCraft, a series of generative AI models for high-quality audio generation.
The AudioCraft model series is a notable achievement that demonstrates the ability to generate high-quality audio with persistent consistency, while providing a user-friendly interface. Compared to previous work in the field, AudioCraft simplifies the overall design of audio generation models, offering a comprehensive toolkit to explore the state-of-the-art models developed by Meta over the years. Additionally, it empowers users to push boundaries and develop their own custom models.
AudioCraft is based on three core models: MusicGen, AudioGen, and EnCodec. AudioCraft showcases its versatility in generating different forms of audio. MusicGen takes meta-owned and specially licensed music training to transform text-based user input into impressive musical compositions. On the other hand, AudioGen, trained on publicly available sound effects, adeptly generates audio from text prompts related to environmental sounds and sound effects, such as dog barks, car honks, or footsteps on wooden floors.
1) Audio Generation: EnCodec
The AudioCraft model faces the complex challenge of generating audio from raw audio signals, as it involves modeling very long sequences. A typical music track lasts several minutes, sampled at 44.1 kHz (the standard for music recording), and contains millions of time steps. In contrast, text-based generation models like Llama and Llama 2 handle text processed as subwords, with each sample representing only a few thousand time steps.
To address this complexity, Meta AI adopts EnCodec, a neural audio codec that learns discrete audio tokens from the raw signal, effectively creating a new fixed "vocabulary" for music samples. With this approach, they can train autoregressive language models on these discrete audio tokens to generate new tokens, sounds, and music. By subsequently using EnCodec's decoder to convert these tokens back into the audio domain, they achieve the desired audio generation.
The AudioCraft model leverages EnCodec, a neural audio codec that learns discrete audio tokens from the raw waveform. The codec performs parallel streams mapping audio signals to one or several discrete tokens. A single autoregressive language model is then employed to recursively model the audio tokens from EnCodec. These generated tokens are then fed back into the codec, converting them back into the audio domain and producing the final output waveform. This flexibility allows for the use of various conditioning models to control the generation process, such as using a pre-trained text encoder for text-to-audio applications.
EnCodec is a lossy neural codec specifically designed to compress various types of audio and reconstruct the original signal with high fidelity. Its architecture includes an autoencoder with a residual vector quantization bottleneck, generating parallel streams of audio tokens with a fixed vocabulary. These different streams capture different levels of information from the audio waveform, ultimately achieving high-fidelity audio reconstruction across all streams.
2) Text-to-Audio: AudioGen
AudioGen delves into the challenging task of generating audio samples conditioned on descriptive text prompts, extending this approach to include both conditioned and unconditioned audio continuations. In the context of generating complex soundscapes, such as "a dog barking while someone plays a trumpet on a busy street," the model faces the complex challenge of generating three different types of acoustic content with distinct background/foreground elements, durations, and relative positions on the timeline. This combination is highly unlikely to occur in the training data and requires deep generalization, acoustic fidelity, production, and mastering capabilities.
AudioGen introduces an autoregressive text-guided audio generation model, which consists of two main stages. In the initial stage, the original audio is encoded into a discrete token sequence using a neural audio compression model designed for end-to-end training to reconstruct the input audio from the compressed representation. To enhance audio generation fidelity, a set of discriminators applies perceptual losses during this process. This compression technique ensures both high-fidelity audio generation and compactness of the representation.
The second stage employs an autoregressive Transformer-decoder language model to operate on the discrete audio tokens obtained from the first stage. Additionally, the language model is conditioned on text input. To represent the text, a separate text encoder model, T5, pretrained on a large corpus of text, is used. This pretrained text encoder significantly aids in generalizing text concepts that are lacking in the current text-audio dataset. This feature becomes particularly important when dealing with diverse and descriptive limited text annotations. By incorporating such a versatile text encoder, AudioGen effectively addresses the challenge of generating rich and diverse audio samples based on text prompts.
3) Text-to-Music: MusicGen
MusicGen is an AudioCraft model dedicated to the challenging task of text-to-music generation, aiming to generate music compositions based on textual descriptions. This process presents unique difficulties as it involves modeling long-range sequences. Unlike speech, music needs to utilize the entire spectrum, resulting in higher sampling rates for music recordings, typically at 44.1 kHz or 48 kHz, compared to 16 kHz for speech. Furthermore, the complexity of music arises from the presence of harmonies and melodies from multiple instruments, forming intricate structures.
MusicGen is a simple yet controllable music generation model where Meta AI proposes a comprehensive framework for modeling multiple parallel streams of acoustic tokens, offering significant advancements over previous research. To enhance the controllability of generating music, the model incorporates unsupervised melody conditioning, facilitating the production of music that adheres to given harmonic and melodic structures.
MusicGen employs a Transformer-based autoregressive decoder, conditioned on either text or melody representation. The language model operates on quantized units obtained from the EnCodec audio tokenizer, ensuring high-fidelity reconstruction from low frame-rate discrete representations. Each stream in this setup contains discrete tokens from various learned codebooks. To address this complexity, previous approaches proposed different modeling strategies. In this work, Meta AI introduces a novel and versatile modeling framework that can adapt to various codebook interleaving patterns, exploring several different variants. By leveraging these patterns, the model can exploit the internal structure of quantized audio tokens, enhancing the music generation process.
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreAlphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Read moreAnthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Read moreApple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Read moreGoogle launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
Read moreYouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Read moreGoogle grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Read moreMistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.
Read moreCustom subliminal audio maker

AI Social Media Manager

AI Search Visibility

Workout and Calorie Tracker
Stock research platform

AI interview assistant

Scientific figure generator

AI courses with real code