Google recently introduced a new model called "SoundStorm: Efficient Parallel Audio Generation" in a paper. This model provides an efficient and high-quality method for audio generation.
SoundStorm addresses the problem of generating longer audio token sequences through two innovative components:
1. Customized architecture design for audio token generation by the SoundStream neural codec.
2. Adoption of a decoding scheme inspired by MaskGIT, a recently introduced image generation method designed specifically for audio token operations.
Compared to the autoregressive decoding method of AudioLM, SoundStorm achieves parallel generation of tokens, reducing the inference time for long sequences by 100 times. Additionally, SoundStorm maintains audio quality while providing higher consistency in speech and acoustic conditions.
Furthermore, the paper demonstrates that by combining SoundStorm with the text-to-semantic modeling stage of SPEAR-TTS, high-quality and natural conversations can be synthesized. This allows for controlling verbal content through transcribed text, selecting speaker voices through short audio prompts, and controlling speaker alternation through transcribed annotations. The provided examples showcase the persuasive dialogue capabilities of SoundStorm and its integration with SPEAR-TTS.
Underlying Technology:
Previous research on AudioLM demonstrated a two-step process for audio generation. The first step is semantic modeling, generating semantic tokens based on previous semantic markers or conditional signals. The second step is acoustic modeling, focusing on generating acoustic tokens from semantic markers.
However, in SoundStorm, researchers specifically addressed the acoustic modeling step and aimed to replace the slower autoregressive decoding method with a faster parallel decoding method.
SoundStorm utilizes the Conformer model architecture with a bidirectional attention mechanism, combining convolution and Transformer. This architecture captures local and global structures in sequence labeling. The model predicts audio tokens generated by SoundStream, which takes semantic token sequences generated by AudioLM as input. SoundStream adopts a method called Residual Vector Quantization (RVQ), representing audio with up to Q tokens at each time step. As the number of tokens generated per time step increases from 1 to Q, the reconstructed audio quality improves gradually.
During the inference process, SoundStorm masks all audio tokens and fills in the masked tokens in multiple iterations. It starts with coarse tokens at RVQ level q = 1 and refines them progressively until reaching level q = Q. This approach enables fast audio generation.
SoundStorm has two key aspects for fast audio generation. Firstly, it predicts tokens in parallel during a single iteration at each RVQ level. Secondly, the design of the model architecture minimizes the computational complexity with respect to the number of levels, Q. To support this inference scheme, a carefully designed masking scheme is used during training to simulate the iterative process used during inference.
Compared to AudioLM, SoundStorm significantly improves generation speed by two orders of magnitude and achieves better consistency when generating longer audio samples. By combining SoundStorm with text-to-semantic modeling models similar to SPEAR-TTS, text-to-speech synthesis can be extended to handle longer contexts.
Furthermore, SoundStorm can generate natural dialogues with multiple speaker turns, allowing users to control the voices of speakers and the generated content. It is worth noting that SoundStorm is not limited to speech synthesis. For example, MusicLM effectively utilizes SoundStorm to synthesize longer music outputs.
Why is this important?
This work addresses the challenge of slow inference time when generating long sequence audio tokens using autoregressive decoding methods. While autoregressive decoding ensures high audio quality, it generates tokens one by one, resulting in significant computational overhead, especially for longer sequences. SoundStorm proposes a new approach by introducing architecture suitable for audio token generation and a decoding scheme inspired by MaskGIT, enabling parallel generation of tokens. In this way, SoundStorm significantly reduces inference time, making audio generation more efficient without sacrificing audio quality and consistency.
Many generative audio models, including AudioLM, use autoregressive decoding, generating tokens one by one. While this approach ensures high audio quality, it may be computationally slow when dealing with longer sequences.
Source: https://analyticsindiamag.com/google-unveils-soundstorm-for-parallel-audio-generation-from-discrete-conditioning-tokens/
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.