More Gemini Features Coming to Google TV
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreLack of labeled music data
Large language models (LLMs) like ChatGPT or Bard are trained on vast amounts of unstructured text data. While collecting content from millions of websites can be computationally expensive, there is a wealth of training data available on the public internet. In contrast, text-to-image models like DALL-E 2 require a completely different dataset consisting of image pairs with corresponding descriptions.
Similarly, text-to-music models rely on songs with music content descriptions. However, unlike images, labeled music is indeed hard to find on the internet. Sometimes, metadata such as instruments, genres, or emotions can be used, but obtaining in-depth descriptions of the entire song is challenging. This poses a serious problem for researchers and companies attempting to collect data to train music generation models.
In early 2023, Google researchers garnered widespread attention in the field of music AI with their groundbreaking models, MusicLM and Noise2Music. However, musicians have little knowledge about how the data for these models was collected. Let's delve into this topic and explore some of the techniques applied in Google's music AI research.
How Google overcame data scarcity
Weakly associated labels
For MusicLM and Noise2Music, Google relies on another model called MuLan, which is trained to calculate the similarity between any music snippet and any textual description. To train MuLan, Google used what they call "weakly associated labels." Instead of carefully curating a high-quality music dataset with descriptive text, they deliberately adopted a different approach.
First, they extracted 30-second snippets from 44 million music videos on YouTube, resulting in 370,000 hours of audio. Then, the music was labeled with various textual tags associated with the videos, such as video titles, descriptions, comments, and playlist names. To reduce noise in this dataset, they employed a large language model to identify which associated textual information had content relevant to the music and discarded all irrelevant content.
Weakly associated labels cannot be considered "fake" datasets since the textual information is still written by humans and undoubtedly has some degree of association with the music. However, this approach prioritizes quantity over quality, which has raised concerns among most machine learning researchers in the past. Google is just getting started...
Pseudo-labels
Noise2Music is a generative music AI based on diffusion techniques, also used in image generation models like DALL-E or Midjourney.
To train Noise2Music, Google took their previous approach to the extreme, transitioning from weakly associated labels to fully human-generated labels. In what they call "pseudo-labels," the authors employed an extraordinary method to collect music description texts. They prompted a large language model (LaMDA) to write multiple descriptions for 150,000 popular songs, resulting in 4 million descriptions. Here are some examples of such descriptions:
Queen's "Don't Stop Me Now": This energetic rock song is based on piano, bass guitar, and drums. The band is full of energy, ready to burst out, and in high spirits.
Subsequently, the researchers removed the song and artist names, generating descriptions that could, in principle, apply to other songs as well. However, even with these descriptions, the researchers still needed to match them with the appropriate songs to obtain a large labeled dataset. This is where their pre-trained model MuLan comes into play, trained on weakly associated labels.
The researchers collected a substantial dataset of unlabeled music, resulting in 340,000 hours of music. For each of these tracks, they used MuLan to identify the artificially generated song descriptions that best matched them. Essentially, each piece of music was not mapped to a textual description of the song itself but to a description encapsulating music similar to it.
Why does this work?
The problem
In traditional machine learning, the labels assigned to each observation (in this case, a piece of music) should ideally represent objective facts. However, music descriptions themselves lack objectivity, which is the first problem. Furthermore, by using audio-to-text mapping techniques, the labels no longer "truthfully" reflect what happens in the songs. They do not provide an accurate description of the music. Given these apparent shortcomings, one might wonder why this approach still yields useful results.
Bias and noise
When labels in a dataset are not accurately assigned, there can be two main reasons: bias and noise. Bias refers to labels consistently being untruthful in a particular way. For example, if a dataset frequently labels instrumental pieces as songs but never labels songs as instrumental pieces, it indicates a bias towards predicting the presence of vocals.
On the other hand, noise represents the general variability in the labels, independent of direction. For example, if every track is labeled as a "sad piano piece," then the dataset has a severe bias as it consistently provides inaccurate labels for many songs. However, since it applies the same label to each track, there is no variability, and thus no noise, in the dataset.
By mapping tracks to descriptive texts written for other tracks, we introduce noise. This is because it is challenging to find perfect descriptions for most tracks in the dataset. Therefore, most labels have some bias, meaning they are untruthful to some extent, which introduces noise. But are the labels biased?
Given that the available descriptions are generated for popular songs, it is reasonable to assume that the pool of descriptions is biased towards (Western) popular music. Despite this, generating 4 million descriptions based on 150,000 unique songs would be expected to provide a range of different descriptions to choose from. Additionally, most labeled music datasets exhibit the same bias, so this approach does not have a unique disadvantage compared to other methods. What truly sets this approach apart is the introduction of more noise.
Why noise is acceptable in machine learning
Training a machine learning model on a biased dataset is generally not an ideal approach as it leads to the model learning and replicating biased task understanding. However, training a machine learning model on a dataset without bias but with noise can still yield impressive results. Let's illustrate this with an example.
Take a look at the chart below, which shows two datasets composed of orange and blue points. In the noiseless dataset, the blue and orange points are completely separable. However, in the noisy dataset, some orange points have moved into the cluster of blue points, and vice versa. Despite this additional noise, if we examine the trained models, we find that both models identify roughly the same patterns. This is because even in the presence of noise, AI learns to recognize the optimal patterns that minimize errors.
This example demonstrates that AI can indeed learn from noisy datasets, such as the ones generated by Google. However, the main challenge lies in the fact that the larger the noisy dataset, the more training data is required to effectively train the model. This rationale is reasonable because noisy datasets inherently contain less valuable information compared to equally sized noiseless datasets.
Conclusion
In conclusion, Google has employed innovative techniques to address the limited availability of labeled music data when training generative music AI models. They leverage weakly associated labels linked to MuLan, utilize textual information from various sources associated with music videos, and employ a language model to filter out irrelevant data. When developing Noise2Music, they introduce pseudo-labels by generating multiple descriptions for popular songs and mapping them to appropriate tracks using their pre-trained model.
Although these methods may deviate from traditional labeling approaches, they have proven to be effective. Despite the introduction of noise, the models are still able to learn and identify optimal patterns. While using fake datasets may be seen as an unconventional approach, it highlights the immense potential of modern language models in creating large-scale and valuable machine learning datasets.
Source: https://towardsdatascience.com/how-google-used-fake-datasets-to-train-generative-music-ai-def6f3f71f19
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreAlphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Read moreAnthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Read moreApple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Read moreGoogle launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
Read moreYouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Read moreGoogle grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Read moreMistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.
Read moreCustom subliminal audio maker

AI Social Media Manager

AI Search Visibility

Workout and Calorie Tracker
Stock research platform

AI interview assistant

Scientific figure generator

AI courses with real code