Teaching AI Composition with Fake Dataset: Unveiling Google's Music Creation Technology

2023-05-29

In this article, we explore Google's innovative approach to training its extraordinary text-to-music models, including MusicLM and Noise2Music. We will delve into the concept of "fake" datasets and how they are used in these groundbreaking models. If you are curious about the inner workings of these technologies and their impact on advancing music AI, then you've come to the right place.

Lack of labeled music data

Large language models (LLMs) like ChatGPT or Bard are trained on vast amounts of unstructured text data. While collecting content from millions of websites can be computationally expensive, there is a wealth of training data available on the public internet. In contrast, text-to-image models like DALL-E 2 require a completely different dataset consisting of image pairs with corresponding descriptions.

Similarly, text-to-music models rely on songs with music content descriptions. However, unlike images, labeled music is indeed hard to find on the internet. Sometimes, metadata such as instruments, genres, or emotions can be used, but obtaining in-depth descriptions of the entire song is challenging. This poses a serious problem for researchers and companies attempting to collect data to train music generation models.

In early 2023, Google researchers garnered widespread attention in the field of music AI with their groundbreaking models, MusicLM and Noise2Music. However, musicians have little knowledge about how the data for these models was collected. Let's delve into this topic and explore some of the techniques applied in Google's music AI research.

How Google overcame data scarcity

Weakly associated labels

For MusicLM and Noise2Music, Google relies on another model called MuLan, which is trained to calculate the similarity between any music snippet and any textual description. To train MuLan, Google used what they call "weakly associated labels." Instead of carefully curating a high-quality music dataset with descriptive text, they deliberately adopted a different approach.

First, they extracted 30-second snippets from 44 million music videos on YouTube, resulting in 370,000 hours of audio. Then, the music was labeled with various textual tags associated with the videos, such as video titles, descriptions, comments, and playlist names. To reduce noise in this dataset, they employed a large language model to identify which associated textual information had content relevant to the music and discarded all irrelevant content.

Weakly associated labels cannot be considered "fake" datasets since the textual information is still written by humans and undoubtedly has some degree of association with the music. However, this approach prioritizes quantity over quality, which has raised concerns among most machine learning researchers in the past. Google is just getting started...

Pseudo-labels

Noise2Music is a generative music AI based on diffusion techniques, also used in image generation models like DALL-E or Midjourney.

To train Noise2Music, Google took their previous approach to the extreme, transitioning from weakly associated labels to fully human-generated labels. In what they call "pseudo-labels," the authors employed an extraordinary method to collect music description texts. They prompted a large language model (LaMDA) to write multiple descriptions for 150,000 popular songs, resulting in 4 million descriptions. Here are some examples of such descriptions:

Queen's "Don't Stop Me Now": This energetic rock song is based on piano, bass guitar, and drums. The band is full of energy, ready to burst out, and in high spirits.

Subsequently, the researchers removed the song and artist names, generating descriptions that could, in principle, apply to other songs as well. However, even with these descriptions, the researchers still needed to match them with the appropriate songs to obtain a large labeled dataset. This is where their pre-trained model MuLan comes into play, trained on weakly associated labels.

The researchers collected a substantial dataset of unlabeled music, resulting in 340,000 hours of music. For each of these tracks, they used MuLan to identify the artificially generated song descriptions that best matched them. Essentially, each piece of music was not mapped to a textual description of the song itself but to a description encapsulating music similar to it.

Why does this work?

The problem

In traditional machine learning, the labels assigned to each observation (in this case, a piece of music) should ideally represent objective facts. However, music descriptions themselves lack objectivity, which is the first problem. Furthermore, by using audio-to-text mapping techniques, the labels no longer "truthfully" reflect what happens in the songs. They do not provide an accurate description of the music. Given these apparent shortcomings, one might wonder why this approach still yields useful results.

Bias and noise

When labels in a dataset are not accurately assigned, there can be two main reasons: bias and noise. Bias refers to labels consistently being untruthful in a particular way. For example, if a dataset frequently labels instrumental pieces as songs but never labels songs as instrumental pieces, it indicates a bias towards predicting the presence of vocals.

On the other hand, noise represents the general variability in the labels, independent of direction. For example, if every track is labeled as a "sad piano piece," then the dataset has a severe bias as it consistently provides inaccurate labels for many songs. However, since it applies the same label to each track, there is no variability, and thus no noise, in the dataset.

By mapping tracks to descriptive texts written for other tracks, we introduce noise. This is because it is challenging to find perfect descriptions for most tracks in the dataset. Therefore, most labels have some bias, meaning they are untruthful to some extent, which introduces noise. But are the labels biased?

Given that the available descriptions are generated for popular songs, it is reasonable to assume that the pool of descriptions is biased towards (Western) popular music. Despite this, generating 4 million descriptions based on 150,000 unique songs would be expected to provide a range of different descriptions to choose from. Additionally, most labeled music datasets exhibit the same bias, so this approach does not have a unique disadvantage compared to other methods. What truly sets this approach apart is the introduction of more noise.

Why noise is acceptable in machine learning

Training a machine learning model on a biased dataset is generally not an ideal approach as it leads to the model learning and replicating biased task understanding. However, training a machine learning model on a dataset without bias but with noise can still yield impressive results. Let's illustrate this with an example.

Take a look at the chart below, which shows two datasets composed of orange and blue points. In the noiseless dataset, the blue and orange points are completely separable. However, in the noisy dataset, some orange points have moved into the cluster of blue points, and vice versa. Despite this additional noise, if we examine the trained models, we find that both models identify roughly the same patterns. This is because even in the presence of noise, AI learns to recognize the optimal patterns that minimize errors.

This example demonstrates that AI can indeed learn from noisy datasets, such as the ones generated by Google. However, the main challenge lies in the fact that the larger the noisy dataset, the more training data is required to effectively train the model. This rationale is reasonable because noisy datasets inherently contain less valuable information compared to equally sized noiseless datasets.

Conclusion

In conclusion, Google has employed innovative techniques to address the limited availability of labeled music data when training generative music AI models. They leverage weakly associated labels linked to MuLan, utilize textual information from various sources associated with music videos, and employ a language model to filter out irrelevant data. When developing Noise2Music, they introduce pseudo-labels by generating multiple descriptions for popular songs and mapping them to appropriate tracks using their pre-trained model.

Although these methods may deviate from traditional labeling approaches, they have proven to be effective. Despite the introduction of noise, the models are still able to learn and identify optimal patterns. While using fake datasets may be seen as an unconventional approach, it highlights the immense potential of modern language models in creating large-scale and valuable machine learning datasets.

Source: https://towardsdatascience.com/how-google-used-fake-datasets-to-train-generative-music-ai-def6f3f71f19

RECENT AI NEWS

RECENT AI TOOLS