AI Training Data Exhaustion Possible by 2026

As the popularity of artificial intelligence (AI) reaches its peak, researchers warn that the industry may run out of training data. This could slow down the growth of AI models, especially large language models, and even alter the trajectory of the AI revolution.

But with so much data available on the internet, why is there a potential shortage of data? Are there any risk-mitigating solutions?

Why High-Quality Data is Important for Artificial Intelligence

We need a large amount of data to train powerful, accurate, and high-quality AI algorithms. For example, ChatGPT was trained on 570G of text data (approximately 300 billion words).

Similarly, the Stable Diffusion algorithm (which serves as the foundation for many AI image generation applications like DALL-E, Lensa, and Midtravel) was trained on the LION-5B dataset, which includes 5.8 billion image-text pairs. Insufficient training data can result in inaccurate or low-quality outputs.

The quality of training data is also crucial. Low-quality data, such as social media posts or blurry photos, is easy to come by but insufficient to train high-performance AI models.

Text obtained from social media platforms may be biased or include false information or illegal content, which can be replicated by the models. For example, when Microsoft attempted to train its AI chatbot using Twitter content, it learned to produce racist and sexist outputs.

That's why AI developers seek high-quality content from books, online articles, scientific papers, Wikipedia, and certain filtered web pages. Google Assistant was trained on 11,000 romance novels selected from its own website, Smashwords, to enhance its conversational abilities.

Do We Have Enough Data?

The AI industry has been training AI systems on increasingly large datasets, which is why we now have high-performance models like ChatGPT or DALL-E3. However, research indicates that the growth of online data storage lags behind the datasets used for training AI.

In a paper published last year, a group of researchers predicted that if the current trend of AI training continues, we will exhaust high-quality text data before 2026. They also estimated that low-quality language data will be depleted between 2030 and 2050, and low-quality image data will be depleted between 2030 and 2060.

According to data from accounting consulting group PwC, AI is projected to contribute up to $15.7 trillion (AUD 24.1 trillion) to the global economy by 2030. However, the depletion of available data could slow down its development.

Should We Be Worried?

While the above points may be alarming to some AI enthusiasts, the situation may not be as dire as it seems. The future development of AI models holds many unknowns, and there are methods to address the risks of data scarcity.

One opportunity is for AI developers to improve algorithms to more efficiently utilize the data they already possess.

In the coming years, they are likely to train high-performance AI systems using less data and smaller computational power. This also helps reduce the carbon footprint of AI.

Another option is to use AI to create synthetic data for training systems. In other words, developers can simply generate the data they need to fit their specific AI models.

Several projects are already using synthetic content, often sourced from data generation services like Mostly AI. This is expected to become increasingly common in the future.

Developers are also exploring content beyond the online space, such as content held by major publishers and offline repositories. Imagine the millions of texts published before the advent of the internet. If these texts are digitized, they can provide new sources of data for AI projects.

News Corp, one of the world's largest content owners, recently stated that it is in negotiations with AI developers for content licensing. Much of News Corp's content is behind paywalls. Such deals would require AI companies to pay for training data, whereas they have mostly obtained training data for free from the internet.

Content creators have been protesting the unauthorized use of their content to train AI models, and some companies have even sued Microsoft, OpenAI, Stability AI, and others. Receiving compensation may help restore some balance of power between creators and AI companies.