The Future of General Artificial Intelligence Chatbots: Less Common in the Future

Artificial intelligence technology is developing rapidly. ChatGPT has become the fastest-growing online service in history. Google and Microsoft are integrating generative AI into their products.

However, we can see that AI chatbots are becoming less general and more specialized. The ability of AI services to serve users is limited by the data they are exposed to.

In the process of AI training, extensive data collection is often chosen, and AI systems absorb thousands of books and web pages. However, a more curated and focused set of training data may make AI chatbots more useful for people working in specific industries or living in specific areas.

The Value of Data

One important factor in this evolution is the increasing cost of collecting training data for large-scale language models (LLMs), which power ChatGPT. Companies know that data is valuable: Meta and Google have made billions of dollars by selling ads targeted at user data. But the value of data is changing. Meta and Google sell "insights" from data; they invest in analytics to turn many data points into predictions about users.

The value of data is slightly different for OpenAI. Imagine a tweet: "A cat is sitting on a mat." This tweet has no value for targeted advertisers. It does not involve information about user interests.

But for OpenAI, which is building LLMs to generate human-like language, this tweet is valuable for demonstrating the patterns of human language. One tweet cannot teach AI to construct sentences, but billions of tweets, blogs, Wikipedia articles, and so on certainly can. For example, the advanced LLM GPT-4 could be built using data collected from websites like X (formerly Twitter), Reddit, and Wikipedia.

The AI revolution is changing business models for companies. Meta and Google, among others, have been leveraging their data resources to invest in AI research and development.

Organizations like X and Reddit have already started charging fees for API access, which is used to scrape data from these websites. Data scraping requires companies like X to spend more money on computing power to complete data queries.

Looking ahead, as organizations like OpenAI seek to build more powerful versions of GPT LLMs, they will face higher costs of data acquisition. One solution to this problem may be synthetic data.

Synthetic Data

Synthetic data is created from scratch by AI systems and used to train more advanced AI systems instead of real training data.

This is a new idea, but it faces many challenges. Good synthetic data needs to have enough differences from the original data it is based on to teach the model something new, while also being similar enough to convey the correct information. This can be difficult to achieve. Synthetic data is just a replica of real-world data, and AI models generated from it may struggle to be creative and may deepen existing biases.

Another problem is the "Hapsburg AI" problem. This suggests that training AI on synthetic data leads to a decrease in the effectiveness of these systems - a parallel can be drawn with the notorious inbreeding of the Hapsburg royal family. Some studies have shown that systems like ChatGPT have already experienced this.

One reason ChatGPT is so impressive is that it uses reinforcement learning from human feedback (RLHF), with people scoring its outputs for accuracy. If the synthetic data generated by AI is inaccurate, then the AI models trained on this data will be inaccurate themselves. Therefore, the need for human feedback to correct inaccuracies may increase.

However, while most people can tell if a sentence is grammatically correct, few can comment on its factual accuracy - especially when the output is technical or specialized content. RLHF is unlikely to discover inaccuracies in professional topics. If synthetic data means more inaccuracies, then the quality of general LLMs may stagnate or decline, even if these models "learn" more.

Small Language Models

Google engineers have revealed that there is nothing stopping third parties from recreating LLMs like GPT-3 or Google's LaMDA. Many organizations can use their own specialized data to build their own internal AI systems for their specific goals. In the long run, this may be more valuable for these organizations than ChatGPT.

Recently, the Japanese government pointed out that developing a ChatGPT version centered around Japan could contribute to their AI strategy, as ChatGPT does not fully represent Japan. Software company SAP has recently launched its AI "roadmap" to provide AI development capabilities for specialized organizations. This will make it easier for companies to build their own customized versions of ChatGPT.

Consulting firms like McKinsey and KPMG are exploring training AI models for "specific purposes." Guides on how to create private, personalized versions of ChatGPT can easily be found online. Open-source systems like GPT4All already exist.

With the development challenges faced by general LLMs, coupled with potential regulatory barriers, the future of AI may be many specific small language models rather than large language models. Small language models may struggle with acquiring large amounts of data.

However, they may also have advantages in RLHF, as few language models can be used for specific purposes. Employees with expertise in their target domain may provide more valuable feedback for such AI systems compared to general feedback for general AI systems. This can overcome the disadvantage of having less data.