Why is ChatGPT Unique and Unmatched in its Imitability?

2023-05-29

Since the release of ChatGPT, the Large Language Models (LLMs) chatbot has attracted a lot of attention. There are users, discussants, developers who are working hard to build their own versions, and other AI companies. Although OpenAI's model is closed-source, many open-source models have given developers hope to build something similar to the best chatbots on the market. One emerging and cost-effective method to improve open-source language models like LLaMa, Alpaca, or Self-Instruct is fine-tuning on outputs from proprietary systems like ChatGPT. This seemed like an effective way to mimic and build the capabilities of stronger models to achieve comparable performance. However, this approach has proven to be unsuccessful. Deficiencies in Inheritance Researchers from UC Berkeley recently published a paper titled "False Promises of Imitating Proprietary LLMs," critically analyzing the effectiveness of this imitation approach. The researchers explained that training models using proprietary language models raises various legal and theoretical concerns. The technical improvements achieved through this approach are also negligible. According to the paper, fine-tuning weaker models to enhance their knowledge capacity has minimal to no impact on language models. Pre-training is the primary source for improving language model capabilities. Therefore, fine-tuning smaller models like LLaMa, Vicuna, Alpaca, and Self-Instruct on outputs from larger models like ChatGPT and Bard does not enhance the models' knowledge since the underlying data remains unchanged. This only alters the style of the model without improving its knowledge. Furthermore, when attempting to mimic the knowledge and capabilities of large models like ChatGPT, weaker models also inherit their deficiencies and biases. Imitating these models through fine-tuning also weakens the ability to directly improve the design decisions of closed AI models (e.g., ChatGPT), resulting in their performance being inferior to models provided directly by AI companies. The models fail to improve in crucial aspects such as factual accuracy, encoding, and problem-solving. Data Imitation is Not the Only Approach Large companies with extensive foundational models like GPT-4 or Google's PaLM have no reason to worry about imitation since there is a significant gap between them and the models attempting to imitate them. Companies with access to abundant data, computing power, and algorithms are more likely to maintain their competitive advantage. Smaller companies attempting to build their own "moats" using open-source language models (LLMs) like LLaMa are more susceptible to the risks of imitation. As explained earlier, fine-tuning models using ChatGPT data does not significantly improve the performance of language models compared to models built and improved by companies using their own pre-training data. However, companies like OpenAI have never disclosed the data used to train their models. It is certain that a significant portion comes from publicly available internet data rather than proprietary data. According to a research paper from UC Berkeley pointed out by users on Hacker News, using this data may be illegal, although this may not be accurate. The conclusions drawn in the paper are not based on the current development of open-source software. Before the release of LLaMa, it was believed that larger models perform better. However, various models built on smaller foundational models like LLaMa have outperformed GPT-4 and PaLM. What if Data Becomes Private? In leaked documents, Google stated that they and OpenAI do not have a "moat" in the AI field. The documents praised the open-source community and mentioned Meta's LLaMa in many aspects. Although Google and OpenAI do not have proprietary rights over the data they train on, the future of language models may not remain the same and become more concerning. Recently, entrepreneur and AI enthusiast Elon Musk prevented OpenAI from accessing his data and threatened to sue them and Microsoft to prohibit their continued use. Now, his new chatbot is likely trained on Twitter data that is exclusive to him and inaccessible even to the open-source community. For the open-source community, this means that if developers rely on ready-made language models like LLaMa, they will not be able to fine-tune on any other models built by large companies. Currently, fine-tuning models using data from ChatGPT or Bard does not involve any legal or ethical issues since this data does not solely belong to the companies. However, if Musk is correct and companies start acquiring exclusive data access rights, the open-source community will cease to thrive. These companies may not want to allow anyone to compete, and the rising open-source community could be limited by restricted data access. Currently, the significance of smaller models lies in their specific use case capabilities. However, if large companies gain exclusive access to other companies' data and keep it confidential, they can easily surpass smaller models. No one will openly disclose or publish the sources of their data, but it is evidently from the internet. If these giants start acquiring intellectual property rights over data from other companies, they will become the most knowledgeable models in the community, and the open-source community may disappear. Source: https://analyticsindiamag.com/why-you-can-never-imitate-chatgpt/

RECENT AI NEWS

RECENT AI TOOLS