OpenAI Abandons Development of GPT-4 Level Model "Arrakis"

OpenAI has previously developed a new AI model called Arrakis, aiming to reduce the cost of AI inference. However, due to quality issues, the company has discontinued the development of this project.

Arrakis was supposed to have similar capabilities as GPT-4 but with lower deployment costs, thanks to the adoption of the so-called "sparse principle." This means that only a portion of the neural network is used to process inputs, whereas in traditional "dense models," the entire neural network is active. For example, Google used the sparse principle in its Pathways AI project.

The development of Arrakis started in the autumn of last year, and the training work began in the spring. However, OpenAI's development team quickly realized that the performance was inadequate, according to two people familiar with the project.

It is reported that the team made some adjustments within a month, and then the leadership team terminated the project. It is currently unclear why the sparse principle performed well in early testing but was not suitable for larger-scale models.

Shifting focus to "GPT-4 Turbo"

Despite encountering setbacks, OpenAI may apply the work of Arrakis to other models, such as the planned Gobi multimodal model. After the failure of Arrakis training, OpenAI researchers started working on developing a faster-responding version of GPT-4.

This could be the "0613" version of GPT-4, which is expected to be much faster than the "0314" version used by OpenAI since March this year. Some users have complained about a decrease in quality in certain areas, although OpenAI has refuted this, resulting in a longer-than-planned time for providing the "0314" version through the API.

OpenAI brings AI to the desert

Gobi, Sahara, and Arrakis: these new AI model prototypes are named after deserts, paying tribute to the sparse principle and hoping that these models can work more efficiently and save costs. In particular, OpenAI's partner Microsoft hopes that Arrakis can reduce the usage cost of generative AI in its products.

According to media reports, Microsoft itself has shifted resources to develop cheaper language models, as the high cost of generative AI threatens emerging business models.

Leaks about Arrakis

There have been previous reports about leaks regarding Arrakis. It is said that this model can generate not only text but also images and videos, with fewer hallucinations compared to GPT-4, making it suitable as an autonomous agent. Additionally, Arrakis will be partially trained using synthetic data. The release is scheduled for next year. According to FeltSteam, Arrakis may be more like a system that generates more efficient models, which could contribute to the development of GPT-3.5 Turbo.

In retrospect, the existence of the multimodal model Arrakis may explain a statement made by Andreas Braun, the Chief Technology Officer of Microsoft Germany, in early March 2023. One week before the official release of GPT-4, he mentioned "multimodal models" that can also generate videos. At that time, OpenAI was likely still developing Arrakis.