Apple Surprises with Open Source Training Method

2023-09-11

Apple recently open-sourced the code for its internal software used to train the new Ajax LLM on GitHub, which has come as a surprise to many.

Two months ago, news about Apple developing a generative AI tool and chatbot caused a positive market sentiment, leading to a record-breaking surge in Apple's stock price to $198.23, a 2.3% increase. However, apart from Apple using Ajax in LLM and internally naming it AppleGPT, no other details about the model have been disclosed.

According to a report by The Information, Apple has recently made an unprecedented move by open-sourcing its code on GitHub. It is believed that Apple is training Ajax GPT with over 200 billion parameters, making it expected to be more powerful than GPT-3.5.

Unprecedented Move

In July of this year, Apple quietly released the code for AXLearn on GitHub, allowing the public to train their own large-scale language models without starting from scratch. AXLearn is an internal software developed by Apple over the past year, which is a pre-built tool for quickly training machine learning models. Ajax is a derivative of JAX, an open-source framework created by Google researchers, and some components of AXLearn are specifically designed for optimization on Google TPU.

While Apple may be at the forefront of bringing innovative solutions, the disadvantage of prioritizing their agenda above everything else cannot be ignored. Apple is known for its closed-source environment, where their technology and code are not open to the public. While major tech companies are releasing superior open-source models, such as Meta's Llama-2, Anthropic's Claude-2, Falcon, Vicuna, Apple has been sticking to its traditional secretive approach, just like OpenAI. Apple's closed-source approach has faced criticism from the tech community, labeling it as a company that benefits from research released by major tech companies but never gives back.

Apple's decision to open-source its training software AXLearn is an important step in its secretive approach. This move may help foster collaboration and innovation within the AI research community and reflects the increasingly open trend in AI development.

Although the exact motivation behind Apple's code release on GitHub has not been disclosed, it is evident that the company's significant investments in AI development, amounting to millions of dollars daily, reflect its determination in the AI competition.

Interestingly, the company applied for the trademark "AXLearn" in Hong Kong last month.

Emulating Google's Culture

Apple's AI chief, John Giannandrea, and the head of its conversational AI team, Ruoming Pang, both have extensive experience from their previous work at Google. Giannandrea brings with him a vision of aligning Apple with Google, where employees have more freedom for various research, publishing papers, and exploring innovative ideas. Apple's previous restrictions in these areas hindered talent growth and recruitment.

Reportedly, Apple has also hired talent from Google and Meta's AI platform teams. Among the 18 contributors who have made contributions to AXLearn on GitHub in the past two years, at least 7 have previously worked at Google or Meta. Apple is likely cultivating talent through the research community, making open-source the right path.

Decoding Clues

From the gathered information, we can conclude that Apple has formed two new teams dedicated to language and image model development. Apple's recent AI research papers suggest that they are working on software capable of generating images, videos, and 3D scenes, indicating their research in multimodal AI.

However, there is still uncertainty about the details of integrating LLM into Apple's products. Apple typically prefers to introduce new software to its devices, but integrating a 200 billion parameter LLM that requires more storage space and computational power into an iPhone is not feasible. The company may focus on developing smaller models for integration into phones, or the model may be used for other purposes, with specific details yet to be clarified.

RECENT AI NEWS

RECENT AI TOOLS