Apple releases CoreNet and OpenELM models to support open AI research.

Apple has launched a new deep neural network training library called CoreNet and an advanced open language model series called OpenELM. These models are now publicly available on the Hugging Face hub, accompanied by a comprehensive framework for training and evaluation on public datasets.

CoreNet, which can be found on GitHub, is a powerful toolkit that supports various tasks including object detection and semantic segmentation. The OpenELM models, built on CoreNet, achieve higher accuracy by optimizing parameter allocation in transformer models. Apple has released four pre-trained and fine-tuned models with different parameter scales: 270 million, 450 million, 1.1 billion, and 3 billion parameters.

What sets OpenELM apart is its unique hierarchical scaling strategy. This strategy gradually reduces the dimensions of attention and feed-forward modules closer to the input in the transformer layers, and gradually expands them as they approach the output. This design improves parameter utilization and enhances model accuracy, offering significant advantages over existing models.

Notably, the 1 billion parameter version of OpenELM outperforms similar OLMo models by 2.36% with only half the pre-training tokens, but still lags behind Microsoft's recently released Phi-3 small language model.

Apple's release of CoreNet and OpenELM undoubtedly contributes to the open research community and marks the tech giant's first attempt in this field. The company not only shares model weights and inference code, but also provides training logs, checkpoints, and pre-training configurations for others to use and improve these models. Additionally, Apple offers code to convert the models into the MLX library, enabling efficient inference and fine-tuning on Apple devices.

Apple's decision to open-source OpenELM may have been influenced by peers such as Google, Microsoft, and Meta. These companies actively contribute advanced open models to the AI community and have received widespread acclaim. Apple's move reflects both the industry's demand for innovation and openness, as well as its efforts to maintain competitiveness and industry relevance.

Furthermore, Apple's emphasis on small parameter models like OpenELM aligns with its strategy of running AI on devices rather than in the cloud. This approach not only protects user privacy but also enables efficient on-device inference, which is crucial for Apple's product and service ecosystem.