H2O AI Launches Ultra-Compact LLM for Mobile Devices

H2O AI, a company dedicated to popularizing artificial intelligence (AI) through a range of open source and proprietary tools, has announced the launch of Danube, a new ultra-compact large language model (LLM) for mobile devices.

This open source model, named after the second largest river in Europe, has 1.8 billion parameters and is said to outperform similar-sized models in a range of natural language tasks, putting it on par with powerful products from Microsoft, Stability AI, and Eleuther AI.

This announcement comes at an opportune time. Companies building consumer devices are actively exploring the potential of offline generative AI, which allows models to run locally on products, providing users with cross-functional quick assistance while eliminating the need to transmit information to the cloud.

"We are excited to release H2O-Danube-1.8B portable LLM on small devices like your smartphone... The proliferation of smaller, lower-cost hardware and more efficient training now enables medium-sized models to reach a wider audience... We believe H2O-Danube-1.8B will be a disruptor in the field of mobile offline applications," said Sri Ambati, CEO and co-founder of H2O, in a statement.

What can we expect from Danube-1.8B LLM?

Although just announced, H2O claims that Danube can handle various natural language applications on small devices, including common sense reasoning, reading comprehension, summarization, and translation, through fine-tuning.

To train this compact model, the company collected trillions of tokens from various web sources and used techniques extracted from the Llama 2 and Mistral models to enhance its generative capabilities.

"We adjusted the architecture of Llama 2 to have a total of approximately 1.8 billion parameters. We (then) used the tokenizer from the original Llama 2 with a vocabulary size of 32,000 and trained our model with a context length of 16,384. We adopted the sliding window attention from Mistral with a size of 4,096," the company described the model architecture on Hugging Face.

In benchmark tests, the model performs comparably or even better than most models in the 1-2 billion parameter category.

For example, in the Hellaswag test, which evaluates common sense natural language reasoning, it achieves an accuracy of 69.58%, second only to Stability AI's Stable LM 2 model with 1.6 billion parameters, which was pre-trained on 200 trillion tokens. Similarly, in the Arc advanced question-answering benchmark, it ranks third with an accuracy of 39.42%, behind Microsoft's Phi 1.5 (1.3 billion parameter model) and Stable LM 2.

To drive adoption of the model, H2O has released related tools.

To simplify the application process of the model, H2O has released Danube-1.8B under the Apache 2.0 license for commercial use. Any team wishing to use the model for mobile use cases can download it from Hugging Face and fine-tune it for their applications.

To further simplify this process, the company plans to release additional tools soon. It has also released a chat-tuned version of the model (H2O-Danube-1.8B-Chat) for use in chat applications.

In the long run, the introduction of Danube and similar compact models is expected to drive the surge of offline generative AI applications on smartphones and laptops, aiding tasks such as email summarization, typing, and image editing. In fact, Samsung has already taken steps in this direction with the launch of its S24 series smartphones.