Finnish AI Startup Silo AI Launches Open Source Large Model Poro

The Finnish Helsinki-based AI startup Silo AI has released a new open-source large language model (LLM) called Poro, aiming to promote the development of multilingual AI capabilities for European languages.

Poro is the first in a planned series of open-source models that will eventually cover all 24 official languages of the European Union. These models are jointly developed by Silo AI's SiloGen, its AI generation division established at the end of 2022, and the TurkuNLP research group at the University of Turku in Finland.

Peter Sarlin, CEO of Silo AI, said in an interview, "This is a matter of digital sovereignty, ensuring that there are models that can capture fundamental values, culture, and language." He added, "Ultimately, this is about value creation, ensuring that not only Europe but any company can create value and create value for proprietary models that stay in Europe and within the organization."

The Poro 34B model has 34.2 billion parameters and is named after the Finnish word for "reindeer." It uses the BLOOM transformer architecture and ALiBi embeddings. The model is trained on a multilingual dataset of 2.1 trillion tokens, covering languages such as English, Finnish, and programming languages like Python and Java.

Poro is being trained on LUMI, the fastest supercomputer in Europe, located in Kajaani, Finland. It provides 512 AMD Instinct MI250X GPUs and has a computing power of 74 petaflops.

According to Sarlin, the design goal of Poro is to address the core challenge of training effective natural language models for resource-scarce European languages like Finnish. By leveraging cross-lingual training methods, the model can utilize data from resource-rich languages like English.

The model is the second significant open-source LLM released in Europe, following Mistral 7B by the French startup Mistral AI at the end of 2023. It showcases Europe's achievements in the rapidly evolving field of generative AI and highlights the increasing competition among different AI labs and companies.

Poro Research Checkpoints

As part of SiloGen's commitment to transparency, the training process of Poro will be documented through the Poro Research Checkpoints initiative.

Sarlin explained, "We will release checkpoints during the model training process, which is a rather novel initiative." He also stated, "There hasn't been this level of detailed openness about model training before."

The initial checkpoint of Poro 34B covers 30% of the training progress. According to benchmark tests released by Silo AI, even after completing only 30% of its extensive training plan, Poro still achieves state-of-the-art results.

In a widely used Finnish language evaluation called FIN-bench, Poro outperforms existing monolingual Finnish AI models designed specifically for the task, such as FinGPT.

Sarlin pointed out, "Even after training 30%, the model's performance in low-resource languages surpasses previous attempts." By leveraging shared patterns among related languages, Poro has an advantage in languages with limited training data.

It is worth noting that Poro's multilingual capability does not compromise its performance in English. Testing on standardized English evaluation sets shows that the model "already surpasses existing models in Finnish benchmarks after 30% training and performs comparably or nearly comparably in English performance."

Open-source models as an alternative for big tech companies

Sarlin believes that open-source models like Poro represent the future of AI, providing a transparent and ethical alternative to closed models by large tech companies.

He stated, "I personally think there will be many open-source alternatives in the end." He also said, "The safest way is actually to choose the open-source way and have a full understanding of how these models are built and architected."

He added, "We are working very hard to ensure both the data and the models comply with regulatory requirements."

Silo AI plans to continue regularly releasing Poro checkpoints throughout the entire training process. The ultimate goal is to create a complete series of open-source models covering all European languages. If preliminary results are any indication, Poro may soon compete with large tech companies.

Collaboration with the University of Turku

Poro represents a part of the ongoing collaboration between Silo AI and the University of Turku in Finland. Researchers from the university's TurkuNLP group have been pioneers in developing open-source resources and models for the Finnish language.

Sarlin said, "My research group joined, and some professors joined, and we basically support ourselves through revenue and self-growth. We are different from other companies." He added, "We have over 300 people, most of whom have PhD degrees in AI-related fields."

This collaboration combines Silo AI's applied AI expertise and computational resources with the university's leadership in multilingual language modeling research. Sarlin stated that this represents a model of how the industry and academia can collaborate to drive AI capabilities, especially for resource-scarce European languages.

Will Europe become a leader in open-source AI?

The release of Poro signifies a new era of openness, collaboration, and transparency in the field of natural language processing. Initiatives like Poro Research Checkpoints provide the community with access to tools and insights that were previously only available to large tech companies.

Sarlin said, "We collaborate with many big brands like Allianz, Rolls-Royce, Honda, and Philips." He also mentioned, "We have been hearing for a while now that these big enterprises are very concerned about the final regulatory requirements and what models they can use."

If Poro delivers on its promises, it will democratize access to high-performance multilingual models and provide Europe with a local alternative, breaking free from systems controlled by US tech companies. While still in its early stages, Poro represents a significant milestone in bringing language AI from proprietary domains into the open-source realm.