"Languages with a larger user base are more challenging for machine learning"

A few months ago, it was unimaginable that an AI-based "language model" could successfully mimic human language. The content written by ChatGPT is often indistinguishable from text generated by humans.

A research team from the Leibniz Institute for the German Language (IDS) in Mannheim, Germany, has now used text materials in 1,293 different languages to study the speed at which different computer language models "write". The surprising result is that languages with a large number of speakers are often more difficult for algorithms to learn, while languages with fewer speakers are relatively easier. This research has been published in the journal "Scientific Reports".

A language model is a computer algorithm that can process and generate human language. Language models can recognize patterns and regularities in large amounts of textual data, gradually learning to predict future text. One specific type of language model is the "Transformer" model, and the famous chatbot service ChatGPT is built on this model.

When the algorithm analyzes the sentence "In the dark night, I heard a sound in the distance...", it can predict words like "barking" or "noise" that can follow the sentence. This prediction is based on some kind of "understanding" of semantic relationships and word combinations in language.

In a new study, a team of linguists from IDS investigated the speed at which computer language models learn by training them on text materials in 1,293 languages. The team used both older, less complex language models and variants of the newer Transformer model mentioned above. They studied the time it took for different algorithms to develop an understanding of patterns in different languages.

The study found that the amount of text the algorithms needed to process, i.e., their ability to learn a language and predict subsequent content, varied depending on the language. The results showed that languages with a larger number of speakers were more difficult for language algorithms to learn compared to languages with fewer speakers.

However, the problem is not as simple as it sounds. To verify the relationship between learning difficulty and the number of language speakers, multiple factors need to be controlled.

The challenge lies in the fact that closely related languages (e.g., German and Swedish) are more similar, while distantly related languages (e.g., German and Thai) are relatively dissimilar. It is necessary to control not only the degree of correlation between languages but also other factors such as the geographical proximity between two languages or the quality of the text materials used for training.

"In our study, we employed various methods using statistics and machine learning to control possible confounding factors as accurately as possible," explained one of the authors of the study, Sascha Wolfer.

However, regardless of the methods used and the type of input text, a stable statistical correlation was found between machine learning ability and the number of speakers.

"The results really surprised us; based on the current state of research, we would have expected the opposite result: that languages with a larger population of speakers would be easier for machines to learn," said the main author of the study, Alexander Koplenig.

Currently, the reasons for this relationship are only speculative. For example, an earlier study led by the same research team indicated that larger languages are generally more complex. Therefore, the increased learning difficulty may make sense for human language learners: once you have learned a complex language, you have more diverse language choices and can express the same content in a more concise form.

However, further research is needed to validate these (or other) explanations. Koplenig pointed out, "We are still in a relatively early stage, and the next step is to explore whether our machine learning findings can be translated into human language acquisition."