Microsoft's UniLM AI Ascends to Top Spot in Abstractive Summarization and Language Generation Field

2019-10-17

Language model pre-training techniques can teach machine learning systems to logically represent text by predicting words based on context. This approach has shown progress in various natural language processing tasks. However, models like Google's BERT are designed to be bidirectional and are not suitable for generating natural language through extensive modifications. To overcome this limitation, researchers at Microsoft Research have developed an alternative method called Unified Language Model (UniLM), which can perform unidirectional, sequence-to-sequence, and bidirectional prediction tasks, as well as fine-tuning for natural language understanding and generation. Microsoft researchers claim that UniLM has achieved outstanding results in the fields of abstract summarization, generative question answering, and language generation data sets. It can compete with BERT on general benchmarks and achieve unprecedented performance, reaching the state-of-the-art level. UniLM is a network based on multi-layer Transformers, which can preprocess a large amount of text and optimize language modeling. Transformers consist of interconnected neurons (functions) that can input data signals and adjust the strength (weights) of each connection. Almost all artificial intelligence systems extract features and make predictions in this way. The difference with Transformers is that each output element can be connected to each input element, and the weights are dynamically calculated. According to the researchers, pre-trained UniLM is similar to BERT and can be fine-tuned. If needed, task-specific layers can be added to adapt to various downstream tasks. However, unlike BERT, UniLM can be configured with different self-attention masks to aggregate context for different types of language models. Additionally, because their pre-training nature is unified, Transformer networks can share parameters, making the learned text representations more general and reducing overfitting to individual tasks. The researchers pre-trained UniLM using articles from the English Wikipedia and open-source book corpora, with a total vocabulary of 28,996 words. They reported impressive performance of UniLM in language tasks, achieving comparable results to BERT on the Glue benchmark and two question answering data sets, and surpassing previous state-of-the-art models on five natural language generation data sets: Gigaword (summarization), Squad (question generation), Coqa (generative question answering), and Dstc7 (dialogue response generation). The Microsoft team stated that in future work, they will train larger models using "network-scale" text corpora to push the limits of current methods. Additionally, they plan to explore expanding UniLM to support cross-lingual tasks. Code and pre-trained model link: [https://github.com/microsoft/unilm](https://github.com/microsoft/unilm)

RECENT AI NEWS

RECENT AI TOOLS