"Huawei Noah's Ark Open Sources Pre-trained Model 'Nezha', Achieving SOTA in Multiple Tasks"

2020-01-14

After BERT, new pre-training language models such as XLnet, RoBERTa, and ERNIE have been continuously introduced. This time, Huawei Noah's Ark Lab has open-sourced NEZHA (Nezha), a Chinese pre-training language model based on BERT. The model is named after Nezha, a mythical character known for having three heads and six arms, symbolizing its ability to handle various natural language tasks effectively. According to the introduction, the current version of NEZHA is based on the BERT model and has undergone multiple optimizations, achieving state-of-the-art performance in a series of Chinese natural language understanding tasks. The NEZHA model was experimented with five Chinese natural language understanding tasks, namely CMRC (Chinese Machine Reading Comprehension), XNLI (Cross-lingual Natural Language Inference), LCQMC (Large-scale Chinese Question Matching), PD-NER (Named Entity Recognition in PeoplesDaily), and ChnSenti (Chinese Sentiment Classification). Researchers trained the NEZHA model on Chinese Wikipedia, Chinese news, and Baidu Baike data. They compared NEZHA with Google's Chinese BERT, BERT-WWM jointly released by Harbin Institute of Technology and iFLYTEK, and ERNIE-Baidu released by Baidu. As shown in the table below, NEZHA achieved state-of-the-art performance (SOTA) in XNLI, LCQMC, PeoplesDaily NER, and ChnSenti tasks. NEZHA, NEZHA-WWM, and NEZHA-Span in the table represent NEZHA models trained from the original BERT pre-training task, with whole-word masking training, and with span prediction task training, respectively (all three models use full functional relative position encoding). In addition, Noah's Ark Lab also open-sourced the TinyBERT pre-training language model. Although language model pre-training, such as BERT, has greatly improved the performance of many natural language processing tasks, pre-training language models are often computationally expensive and require a large amount of memory, making it difficult to execute them effectively on resource-constrained devices. To accelerate inference speed, reduce model size, and maintain accuracy, Huawei researchers proposed a novel transformer distillation method specifically designed for transformer-based models, called knowledge distillation (KD). By utilizing this new knowledge distillation method, a significant amount of knowledge encoded in BERT can be effectively transferred to TinyBERT. Additionally, they introduced a new two-stage learning framework for TinyBERT, which applies the transformer distillation method in both the pre-training stage and the task-specific learning stage. This framework ensures that TinyBERT can capture both general-domain knowledge and task-specific knowledge from BERT. In the GLUE benchmark test, TinyBERT is 7.5 times smaller than BERT and 9.4 times faster in inference speed, while still maintaining competitive advantages in natural language understanding tasks.

RECENT AI NEWS

RECENT AI TOOLS