Baichuan Intelligence Launches Global Longest Context Window Large Model Baichuan2-192K, Leading Claude2

Baichuan Intelligence released the Baichuan2-192K large model on October 30th, with a context window length of 192K, the longest in the world. Baichuan2-192K can process about 350,000 Chinese characters, which is 4.4 times that of the current excellent long-window large model Claude2 and 14 times that of GPT-4.

Baichuan2-192K also leads Claude2 comprehensively in terms of long-window text generation quality, long-context understanding, long-text question answering, summarization, and other aspects. It has achieved the State-of-the-Art (SOTA) in 7 out of 10 long-text evaluation sets in Chinese and English, including Dureader, NarrativeQA, LSHT, and TriviaQA.

Baichuan Intelligence has achieved a balance between window length and model performance through extreme optimization of algorithms and engineering. In terms of algorithms, Baichuan Intelligence proposes an extrapolation solution for RoPE and ALiBi dynamic position encoding, which can dynamically interpolate different degrees of attention mask for ALiBi position encoding of different lengths, enhancing the model's ability to model long sequence dependencies while ensuring resolution. In terms of engineering, Baichuan Intelligence integrates tensor parallelism, pipeline parallelism, sequence parallelism, recomputation, and Offload functions on the basis of its self-developed distributed training framework, creating a comprehensive 4D parallel distributed solution. This solution can automatically find the most suitable distributed strategy based on the specific workload of the model, greatly reducing the GPU memory usage during long-window training and inference processes.

Baichuan Intelligence has opened the API interface of Baichuan2-192K and launched API internal testing, making it available to core partners in industries such as law, media, and finance. Baichuan2-192K can process and analyze hundreds of pages of materials at once, providing significant assistance in extracting and analyzing key information from long documents, generating summaries for long documents, conducting long document reviews, writing long articles or reports, and assisting in complex programming tasks.