Meta releases Llama 2 Long, surpassing GPT-3.5 Turbo and Claude 2

At the Meta Connect conference held this week, Meta showcased a series of new artificial intelligence features for its consumer-facing services Facebook, Instagram, and WhatsApp.

However, the most significant news from the Mark Zuckerberg company may be a computer science paper published by Meta researchers on the open access and non-peer-reviewed website arXiv.org, without any public promotion.

The paper introduces Llama 2 Long, a new artificial intelligence model based on Meta's open-source Llama 2 released this summer, which has been "continuously pre-trained from Llama 2 using long training sequences and upsampled long texts," the researchers wrote in the paper.

As a result, this new extended artificial intelligence model from Meta surpasses some leading competitors in generating replies to long user prompts, including OpenAI's GPT-3.5 Turbo (with a 16,000-character context window) and Claude 2 (with a 100,000-character context window), according to the paper.

The Birth of Llama 2 Long

Meta researchers selected different training parameter sizes for the original Llama 2 - algorithms that can change their data and information values during the learning process. For Llama 2, these values were 7 billion, 13 billion, 34 billion, and 70 billion variants, and additional 400 billion tokens of long text data sources were added to the original Llama 2 training dataset.

Then, the researchers kept the structure of the original Llama 2 unchanged and made only "necessary modifications" to the positional encoding, which is crucial when the model needs to pay attention to longer content.

This modification involves changes to the Relative Positional Encoding (RoPE) encoding, a method of programming transformer models in language models like Llama 2 (and Llama 2 Long) that maps their token embeddings (numbers used to represent words, concepts, and ideas) onto a 3D grid, showing their positions relative to other tokens, even when rotated. This allows the model to generate accurate and useful replies with less required information (and therefore less computational storage space).

Meta researchers reduced the rotation angle of the RoPE encoding from Llama 2 to Llama 2 Long, enabling them to ensure that the model's knowledge base includes more "remote tokens" - tokens that occur less frequently or have less relation to other information.

By using human feedback reinforcement learning (RLHF) and synthetic data generated by Llama 2 chat, a common training method for AI models where AI is rewarded with human supervision for answering questions correctly, the researchers were able to improve its performance in common language model tasks, including encoding, mathematics, language understanding, common-sense reasoning, and answering user questions.

Considering the outstanding performance of Llama 2 compared to regular Llama 2 as well as Anthropic's Claude 2 and OpenAI's GPT-3.5 Turbo, it is not difficult to understand why the open-source AI community on Reddit, Twitter, and Hacker News expressed their admiration and excitement for Llama 2 from the moment the paper was released. This is a major validation of Meta's "open-source" approach in the field of AI model generation and demonstrates that open-source can compete with closed-source models provided by well-funded startups that require payment to use.