LLM System to Acquire Unlimited Context Length

It is well known that LLM forgets previous contextual information. The main reason for this is the limited context length of the model. Some even say that this is the biggest bottleneck in achieving Artificial General Intelligence (AGI).

Soon, the debate over which model has the maximum context length seems to be irrelevant. Microsoft, Google, and Meta are all moving towards the direction of making context length unlimited.

The end of Transformer?

Although all LLMs currently run on Transformers, it may soon become a thing of the past. For example, Meta has introduced MEGALODON, a neural network architecture designed specifically for efficient sequence modeling with unlimited context length.

MEGALODON aims to overcome the limitations of the Transformer architecture, such as quadratic computational complexity and limited inductive bias for length generalization. The model demonstrates excellent efficiency on a scale of 7 billion parameters and 20 trillion training tokens, outperforming other models like Llama 2 in terms of training loss.

It introduces key innovations such as the Complex Exponential Moving Average (CEMA) component and the Time Step Normalization layer, which improve long-context pretraining and data efficiency. These improvements enable MEGALODON to perform well in various tasks such as instruction fine-tuning, image classification, and autoregressive language modeling.

Most likely, Meta's Llama 3 will be based on the MEGALODON architecture, giving it unlimited context length.

Similarly, Google researchers have proposed a method called "Infini-Attention," which incorporates compressed memory into the regular attention mechanism. The paper titled "Not Missing Any Context" points out that Infini-Attention integrates compressed memory into the regular attention mechanism and combines masked local attention and long-term linear attention mechanisms in a single Transformer block.

This approach combines masked local attention and long-term linear attention mechanisms in a single Transformer block, allowing existing LLMs to handle infinitely long contexts with limited memory and computational resources.

This approach naturally extends to handling input sequences of up to millions in length and outperforms baselines on long-context language modeling benchmarks and book summarization tasks. After fine-tuning on up to 5K sequence length passphrase instances, the 1B model successfully solves problems of length 1M.

No more forgetting

In a similar vein, another group of Google researchers introduced Feedback Attention Memory (FAM). This is a novel Transformer architecture that utilizes feedback loops to enable the network to attend to its own latent representations, thereby generating working memory within the Transformer and enabling it to handle infinitely long sequences.

The introduction of FAM provides a new approach by adding feedback activations that feed back context representations to each sliding window attention block. This achieves integrated attention, block-level updates, information compression, and global context storage.

In addition, researchers from the Beijing Academy of Artificial Intelligence have proposed Activation Beacon, a method to extend LLM context length by compressing original activations into a compact form. This plugin component allows LLM to maintain its performance in shorter contexts while perceiving long contexts.

Activation Beacon employs a sliding window approach for streaming processing, improving training and inference efficiency. By training and changing compression rates using short sequence data, Activation Beacon can support different context lengths at a lower training cost. Experiments have verified that Activation Beacon is an effective, efficient, and low-cost solution for extending LLM context length.

Do we really need tokens?

In February of this year, Microsoft Research published a paper titled "LongRoPE: Extending LLM Context Window to Over 2 Million Tokens." This technology significantly increases the context length of LLM to an unprecedented 2048k tokens while maintaining their original performance within shorter context windows.

In addition, another research team from Microsoft challenged the traditional approach to LLM pretraining, which applies the next token prediction loss uniformly to all tokens in the training corpus. Instead, they proposed a new language model called RHO-1, which utilizes Selective Language Modeling (SLM).

The SLM approach addresses this issue by directly attending to token-level and eliminating unnecessary token loss during pretraining.

SLM first trains a reference language model on high-quality corpora to establish utility metrics for tokens based on the desired distribution. Tokens with high additional loss between the reference model and the training model are selected for training, allowing the language model to focus on tokens that are most beneficial for downstream applications.

No more "getting lost in the middle"?

For a long time, there has been a discussion about how longer context length window models get lost in the middle. Even with long-context LLMs, for accuracy, it is still recommended to choose shorter context length inputs. It is worth noting that facts at the beginning and end of the input are better preserved than those in the middle.

Jim Fan from NVIDIA AI explained that when it comes to improving LLM, claiming to have millions or billions of tokens is not helpful. "What really matters is how the model actually uses the context. Making wild claims is easy, but solving the real problem is harder," he said.

Meanwhile, to measure the efficiency of these longer context lengths, NVIDIA researchers developed RULER, a synthetic benchmark designed to evaluate long-context language models across multiple task categories, including retrieval, multi-hop reasoning, aggregation, and question answering.

All of this suggests that future LLM systems will have unlimited context length.