Microsoft Researchers: Not All Tokens Are Indispensable

Microsoft researchers have challenged the traditional approach of pre-training language models (LM) by applying the next token prediction loss uniformly to all tokens in the training corpus. Instead, they propose a new language model called RHO-1, which utilizes Selective Language Modeling (SLM). This approach selectively trains useful tokens that align with the desired distribution, rather than attempting to predict every next token. They introduce the Rho-Math-v0.1 model, with Rho-Math-1B and Rho-Math-7B achieving 15.6% and 31.0% few-shot accuracy on the MATH dataset, respectively - matching DeepSeekMath but with only 3% pre-trained tokens. Rho-Math-1B-Interpreter is the first 1B LLM to achieve over 40% accuracy on MATH. Rho-Math-7B-Interpreter achieves 52% accuracy on the MATH dataset, with fine-tuning using only 69k samples. The SLM method of RHO-1 involves scoring pre-trained tokens using a reference model and training the language model on tokens with higher residual loss, focusing on the loss. This selection process allows RHO-1 to improve few-shot accuracy for 9 mathematical tasks by up to 30% during continuous pre-training on the 15B OpenWebMath corpus. After fine-tuning, the model also achieves state-of-the-art results on the MATH dataset and an average improvement of 6.8% across 15 different tasks when pre-trained on 80B general tokens. Traditional training methods often use heuristics and classifiers to filter data at the document level to improve data quality and model performance. However, even high-quality datasets may contain noise tokens that negatively impact training. The SLM method directly addresses this issue by focusing on the token level and canceling the loss of unnecessary tokens during pre-training. SLM first trains a reference language model on high-quality corpora to establish utility metrics for scoring tokens based on the desired distribution. Tokens with higher residual loss are selected for training between the reference model and the training model, allowing the language model to focus on the most beneficial tokens for downstream applications. In the study, SLM selected tokens during pre-training that were closely related to mathematics, effectively focusing the model on the relevant parts of the original corpus. By examining the token filtering at various checkpoints, researchers found that tokens selected in later checkpoints had higher perplexity in the later stages of training, while lower perplexity in the early stages. The discussion section focuses on introducing future work, including the potential extension of SLM beyond the field of mathematics, scalability of the technique to larger models and datasets, and exploring the need to train reference models for token scoring. Improvements to SLM may include reweighting tokens instead of selecting them and using multiple reference models to reduce overfitting. SLM may be extended to supervised fine-tuning to address noise and distribution mismatch in datasets and potentially achieve a naturally aligned base model for alignment tasks by training reference models that emphasize usefulness, authenticity, and harmlessness during pre-training.