Microsoft Introduces XOT Method to Enhance Language Model Reasoning Reliability

Microsoft has launched "Everything on Thinking" (XOT), which can integrate knowledge from external domains and generate more reliable reasoning in language models.

Complex prompting engineering methods are usually aimed at enhancing the reasoning ability of large language models. From simple chain thinking prompts to more complex tree thinking prompts, they attempt to decompose problems into so-called "thinking". A thinking is a simple sentence that describes the result of a simpler sub-problem or conclusion, along with a related action, such as a simple solution to solve a sub-problem and obtain a new result.

The new method "Everything on Thinking" (XOT) developed by researchers from Microsoft, Georgia Institute of Technology, and East China Normal University aims to extend the capabilities of language models through an AlphaZero-inspired external module. XOT integrates external domain knowledge into thinking using reinforcement learning and Monte Carlo tree search (MCTS). The researchers say that this can enable language models to efficiently generalize to unknown problems.

AlphaZero-inspired XOT method outsources the search for thinking structures

Specifically, XOT utilizes MCTS search to help solve the thinking structure of problems. During the training phase, MCTS is used to explore possible solutions - thinking structures - for specific tasks such as puzzles. This process involves recording the state, value, and visit frequency of thinking nodes in the search. Then, the recorded data is used to train the model through reinforcement learning to predict potentially successful solution paths - eliminating the need to search the entire solution tree for each problem. Ideally, the model can generalize to new problems in games.

Next, the team links the model with the language model, providing it with thinking structures that can solve problems posed by the language model. In a collaborative process, the language model reviews thinking and thinking structures and can request revisions to improve the quality of solutions. By using an external model, XOT eliminates the need for the language model to explore and evaluate thinking on its own. Compared to other methods, using an external model greatly reduces the demand on the language model.

XOT demonstrates breakthrough performance in test scenarios

The researchers tested XOT on several challenging problem-solving tasks, including the 24-point game, 8-puzzle game, and pocket cube. The results showed that XOT outperformed other methods significantly and even solved problems that other methods failed to solve. However, XOT did not achieve 100% reliability.

Nevertheless, the team believes that the XOT framework is a promising approach to integrating external domain knowledge into language model reasoning. They state that it offers improvements in performance, efficiency, and flexibility, which are a combination that other methods cannot achieve.

It is currently unclear whether Microsoft intends to use this method in its own products. Google Gemini may employ a similar approach: Demis Hassabis, CEO of Google DeepMind, revealed in an interview that they hope to apply the ideas of AlphaGo to Gemini.