Model Quiet-STaR Enables AI to "Think" Like Humans

Humans are naturally equipped with the ability to reason: the ability to think about "if," "why," and the implied meaning, as well as the ability to infer unspoken information. These are all key to our problem-solving abilities.

So far, artificial intelligence models have struggled in this regard. However, researchers from Stanford University and Notbad AI have now revealed that they have taught an AI model to think before responding to prompts, just like (most) people consider what to say before speaking.

The researchers introduced Quiet-STaR, an extension of the Self-Taught Reasoner (STaR) model, which has been trained on a large amount of internet data to generate reasons at each token to explain future text and improve predictions.

Quiet-STaR was applied to Mistral 7B and showed improved zero-shot reasoning capabilities on the CommonsenseQA question-answering challenge (from a baseline of 36.3% to 47.2%) and the GSM8K elementary school math word problems dataset (from a baseline of 5.9% to 10.9%). Furthermore, these improvements consistently increased as the number of tokens used in the "internal thinking" of the model increased.

"Quiet-STaR represents a step towards language models that can learn reasoning in a more general and scalable way," the researchers wrote.

Shortcomings of AI reasoning so far

Previous methods that helped language models learn reasoning from their inference focused more on specificity and lacked generality: AI was trained to solve individual tasks or relied on carefully curated datasets for predetermined tasks.

For example, the developers of Quiet-STaR pointed out that a pre-trained language model performed better in outputting human-like reasoning trajectories before answering multiple-choice questions compared to AI trained directly on answers. Other models were able to generate a series of solution approaches without additional supervision when provided with a "scaffold." Additionally, the models were "forced" to use a series of reasoning steps and were not allowed to answer unless they were fully confident.

"However, these methods again only apply to question-answering datasets," argued the researchers from Stanford University and Notbad AI.

STaR, in particular, demonstrated that models can be "guided" in their reasoning abilities on question-answering datasets. They can sample reasoning attempts to answer questions and if these reasoning attempts lead to the correct answer, they are trained on these reasoning attempts and iteratively solve increasingly difficult problems.

However, the researchers behind Quiet-STaR pointed out that training from curated datasets limits the "scale and generalizability" of reasoning. High-quality datasets "essentially cover only a subset of reasoning tasks."

The researchers assert that inferring reasoning from a few examples in question-answering is a "highly constrained setting." "Ideally, language models should learn to infer unspoken reasoning from arbitrary text."

By extending STaR, "we allow language models to learn from diverse reasoning tasks in language. To our knowledge, this is the first explicit work training language models to reason generally from text, rather than training on curated reasoning tasks or sets of reasoning tasks," they wrote.

"Quietly" thinking

The researchers from Stanford University and Notbad AI refer to their technique as Quiet-STaR because it "quietly" applies STaR.

The method generates many internal thoughts in parallel at each token to explain future text (i.e., the "thinking" process) before responding to prompts. When the AI finally answers, it produces a mixture of reasoned and unreasoned predictions.

The REINFORCE algorithm is then applied; in reinforcement learning, this collects samples in an episode to update policy parameters as well as embeddings at the start and end of thinking. The researchers explained that this helps increase the likelihood of the AI accurately predicting future text. As part of this process, the model also discards incorrect predictions.

"By iteratively optimizing these parameters, Quiet-STaR trains the model to generate more useful reasons during the training process," the researchers wrote.

Because their goal is general reasoning, they used zero-shot prompts ("Let's think step by step") without contextual examples. Quiet-STaR was applied to Mistral 7B, using the OpenWebMath and Colossal Clean Crawled Corpus datasets.

"Quiet-STaR... allows the model to think quietly at each token, and the trained distribution is useful," the researchers added.

They further noted, "By training on diverse reasoning tasks implicit in diverse web text, rather than narrowly specialized on specific datasets, Quiet-STaR points the way towards more robust and adaptive language models."

Narrowing the gap between models and human reasoning abilities

It is worth noting that the researchers created a parallel sampling algorithm that generates reasoning from all tokens in a string. This allows tokens to "attend to themselves" as well as all preceding tokens and preceding text with the same thinking. This enables "parallel continuations of all thoughts," and each reasoning call generates an additional token for all tokens.

The researchers introduced custom meta-tokens at the start and end of each thought: <|startofthought|> and <|endofthought|>, initialized as dashes ("—"), which are commonly used to indicate pauses.

"Intuitively, the start-of-thought token can be understood as putting the model in 'thinking mode,'" the researchers explained, "while the end-of-thought token can be understood as telling the model when to stop thinking."

The next step is to introduce so-called "mixing heads," which are "shallow" multi-layer perceptrons. This helps the researchers retrospectively determine the extent to which predictions from the next token in a given thought should be included in the current next token prediction.

Finally, the researchers optimized the parameters to increase the likelihood of future text. Reinforcement techniques provide "learning signals" for reasoning based on their impact on future predictions. To help reduce variance, the researchers also introduced a "teacher forcing" technique to ensure that the neural network stays as close as possible to the true sequence.

In conclusion, the researchers stated, "Quiet-STaR represents a step towards language models that can learn reasoning in a more general and scalable way. Future work can further narrow the gap between language models and human-like reasoning abilities based on these insights."