The Phenomenon of Overconfidence in LLM

2023-09-11

Developers trying to optimize LLM often encounter a series of problems. An experiment by Jonathan Whitaker and Jeremy Howard from fast.ai revealed a problem in the LLM model that has been overlooked, which should not be confused with the widely discussed LLM illusion.

Overconfidence refers to the model's insistence that certain information provided in the dataset is correct, even for the proposed problem. This may be caused by the notorious terms of underfitting and overfitting.

First, overfitting refers to the model being too fine-tuned and overly close to the training data. On the contrary, underfitting means that the model does not have enough training data for prediction. This balance is often referred to as the bias-variance tradeoff.

To address these issues, developers have applied various techniques, some of which are effective and some may bring other problems. In the case of fast.ai researchers, they attempted to train the model on a single example, and they were surprised to find that the results were very different from what they expected.

Overconfidence in LLM

When the model faces new unseen data, even if it is incorrect, it may express unnecessary confidence in its predictions. This is contrary to the traditional notion that neural networks usually require a large number of examples because the loss surface during training is rugged.

Imagine a language model that performs disease diagnosis based on refined medical datasets using patient descriptions. When presented with cases with obvious symptoms and clear diagnostic criteria, the model assigns a high probability to a specific disease. For example, if a patient describes typical symptoms of the flu, the model may assign a probability close to 1.0 for the diagnosis of the flu.

However, when faced with complex medical cases with ambiguous symptoms or the possibility of multiple diseases, the model may evenly distribute probabilities among different diagnostic options, indicating its uncertainty in the correct diagnosis.

Similarly, when training neural network classifiers (which usually encounter large datasets repeatedly), Howard and Whitaker noticed that even a single example of an input-output pair can have a significant impact on these models. The research found that these models exhibit overconfidence during the training process. As their confidence increases, they assign probabilities close to 1.0 to predictions that are related to their own predictions, even if these predictions are incorrect.

Especially in the early stages of training, this overconfidence raises questions about how neural networks handle new information and adapt to it.

They found that the model can learn to make accurate predictions after seeing a single example, which actually means memorizing the training data (a singleton) and demonstrating strong generalization ability, making it unlikely to overfit. The goal is to make machine learning efficient and make reliable predictions by adjusting its confidence scores.

Is overfitting the cause of overconfident models?

Although overfitting, the phenomenon of models becoming too specific to the training data, is a well-known challenge in machine learning, the real problem here seems to be overconfidence. These predictions lead to an unexpected result: the validation loss, which measures the model's performance on unseen data, becomes worse even if the model's training loss improves.

As expected, this experiment sparked several discussions in a HackerNews thread. When the model knows the training data too well, its performance on new data will be poor. The researchers of the model explained that they did not point out any problems, but only pointed out an opportunity to train the model with a single example.

Interestingly, these two terms are closely related, and overconfidence may be a symptom of overfitting. When the model overfits, it learns statistical noise and potential patterns in the training data. This may cause the model to be overly confident in its predictions, even if these predictions are not accurate.

However, overconfidence is not always caused by overfitting. If the model has not been trained with enough data or the data does not represent the real world, the model may also be overconfident.

Lucas Beyer, a researcher at Google AI, clarified that these findings are specific to fine-tuning pre-trained models and may not necessarily change the initial pre-training approach of the model. He also pointed out that these findings are more applicable to fine-tuning scenarios and may not be relevant to training models from scratch.

Although this experiment has other issues and criticisms, one oversight that no one missed is the lack of providing the base model or any details of the model used in this experiment. It is not even clear whether they repeatedly used the same dataset to fine-tune the model, leading to the phenomena of overfitting and overconfidence.

RECENT AI NEWS

RECENT AI TOOLS