AnomalyGPT: A Novel IAD Method Based on Large-scale Vision-Language Model (LVLM)

2023-09-04

In various natural language processing (NLP) tasks, large language models (LLMs) such as GPT-3.5 and LLaMA have demonstrated outstanding performance. Recently, advanced techniques that align visual information with textual features, such as MiniGPT-4, BLIP-2, and PandaGPT, have expanded the ability of LLMs to interpret visual information, leading to significant transformations in the field of artificial general intelligence (AGI). However, despite being pretrained on a large amount of data obtained from the internet, LVLMs still have limited potential in industrial anomaly detection (IAD) tasks. Additionally, their mastery of domain-specific knowledge is only moderate and they need to be more sensitive to local features within objects. The goal of IAD tasks is to identify and locate anomalies in photos of industrial products.

Since real-world examples are rare and unpredictable, models must be trained only on normal samples to recognize abnormal samples that differ from the normal ones. Currently, most IAD systems only provide anomaly scores for test samples and require manual definition of standards for distinguishing normal and abnormal instances for each object category, making them unsuitable for practical production scenarios. Researchers from the Chinese Academy of Sciences, University of Chinese Academy of Sciences, Objecteye Company, and Wuhan AI Research Institute have proposed a unique IAD method based on LVLM, as shown in Figure 1, because existing IAD methods and LVLMs cannot handle IAD problems well. AnomalyGPT can identify anomalies and their locations without the need for manual threshold adjustment.

Furthermore, their method can provide image information and facilitate interactive interactions, allowing users to make subsequent queries based on their needs and responses. With just a few normal samples, AnomalyGPT can also learn in context to quickly adapt to new objects. They optimize LVLM using synthetic visual-text data and incorporate IAD domain knowledge. However, training directly on IAD data still needs improvement. The first challenge is data scarcity. They pretrained on 160,000 photos with multi-turn dialogues, including technologies like LLaVA and PandaGPT. However, the available IAD datasets currently have a small number of samples, making direct fine-tuning prone to overfitting and catastrophic forgetting.

To address this issue, they fine-tune LVLM using query embeddings instead of parameter fine-tuning. After the image input, additional query embeddings are inserted to add extra IAD information to LVLM. The second challenge involves fine-grained semantics. They propose using a decoder based on visual-text feature matching to obtain pixel-level anomaly localization results. The decoder's output is available for both LVLM and the original test image through query embeddings. This allows LVLM to simultaneously use the original image and the decoder's output to identify anomalies, improving its judgment accuracy. They conducted comprehensive experiments on the MVTec-AD and VisA databases.

They achieved unsupervised training on the MVTec-AD dataset, obtaining an accuracy of 93.3%, image-level AUC of 97.4%, and pixel-level AUC of 93.1%. After transferring a shot to the VisA dataset, they achieved an accuracy of 77.4%, image-level AUC of 87.4%, and pixel-level AUC of 96.2%. On the other hand, after unsupervised training a shot from the VisA dataset and transferring it to the MVTec-AD dataset, they achieved an accuracy of 86.1%, image-level AUC of 94.1%, and pixel-level AUC of 95.3%.

Their contributions can be summarized as follows:

  • They propose an innovative approach of using LVLM for handling IAD tasks. Their method allows for multi-turn discussions and can detect and locate anomalies without manual threshold adjustment. Their lightweight decoder based on visual-text feature matching addresses the weak fine-grained semantic understanding capability of LLMs. It alleviates the limitation of LLMs only being able to generate textual outputs. To their knowledge, they are the first to successfully apply LVLM to industrial anomaly detection research.
  • To maintain the intrinsic capabilities of LVLM and achieve multi-turn dialogue, they train the model simultaneously with the data used during LVLM pretraining and fine-tune it using query embeddings.
  • Their method exhibits strong transferability and can perform few-shot learning in context on new datasets, yielding excellent results.

RECENT AI NEWS

RECENT AI TOOLS