Visual Analogy Study: Revealing the Gap in Visual Intelligence of Artificial Intelligence

The field of artificial intelligence has long been plagued by a question: what exactly is intelligence? Artificial intelligence models like GPT-4 highlight this uncertainty: some researchers believe that the GPT model shows signs of true intelligence, while others hold different views.

To address these debates, researchers from the Santa Fe Institute (SFI), Arseny Moskvitch, Melanie Mitchell, and Victor Vikram Oduya, have proposed specific tasks in a forthcoming paper in the Journal of Machine Learning Research to determine and test the concept of intelligence. They have released this paper on the arXiv preprint server. The authors provide these tasks and find that even state-of-the-art artificial intelligence still lags behind humans in abstract and generalizable concepts.

The team created evaluation problems based on the domain developed by Google researcher François Chollet, focusing on visual analogical reasoning that captures basic concepts such as above, below, center, inside, and outside. Human and artificial intelligence participants were shown patterns of several demonstration concepts and then asked to apply the concept to different images. The figure below shows a test about similarity.

For humans, these visual problems are very easy: for example, their accuracy on similarity concepts reaches 88%. However, GPT-4 struggled, with only 23% of the problems answered correctly. Therefore, researchers believe that current artificial intelligence programs are still weak in visual abstract reasoning.

Moskvitch said, "We reason mostly by analogy, so it's a very interesting question." The team used novel visual problems to ensure that the machines had not encountered them before. GPT-4 underwent extensive training on the internet, so it was necessary to avoid any content it may have already encountered, ensuring that it was not just imitating existing text but demonstrating its understanding. That's why results like excelling on the Bar exam are not a good measure of its true intelligence.

The team believes that as time goes on and artificial intelligence algorithms improve, developing evaluation procedures will become increasingly difficult and important. Rather than trying to create a method to test artificial intelligence intelligence, we should design more carefully curated datasets that focus on specific aspects of intelligence. Moskvitch said, "The better our algorithms get, the harder it is to tell what they can and cannot do. So we need to be very careful in developing evaluation datasets."