Reconstructing Mental Images from Brain Scans using AI

In the near future, the computing interface may surpass touch screens and keyboards, even surpassing eyes and gestures, and enter our inner world.

Although society has not reached this point yet, we are gradually getting closer. Researchers from Meta Platforms announced today a new deep learning application called Image Decoder, which is based on Meta's open-source base model DINOv2. It can convert brain activity into highly accurate images, almost in real-time, showing what the observer sees or imagines.

In other words, if Meta's researchers are sitting behind a barrier and cannot see the observer, even if the observer is on the other side of the world, Image Decoder can allow Meta's researchers to see what they are looking at or imagining based on the observer's brain activity - provided that the observer is in a neuroimaging facility and undergoing scanning with MEG instruments.

These researchers work at Facebook AI Research (FAIR) and Paris PSL University, and they provide more detailed information about their work and the Image Decoder system in a new paper.

Meta wrote in an email, "This research strengthens Meta's long-term research plan to understand the foundations of human intelligence, identify its similarities and differences from current machine learning algorithms, and ultimately help build AI systems with human-like learning and reasoning abilities."

How Meta's Image Decoder Works

Meta's researchers describe the technical foundation of Image Decoder in the paper.

Essentially, it combines two previously quite independent fields: machine learning (especially deep learning, where computers learn by analyzing labeled data and examining new data to try to correctly label it) and magnetoencephalography (MEG), a non-invasive system for measuring and recording brain activity that captures tiny changes in the brain's magnetic field to infer what a person is thinking.

Meta's researchers trained the deep learning algorithm using 63,000 previous MEG results from four patients (two females and two males, with an average age of 23) in 12 sessions, where the patients saw 22,448 unique images and 200 repeated images.

The Meta team used DINOv2, a self-supervised learning model designed to train other models, which Meta publicly released in April 2023.

The researchers instructed the Image Decoder algorithm to simultaneously observe the raw data and the images that the subjects actually saw when their brains produced MEG activity.

By comparing the MEG data with the actual source images, the algorithm learned to interpret specific shapes and colors in the brain and how they are represented.

Promising Results and Ethical Considerations

Although the Image Decoder system is far from perfect, the results are encouraging for the researchers, as it achieves an accuracy of 70% in the highest-performing cases of accurately retrieving or reproducing images based on MEG data, which is seven times higher than existing methods.

Some of the images successfully retrieved from a set of potential images by Image Decoder include broccoli, caterpillar, and speaker cabinet. However, its performance is poorer for decoding more complex and diverse images such as corn tortilla, avocado, and beans.

"Overall, our findings provide a promising avenue for real-time decoding of visual representations in laboratory and clinical settings," the researchers wrote.

However, they point out that this technology raises "several ethical considerations."

Among the ethical considerations proposed by the researchers, "the most important is protecting mental privacy," although they do not specify how to achieve this.

The funding for this work comes from a parent company that has been fined billions of dollars for violating consumer privacy, which is also a noteworthy issue, although the researchers do not directly mention it.

However, due to technical limitations, it is currently not possible to use this technology to read a person's thoughts without their consent. Image Decoder is best at interpreting specific objects and scenes that people have previously seen.

"In contrast, decoding accuracy is significantly reduced when participants are asked to imagine images," the researchers noted.

Furthermore, the researchers pointed out, "Decoding performance is also severely affected when participants engage in interfering tasks, such as counting down. In other words, the participant's consent is not only a legal requirement but also a primary technical requirement."

Therefore, if the decoding of a person's brain activity is done without their consent, the participant can prevent this process through certain techniques, such as counting down, provided they are aware of this option and their situation.