AI can reproduce your vision: How the Mind's Eye model turns brain scans into images

2023-06-15

We have always wanted to understand how our brains work. The field of neuroscience has made a lot of progress, but we still know very little about the detailed workings of the brain. We are working hard to find answers, but there are still many challenges waiting for us to solve. Neuroscience has been dedicated to deciphering the complex relationship between brain activity and cognitive states. By gaining a deeper understanding of how environmental inputs are encoded in neural processes, we can greatly enhance our knowledge of the brain and its mechanisms. Recently, the development of computational methods has provided new opportunities to uncover these mysteries, with functional magnetic resonance imaging (fMRI) being a particularly powerful tool. fMRI measures neural activity by detecting changes in blood oxygen levels and has been applied in real-time clinical settings. One particularly promising application of fMRI is its ability to achieve mind reading in brain-computer interfaces. By decoding patterns of neural activity, we can infer a person's mental state and even reconstruct images from their brain activity. Previous research in this field has mainly employed simple mapping methods, such as ridge regression, to link fMRI activity with image generation models. MindEye aims to decode environmental inputs and cognitive states from brain activity. It utilizes a combination of large-scale MLP, contrastive learning, and diffusion models to map fMRI voxels to the image embeddings of a pre-trained CLIP model. The model consists of two pipelines: a high-level (semantic) pipeline and a low-level (perceptual) pipeline. In the high-level pipeline, fMRI voxels are mapped to the CLIP image space, which is more semantic in nature. Contrastive learning is then used to train the model, with fMRI introduced as an additional modality into the embedding space of the pre-trained CLIP model. To improve model performance, a bidirectional version of mixup contrastive data augmentation is employed. The low-level pipeline maps fMRI voxels to the embedding space of a stable diffusion variational autoencoder (VAE). The output of this pipeline can be used to reconstruct blurry images that achieve state-of-the-art performance in low-level image metrics. Due to the low quality of the output, an img2img method is used to further improve the image reconstruction while maintaining high-level metrics. MindEye has achieved state-of-the-art results in both image reconstruction and retrieval tasks. It produces high-quality reconstructions that match the low-level features of the original images and performs well in both low-level and high-level image metrics. The discontinuous CLIP fMRI embeddings obtained by MindEye also demonstrate excellent performance in image and brain retrieval tasks. Source: https://www.marktechpost.com/2023/06/14/ai-see-what-you-see-minds-eye-is-an-ai-model-that-can-reconstruct-brain-scans-into-images/

RECENT AI NEWS

RECENT AI TOOLS