Mineralogy meets zero-shot computer vision

2023-08-28

For geologists, identifying minerals is a complex and time-consuming task, with the identification of a single sample typically taking anywhere from 30 minutes to several days. Complicating matters further is the fact that a significant portion of minerals still lack comprehensive research. Out of the currently identified 6,000 minerals, only a few hundred have been fully identified.

Visual diagnosis of minerals and rocks is a widely used method in geology, as it is much cheaper and faster compared to other methods such as spectroscopy and chemical analysis. However, this method is less accurate compared to more expensive methods. Even experienced mineralogists can make mistakes when dealing with rare materials or low-quality samples. Integrating machine intelligence into this process can help identify errors and reduce the time spent by experts on routine tasks.

Although research in this field is ongoing, there is a lack of clear benchmarks for mineral image analysis in scientific literature. To fill this gap, the Artificial Intelligence Research Institute collaborated with Sber AI and Lomonosov Moscow State University to create a benchmark dataset for computer vision models focused on mineral identification.

This dataset is named MineralImage5k and is based on the collection of the Fersman Mineralogical Museum, consisting of 44,000 samples. While smaller than the Mindat dataset, MineralImage5k provides greater photo condition homogeneity and includes untreated samples that closely resemble natural minerals.

The MineralImage5k dataset is divided into three subsets of varying complexity, presenting challenges for researchers in mineral classification, segmentation, and size estimation. The simplest classification task proposed in the benchmark consists of 10 minerals, with at least 462 examples per mineral. The most challenging task is to classify minerals into 5,000 categories with only one image available.

One issue that artificial intelligence may face when dealing with mineral photos is determining which part of the presented rock is the actual mineral. To address this problem, in addition to classification, we also share a set of approximately 100 images with additional labels for segmentation tasks. Integrating segmentation tasks into the classification pipeline can provide additional insights and reduce the occurrence of such cases when the model makes mistakes.

In addition to classification and segmentation, we also studied zero-shot mineral size estimation. Automatic estimation of specimen size is highly useful in the process of storing museum specimens. With this data on the samples, we can plan the optimal storage system and purchase or manufacture the correct number of appropriately sized boxes. Therefore, in our benchmark, we provide over 18,000 labeled samples for regression tasks.

To demonstrate the effectiveness of the benchmark, we evaluated a visual language model pre-trained on general domain data. We found that fine-tuning the model on specific domain datasets, such as MineralImage5k, significantly improves its accuracy. We also highlight the potential for cross-dataset evaluation of mineral recognition models.

RECENT AI NEWS

RECENT AI TOOLS