Exploring the Capabilities and Challenges of Google AI Chatbot Bard in Computer Vision Tasks

Bard, Google's AI chatbot, was launched in March 2023 and achieved moderate success. It was later expanded globally in May. Bard is a generative AI that can accept prompts and perform text-based tasks such as providing answers, summaries, and creating various forms of textual content.

On July 13, 2023, Google's Bard announced a major update that allows the use of images as input along with text prompts. It is claimed that Bard can provide descriptions (e.g., image captions) or answer questions using visual content analysis.

It is worth noting that while other models like GPT4 claim to have the ability to accept and understand visual input prompts, they have not been tested publicly. Access to Bard provides the computer vision community with the first opportunity to evaluate its soundness and robustness and understand its existing advantages and limitations. In this study, the researchers aim to analyze Bard's ability to address long-standing computer vision problems in image understanding.

This research, published in Machine Intelligence Research, identifies several interesting scenarios based on computer vision problems for qualitative evaluation of Bard. As Bard's API-based access is still unavailable, the evaluation by the researchers does not include quantitative results from large-scale benchmark tests.

Instead, the goal is to identify some insightful scenarios and corresponding visual-text prompts to evaluate Bard's visual understanding capabilities and those of future large-scale multimodal models like GPT4. The researchers specifically focus on Bard's motivation, as it outperforms all open-source and closed-source multimodal dialogue models, including Bing-Chat launched on July 18, 2023, as demonstrated by LLaVA-Bench.

To evaluate Bard's capabilities such as visual perception and contextual understanding based on given text prompts, the researchers designed a series of visual language task scenarios.

Subsequently, they delved into several examples derived from these empirical studies, covering a total of 15 visual question answering (VQA) scenarios, including object detection and localization, analyzing object attributes, counting, adaptation, and fine-grained recognition in natural images, among other tasks. They also attempted some challenging cases such as identifying camouflaged objects and images from different domains like medical, underwater, and remote sensing. These scenarios are explained below.

The first scenario is object attributes, which indicates that Bard faces challenges in identifying attributes that require a deep understanding of each object and its properties.

The second scenario is object presence, which suggests that Bard's basic understanding of visual content is still limited. The researchers further note that Bard is currently optimized for images without any humans and removes visual inputs containing faces or people.

The third scenario is object localization, which indicates that Bard's ability to locate visual context can be further improved.

The fourth scenario is relational reasoning, which suggests that Bard has room for improvement in reasoning relationships.

The fifth scenario is adaptation, implying that Bard still needs to better capture visual semantics strictly guided by text and more effectively associate these semantics with objects in recognized scenes.

The sixth scenario is adversarial examples, where Bard's outputs indicate its inability to understand adversarial examples.

The seventh scenario is rainy conditions, and the results show that Bard performs poorly when there are rainy conditions in the image.

The eighth scenario is emotion understanding, and when queried by the researchers, Bard's answers are incorrect.

The ninth scenario is fine-grained recognition, which involves identifying specific subclasses within given object categories. This task is more complex than general object recognition due to increased intra-class variation, subtle inter-class differences, and the need for domain-specific knowledge. Bard provides both correct and incorrect answers.

The tenth scenario is identifying camouflaged objects, indicating Bard's need to improve its ability to parse camouflage patterns and similar textures.

The eleventh scenario is object counting, and the researchers note that while Bard excels at describing scenes, it seems less proficient in understanding advanced content in challenging scenarios.

The twelfth scenario is detecting industrial defects. The researchers observe difficulties for Bard in recognizing these hard-to-perceive defects, resulting in incorrect responses to users.

The thirteenth scenario is recognizing optical characters. Bard encounters difficulties in various text recognition scenarios, finding it challenging to understand text in natural images.

The fourteenth scenario is analyzing medical data. No meaningful content is outputted in the experiment.

The fifteenth scenario is interpreting remote sensing data. The researchers' findings suggest that Bard has a tendency to excel in overall understanding of visual scenes but faces challenges in distinguishing fine-grained visual patterns, particularly in accurately counting objects like commercial buildings.

Bard's emergence in the conversational AI field has garnered widespread attention due to its significant success. Building on this momentum, this study aims to comprehensively evaluate Bard's performance in various task scenarios, including general, camouflaged, medical, underwater, and remote sensing images. The survey results show that while Bard excels in many areas, it still faces challenges in certain visual-based scenarios.

This finding highlights Bard's immense potential in various applications and emphasizes the ample room for growth and improvement in visual-related tasks. The empirical insights from this study are expected to be valuable for the development of future models, particularly in bridging the gap in visual performance. By addressing the observed limitations in visual scenarios, researchers anticipate that subsequent models will have stronger visual understanding capabilities, ultimately driving further advancements in conversational AI.