Google Releases AVIS: A New Method for Autonomous Image Information Retrieval

2023-08-22

Google's AVIS attempts to answer questions about images that are not easily obtainable, such as the date of establishment of an airline or the year of car manufacturing.

Recent advancements in large language models (LLMs) have made multimodal capabilities possible, such as image captioning and visual question answering. However, these visual language models (VLMs) still face difficulties in complex real-world visual reasoning that requires external knowledge, known as "visual information seeking".

To address this limitation, Google researchers propose a new approach called AVIS, which combines Google's PALM with computer vision, web search, and image search tools. With the help of these tools, AVIS autonomously searches for visual information within a dynamic framework using language models.

Google's AVIS learns from humans

Unlike previous systems that combine large language models with tools in a strict two-step process, AVIS is more flexible for planning and reasoning, enabling actions to be adjusted based on real-time feedback.

AVIS consists primarily of the following three components:

  • A planner that determines the next action (API calls and queries) using LLM
  • A working memory to retain past API execution information
  • An inferencer that processes API outputs using LLM and extracts useful information

The planner and inferencer are iteratively used, with the planner determining the next tool and query based on the updated state by the inferencer. This process continues until the inferencer determines that there is enough information to provide a final answer.

Additionally, three types of tools are integrated:

  • Computer vision tools for extracting visual information from images
  • Web search tools for retrieving open-world knowledge and facts
  • Image search tools for obtaining relevant information from metadata associated with visually similar images

To fully leverage these capabilities, researchers conducted a user study capturing common sequences of actions taken by humans using visual reasoning tools for decision-making and constructed a transition graph guiding AVIS behavior.

AVIS achieves state-of-the-art performance without fine-tuning

On the Infoseek dataset, AVIS achieves an accuracy of 50.7%, significantly outperforming fine-tuned visual language models such as OFA and PaLI. On the OK-VQA dataset, AVIS achieves an accuracy of 60.2% with fewer examples, surpassing most previous achievements and approaching fine-tuned models, according to Google.

In the future, the research team aims to explore their framework on other reasoning tasks and attempt to use lighter language models, as the current PALM model used is computationally intensive with 540 billion parameters.

Summary

  • Google's Autonomous Visual Information Seeking (AVIS) is a new approach that integrates large language models with computer vision, web search, and image search tools to autonomously search for visual information and answer complex questions about images.
  • The AVIS system consists primarily of a planner (using language models to determine the next action), working memory (for storing information), and inferencer (processing outputs and extracting useful information), enabling flexible and dynamic planning and reasoning. Multiple tools are used simultaneously.
  • AVIS achieves high accuracy on the Infoseek and OK-VQA datasets, outperforming fine-tuned visual language models and approaching the state-of-the-art without the need for fine-tuning. The research team plans to apply this framework to other reasoning tasks and explore the use of lighter language models due to the current model's computational load.

RECENT AI NEWS

RECENT AI TOOLS