Google DeepMind Releases RT-2 Based on LLM: Achieving Visual Language Action Control

Google DeepMind recently announced Robot Transformer 2 (RT-2), an AI model for controlling robots using Visual Language Action (VLA). RT-2 utilizes fine-tuned LLM to output motion control commands. It can perform tasks that are not explicitly included in the training data and achieve a three-fold improvement over the benchmark model in new skill evaluations.

DeepMind trained two variants of RT-2 based on two different visual LLM base models: one based on PaLM-E with 12B parameters and another based on PaLI-X with 55B parameters. LLM was jointly fine-tuned on a mixture of general visual language datasets and robot-specific data. The model learns to output vectors representing robot motion commands, treating them as a sequence of integers, effectively a new language learned by the model. The final model is capable of receiving images of the robot's workspace and user commands such as "pick up the package about to fall off the table" and generating motion commands to execute the task. DeepMind states:

"RT-2 not only demonstrates how artificial intelligence rapidly advances robot technology but also brings tremendous potential to more general-purpose robots. While creating useful robots in human-centered environments still requires a lot of work, RT-2 shows us an exciting future of robots that is soon to be realized."

Google Robotics and DeepMind have released several systems that use LLM to control robots. In 2022, Google's SayCan uses LLM to generate high-level action plans for robots, and the Code-as-Policies system uses LLM to generate Python code for robot control execution. These systems use pure text LLM to process user inputs, while the visual components are handled by separate robot modules. Earlier this year, Google's PaLM-E system processed multimodal input data from robot sensors and output a series of high-level action steps.

RT-2 is an upgraded version of RT-1. The key idea behind the RT series is to train models to directly output robot commands, without outputting higher-level motion abstractions as in previous approaches. Both RT-2 and RT-1 take images and textual descriptions of tasks as input. However, RT-1 uses a series of different visual modules to generate visual annotations for input LLM, while RT-2 uses a single visual language model like PaLM-E.

DeepMind conducted over 6000 experiments to evaluate RT-2. The researchers focused particularly on its novelty capability: performing tasks that are not in the robot-specific training data but induced from its visual language pre-training. RT-2 was tested on three task categories: symbolic understanding, reasoning, and human recognition. It achieved an "average success rate of over three times" compared to the benchmark model. However, the model did not acquire any physical skills not included in the robot training data.

In a discussion on Hacker News about this work, a user commented:

"It seems like this work (and much of robot learning work) is still limited to position/velocity control rather than impedance control. This essentially outputs where to go, which can be handled by a closed-loop controller or an open-loop motion planner. This seems to greatly reduce the data requirements but appears to be a fundamental limitation on the tasks we can accomplish. Manipulation is difficult because we need to consider not only what is happening in the world but also how our interaction changes it and how we need to react to it."

Although RT-2 has not been open-sourced yet, the code and data for RT-1 are already available.