Google's PaLM-E integrates visual and language artificial intelligence for robot control.

2023-06-28

Google's robot research team recently announced PaLM-E, a combination of their PaLM and Vision Transformer (ViT) models specifically designed for robot control. PaLM-E processes multimodal input data from robot sensors and outputs text commands to control the robot's actuators. In addition to performing well on several robot tasks, PaLM-E outperformed other models on the OK-VQA benchmark test. PaLM-E addresses the issue of rounding or embodying large language models (LLMs) by incorporating multimodal sensor data into the input of the LLM. These inputs are first processed by an encoder, projecting them into the embedding space of the LLM for language input tokens, resulting in multimodal sentences composed of text and other data. Then, PaLM-E generates text outputs such as answers to input questions or high-level instructions for the robot. According to Google, "PaLM-E pushes the boundaries of training general-purpose models that can handle visual, language, and robot technologies, and has the potential to be a key driver for other broader applications using multimodal learning, including unifying seemingly separate task capabilities." Google's robot researchers have open-sourced several previous LLM systems for robot control. In 2022, InfoQ reported on two projects, SayCan, which uses LLM to output high-level action plans, and Code-as-Policies, which uses LLM to output low-level robot control code. PaLM-E is based on a pretrained PaLM language model. Robot sensor data is injected into a text input; for example, the model can handle input questions like "What happened between and ?" where "img_1" and "img_2" are images encoded by ViT and mapped to the same embedding space as the text input tokens. In this case, the model's output would be an answer to the question. Google has created encoders for various input modalities, including robot state vectors (e.g., 3D pose information), 3D scene representations, and entity references of objects in the robot's environment. The researchers evaluated PaLM-E by using it to control robots in multiple tasks in simulated and real-world environments, including object grasping and stacking, pushing objects in a tabletop environment, and robot movement in a kitchen environment. PaLM-E was able to generate "long-term plans" for robots and generalize to unseen tasks in the tabletop pushing task. In the kitchen, the robot was able to complete long-term planning tasks "even in the presence of adversarial interference." Some users commented on this work in a discussion on Hacker News. One user wondered if the model's performance scales with the number of parameters. Another user responded, "Performance does improve with an increase in parameters, although it's not a linear relationship. As Google found in their research on Chinchilla LLM, performance also improves with an increase in the size of the training set. They were able to determine the optimal training amount for a given size of the model to make the most of the budget. So even if we don't discover any better model architectures, although we might, we should be able to achieve better model performance by increasing the scale of the model, training corpus, and budget." The PaLM-E website features several demonstration videos showcasing scenarios where the model controls robots to perform tasks. Source: https://www.infoq.com/news/2023/06/google-palm-e-robot/?topicPageSponsorship=b2206c17-c7cf-47e8-aee9-0514a0817c31&itm_source=presentations_about_ai-ml-data-eng&itm_medium=link&itm_campaign=ai-ml-data-eng

RECENT AI NEWS

RECENT AI TOOLS