"Empowering Robots to Better Navigate the Unknown World"

Inspired by the ability of humans to manipulate unfamiliar objects, a team from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) has designed a Feature Field for Robotic Manipulation (F3RM) system that combines 2D images with basic model features into a 3D scene to help robots recognize and grasp nearby objects. F3RM is able to interpret queries provided by humans, making it highly useful in real-world environments with thousands of objects, such as warehouses and homes.

F3RM provides robots with the ability to understand natural language and manipulate objects. As a result, robots can comprehend less specific requests from humans and complete the required tasks. For example, if a user asks the robot to "pick up a cup," the robot can locate and grasp the object that best matches the description.

Learning "where objects are through observation"

This approach can help robots pick items in inevitably cluttered and unpredictable large-scale distribution centers. In these warehouses, robots often receive inventory descriptions and need to identify these items. Robots must match the provided text with the items, regardless of packaging variations, to ensure accurate order fulfillment for customers.

For instance, a large online retailer's distribution center may contain millions of items, many of which the robots have never encountered before. To operate in such a massive environment, robots need to understand the geometric and semantic information of different objects, some of which are in tight spaces. With F3RM's advanced spatial and semantic perception capabilities, robots can more efficiently locate objects, place them in containers, and then send them to the packaging area. Ultimately, this will help factory workers process customer orders more efficiently.

Yang said, "One surprising thing about F3RM is that the same system also works at the scale of rooms and buildings and can be used to build simulation environments for robot learning and large-scale mapping. But before further expanding this work, we want to first speed up the system's runtime. This way, we can use this type of representation for more dynamic robot control tasks, hoping to use it in real-time so that robots handling more dynamic tasks can use it for perception."

The MIT team pointed out that F3RM's understanding capabilities in different scenarios make it highly useful in urban and home environments. For example, the method can assist personalized robots in identifying and picking up specific items. The system helps robots understand the surrounding environment at both the physical and perceptual levels.

Phillip Isola, an associate professor of electrical engineering and computer science at MIT, chief research scientist at CSAIL, and senior author of the paper, said, "The problem of visual perception can be defined, in David Marr's words, as understanding 'where objects are through observation.' Recent foundational models have excelled at understanding 'what objects are'; they can recognize tens of thousands of object categories and provide detailed textual descriptions of images. Meanwhile, radiance fields have excelled at representing where scenes are in space. The combination of these two approaches creates a representation of where objects are in 3D space, and our work suggests that this combination is particularly useful for robot tasks that involve manipulating objects in 3D."

Creating a "digital twin"

F3RM understands the surrounding environment by capturing photos with a camera mounted on a selfie stick. This fixed camera takes 50 images in different poses to construct a Neural Radiance Field (NeRF), a deep learning method for building 3D scenes from 2D images. The collage of these RGB photos creates a "digital twin" of the surrounding environment, forming a representation of the area within a 360-degree view.

In addition to the highly detailed neural radiance field, F3RM constructs a feature field to enhance the combination of geometry and semantic information. The system uses CLIP, a visual foundation model trained on billions of images, to efficiently learn visual concepts. By reconstructing the 2D CLIP features of the images captured by the selfie stick, F3RM effectively elevates these 2D features to a 3D representation.

Maintaining openness

After several demonstrations, the robot applies its understanding of geometry and semantics to grasp objects it has never encountered before. Once a user submits a text query, the robot searches for possible grasping methods to identify the most likely way to successfully pick up the requested object. Each potential option is scored based on its relevance to the prompt, similarity to demonstrations the robot has been trained on, and whether it would result in any collisions. The highest-scoring grasping method is then selected and executed.

To demonstrate the system's ability to execute human commands, researchers asked the robot to pick up a toy of the character Baymax from the Disney movie "Big Hero 6." Although F3RM has never been directly trained to pick up this cartoon superhero toy, the robot uses its spatial perception and visual language features from its foundational model to decide which object to choose and how to grasp it.

F3RM also allows users to specify the level of detail they want the robot to handle objects with in different languages. For example, if there is a metal cup and a glass cup, a user can ask the robot to pick up the "glass cup." If the robot sees two glass cups, one filled with coffee and the other with juice, the user can ask the robot to pick up the "glass cup with coffee." The embedded foundational model features in the feature field make this open-ended understanding possible.

William Shen, a PhD student at MIT, member of CSAIL, and co-author of the paper, said, "If I show someone how to grasp a cup by the rim, they can easily transfer that knowledge to similar geometric shapes like bowls, measuring cups, or even rolls of tape. For a robot, achieving this adaptability is very challenging. F3RM combines geometric understanding with the semantics of foundational models trained on internet-scale data, making this radical generalization from a few demonstrations possible."