Google DeepMind collaborates with universities to develop UniSim model, creating realistic AI system training simulator.

Google DeepMind has collaborated with researchers from UC Berkeley, MIT, and the University of Alberta to develop a new machine learning model for creating realistic simulation environments to train various types of AI systems.

The researchers wrote, "The next milestone for generative models is to produce realistic responses to actions by humans, robots, and other interacting agents." This is what they hope to achieve with UniSim, a generative AI system that can create a "real-world interactive" simulator.

Although UniSim is still in its early stages, it demonstrates the first step towards achieving this milestone. UniSim could be a valuable asset for fields that require complex real-world interactions, such as robotics and autonomous vehicles.

What is UniSim?

UniSim is a generative model that simulates interactions between humans and agents, as well as interactions between agents and the world. It can simulate the visual effects of both high-level instructions (e.g., "open the drawer") and low-level controls (e.g., "move by x, y"). These simulated data can then be used as training examples for other models that need to collect data from the real world.

The researchers wrote, "We plan to combine various data in a conditional video generation framework, including internet text-image pairs, navigation, manipulation, human activities, robots, and motion-rich data from simulation and rendering." The researchers state that UniSim can successfully integrate a large amount of knowledge contained in its training data and can generalize beyond its training examples, "achieving rich interactions through fine-grained motion control of static scenes and objects."

The ability of UniSim to simulate realistic experiences has far-reaching implications. It can be used to train low-level control policies, video caption models, and other machine learning models that require high-quality and consistent visual data.

Integrating Diverse Data Sources

UniSim is trained using datasets collected from simulation engines, real-world robot data, human activity videos, and image descriptions. However, the diversity of different data formats poses a significant challenge to the model's training.

The researchers wrote, "Due to the natural information divergence caused by different datasets designed by different industrial or research communities for different tasks, it is difficult to overcome and has made it challenging to build a real-world simulator aimed at capturing the realistic experience of the world we live in."

These datasets have been annotated in different ways and serve different purposes. For example, paired text-image data provides rich scenes and objects but lacks actions. Video captions and question-answer data provide high-level activity descriptions but lack details on low-level actions. Human activity data enriches human behavior but lacks mechanical movements, while robot data, although rich in robot actions, is limited in quantity.

To address this challenge, the researchers first convert all the different datasets into a unified format. They use transformer models, a deep learning architecture used for large-scale language models, to create embeddings from text descriptions and non-visual modalities such as motion control and camera angles. They train diffusion models to encode visual observations that describe actions. Then, they connect the diffusion models with the embeddings conditionally, connecting observations, actions, and outcomes.

Once the training is complete, UniSim can generate various realistic videos, including human actions and environment navigation.

It can also perform long-term simulations, such as scenarios where a robot hand performs a series of multiple actions. The generated examples demonstrate that UniSim successfully preserves the structure of scenes and objects involved in these long-term simulations.

In addition, UniSim can generate "random environment transitions," such as revealing different objects under a cloth or towel. This capability is particularly useful in computer vision applications for simulating contrasting facts and different scenarios.

Bridging the Gap between Simulation and Reality

The ability of UniSim to generate realistic videos from text descriptions is impressive, but its true value lies in its integration with reinforcement learning environments. Here, UniSim can simulate the outcomes of various application scenarios, such as robotics, enabling offline training of models and agents without the need for real-world training.

The researchers emphasize the benefits of this approach: "Using UniSim as an environment for training has several advantages, including unlimited environment access (via parallelizable video servers), real-world-like observation outcomes (via realistic diffusion outputs), and flexible time control frequencies (via time-stretched actions across low-level robot control and high-level text actions)."

Simulation environments are a crucial component of reinforcement learning. However, the high visual quality of UniSim can help bridge the gap between simulated learning and real-world learning, which is a challenge commonly referred to as the "sim-to-real gap."

The researchers state that models trained using UniSim can "generalize in a zero-shot manner to real-world robot environments, taking a step towards bridging the gap between simulation and reality in specific learning."

Applications of UniSim

Real-world simulators like UniSim have many potential applications, ranging from controlled content creation to games and movies, to directly deploying embodied agents trained in simulated environments to the real world. UniSim can also complement recent advancements in visual language models (VLM) such as DeepMind's RT-X model.

VLM agents require a large amount of real-world data, especially when performing complex multi-step tasks. The researchers demonstrate that UniSim can generate a significant amount of training data for VLM strategies.

The researchers note, "We use UniSim to train high-level visual-language planners and low-level reinforcement learning policies, which demonstrate zero-shot real-world transfer after training in purely learned real-world simulators." This approach also applies to other types of models, such as video caption models, which can benefit from training in UniSim's simulated experiences. UniSim can also simulate rare events, which is particularly useful in applications such as robotics and autonomous driving, where data collection can be expensive and risky.

Although the paper does not provide exact data on the cost of training UniSim, the researchers acknowledge that "UniSim, like other modern foundational models, requires significant computational resources for training. Despite this disadvantage, we hope that UniSim will generate widespread interest in improving machine intelligence and applying it to learning in real-world simulators."