Nvidia DreamDojo: Open-Source World Model for Robot Training

Nvidia's artificial intelligence research division has introduced DreamDojo, an open-source interactive world model designed specifically for robotics.

By leveraging robot motor control, the system generates pixel-based future simulations without the need for game engines, mesh grids, or manually crafted animations. Jim Fan, Nvidia's Director of AI and Principal Scientist, describes this advancement as "Simulation 2.0."

DreamDojo Learns from Human Videos Instead of Robot Data

Fan explains that real-world robot learning encounters bottlenecks related to time, equipment wear, safety concerns, and reset procedures. DreamDojo aims to overcome these limitations by prioritizing learning from human interactions first.

The model undergoes pre-training on 44,000 hours of first-person human video without any robot intervention. The concept of "latent actions" serves as a unified representation inferred directly from video, capturing changes between world states without knowledge of underlying hardware. This enables the model to train on any first-person video as if it were equipped with motor commands.

In the second phase, the model is fine-tuned on specific robots to adapt to their hardware. Fan describes this process as separating the "appearance and behavior of the world" from the "operational methods of the specific robot." The base model learns general physical rules before adapting to the robot's unique mechanical structure.

The real-time version of DreamDojo operates at 10 frames per second, maintaining stability for over a minute of continuous rollout. It supports real-time VR teleoperation within the dream space, policy evaluation in neural simulators, and model-based planning, all utilizing the world model.

According to Fan, all weights, code, post-training datasets, evaluation sets, and the whitepaper are publicly available. DreamDojo is built upon Nvidia Cosmos, which also features open weights. Further details can be found on the project page and in the associated paper.