Developing Multiple Robot Environments with NVIDIA Isaac Sim, ROS, and Nimbus

2023-05-30

Using Unreal Tournament Game Engine for Multi-Robot Simulation

About 20 years ago, Cogniteam embarked on a journey of robot development with the idea of creating a robot framework for multi-robot task allocation and collaboration. Initially called CogniTAO, a simplified version of this system was later released as the ROS decision module.

At that time, there were few use cases for multiple robots, and 3D simulation of these robots was not feasible. Therefore, Yaki Alon wrote a modified version of the 2000-2004 Unreal Tournament game engine to simulate four robots. Yaki Alon's small team spent about 3 years developing a simulation environment that could reliably run for 15 minutes.


Figure 1. Simulation of four robots (top) and video of the robots (bottom)


This environment could simulate four robots on five state-of-the-art desktop computers, equipped with a camera, Hokuyo LiDAR, rangefinder, and map-making capabilities, and receive video data remotely from each robot. An engineer from Yaki Alon's team developed a C++ TCP client to stream data directly from the game engine over the local network and display it in full screen. Yaki Alon's team had to run the code in a strict sequence to ensure that the robots would generate in the correct positions on time.

Using Gazebo for Multi-Robot Simulation

Fast forward to 2013, 10 years later, when Yaki Alon's team moved their work to Gazebo, which became the standard platform for robot simulation. Three programmers spent about 2 years simulating 10 robots on two Intel Xeon machines. They used the ROS move_base navigation stack and OpenCV Hough circle transform for object detection, a tool used by the robot team for demonstrations before TensorFlow was available. Igor Makhtes from the team developed an RQT plugin for controlling and displaying data streams from multiple robots (Figure 2), which took him 6 months to complete.


Figure 2. Video feed and map view of 10 robots with RQT plugin


These robots needed to communicate with each other but also operate when the connection was unavailable. To achieve this, each robot had to run its own ROS master node and synchronize through the ROS multi-master network.

Using NVIDIA Isaac Sim for Multi-Robot Simulation

A few months ago, Yaki Alon asked Saar Moseri, a computer science student from Cogniteam's algorithm team, to set up a multi-robot simulation scenario using the cloud robotics ecosystem Nimbus and NVIDIA Isaac Sim. The internal testing team of Yaki Alon's team and I wanted to use Nimbus agents to control Yaki Alon's team's robots and view the data they generated.

Saar spent about 2 weeks familiarizing himself with the environment and configuring the system. Figure 3 shows the result of this effort, running on a standard desktop computer with an NVIDIA GeForce RTX 3080 in Cogniteam's lab.

Figure 3. NVIDIA Isaac Sim multi-robot default setup


Saar used the Isaac Sim documentation provided by NVIDIA NGC to install and set up the environment. He installed agents on the simulated robots using Nimbus and created a gateway node to receive simulation data through ROS.


Figure 4. Nimbus robot editor (top) and Nimbus configuration editor (bottom)


Then, Yaki Alon's team created a node configuration as shown in Figure 5.


Figure 5. Nimbus simple task configuration with move_base navigation


These two building blocks, already containerized, are a gateway node and a node for move_base navigation. This configuration was deployed on an agent running on a simulation desktop computer in Cogniteam's lab. Nimbus hub also provides more complex configurations, including nodes for GMapping, path tracking, and more.


Yaki Alon's team and Yaki Alon were amazed by the possibilities this approach brought. In the described configuration, simulated sensor data is transmitted from Isaac Sim through the ROS gateway, supporting both ROS and ROS 2. Nimbus enables viewing and control functionalities.


Such a setup allows Yaki Alon's team to perform basic simulation tasks and control a fleet of simulated robots in their local lab, among many other capabilities. Yaki Alon's team can now record simulated runs and sensor data of the robots, SSH into the simulated robots remotely, globally monitor simulation data, and even send email and SMS notifications to Yaki Alon's team's validation team about simulation progress through a web browser.


By combining Isaac Sim and Nimbus, a unified system is formed that shares similarities with existing cloud simulation products but runs on local machines without additional cloud simulation computing costs. Additionally, it opens up new cutting-edge simulation workflows, such as using hardware in the loop. This is not possible when simulations run in the cloud. Figure 6 shows the appearance of control, navigation, and mapping in Nimbus.



Figure 6. Nimbus WebRTC video monitoring (top) and map view with autonomous control (bottom)


To replicate the setup described, refer to the Isaac Sim documentation. Then, visit Nimbus, create a free account, log in, and follow the instructions to create a robot with a free license.


Visit the Nimbus dashboard to deploy the Isaac Sim configuration. Since everything is containerized, including Isaac Sim, and the control is browser-based, you don't need to install any applications. The agent on your computer will set up everything required for execution.

Then, on the monitoring page of that agent, add monitoring for any data relevant to your setup. In the agent settings, you can define notifications based on conditions on the ROS stream, such as:

"if GoalStatus == ABORTED" send sms/mail to simulation@your-company.com

Successful deployment and simulation are crucial for autonomous robots. For testing, running the same scenario multiple times is essential, but multi-robot simulation is different. Developing a high-fidelity multi-robot simulation environment is complex and time-consuming, but it can be simplified with NVIDIA Isaac Sim and Nimbus, as described in this article. Yaki Alon's team and Yaki Alon will showcase their browser interface and remote simulation with Israeli robots at ICRA2023 (Booth C22) in London from May 29 to June 2.



Source: https://developer.nvidia.com/blog/develop-a-multi-robot-environment-with-nvidia-isaac-sim-ros-and-nimbus/

RECENT AI NEWS

RECENT AI TOOLS