NVIDIA Omniverse Technology Debuts with Apple Vision Pro Headset

Nvidia showcases its Omniverse digital twin technology, demonstrated through the new Apple Vision Pro headset. Nvidia engineers have utilized the Omniverse Cloud API to stream interactive industrial digital twin technology to the Apple Vision Pro headset. During the demonstration, cloud-based images were streamed at a resolution higher than what the Vision Pro headset supports. This presents an exciting application scenario for the Apple Vision Pro headset, which is priced at a high $3,500 for consumers. With the cost-saving potential of digital twin technology, industrial and enterprise companies can afford to equip their teams with such headsets. "Omniverse Cloud is now available on Apple Vision Pro," said Nvidia CEO Jensen Huang during his keynote speech at Nvidia GTC 2024, earning a round of applause. The concept of digital twin involves building a factory in digital form first, simulating all aspects realistically before anyone starts constructing the physical factory. Once the factory is built, digital twin technology can be used to quickly reconfigure the factory. Additionally, by capturing operational data through sensors, designers can modify the digital twin technology to make the simulation more accurate. This feedback loop can save companies a significant amount of money. At the Nvidia GTC conference, a new software framework built on the Omniverse Cloud API was announced, enabling developers to easily send their universal scene description (OpenUSD) industrial scenes from content creation applications to the Nvidia Graphics Delivery Network (GDN), a global graphics-ready data center network that can stream advanced 3D experiences to the Apple Vision Pro headset. This network is the same one Nvidia created for its GeForce Now cloud gaming service. Live Demo Nvidia showcased a demonstration of this technology, which made its debut at GTC. In the demo, an interactive, physically accurate digital twin of a car was streamed in full fidelity to the high-resolution display of the Apple Vision Pro headset. The scene consisted of approximately 100 billion triangles, all rendered with ray tracing, global illumination, and dynamic lighting, without precomputation, through cloud-based interactive rendering. It was like watching animation on a big-screen TV, but inside the headset. The demo showcased a car configurator application developed by CGI studio Katana on the Omniverse platform. Nvidia demonstrated how to switch paint and decoration options and even enter the vehicle, utilizing the powerful capabilities of spatial computing to blend the 3D realistic environment with the physical world. One could reach out and grab the car model, pinch it to shrink it, or expand fingers to enlarge it to a size much larger than myself. The image quality remained stunning regardless of the method used. Typically, the visual quality on the Apple Vision Pro is limited due to the memory of the application (8GB or 16GB). However, Rev Lebaredian, Nvidia's Vice President of Simulation, stated in an interview that cloud streaming allows Nvidia to send images from the cloud data center to the headset, streaming on demand. "We've been working to bring our RTX and rendering technology to Vision Pro," said Lebaredian. "There are limitations to doing graphics processing on the device. What you're seeing is essentially rendering happening in the cloud, using Omniverse and Omniverse RTX rendering technology, as well as more complex and larger USD content than the device itself or most ordinary computers can handle. It's running on the same infrastructure as our GeForce Now cloud gaming service. It's on the GDN and streamed directly to the device." For engineers or anyone needing accurate 3D imagery like digital twins, this allows them to design from within the virtual world and iterate simulations of the digital world faster with a realistic feel. One can switch to the environment tab and slide it to change the time of day, and the lighting changes after resetting the tab. Occasionally, there were some artifacts, such as seeing purple through the side window. But most of the time, it was incredibly impressive. Entering the Quanta Cloud Technology factory Nvidia also showcased a scene from Quanta Cloud Technology's server factory in Taiwan. The factory displayed a set of tracks that transport Nvidia DGX supercomputers from one part of the factory to another. Reflections, shadows, and lighting all looked great. Using Vision Pro's pinch-to-zoom feature, one could zoom in real-time to view details or zoom out for a wider view. The factory model used actual computer-aided design (CAD) models with billions of polygons. "The idea here is that people building the factory can plan the placement of equipment in advance, ensuring there are no inefficiencies in the production line before the actual placement of major equipment," said Lebaredian. The cloud essentially allows viewers to see about 20GB of data, while Apple Vision Pro typically only allows a few GB of data to be seen, as the headset must run on low-power and limited processors. This demo allows you to see 96GB of memory per eye instead of 16GB. The processing power can reach approximately 500 trillion floating-point operations per eye, a level of image processing that Vision Pro cannot achieve to render images. The data for this demo comes from Katana, a partner of Omniverse, which Nissan has used to create promotional materials for its cars. "We're thrilled to show that we can stream the exact same data losslessly to Vision Pro and actually improve the clarity," said Lebaredian. Bringing the Power of RTX Enterprise Cloud Rendering to Spatial Computing Spatial computing has become a powerful technology that provides immersive experiences and seamless interactions between people, products, processes, and physical spaces. Use cases in industrial enterprises require high-resolution displays and powerful sensors capable of running at high frame rates to make the manufacturing experience as close to reality as possible. This new Omniverse-based workflow combines the groundbreaking high-resolution display of the Apple Vision Pro with Nvidia's powerful RTX cloud rendering, delivering spatial computing experiences with just the device and an internet connection. This cloud-based approach allows real-time physically accurate rendering to seamlessly stream to the Apple Vision Pro, providing high-fidelity visual effects without sacrificing the details of large-scale, high-fidelity engineering datasets. "The Apple Vision Pro's groundbreaking ultra-high-resolution display, combined with the realistic rendering of OpenUSD content streamed from Nvidia's accelerated computing, presents incredible opportunities for the development of immersive experiences," said Mike Rockwell, Apple's Vice President of the Visual Products Group, in a statement. "Spatial computing will redefine how designers and developers create captivating digital content, ushering in a new era of creativity and interactivity." "Apple Vision Pro is the first unrestricted device that allows enterprise customers to realize their work without limitations," said Lebaredian. "We look forward to our customers using these amazing tools." The workflow also introduces hybrid rendering, a groundbreaking technique that combines local and remote rendering on the device. Users can interactively render experiences fully within a single application, using Apple's native SwiftUI and Reality Kit, as well as the Omniverse RTX Renderer streamed from GDN. Nvidia GDN is available in over 130 countries worldwide, leveraging Nvidia's global cloud-to-edge streaming infrastructure to deliver smooth, high-fidelity, and interactive experiences. By offloading intensive computational tasks to GDN, users can handle the most demanding rendering use cases, regardless of the size or complexity of the dataset. Enhancing Spatial Computing Workloads Across Applications The Omniverse-based workflow demonstrates the potential for a wide range of use cases. For example, designers can use this technology to view their 3D data in full fidelity without compromising quality or model accuracy. This means designers can interact with a credible simulation that looks and behaves like a real physical product. It also opens up new channels and opportunities for e-commerce experiences. In an industrial environment, factory planners can view and interact with their complete engineering factory dataset, optimizing their workflows and identifying potential bottlenecks. For developers and independent software vendors, the capabilities Nvidia is building will allow them to seamlessly interact with data from existing applications using native tools on the Apple Vision Pro.