"NVIDIA Develops AI that Can Create 3D Models from 2D Images"

2019-12-10

To turn a photo into a complete 3D model, you can use a 3D printer for printing or perform digital modeling from a set of images to achieve lifelike visual effects in movies. However, Nvidia has successfully trained a neural network that can generate fully textured 3D models based solely on a single photo.


We have seen similar methods for automatically generating 3D models before, but they either require capturing a series of photos from different angles for accurate results or rely on manual input to help the software determine the object's size and shape. These problem-solving methods are not necessarily wrong, but they do limit the potential applications of such software.


At the annual Conference on Neural Information Processing Systems, researchers from Nvidia will present a new concept: Differentiable Interpolation-Based Renderer (DIB-R). DIB-R combines learning-based prediction and interpolation to render 3D objects. It combines the "seen" content with 2D images and makes inferences based on its "understanding" of the 3D world. This is similar to how humans transform the 2D input from our eyes into 3D mental images.


Nvidia researchers trained their DIB-R neural network on multiple datasets, including images previously turned into 3D models, 3D models presented from various angles, and image sets focusing on specific subjects from multiple angles. It took approximately two days to train the neural network to infer additional dimensions of a given object (e.g., birds), but once completed, it can generate 3D models based on 2D photos in just 100 milliseconds, achieving unprecedented results.It is worth noting that the processing speed makes this tool particularly interesting, as it has the potential to greatly improve how machines such as robots or autonomous vehicles perceive the world and understand their surroundings. Still images extracted from real-time videos can be instantly transformed into 3D models, enabling autonomous vehicles to accurately determine the size of large trucks to avoid or allowing robots to predict how to correctly pick up random objects based on their estimated shapes. DIB-R can even enhance the performance of safety cameras responsible for identifying and tracking individuals, as the instantly generated 3D models make it easier to perform image matching when people move within the field of view. Each new technology is truly amazing.


With further development, researchers hope to expand DIB-R to include features that essentially turn it into a virtual reality renderer. The team envisions a future where AI can create fully immersive 3D worlds in milliseconds using only photos.

RECENT AI NEWS

RECENT AI TOOLS