Simplifying 3D Deep Learning Research with NVIDIA's Kaolin PyTorch Library

2019-12-02

Research in 3D computer vision and AI is rapidly advancing like two parallel skyscrapers. However, the gap between them remains difficult to bridge. To bridge this divide, NVIDIA has released Kaolin today, which allows 3D models to be integrated into the field of neural networks in just a few steps. Kaolin simplifies the process of preparing 3D models for deep learning, reducing it from 300 lines of code to just 5 lines.


The field of 3D modeling is booming, and Kaolin can have a significant impact. Online repositories already have many 3D datasets, thanks in part to around 30 million depth cameras capable of capturing 3D images, which are now being used globally from laboratories to living rooms.Complex 3D datasets can be loaded into machine learning frameworks regardless of how they are represented or presented. Such tools can benefit researchers in fields such as robotics, autonomous vehicles, medical imaging, and virtual reality. Until now, researchers have lacked good tools to make these models work with deep learning tools, instead being forced to spend a lot of time writing code from scratch.


For a wider audience, Kaolin is a software library that supports various interesting 3D applications. Imagine being able to take pictures and turn them into 3D models that can be shared.



Accelerating Research Interface


The core of Kaolin is a set of efficient geometric functions that manipulate 3D content. It can be wrapped as PyTorch tensors for 3D datasets, implemented as polygon meshes, point clouds, signed distance functions, or voxel grids.


Researchers can prepare their 3D datasets for deep learning and then choose a neural network model from the alternative solutions provided by Kaolin. The interface offers a rich model repository, including models and base models, for tasks such as classification, segmentation, 3D reconstruction, and super-resolution.



Some Examples of Practical Applications


3D part segmentation can automatically identify different parts of a 3D model, making it easy to assemble characters for animation or customize models to generate object variants.


It builds 3D models based on product images identified by trained neural networks. Conversely, it can use 3D model search to find the most suitable 3D model database. In addition to the source code, we will also release pre-trained models for these tasks on popular benchmarks. We hope they can serve as benchmarks for future research, simplifying the task of model comparison.


Kaolin's modular approach allows users to easily perform differentiable rendering, a hot new technology in 3D deep learning. Users can simply modify the components of the interface materials without having to write an entire renderer from scratch.



Combining AI with 3D


NVIDIA researchers stated, "We have conducted a lot of 3D-related research at NVIDIA. Sometimes, we spend days browsing open-source code written by others to find the best methods and then put them all into a library for internal use. After writing boilerplate code for several of our projects, one of our interns suggested creating a more comprehensive toolset for PyTorch. Researchers have had utilities for 2D images for some time. A technique applicable to 3D can expand the community. We hope it can help many current and new 3D researchers use AI to create amazing things."

RECENT AI NEWS

RECENT AI TOOLS