Facebook's New Developments in the 3D Field

2019-10-30

The most advanced machine learning algorithms can extract two-dimensional objects from photos and present them realistically in three dimensions. This technology is an important research area at Facebook and is applicable to augmented reality applications, robotics, and navigation. In a blog post, Facebook emphasized its latest progress in intelligent content understanding: their system can detect complex foreground and background objects, such as chair legs or overlapping furniture. The researchers wrote in the blog post, "Our research builds on the latest advancements in using deep learning to predict and locate objects in images, as well as new tools and architectures for 3D shape understanding. 3D understanding will play a crucial role in improving AI interpretation and transforming the real world, making it particularly important." Within the complex and extensive research framework, Mesh R-CNN is a highly important method that predicts three-dimensional shapes from cluttered and occluded object images. Facebook's researchers stated that they added a mesh prediction branch to the two-dimensional object segmentation system of Mesh R-CNN and further strengthened this branch by using a library called Torch3d, which contains highly optimized 3D operators. Mask R-CNN can detect and classify different objects in images and then infer their three-dimensional shapes using the aforementioned predictors. Mesh R-CNN successfully detected objects of all categories and estimated their complete three-dimensional shapes in furniture scenes in the publicly available Pix3d dataset. Canonical 3D Pose Networks, or C3DPO for short, is a system of equal importance to Mesh R-CNN. It can establish three-dimensional keypoint models and achieve state-of-the-art model reconstruction results using two-dimensional keypoint supervision. This kind of reconstruction was previously achievable but limited by memory constraints. The emergence of the C3DPO architecture perfectly solves this problem. It can use the reconstructed model to predict corresponding camera viewpoint parameters and three-dimensional keypoint positions, enabling the components to learn together with the model to address the ambiguity introduced in decomposing three-dimensional viewpoints and shapes. In conclusion, the researchers stated, "Computer vision has many open research questions, and we are exploring them one by one to drive the field forward. As the digital world adapts and transitions to using 3D photos, immersive AR and VR experiences, and other products, we need to continuously advance complex systems to meet people's imagination in the visual field in this new era."

RECENT AI NEWS

RECENT AI TOOLS