POCO: A Novel AI Framework for 3D Human Pose and Shape Estimation

Estimating 3D human pose and shape (HPS) from photos and dynamic images is a necessary condition for reconstructing human actions in real scenes. However, inferring 3D information from 2D images faces numerous challenges, such as depth ambiguity, occlusion, unusual clothing, and motion blur. Even state-of-the-art HPS methods can produce errors and often remain unaware of these errors. HPS serves as an intermediate task that provides output consumed by downstream tasks, such as understanding human behavior or 3D graphics applications. These downstream tasks require a mechanism to assess the accuracy of HPS results, thus these methods must generate an uncertainty (or confidence) value related to the quality of HPS. One approach to address this uncertainty is to output multiple bodies, but this still lacks a clear uncertainty measure. There are also exceptions that can estimate the distribution of body parameters. One method is to calculate uncertainty by sampling from the body distribution and computing the standard deviation of these samples. While effective, this method has two limitations: it is slow because it requires multiple forward passes through the network to generate samples, and it trades speed for accuracy. More samples can improve accuracy but also increase computational requirements. Recently, a method has been developed to bypass explicit supervision by training a network to simultaneously output body parameters and uncertainty. Inspired by semantic segmentation work, it uses a Gaussian-based density function but recognizes the need for a more complex distribution for modeling human pose. Methods that directly estimate uncertainty typically include a density function and a scale network. Existing methods use an unconditional bDF and rely solely on image features for the scale network. While this method works well when samples have similar distributions, it performs poorly when dealing with diverse datasets that require robust 3D HPS models. POCO ("POse and shape estimation with COnfidence") is a novel framework applicable to standard HPS methods that addresses these challenges. POCO extends these methods to estimate uncertainty. In a single forward pass, POCO directly infers Skinned Multi-Person Linear Model (SMPL) body parameters and their regression uncertainty, which are highly correlated with reconstruction quality. The key innovation of this framework is the Dual Calibration Strategy (DCS), which enhances the density function and scale network. An overview of this framework is shown in the figure below. Unlike previous methods, POCO introduces a conditional vector (Cond-bDF) to model the density function of inferred pose errors. POCO does not use simple one-hot data source encoding but instead uses image features for calibration, enabling more scalable training on diverse and complex image datasets. Additionally, the authors of POCO introduce an improved method for estimating uncertainty in HPS models. They use image features and condition the network on SMPL poses, resulting in better pose reconstruction and uncertainty estimation. Their method seamlessly integrates into existing HPS models, improving accuracy without any drawbacks. The research claims that this approach surpasses state-of-the-art methods in associating uncertainty with pose errors. The results showcased in their work are shown below. This is an overview of POCO, an innovative artificial intelligence framework for 3D human pose and shape estimation.