Evaluation of hybrid deep learning and optimization method for 3D human pose and shape reconstruction in simulated depth images

International audience; In this paper, we address the problem of capturing both the shape and the pose of ahuman character using a single depth sensor. Some previous works proposed to fi taparametric generic human template into the depth image, while others developed deeplearning (DL) approaches to fi nd the correspondence between depth pixels and ver-tices of the template. We designed a hybrid approach, combining the advantages ofboth methods, and conducted extensive experiments on the SURREAL, DFAUSTdatasets and a subset of AMASS. Results show that this hybrid approach en-ables us to enhance pose and shape estimation compared to using DL or model fi ttingseparately. We also evaluated the ability of the DL-based dense correspondence methodto segment also the background - not only the body parts. We also evaluated 4 di ff er-ent methods to perform the model fi tting based on a dense correspondence, where thenumber of available 3D points di ff ers from the number of corresponding template ver-tices. These two results enabled us to better understand how to combine DL and modelfi tting, and the potential limits of this approach to deal with real depth images. Futureworks could explore the potential of taking temporal information into account, whichhas proven to increase the accuracy of pose and shape reconstruction based on a uniquedepth or RGB image.