A General Method of Realistic Avatar Modeling and Driving for Head-Mounted Display Users

The head-mounted displays (HMDs) provide immersive experiences in virtual reality (VR). However, the face interactions are limited due to the serious occlusion of the user’s face. Existed approaches try to recover the user’s facial expression by adding additional sensors to HMD. In this paper, we develop a novel framework to reconstruct the userb’s 3D face in VR only using an RGB camera. Given a reference face, a realistic full-textured avatar is created by fitting a 3D Morphable Model (3DMM). A self-supervised UV map Generative Adversarial Network (GAN) is proposed to make the facial texture look more realistic. Next, we propose a novel landmark detection method to locate the landmark positions under HMD occlusion since facial landmarks are commonly used for driving the avatar. To this end, we synthesize a face dataset with HMD. Our method is easy to build and popularize with low cost. The experiments on the synthetic and real HMD data demonstrate that the proposed method can detect landmark accurately and restore facial expressions faithfully despite the large occlusion of HMD.