Face analysis using 3D morphable models

Face analysis aims to extract valuable information from facial images. One effective approach for face analysis is the analysis by synthesis. Accordingly, a new face image synthesised by inferring semantic knowledge from input images. To perform analysis by synthesis, a genera- tive model, which parameterises the sources of facial variations, is needed. A 3D Morphable Model (3DMM) is commonly used for this purpose. 3DMMs have been widely used for face analysis because the intrinsic properties of 3D faces provide an ideal representation that is immune to intra-personal variations such as pose and illumination. Given a single facial input image, a 3DMM can recover 3D face (shape and texture) and scene properties (pose and illumination) via a fitting process. However, fitting the model to the input image remains a challenging problem. One contribution of this thesis is a novel fitting method: Efficient Stepwise Optimisation (ESO). ESO optimises sequentially all the parameters (pose, shape, light direction, light strength and texture parameters) in separate steps. A perspective camera and Phong reflectance model are used to model the geometric projection and illumination respectively. Linear methods that are adapted to camera and illumination models are proposed. This generates closed-form solu- tions for these parameters, leading to an accurate and efficient fitting. Another contribution is an albedo based 3D morphable model (AB3DMM). One difficulty of 3DMM fitting is to recover the illumination of the 2D image because the proportion of the albedo and shading contributions in a pixel intensity is ambiguous. Unlike traditional methods, the AB3DMM removes the illumination component from the input image using illumination normalisation methods in a preprocessing step. This image can then be used as input to the AB3DMM fitting that does not need to handle the lighting parameters. Thus, the fitting of the AB3DMM becomes easier and more accurate. Based on AB3DMM and ESO, this study proposes a fully automatic face recognition (AFR) system. Unlike the existing 3DMM methods which assume the facial landmarks are known, our AFR automatically detects the landmarks that are used to initialise our fitting algorithms. Our AFR supports two types of feature extraction: holistic and local features. Experimental results show our AFR outperforms state-of-the-art face recognition methods.