Facial action unit detection with local key facial sub-region based multi-label classification for micro-expression analysis

This work is partially funded by National Key Research and Development Project of China under Grant 2019YFB1312000, and by National Natural Science Foundation of China under Grant No. 62076195. L. Zhang is funded by the China Scholarship Council – University of St Andrews Scholarship (No.201908060250). Micro-expressions describe unconscious facial movements which reflect a person's psychological state even when there is an attempt to conceal it. Often used in psychological and forensic applications, their manual recognition requires professional training and is time consuming. Therefore, achieving automatic recognition by means of computer vision would confer enormous benefits. Facial Action Unit (AU) is a coding of facial muscular complexes which can be independently activated. Each AU represents a specific facial action. In the present paper, we propose a method for the challenging task that is the detection of activated AUs when the micro-expression occurs, which is crucial in the inference of emotion from a video capturing a micro-expression. This specific problem is made all the more difficult in the light of limited amounts of data available and the subtlety of micro-movements. We propose a segmentation method for key facial sub-regions based on the location of AUs and facial landmarks, which extracts 11 facial key regions from each sequence of micro-expression images. AUs are assigned to different local areas for multi-label classification. Considering that there is little prior work on the specific task of detection of AU activation in the existing literature on micro-expression analysis, for the evaluation of the proposed method we design an AU independent cross-validation method and adopt Unweighted Average Recall (UAR), Unweighted F1-score (UF1), and their average as the scoring criteria. Evaluated using the established standards in the field and compared with previous work, our approach is shown to exhibit state-of-the-art performance. Postprint