Search

Large-scale facial expression datasets are primarily composed of real-world facial expressions. Expression occlusion and large-angle faces are two important problems affecting the accuracy of expression recognition. Moreover, because facial expression data in natural scenes commonly follow a long-tailed distribution, trained models tend to recognize the majority classes while recognizing the minority classes with low accuracies. To improve the robustness and accuracy of expression recognition networks in an uncontrolled environment, this paper proposes an efficient network structure based on an attention mechanism that fuses global and local features (AM-FGL). We use a channel spatial model and local feature convolutional neural networks to perceive the global and local features of the human face, respectively. Because the distribution of real-world scene field expression datasets commonly follows a long-tail distribution, where neutral and happy expressions account for the tail expressions, a trained model exhibits low recognition accuracy for tail expressions such as fear and disgust. CutMix is a novel data enhancement method proposed in other fields; thus, based on the CutMix concept, a simple and effective data-balancing method is proposed (BC-EDB). The key idea is to paste key pixels (around eyes, mouths, and noses), which reduces the influence of overfitting. Our proposed method is more focused on the recognition of tail expression, occluded expression, and large-angle faces, and we achieved the most advanced results in occlusion-RAF-DB, 30∘ pose-RAF-DB, and 45∘ pose-RAF-DB with accuracies of 86.96%, 89.74%, and 88.53%.

There has been contention in the previous research regarding the association between face processing abilities (e.g. face detection, face perception, face memory and expression recognition) and cognitive abilities (i.e. intelligence). Some would contend that face processing abilities are unique and distinct from other cognitive abilities (Shakeshaft & Plomin, 2015; Wilmer et al., 2010). Whilst others have found moderate associations between these abilities (Connolly et al., 2019; Gignac et al., 2016; Hildebrandt et al., 2011; Schlegel et al., 2019). It would be greatly beneficial to explore the shared variance between face processing abilities and cognitive abilities, not only in improving our understanding of these abilities, but also the possibility of conceptualising face processing abilities within the contemporary models of cognitive ability, such as the Cattell-Horn-Carroll (CHC) model (Meyer et al., 2021). The majority of the recent research on the association between cognitive abilities and face identity processing has centred around face memory. Face memory is the ability to recognise a face that has previously been seen, a vital skill for socialisation (Oruc et al., 2019). Some studies have found a moderate, positive association between short-term face memory and other cognitive abilities, including general intelligence (Gignac et al., 2016; Hildebrandt et al., 2011; Shakeshaft & Plomin, 2015). However, other studies have failed to report a significant association (Davis et al., 2011; Palermo, Connor, Davis, Irons, & McKone, 2013; Peterson & Miller, 2012). The mixed results reported in the literature may be due to several factors, including small and range restricted samples (e.g., Davis et al., 2011; Peterson & Miller, 2012). Additionally, several studies measured cognitive abilities with only one or two subtests, which would only be described as ‘poor’ to ‘fair’ intelligence measurement, according to Gignac and Bate’s (2017) guidelines. Therefore, it is important to conduct a study that includes a large, representative sample with a balanced number of good quality cognitive abilities measures across the four core dimensions of cognitive ability: crystallised intelligence, visual/fluid intelligence, memory span, and processing speed. In comparison to face memory, there has been very minimal research conducted on face detection and cognitive abilities. Face detection is the process of locating a face within a visual scene. This ability is an essential prerequisite for higher-order face-specific processes, such as face memory (Garrido et al., 2008; Verhallen et al., 2017). To the authors’ knowledge, only one study has investigated the association between intelligence and face detection (Vigen et al., 1982). They found a moderate, positive correlation (r = .25) between the WAIS-R and a commonly used face detection task, the Mooney task. Therefore, it remains unknown the degree to which individual differences in face detection arise from a process specific to faces versus individual differences in visual processing more generally. On the one hand, there is neurophysiological evidence to suggest that face detection may be, at least to some degree, unique. For example, studies have shown that detection of a face in the visual space activates the highly face-selective fusiform face area, compared to when a face is not detected (Andrews & Schluppeck, 2004; Kriegeskorte et al., 2007; McKeeff & Tong, 2007). Whilst on the other hand face detection tasks are remarkably similar to tasks of visual closure, and an association between face detection and visual closure has been established (Walker et al., 2022). Therefore, an important aspect of this current study is to examine the association between face detection, other face processing abilities, specific cognitive abilities, and general cognitive ability. Face perception is considered a higher-level process than face detection (Verhallen et al., 2017). It’s the ability to either discriminated or individualised a person’s identity when viewing faces (Oruc et al., 2019). The ability to tell faces apart or individualise a face is important for day-to-day life. For example, being able to scan faces within a crowd and see each face as being distinct from each other. Previous research has shown a positive association between face perception and cognitive abilities (Connolly et al., 2019; Hildebrandt et al., 2011). However, only a small amount of empirical research has correlated individual differences in face perception with other cognitive ability dimensions, whether general or narrow in nature. This study aims to provide further evidence of the association between face perception and general cognitive ability, and specific cognitive abilities. Another very important face processing ability, especially for socialisation, is expression recognition, also known as face emotion recognition. The ability to accurately and efficiently recognise facial expressions is essential for effective interpersonal interaction and social cognition (Fox & Zougkou, 2011). Empirically, individual differences in expression recognition have been linked to cognitive abilities (Borod et al., 2000; Connolly et al., 2019). Schlegel et al. (2019) conducted a meta-analysis on the association between expression recognition and cognitive abilities. They found that general cognitive ability is positively correlated with expression recognition (r = .19). They also found that specific cognitive abilities (crystallized, fluid, spatial, memory, information processing speed and efficiency) also positively correlated with expression recognition to a similar extent. Schlegel et al. (2019) concluded that the effect size reported in the meta-analysis may be on the lower end of the true associations, due to psychometric limitations and limited complexity of the measures of expression recognition ability. Therefore, it is important that the association between expression recognition and cognitive abilities be examined when the measures have good psychometric properties and follow all steps outlined in recent test development guidelines (e.g. DeVellis, 2017). In doing so, the true association between expression recognition and cognitive abilities could be observed. It's also important to investigate the inter-association between the face processing abilities. Verhallen et al. (2017) and McCaffery et al. (2018) found highly significant correlations between their measures of face detection, face perception, and face memory. The correlations between measures differed in magnitude with shared variance ranging between 4% and 23%. Both studies concluded that a general ability of face processing may exist and underlies performance with a wide range of face tasks. Similarly, research into expression recognition and the other face processing abilities has seen considerable correlations between measures (Gignac et al., 2020; Hildebrandt et al., 2015; Palermo et al., 2013). Therefore, this study will aim to investigate the unique construct related variance associated with each of the four face processing abilities. One final aspect of this study is to investigate the shared variance between trait-autism and face processing abilities. To the authors’ knowledge, no individual difference study has yet examined all four dimensions of face processing (face detection, face perception, face memory and expression recognition) and trait-autism, simultaneously. There is evidence to suggest that higher trait-autism is associated with poor face processing abilities (Bothe et al., 2019; Davis et al., 2017; Halliday et al., 2014). Autistic-like traits (also known as the Broader Autism Phenotype) refers to milder forms of traits seen in neuro-typical individuals that are associated with the Autism Spectrum Disorder (ASD), such as communication deficits, restricted activities and poorer executive functioning (Deisinger, 2015). Specifically, previous research suggests that nonverbal communication, one dimension of trait-autism, is most likely to be associated with face processing ability compared to the other dimensions. For example, Bothe et al. (2019) found that nonverbal communication was negatively associated with discriminating and labelling emotions (r = -.24 & r = -.29, respectively). Davis et al. (2017) and Lewis et al. (2018) found that nonverbal communication and social aspects of trait-autism are negatively associated with face identity recognition. Therefore, it is predicted that the communication dimension of trait-autism, and potentially the social dimensions of trait-autism, is more likely to be negatively associated with face processing abilities. In conclusion, this study will provide the opportunity to determine the unique effects associated with each of the four face processing abilities, with respect to the dimensions of the trait-autism and cognitive abilities, in neuro-typical adults. In addition to enhancing our understanding of face processing, trait-autism, and cognitive abilities; the observation of one or more unique effects would suggest the possible benefits of experimental manipulation (i.e., face processing training may help ameliorate certain trait-autism behaviours). It will also be noted that cognitive abilities may be expected to associate positively with face processing ability, but unlikely with the trait-autism, thus, controlling for the effects of cognitive abilities on face processing abilities may increase the observed effects between face processing abilities and trait-autism (i.e., a suppressor effect).

In 21st-century society, with the rapid development of information technology, the scientific and technological strength of all walks of life is increasing, and the field of education has also begun to introduce high and new technologies gradually. Affected by the epidemic, online teaching has been implemented all over the country, forming an education model of “dual integration” of online and offline teaching. However, the disadvantages of online teaching are also very obvious; that is, teachers cannot understand the students’ listening status in real-time. Therefore, our study adopts automatic face detection and expression recognition based on a deep learning framework and other related technologies to solve this problem, and it designs an analysis system of students’ class concentration based on expression recognition. The students’ class concentration analysis system can help teachers detect students’ class concentration and improve the efficiency of class evaluation. In this system, OpenCV is used to call the camera to collect the students’ listening status in real-time, and the MTCNN algorithm is used to detect the face of the video to frame the location of the student’s face image. Finally, the obtained face image is used for real-time expression recognition by using the VGG16 network added with ECANet, and the students’ emotions in class are obtained. The experimental results show that the method in our study can more accurately identify students’ emotions in class and carry out a teaching effect evaluation, which has certain application value in intelligent education fields, such as the smart classroom and distance learning. For example, a teaching evaluation module can be added to the teaching software, and teachers can know the listening emotions of each student in class while lecturing.

Recurrent Neural Networks are powerful machine learning frameworks that allow for data to be saved and referenced in a temporal sequence. This opens many new possibilities in fields such as handwriting analysis and speech recognition. This paper seeks to explore current research being conducted on RNNs in four very important areas, being biometric authentication, expression recognition, anomaly detection, and applications to aircraft. This paper reviews the methodologies, purpose, results, and the benefits and drawbacks of each proposed method below. These various methodologies all focus on how they can leverage distinct RNN architectures such as the popular Long Short-Term Memory (LSTM) RNN or a Deep-Residual RNN. This paper also examines which frameworks work best in certain situations, and the advantages and disadvantages of each proposed model.

Data and visual stimuli from the publication: Long, H.*, Peluso, N.*, Baker, C.I. et al. A database of heterogeneous faces for studying naturalistic expressions. Sci Rep 13, 5383 (2023). *equally contributing first authors https://doi.org/10.1038/s41598-023-32659-5

Depression is a mental illness that may be harmful to an individual’s health. Using deep learning models to recognize the facial expressions of individuals captured in videos has shown promising results for automatic depression detection. Typically, depression levels are recognized using 2D-Convolutional Neural Networks (CNNs) that are trained to extract static features from video frames, which impairs the capture of dynamic spatio-temporal relations. As an alternative, 3D-CNNs may be employed to extract spatiotemporal features from short video clips, although the risk of overfitting increases due to the limited availability of labeled depression video data. To address these issues, we propose a novel temporal pooling method to capture and encode the spatio-temporal dynamic of video clips into an image map. This approach allows fine-tuning a pre-trained 2D CNN to model facial variations, and thereby improving the training process and model accuracy. Our proposed method is based on two-stream model that performs late fusion of appearance and dynamic information. Extensive experiments on two benchmark AVEC datasets indicate that the proposed method is efficient and outperforms the state-of-the-art schemes.

The accurate recognition of emotion is important for interpersonal interaction and when navigating our social world. However, not all facial displays reflect the emotional experience currently being felt by the expresser. Indeed, faces express both genuine and posed displays of emotion. In this article, we summarize the importance of motion for the recognition of face identity before critically outlining the role of dynamic information in determining facial expressions and distinguishing between genuine and posed expressions of emotion. We propose that both dynamic information and face familiarity may modulate our ability to determine whether an expression is genuine or not. Finally, we consider the shared role for dynamic information across different face recognition tasks and the wider impact of face familiarity on determining genuine from posed expressions during real-world interactions.

We describe a deep learning based method for estimating 3D facial expression coefficients. Unlike previous work, our process does not relay on facial landmark detection methods as a proxy step. Recent methods have shown that a CNN can be trained to regress accurate and discriminative 3D morphable model (3DMM) representations, directly from image intensities. By foregoing facial landmark detection, these methods were able to estimate shapes for occluded faces appearing in unprecedented in-the-wild viewing conditions. We build on those methods by showing that facial expressions can also be estimated by a robust, deep, landmark-free approach. Our ExpNet CNN is applied directly to the intensities of a face image and regresses a 29D vector of 3D expression coefficients. We propose a unique method for collecting data to train this network, leveraging on the robustness of deep networks to training label noise. We further offer a novel means of evaluating the accuracy of estimated expression coefficients: by measuring how well they capture facial emotions on the CK+ and EmotiW-17 emotion recognition benchmarks. We show that our ExpNet produces expression coefficients which better discriminate between facial emotions than those obtained using state of the art, facial landmark detection techniques. Moreover, this advantage grows as image scales drop, demonstrating that our ExpNet is more robust to scale changes than landmark detection methods. Finally, at the same level of accuracy, our ExpNet is orders of magnitude faster than its alternatives., Comment: Accepted to the IEEE International Conference on Automatic Face and Gesture Recognition, 2018

Altering Facial appearances using surgical procedures are common now days. But it raised challenges for face recognition algorithms. Plastic surgery introduces non linear variations. Because of these variations it is difficult to be modeled by the existing face recognition system. Here presents a multi objective evolutionary granular algorithm. It operates on several granules extracted from a face images at multiple level of granularity. This granular information is unified in an evolutionary manner using multi objective genetic approach. Then identify the facial expression from the face images. For that 3D facial shapes are considering here. A novel automatic feature selection method is proposed based on maximizing the average relative entropy of marginalized class-conditional feature distributions and apply it to a complete pool of candidate features composed of normalized Euclidian distances between 83 facial feature points in the 3D space. A regularized multi-class AdaBoost classification algorithm is used here to get the highest average recognition rate.

The vast increase in the use of social networks and other internet-based communication tools contributed to the escalation of the problem of exchanging child pornographic material over the internet. The problem of dissemination of child pornographic material could be addressed using dedicated image detection algorithms capable of rating the inappropriateness level of images exchanged through computer networks so that images with inappropriate content involving children are blocked. However, the complexity of the image detection task coupled with the nonexistence of suitable datasets, inhibit the development of efficient algorithms that can be used for detecting offensive images containing children. To deal with the problem, we propose a methodological approach that can be used for supporting the development of child pornography detectors through the generation of synthetic datasets and through the decomposition of the task into a set of simpler tasks for which training data is available. Preliminary results show the promise of the proposed approach.

Despite lots of effort being exerted in designing feature descriptors, it is still challenging to find generalized feature descriptors, with acceptable discrimination ability, which are able to capture prominent features in various image processing applications. To address this issue, we propose a computationally feasible discriminative ternary census transform histogram (DTCTH) for image representation which uses dynamic thresholds to perceive the key properties of a feature descriptor. The code produced by DTCTH is more stable against intensity fluctuation, and it mainly captures the discriminative structural properties of an image by suppressing unnecessary background information. Thus, DTCTH becomes more generalized to be used in different applications with reasonable accuracies. To validate the generalizability of DTCTH, we have conducted rigorous experiments on five different applications considering nine benchmark datasets. The experimental results demonstrate that DTCTH performs as high as 28.08% better than the existing state-of-the-art feature descriptors such as GIST, SIFT, HOG, LBP, CLBP, OC-LBP, LGP, LTP, LAID, and CENTRIST.

Facial expression has many applications in human-computer interaction. Although feature extraction and selection have been well studied, the specificity of each expression variation is not fully explored in state-of-the-art works. In this work, the problem of multiclass expression recognition is converted into triplet-wise expression recognition. For each expression triplet, a new feature optimization model based on action unit (AU) weighting and patch weight optimization is proposed to represent the specificity of the expression triplet. The sparse representation-based approach is then proposed to detect the active AUs of the testing sample for better generalization. The algorithm achieved competitive accuracies of 89.67% and 94.09% for the Jaffe and Cohn–Kanade (CK+) databases, respectively. Better cross-database performance has also been observed.

International audience; We present a novel and generic framework for the recognition of body expressions using human postures. Motivated by the state of the art from the domain of psychology, our approach recognizes expression by analyzing sequence of pose. Features proposed in this article are computationally simple and intuitive to understand. They are based on visual cues and provide in-depth understanding of body postures required to recognize body expressions. We have evaluated our approach on different databases with heterogeneous movements and body expressions. Our recognition results exceeds state of the art for some database and for others we obtain results at par with state of the art.

Background Facial emotion recognition (FER) is essential to guide social functioning and behaviour for interpersonal communication. FER may be altered in severe mental illness such as in psychosis and in borderline personality disorder patients. However, it is unclear if these FER alterations are specifically related to psychosis. Awareness of FER alterations may be useful in clinical settings to improve treatment strategies. The aim of our study was to examine FER in patients with severe mental disorder and their relation with psychotic symptomatology. Materials and Methods Socio-demographic and clinical variables were collected. Alterations on emotion recognition were assessed in 3 groups: patients with first episode psychosis (FEP) (n = 64), borderline personality patients (BPD) (n = 37) and healthy controls (n = 137), using the Degraded Facial Affect Recognition Task. The Positive and Negative Syndrome Scale, Structured Interview for Schizotypy Revised and Community Assessment of Psychic Experiences scales were used to assess positive psychotic symptoms. WAIS III subtests were used to assess IQ. Results Kruskal-Wallis analysis showed a significant difference between groups on the FER of neutral faces score between FEP, BPD patients and controls and between FEP patients and controls in angry face recognition. No significant differences were found between groups in the fear or happy conditions. There was a significant difference between groups in the attribution of negative emotion to happy faces. BPD and FEP groups had a much higher tendency to recognize happy faces as negatives. There was no association with the different symptom domains in either group. Conclusions FEP and BPD patients have problems in recognizing neutral faces more frequently than controls. Moreover, patients tend to over-report negative emotions in recognition of happy faces. Although no relation between psychotic symptoms and FER alterations was found, these deficits could contribute to a patient's misinterpretations in daily life.

Copyright © 2011 Society for Neuroscience and the authors. The The Journal of Neuroscience uses a Creative Commons Attribution-NonCommercial-ShareAlike licence: http://creativecommons.org/licenses/by-nc-sa/4.0/. Simulation models of expression recognition contend that to understand another's facial expressions, individuals map the perceived expression onto the same sensorimotor representations that are active during the experience of the perceived emotion. To investigate this view, the present study examines facial expression and identity recognition abilities in a rare group of participants who show facilitated sensorimotor simulation (mirror-touch synesthetes). Mirror-touch synesthetes experience touch on their own body when observing touch to another person. These experiences have been linked to heightened sensorimotor simulation in the shared-touch network (brain regions active during the passive observation and experience of touch). Mirror-touch synesthetes outperformed nonsynesthetic participants on measures of facial expression recognition, but not on control measures of face memory or facial identity perception. These findings imply a role for sensorimotor simulation processes in the recognition of facial affect, but not facial identity. The Medical Research Council, the Economic and Social Research Coucnil and the British Academy.

Micro-expressions are brief spontaneous facial expressions that appear on a face when a person conceals an emotion, making them different to normal facial expressions in subtlety and duration. Currently, emotion classes within the CASME II dataset are based on Action Units and self-reports, creating conflicts during machine learning training. We will show that classifying expressions using Action Units, instead of predicted emotion, removes the potential bias of human reporting. The proposed classes are tested using LBP-TOP, HOOF and HOG 3D feature descriptors. The experiments are evaluated on two benchmark FACS coded datasets: CASME II and SAMM. The best result achieves 86.35\% accuracy when classifying the proposed 5 classes on CASME II using HOG 3D, outperforming the result of the state-of-the-art 5-class emotional-based classification in CASME II. Results indicate that classification based on Action Units provides an objective method to improve micro-expression recognition., Comment: 11 pages, 4 figures and 5 tables. This paper will be submitted for journal review

Facial expression is one of the most natural way of human beings to communicate his-her internal feeling, to stress his-her words, to agree or disagree with the interlocutor, to regulate interaction with the environment and nearby people. This paper challenges the classification experiment run by human beings on the ADFES-BIV database, which is a recently introduced collection of videos expressing low, middle, and high intensity emotions. The proposed automatic system uses the Sparse Representation based Classifier and reaches the top performance of 80 % by considering the temporal information intrinsically present in the videos.

Several models have been proposed that attempt to explain how the brain identifies people by looking at their faces. However, to date, it is still not clear by which mechanism the brain successfully accomplishes the matching of two or more face images when differences in facial expression make the (local and global) appearance of these images different from one another. There seems to be a consensus that faces are processed holistically rather than locally, but there is not yet consensus on whether information on facial expression is passed to the identification process to aid recognition of individuals or not. Models have been proposed that exploit each of these two views, and psychophysical data exist in favor of and against each view. In this article, we show how the experimental data of these two opposite views can be explained by incorporating a key process of motion estimation in the classical feedforward model of face processing. This new model will then lead us to hypothesize that to successfully match expression variant faces, it is convenient to use the information supplied by this motion estimation process within the matching task. We will show experimental results in favor of this hypothesis. Finally, we will show how we can also use the same motion estimator to recognize facial expressions.

Guangheng Dong1, Shenglan Lu21Department of Psychology, 2Department of International Education, Zhejiang Normal University, Jinhua, ChinaAbstract: The present study investigates the relationship of expression recognition and affective experience during facial expression processing by event-related potentials (ERP). Facial expressions used in the present study can be divided into three categories: positive (happy), neutral (neutral), and negative (angry). Participants were asked to finish two kinds of facial recognition tasks: one was easy, and the other was difficult. In the easy task, significant main effects were found for different valence conditions, meaning that emotions were evoked effectively when participants recognized the expressions in facial expression processing. However, no difference was found in the difficult task, meaning that even if participants had identified the expressions correctly, no relevant emotion was evoked during the process. The findings suggest that emotional experience was not simultaneous with expression identification in facial expression processing, and the affective experience process could be suppressed in challenging cognitive tasks. The results indicate that we should pay attention to the level of cognitive load when using facial expressions as emotion-eliciting materials in emotion studies; otherwise, the emotion may not be evoked effectively.Keywords: affective experience, expression recognition, cognitive load, event-related potential