Search

At a time of a rapid growth in the population of elderly individuals and at a time of decreased/pressed availability of human healthcare-resources, automated face analysis has the potential to offer efficient and cost-effective methods for monitoring of a number of pathologies. The author revisits works in automated face analysis, which have focused on designing computer vision algorithms deducing the health state of individuals. Current limitations and benefits are discussed, placing emphasis on the potential that such technology can bring. Computer vision algorithms, most recently based on deep neural networks have been trained with facial images or videos, jointly with health state annotations from clinical experts, in order to learn such algorithms to deduce facets of health states. Examples of such notable algorithms include approaches detecting stress, depression, apathy, pain, neurological disorder, as well as classification of expressions and phenotypes of genetic disorders. Such algorithms are evolving rapidly, providing increasingly reliable accuracy and can support clinicians by providing objective measures. [ABSTRACT FROM AUTHOR]

• Integrating multiple sources of information to address the errors in emotion recognition caused by variations in posture and environment in psychological assessment. • The target detection model extracts human features for real-time recognition and assessment, combining scene information to recognize emotions in the environment and predict a person's apparent emotional state. • Contextual information such as body posture and current scene is crucial for accurately predicting human emotions. • Drawing from psychology, neuroscience, and computer vision, it is essential to design appropriate context extraction and detection methods. Expression is the main method for judging the emotional state and psychological condition of the human body, and the prediction of changes in facial expressions can effectively determine the mental health of a person, thus avoiding serious psychological or psychiatric disorders due to early negligence. From a computer vision perspective, most researchers have focused on studying facial expression analysis, and in some cases, body posture is also considered. However their performance is more limited under unconstrained natural conditions, which requires more information to be used in human emotion analysis. In this paper, we design an Adaptive Multi-End Fusion Attention Mechanism suitable for extracting human body information based on the deep learning framework, depending on human expressions, postures and the environment they are in and add it to an object detection model to obtain the information we need from different regions of the human body and face and features of different sizes and use fusion networks for feature fusion and classification, and from different test methods to confirm that this fusion approach for expression recognition and prediction is feasible. This model achieves an average accuracy of 34.51 % in the Emotic contextual expression recognition dataset. [ABSTRACT FROM AUTHOR]

Facial expression recognition plays an important role in face recognition. Face recognition helps computer to figure out people faces from a simple scenery and to recognize who they are. But facial expression recognition assists computer to analyze the emotion state of one single person in order to improve the human-computer interaction experience. For facial expressions, there are certainly lots of conspicuous features for observers to look on, like shapes of eyes, mouth. When people smile, their lips turn up and their eyebrows bend downward. In this way, we get features of smile. And in same way, we can also do the same to anger, sadness, and surprise etc. Single neural network can make it. However, the accuracy is always low, nearly 70% up and down, which always companied with fluctuation, like electrocardiogram, though not that severe, according to experiments. Therefore, we choose to include Convolutional Block Attention Module (CBAM) into some layers of VGG network we used to improve the accuracy and also the stability. CBAM is an effective attention module for neural networks to concentrate on features. With CBAM, we simplify the scenery that networks need to analyze, and then we can finally make an improvement. This way, we can build better human-computer interfaces. [ABSTRACT FROM AUTHOR]

Computing environment is moving towards human-centered designs instead of computer centered designs and human's tend to communicate wealth of information through affective states or expressions. Traditional Human Computer Interaction (HCI) based systems ignores bulk of information communicated through those affective states and just caters for user's intentional input. Generally, for evaluating and benchmarking different facial expression analysis algorithms, standardized databases are needed to enable a meaningful comparison. In the absence of comparative tests on such standardized databases it is difficult to find relative strengths and weaknesses of different facial expression recognition algorithms. In this article we present a novel video database for Children's Spontaneous facial Expressions (LIRIS-CSE). Proposed video database contains six basic spontaneous facial expressions shown by 12 ethnically diverse children between the ages of 6 and 12 years with mean age of 7.3 years. To the best of our knowledge, this database is first of its kind as it records and shows spontaneous facial expressions of children. Previously there were few database of children expressions and all of them show posed or exaggerated expressions which are different from spontaneous or natural expressions. Thus, this database will be a milestone for human behavior researchers. This database will be a excellent resource for vision community for benchmarking and comparing results. In this article, we have also proposed framework for automatic expression recognition based on Convolutional Neural Network (CNN) architecture with transfer learning approach. Proposed architecture achieved average classification accuracy of 75% on our proposed database i.e. LIRIS-CSE. [ABSTRACT FROM AUTHOR]

This study examined whether three subtypes of anxiety (trait anxiety, state anxiety, and social anxiety) have different effects on recognition of facial expressions. One hundred and thirty-eight participants matched facial expressions of three intensity levels (20 %, 40 %, 100 %) with one of the six emotion labels ("happy", "sad", "fear", "angry", "disgust", and "surprise"). While using a conventional method of analysis we were able to replicate some significant correlations between each anxiety type and recognition performance found in the literature. However, when we used partial correlation to isolate the effect of each anxiety type, most of these correlations were no longer significant, apart from the negative correlations between Beck Anxiety Inventory and reaction time to fearful faces displayed at 40 % intensity level, and the correlations between anxiety and categorisation errors. Specifically, social anxiety was positively correlated with misidentifying a happy face as a disgust face at 40 % intensity level, and state anxiety negatively correlated with misidentifying a happy face as a sad face at 20 % intensity level. However, these partial correlation analyses became non-significant after p value adjustment for multiple comparisons. Our eye tracking data also showed that state anxiety may be associated with reduced fixations on the eye regions of low-intensity sad or fearful faces. These analyses cast doubts on some effects reported in the previous studies because they are likely to reflect a mixture of influences from highly correlated anxiety subtypes. • A person with trait anxiety tends to react fast to fearful faces. • A person with social anxiety tends to see a happy face as a disgust face • A person with state anxiety tends not to see a happy face as a sad face • These effects are weak and only occur for low intensity facial expressions • Each type of anxiety differentially influences recognition of facial expressions [ABSTRACT FROM AUTHOR]

Abstract Facial expression recognition in a wild situation is a challenging problem in computer vision research due to different circumstances, such as pose dissimilarity, age, lighting conditions, occlusions, etc. Numerous methods, such as point tracking, piecewise affine transformation, compact Euclidean space, modified local directional pattern, and dictionary-based component separation have been applied to solve this problem. In this paper, we have proposed a deep learning–based automatic wild facial expression recognition system where we have implemented an incremental active learning framework using the VGG16 model developed by the Visual Geometry Group. We have gathered a large amount of unlabeled facial expression data from Intelligent Technology Lab (ITLab) members at Inha University, Republic of Korea, to train our incremental active learning framework. We have collected these data under five different lighting conditions: good lighting, average lighting, close to the camera, far from the camera, and natural lighting and with seven facial expressions: happy, disgusted, sad, angry, surprised, fear, and neutral. Our facial recognition framework has been adapted from a multi-task cascaded convolutional network detector. Repeating the entire process helps obtain better performance. Our experimental results have demonstrated that incremental active learning improves the starting baseline accuracy from 63% to average 88% on ITLab dataset on wild environment. We also present extensive results on face expression benchmark such as Extended Cohn-Kanade Dataset, as well as ITLab face dataset captured in wild environment and obtained better performance than state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

In this paper we present architectures based on deep neural nets for expression recognition in videos, which are invariant to local scaling. We amalgamate autoencoder and predictor architectures using an adaptive weighting scheme coping with a reduced size labeled dataset, while enriching our models from enormous unlabeled sets. We further improve robustness to lighting conditions by introducing a new adaptive filter based on temporal local scale normalization. We provide superior results over known methods, including recent reported approaches based on neural nets. [ABSTRACT FROM AUTHOR]

Altering Facial appearances using surgical procedures are common now days. But it raised challenges for face recognition algorithms. Plastic surgery introduces non linear variations. Because of these variations it is difficult to be modeled by the existing face recognition system. Here presents a multi objective evolutionary granular algorithm. It operates on several granules extracted from a face images at multiple level of granularity. This granular information is unified in an evolutionary manner using multi objective genetic approach. Then identify the facial expression from the face images. For that 3D facial shapes are considering here. A novel automatic feature selection method is proposed based on maximizing the average relative entropy of marginalized class-conditional feature distributions and apply it to a complete pool of candidate features composed of normalized Euclidian distances between 83 facial feature points in the 3D space. A regularized multi-class AdaBoost classification algorithm is used here to get the highest average recognition rate. [ABSTRACT FROM AUTHOR]

It is well known that some facial attributes –like soft biometric traits– can increase the performance of traditional biometric systems and help recognition based on human descriptions. In addition, other facial attributes, such as facial expressions, can be used in human–computer interfaces, image retrieval, talking heads and human emotion analysis. This paper addresses the problem of automated recognition of facial attributes by proposing a new general approach called Adaptive Sparse Representation of Random Patches (ASR+). The proposed method consists of two stages: in the learning stage, random patches are extracted from representative face images of each class ( e.g. , in gender recognition –a two-class problem–, images of females/males) in order to construct representative dictionaries. A stop list is used to remove very common words of the dictionaries. In the testing stage, random test patches of the query image are extracted, and for each non–stopped test patch a dictionary is built concatenating the ‘best’ representative dictionary of each class. Using this adapted dictionary, each non–stopped test patch is classified following the Sparse Representation Classification (SRC) methodology. Finally, the query image is classified by patch voting. Thus, our approach is able to learn a model for each recognition task dealing with a larger degree of variability in ambient lighting, pose, expression, occlusion, face size and distance from the camera. Experiments were carried out on eight face databases in order to recognize facial expression, gender, race, disguise and beard. Results show that ASR+ deals well with unconstrained conditions, outperforming various representative methods in the literature in many complex scenarios. [ABSTRACT FROM AUTHOR]

Deriving an effective image representation is a critical step for a successful automatic image recognition application. In this paper, we propose a new feature descriptor named Local Directional Texture Pattern (LDTP) that is versatile, as it allows us to distinguish person’s expressions, and different landscapes scenes. In detail, we compute the LDTP feature, at each pixel, by extracting the principal directions of the local neighborhood, and coding the intensity differences on these directions. Consequently, we represent each image as a distribution of LDTP codes. The mixture of structural and contrast information makes our descriptor robust against illumination changes and noise. We also use Principal Component Analysis to reduce the dimension of the multilevel feature set, and test the results on this new descriptor as well. [ABSTRACT FROM AUTHOR]

Highlights: [•] First work to identify the problem of transfer learning with one-class data. [•] A novel regression-based algorithm to address this problem. [•] A new approach to select the most transferable features for one-class data. [•] Experiment on two new application scenarios with one-class data. [ABSTRACT FROM AUTHOR]

Abstract: This paper presents a method for the recognition of the six basic facial expressions in images or in image sequences using landmark points. The proposed technique relies on the observation that the vectors formed by the landmark point coordinates belong to a different manifold for each of the expressions. In addition experimental measurements validate the hypothesis that each of these manifolds can be decomposed to a small number of linear subspaces of very low dimension. This yields a parameterization of the manifolds that allows for computing the distance of a feature vector from each subspace and consequently from each one of the six manifolds. Two alternative classifiers are next proposed that use the corresponding distances as input: the first one is based on the minimum distance from the manifolds, while the second one uses SVMs that are trained with the vector of all distances from each subspace. The proposed technique is tested for two scenarios, the subject-independent and the subject-dependent one. Extensive experiments for each scenario have been performed on two publicly available datasets yielding very satisfactory expression recognition accuracy. [Copyright &y& Elsevier]

Abstract: Facial expression analysis is essential for human–computer interface. For different expressions, different parts of the face play different roles due to distinct movement of facial muscles. In this work, we propose to learn the weight associated with different facial regions for different expressions. The facial feature points are first located accurately based on a graphical model. Based on using the optical flow to represent the motion information due to facial expression, a quadratic programming problem is formulated to learn the optimal spatial weighting from training data such that faces of the same expression category are closer than those of different categories in the weighted optical flow space. We demonstrate the advantages of applying the learned weight to facial expression recognition and intensity estimation through experiments on several well-known facial expression databases. [Copyright &y& Elsevier]

Abstract: A key assumption of traditional machine learning approach is that the test data are draw from the same distribution as the training data. However, this assumption does not hold in many real-world scenarios. For example, in facial expression recognition, the appearance of an expression may vary significantly for different people. As a result, previous work has shown that learning from adequate person-specific data can improve the expression recognition performance over the one from generic data. However, person-specific data is typically very sparse in real-world applications due to the difficulties of data collection and labeling, and learning from sparse data may suffer from serious over-fitting. In this paper, we propose to learn a person-specific model through transfer learning. By transferring the informative knowledge from other people, it allows us to learn an accurate model for a new subject with only a small amount of person-specific data. We conduct extensive experiments to compare different person-specific models for facial expression and action unit (AU) recognition, and show that transfer learning significantly improves the recognition performance with a small amount of training data. [Copyright &y& Elsevier]

Abstract: The face is an important medium used by humans to communicate, and facial articulation also reflects a person''s emotional and awareness states, cognitive activity, personality or wellbeing. With the advances in 3-D imaging technology and ever increasing computing power, automatic analysis of facial articulation using 3-D sequences is becoming viable. This paper describes Hi4D-ADSIP — a comprehensive 3-D dynamic facial articulation database, containing scans with high spatial and temporal resolution. The database is designed not only to facilitate studies on facial expression analysis, but also to aid research into clinical diagnosis of facial dysfunctions. The database currently contains 3360 facial sequences captured from 80 healthy volunteers (control subjects) of various age, gender and ethnicity. The database has been validated using psychophysical experiments used to formally evaluate the accuracy of the recorded expressions. The results of baseline automatic facial expression recognition methods using Eigen- and Fisher-faces are also presented alongside some initial results obtained for clinical cases. This database is believed to be one of the most comprehensive repositories of facial 3-D dynamic articulations to date. The extension of this database is currently under construction aiming at building a comprehensive repository of representative facial dysfunctions exhibited by patients with stroke, Bell''s palsy and Parkinson''s disease. [Copyright &y& Elsevier]

Abstract: Facial expression analysis has interested many researchers in the past decade due to its potential applications in various fields such as human–computer interaction, psychological studies, and facial animation. Three-dimensional facial data has been proven to be insensitive to illumination condition and head pose, and has hence gathered attention in recent years. In this paper, we focus on discrete expression classification using 3D data from the human face. The paper is divided in two parts. In the first part, we present improvement to the fitting of the Annotated Face Model (AFM) so that a dense point correspondence can be found in terms of both position and semantics among static 3D face scans or frames in 3D face sequences. Then, an expression recognition framework on static 3D images is presented. It is based on a Point Distribution Model (PDM) which can be built on different features. In the second part of this article, a systematic pipeline that operates on dynamic 3D sequences (4D datasets or 3D videos) is proposed and alternative modules are investigated as a comparative study. We evaluated both 3D and 4D Facial Expression Recognition pipelines on two publicly available facial expression databases and obtained promising results. [Copyright &y& Elsevier]

Abstract: We propose a novel approach to human facial expression recognition using only the shape information at a finite set of fiducial points, extracted from the 3D neutral and expressive faces. In the course of applying the technique to the facial database, BU-3DFE, which contains facial shape and 2D color (“texture”) information, we extract from the images of neutral and expressive faces, salient contours in the facial interest-regions around the eyebrows, eyes, nose and mouth by invoking an active contour algorithm. The contours are then uniformly sampled and mapped onto the 3D-mesh dataset in order to generate a shape (and color) description of the interest-regions. By a matrix–algebraic operation on the shape of the neutral and expressive faces, a shape feature-matrix is computed for each expression and for each person, which is then subjected to the proposed modified PCA approach to recognize expressions. Classification results are presented to demonstrate the effectiveness of the proposed approach. It is also found that accuracy estimates compare favorably with those in the literature on facial expression recognition from 3D-mesh datasets. [Copyright &y& Elsevier]

Several models have been proposed that attempt to explain how the brain identifies people by looking at their faces. However, to date, it is still not clear by which mechanism the brain successfully accomplishes the matching of two or more face images when differences in facial expression make the (local and global) appearance of these images different from one another. There seems to be a consensus that faces are processed holistically rather than locally, but there is not yet consensus on whether information on facial expression is passed to the identification process to aid recognition of individuals or not. Models have been proposed that exploit each of these two views, and psychophysical data exist in favor of and against each view. In this article, we show how the experimental data of these two opposite views can be explained by incorporating a key process of motion estimation in the classical feedforward model of face processing. This new model will then lead us to hypothesize that to successfully match expression variant faces, it is convenient to use the information supplied by this motion estimation process within the matching task. We will show experimental results in favor of this hypothesis. Finally, we will show how we can also use the same motion estimator to recognize facial expressions. [Copyright &y& Elsevier]

The video analysis system described in this paper aims at facial expression recognition consistent with the MPEG4 standardized parameters for facial animation, FAP. For this reason, two levels of analysis are necessary: low-level analysis to extract the MPEG4 compliant parameters and high-level analysis to estimate the expression of the sequence using these low-level parameters.The low-level analysis is based on an improved active contour algorithm that uses high level information based on principal component analysis to locate the most significant contours of the face (eyebrows and mouth), and on motion estimation to track them. The high-level analysis takes as input the FAP produced by the low-level analysis tool and, by means of a Hidden Markov Model classifier, detects the expression of the sequence. [Copyright &y& Elsevier]

• A new iterative framework for the hyper-parameter optimization in deep sparseness strategies is proposed to adapt hyper-parameter setting to different databases in FER. • A simplified network is deployed to surrogate the original network for hyper-parameter optimization, where Euclidean losses with unilateral back propagation are introduced to approximate the original network. • The proposed algorithm can automatically adapt deep network metrics to different databases with a reasonable time complexity. • The hyper-parameter optimization algorithm achieved competitive performances on six public benchmark expression databases. For facial expression recognition, the sparseness constraints of the features or weights can improve the generalization ability of a deep network. However, the optimization of the hyper-parameters in fusing different sparseness strategies demands much computation, when the traditional gradient-based algorithms are used. In this work, an iterative framework with surrogate network is proposed for the optimization of hyper-parameters in fusing different sparseness strategies. In each iteration, a network with significantly smaller model complexity is fitted to the original large network based on four Euclidean losses, where the hyper-parameters are optimized with heuristic optimizers. Since the surrogate network uses the same deep metrics and embeds the same hyper-parameters as the original network, the optimized hyper-parameters are then used for the training of the original deep network in the next iteration. While the performance of the proposed algorithm is justified with a tiny model, i.e. LeNet on the FER2013 database, our approach achieved competitive performances on six publicly available expression datasets, i.e., FER2013, CK+, Oulu-CASIA, MMI, AFEW and AffectNet. [ABSTRACT FROM AUTHOR]

• A new framework of facial expression recognition is proposed, where different feature sparseness strategies are embedded in deep networks and further investigated. • Feature sparseness of the fully-connected layer input is embedded into a deep network to boost the feature generalization ability, which includes less parameters, while achieves better performance than other network sparseness. • The advantage of the feature sparseness is evaluated with a toy model based on a quantitative metric. • The deep metric learning achieved competitive recognition rates and state-of-the-art cross-database performance on four benchmark expression databases, i.e. FER2013, CK+, Oulu-CASIA and MMI. While weight sparseness-based regularization has been used to learn better deep features for image recognition problems, it introduced a large number of variables for optimization and can easily converge to a local optimum. The L2-norm regularization proposed for face recognition reduces the impact of the noisy information, while expression information is also suppressed during the regularization. A feature sparseness-based regularization that learns deep features with better generalization capability is proposed in this paper. The regularization is integrated into the loss function and optimized with a deep metric learning framework. Through a toy example, it is showed that a simple network with the proposed sparseness outperforms the one with the L2-norm regularization. Furthermore, the proposed approach achieved competitive performances on four publicly available datasets, i.e., FER2013, CK+, Oulu-CASIA and MMI. The state-of-the-art cross-database performances also justify the generalization capability of the proposed approach. [ABSTRACT FROM AUTHOR]