Search

In order to solve the problem of low detection efficiency and long working time in the traditional video surveillance system for abnormal behavior detection and identification methods. A multimodal abnormal behavior detection and identification method based on video surveillance is proposed and applied to an online video classroom concentration evaluation task for college students in English. The model works by capturing abnormal behaviors and facial expressions and building a joint network that fuses abnormal behaviors and facial expressions. By testing on two open-source datasets and self-built classroom real-time datasets, the results verify that the model in this paper has better recognition performance compared to current mainstream models while maintaining real-time performance. The model proposed in this paper provides a new way of thinking about building smart classrooms. [ABSTRACT FROM AUTHOR]

In recent years, a rise in missing person cases has posed challenges for law enforcement. This paper explores the various issues surrounding many unresolved cases and aims to uncover the contributing factors. Using a detailed analysis of police records, the study identifies patterns and challenges in resolving missing person cases. By understanding these dynamics, law enforcement agencies can refine strategies to enhance the likelihood of resolution, emphasizing the critical need for effective measures in addressing the growing issue of missing persons. Additionally, the paper proposes an innovative approach to locating missing persons using machine learning (ML) algorithms, specifically support vector machine (SVM) and K-nearest neighbors (KNN). Utilizing facial expressions as the basis for model training, the system swiftly and accurately identifies known individuals. The system, fed with a missing person dataset from Kaggle, outputs the person's identity based on features like gender, age and location. The results are then communicated to the police for further investigation. This streamlined approach enhances the efficiency of the search and identification process, contributing to more effective resolutions of missing person cases. The proposed system serves as a valuable tool for law enforcement in expediting investigations and addressing the critical issue of missing persons in a timely and efficient manner. [ABSTRACT FROM AUTHOR]

Pain assessment has become an important component in modern healthcare systems. It aids medical professionals in patient diagnosis and providing the appropriate care and therapy. Conventionally, patients are asked to provide their pain level verbally. However, this subjective method is generally inaccurate, not possible for non-communicative people, can be affected by physiological and environmental factors and is time-consuming, which renders it inefficient in healthcare settings. So, there has been a growing need to build objective, reliable and automatic pain assessment alternatives. In fact, due to the efficiency of facial expressions as pain biomarkers that accurately expand the pain intensity and the power of machine learning methods to effectively learn the subtle nuances of pain expressions and accurately predict pain intensity, automatic pain assessment methods have evolved rapidly. This paper reviews recent spatial facial expressions and machine learning-based pain assessment methods. Moreover, we highlight the pain intensity scales, datasets and method performance evaluation criteria. In addition, these methods' contributions, strengths and limitations will be reported and discussed. Additionally, the review lays the groundwork for further study and improvement for more accurate automatic pain assessment. [ABSTRACT FROM AUTHOR]

There are several factors for vehicle accidents during driving such as drivers' negligence, drowsiness, and fatigue. These accidents can be avoided, if drivers are warned in time. Moreover, recent developments in computer vision and artificial intelligence (AI) have helped to monitor drivers and alert them in case they are not concentrating on driving. The AI techniques can extract relevant features from expressions of driver's face, such as eye closure, yawning, and head movements to infer the level of sleepiness. In addition, they can acquire biological signals from the driver's body, and indications from the vehicle behavior. This paper provides a comprehensive review of the detection techniques of drowsiness and fatigue of drivers using machine learning (ML) and deep learning (DL). The current techniques for this application are classified into four categories: image- or video-based analysis during the driving, biological signal analysis for drivers, vehicle movement analysis, and hybrid techniques. A review of supervised techniques is presented for detecting fatigue and drowsiness on different datasets, with a comparison of the various techniques in terms of pros and cons. Results are presented in terms of accuracy of detection for each technique. The results are discussed according to the recent problems and challenges in this field. The paper also highlights the applicability and reliability of the different techniques. Furthermore, some suggestions are presented for the future work in the field of driver drowsiness detection (DDD). [ABSTRACT FROM AUTHOR]

Given their easy accessibility and dual‐channel model of content presentation, instructional videos have become a favoured tool for EFL vocabulary learning tool among many students. Teachers often use various nonverbal behaviours to elicit social reactions and guide learners' attention in instructional videos. The current study conducted three eye‐tracking experiments to examine the circumstances under which a teacher's happy facial expressions are beneficial in instructional videos, with or without pointing gestures and mouse pointing. Experiments 1 and 2 demonstrated that the combination of happy facial expressions and pointing gestures attracted learners' attention to the teacher and hindered students' learning performance, regardless of the complexity of slides. Experiment 3 showed that in instructional videos with complex slides, using happy facial expressions along with mouse pointing can enhance students' learning performance. Teachers are advised to show happy facial expressions and avoid using pointing gestures when designing instructional videos. Practitioner notes What is already known about this topic Given easy accessibility and dual‐channel model of content presentation, instructional videos have become a favoured tool for EFL vocabulary learning. When teachers record instructional videos while standing alongside slides, they often use nonverbal cues to support their speech. Teachers' social and attentional cues interactively influence students' learning processes and performance. What this paper adds A teacher's happy facial expressions evoke more positive emotions and greater motivation in learners compared to bored expressions. A teacher's pointing gestures, when combined with happy facial expressions, divert students' attention away from slides and towards the teacher. A teacher's happy facial expressions enhance students' learning performance when no pointing gestures are used in videos with simple slides. Implications for practice/policy Teachers are advised to display happy facial expressions and avoid using pointing gestures in instructional videos, regardless of the complexity of the slides. Practitioners should consider how to incorporate teachers' facial expressions pointing gestures and mouse pointing effectively. What is already known about this topic Given easy accessibility and dual‐channel model of content presentation, instructional videos have become a favoured tool for EFL vocabulary learning. When teachers record instructional videos while standing alongside slides, they often use nonverbal cues to support their speech. Teachers' social and attentional cues interactively influence students' learning processes and performance. What this paper adds A teacher's happy facial expressions evoke more positive emotions and greater motivation in learners compared to bored expressions. A teacher's pointing gestures, when combined with happy facial expressions, divert students' attention away from slides and towards the teacher. A teacher's happy facial expressions enhance students' learning performance when no pointing gestures are used in videos with simple slides. Implications for practice/policy Teachers are advised to display happy facial expressions and avoid using pointing gestures in instructional videos, regardless of the complexity of the slides. Practitioners should consider how to incorporate teachers' facial expressions pointing gestures and mouse pointing effectively. [ABSTRACT FROM AUTHOR]

This paper proposes a model-based method for real-time automatic mood estimation in video sequences. The approach is customized by learning the person's specific facial parameters, which are transformed into facial Action Units (AUs). A model mapping for mood representation is used to describe moods in terms of the PAD space: Pleasure, Arousal, and Dominance. From the intersection of these dimensions, eight octants represent fundamental mood categories. In the experimental evaluation, a stimulus video randomly selected from a set prepared to elicit different moods was played to participants, while the participant's facial expressions were recorded. From the experiment, Dominance is the dimension least impacted by facial expression, and this dimension could be eliminated from mood categorization. Then, four categories corresponding to the quadrants of the Pleasure–Arousal (PA) plane, "Exalted", "Calm", "Anxious" and "Bored", were defined, with two more categories for the "Positive" and "Negative" signs of the Pleasure (P) dimension. Results showed a 73% of coincidence in the PA categorization and a 94% in the P dimension, demonstrating that facial expressions can be used to estimate moods, within these defined categories, and provide cues for assessing users' subjective states in real-world applications. [ABSTRACT FROM AUTHOR]

Facial expression is an important way to reflect human emotions and it represents a dynamic deformation process. Analyzing facial movements is an effective means of understanding expressions. However, there is currently a lack of methods capable of analyzing the dynamic details of full-field deformation in expressions. In this paper, in order to enable effective dynamic analysis of expressions, a classic optical measuring method called stereo digital image correlation (stereo-DIC or 3D-DIC) is employed to analyze the deformation fields of facial expressions. The forming processes of six basic facial expressions of certain experimental subjects are analyzed through the displacement and strain fields calculated by 3D-DIC. The displacement fields of each expression exhibit strong consistency with the action units (AUs) defined by the classical Facial Action Coding System (FACS). Moreover, it is shown that the gradient of the displacement, i.e., the strain fields, offers special advantages in characterizing facial expressions due to their localized nature, effectively sensing the nuanced dynamics of facial movements. By processing extensive data, this study demonstrates two featured regions in six basic expressions, one where deformation begins and the other where deformation is most severe. Based on these two regions, the temporal evolutions of the six basic expressions are discussed. The presented investigations demonstrate the superior performance of 3D-DIC in the quantitative analysis of facial expressions. The proposed analytical strategy might have potential value in objectively characterizing human expressions based on quantitative measurement. [ABSTRACT FROM AUTHOR]

This paper offers a thorough analysis of the state of deceit detection in criminal justice and law enforcement settings as of right now. The study, which synthesizes findings from multiple investigations, emphasizes the progress made as well as the ongoing difficulties in accurately distinguishing deception from truth. The limitations of conventional techniques like behavior analysis interviews and polygraph exams, the potential of alternative strategies like voice tone analysis and facial expression analysis, and the moral ramifications of using emotional AI systems for deception detection are some of the main subjects covered. The review highlights the necessity for ongoing interdisciplinary research efforts and ethical concerns through critical analysis and discussion, in order to progress the field of deception detection while maintaining justice and respect to human rights values. [ABSTRACT FROM AUTHOR]

This paper considers the automatic labeling of emotions in face images found on social media. Facial landmarks are commonly used to classify the emotions from a face image. However, it is difficult to accurately segment landmarks for some faces and for subtle emotions. Previous authors used a Gaussian prior for the refinement of landmarks, but their model often gets stuck in a local minima. Instead, the calibration of the landmarks with respect to the known emotion class label using principal component analysis is proposed in this paper. Next, the face image is generated from the landmarks using an image translation model. The proposed model is evaluated on the classification of facial expressions and also for fish identification underwater and outperforms baselines in accuracy by over 20 % . [ABSTRACT FROM AUTHOR]

Autism spectrum disorder (A.S.D.) is considered a heterogeneous mental disorder, which is notoriously difficult to identify for a better diagnosis, especially among children. The current diagnosis methodology is purely based on the behavioural observation of symptoms prone to misdiagnosis. Several hybrid methods were explored, which also needs its improvisation in better prediction and diagnosis to move this field towards intelligent and accurate diagnosis. The main objective of this research paper was to develop the new diagnosis software which integrates the novel fuzzy hybrid deep convolutional neural networks and fusion of facial expressions and human gaits based on input video sequences. The algorithm has been trained and validated with the different video datasets such as Kaggle FER2013 and Karolinska Directed Emotional Faces (KDEF) datasets with real-time test scenarios, and various parameters such as accuracy, recall and F1-score were evaluated. Our proposed deep learning model outperforms another state-of-the-art method with an increase in prediction accuracy up to 30% with maximum accuracy of 95%. The model presented in this paper yields more advantages in terms of prediction time and accuracy also. However, the speech therapists, teachers, caretakers, and parents can use the software as a technological tool when working with children with A.S.D. [ABSTRACT FROM AUTHOR]

In static advertisements, images instead of videos are used to promote products, and it is most often the faces of the models that get the consumers' attention. How facial expressions transfer emotions from the producer to the consumer has been explained by the emotional contagion theory, which suggests the recipient replicates the emitter through motor mimicry which changes the recipient's emotional state. This paper investigates the effects of facial expressions in advertisements on adolescents in comparison to adults. It was hypothesized that participants would give a higher product evaluation if the model displayed a real smile, versus a fake smile or neutral facial expression. It was also hypothesized that adolescents would be more susceptible to facial expressions and therefore pay more for the same product than adults. High school and adult participants completed a survey in which they evaluated a product (a white t-shirt) when the model displayed a neutral face, fake smile, and real smile. The results showed that participants gave higher product evaluation scores and were willing to pay more when the model displayed a genuine smile over the other facial expressions. In addition, adolescents were less influenced by the model's expressions than adults which may be due to their lack of experience in spending. To extend this study, the effects of emotional contagion theory on other factors including how the product attracts attention or lasts in the consumers' memory could be tested. [ABSTRACT FROM AUTHOR]

Wearable technologies and digital phenotyping foster unique opportunities for designing novel intelligent electronic services that can address various well-being issues in patients with mental disorders (i.e., schizophrenia and bipolar disorder), thus having the potential to revolutionize psychiatry and its clinical practice. In this paper, we present e-Prevention, an innovative integrated system for medical support that facilitates effective monitoring and relapse prevention in patients with mental disorders. The technologies offered through e-Prevention include: (i) long-term continuous recording of biometric and behavioral indices through a smartwatch; (ii) video recordings of patients while being interviewed by a clinician, using a tablet; (iii) automatic and systematic storage of these data in a dedicated Cloud server and; (iv) the ability of relapse detection and prediction. This paper focuses on the description of the e-Prevention system and the methodologies developed for the identification of feature representations that correlate with and can predict psychopathology and relapses in patients with mental disorders. Specifically, we tackle the problem of relapse detection and prediction using Machine and Deep Learning techniques on all collected data. The results are promising, indicating that such predictions could be made and leading eventually to the prediction of psychopathology and the prevention of relapses. [ABSTRACT FROM AUTHOR]

With the introduction of concepts for virtual interaction and digital doubles, a rich scenario has been created for embodied avatars to strive. These avatars, more recently referred to as digital humans, have become a popular area of research, resulting in various techniques and methods that focus on improving the perception of their realism, fidelity, emphatic response, and interactivity. This survey aims to explore the literature and recent advancements on the key processes behind the creation and animation of digital human faces through the view of a general pipeline. The extensive review carried out in this study explores the usual data collection protocols, the main facial codification paradigms and databases, the approaches for digital human asset creation, facial tracking solutions for performance-driven animation, the solving process, and the final rendering delivery. Different quantitative evaluation methods, visual perception tests, and empathetic response evaluations for digital humans are also included in the survey. Additionally, the paper presents an updated summary of public and private frameworks for digital humans that go through the complete general pipeline presented. Finally, the condensed knowledge is discussed, inquiring into the possible direction of future developments in the field. [ABSTRACT FROM AUTHOR]

Significant advancements have been made in developing parametric models for digital humans, with various approaches concentrating on parts such as the human body, hand, or face. Nevertheless, connectors such as the neck have been overlooked in these models, with rich anatomical priors often unutilized. In this paper, we introduce HACK (Head-And-neCK), a novel parametric model for constructing the head and cervical region of digital humans. Our model seeks to disentangle the full spectrum of neck and larynx motions, facial expressions, and appearance variations, providing personalized and anatomically consistent controls, particularly for the neck regions. To build our HACK model, we acquire a comprehensive multi-modal dataset of the head and neck under various facial expressions. We employ a 3D ultrasound imaging scheme to extract the inner biomechanical structures, namely the precise 3D rotation information of the seven vertebrae of the cervical spine. We then adopt a multi-view photometric approach to capture the geometry and physically-based textures of diverse subjects, who exhibit a diverse range of static expressions as well as sequential head-and-neck movements. Using the multi-modal dataset, we train the parametric HACK model by separating the 3D head and neck depiction into various shape, pose, expression, and larynx blendshapes from the neutral expression and the rest skeletal pose. We adopt an anatomically-consistent skeletal design for the cervical region, and the expression is linked to facial action units for artist-friendly controls. We also propose to optimize the mapping from the identical shape space to the PCA spaces of personalized blendshapes to augment the pose and expression blendshapes, providing personalized properties within the framework of the generic model. Furthermore, we use larynx blendshapes to accurately control the larynx deformation and force the larynx slicing motions along the vertical direction in the UV-space for precise modeling of the larynx beneath the neck skin. HACK addresses the head and neck as a unified entity, offering more accurate and expressive controls, with a new level of realism, particularly for the neck regions. This approach has significant benefits for numerous applications, including geometric fitting and animation, and enables inter-correlation analysis between head and neck for fine-grained motion synthesis and transfer. [ABSTRACT FROM AUTHOR]

Facial emotion recognition (FER) is an emerging and significant research area in the pattern recognition domain. In daily life, the role of non-verbal communication is significant, and in overall communication, its involvement is around 55% to 93%. Facial emotion analysis is efficiently used in surveillance videos, expression analysis, gesture recognition, smart homes, computer games, depression treatment, patient monitoring, anxiety, detecting lies, psychoanalysis, paralinguistic communication, detecting operator fatigue and robotics. In this paper, we present a detailed review on FER. The literature is collected from different reputable research published during the current decade. This review is based on conventional machine learning (ML) and various deep learning (DL) approaches. Further, different FER datasets for evaluation metrics that are publicly available are discussed and compared with benchmark results. This paper provides a holistic review of FER using traditional ML and DL methods to highlight the future gap in this domain for new researchers. Finally, this review work is a guidebook and very helpful for young researchers in the FER area, providing a general understating and basic knowledge of the current state-of-the-art methods, and to experienced researchers looking for productive directions for future work. [ABSTRACT FROM AUTHOR]

Continuous affect estimation from facial expressions has attracted increased attention in the affective computing research community. This paper presents a principled framework for estimating continuous affect from video sequences. Based on recent developments, we address the problem of continuous affect estimation by leveraging the Bayesian filtering paradigm, i.e., considering affect as a latent dynamical system corresponding to a general feeling of pleasure with a degree of arousal, and recursively estimating its state using a sequence of visual observations. To this end, we advance the state-of-the-art as follows: (i) Canonical face representation (CFR): a novel algorithm for two-dimensional face frontalization, (ii) Convex unsupervised representation learning (CURL): a novel frequency-domain convex optimization algorithm for unsupervised training of deep convolutional neural networks (CNN)s, and (iii) Deep extended Kalman filtering (DEKF): an extended Kalman filtering-based algorithm for affect estimation from a sequence of CNN observations. The performance of the resulting CFR-CURL-DEKF algorithmic framework is empirically evaluated on publicly available benchmark datasets for facial expression recognition (CK+) and continuous affect estimation (AVEC 2012 and 2014). [ABSTRACT FROM AUTHOR]

Facial expressions play an important role in human communication and integrating deep learning techniques into Internet of Things (IoT) scenarios enhances the understanding of this data, enabling applications in industries such as healthcare, security, and human-computer interaction. The existing methods suffer from lower accuracy and higher computational complexity compared to the proposed Deep Convolutional Neural Network using Galactic Swarm Optimization (DCNN-GSO) approach hinder their practical applicability in real-time image processing tasks. This paper proposes a comprehensive framework for facial expression analysis in IoT environments. The preprocessing phase uses a Gaussian filter to improve image quality and reduce noise. Feature extraction is performed using a spatial temporal interest point (STIP), which captures the spatial and temporal cue of facial expressions. The proposed method leverages Deep Convolutional Neural Networks (DCNN) to extract discriminative features from facial images captured by IoT devices. Galactic Swarm Optimization (GSO) is employed to optimize the hyperparameters of the DCNN model, thereby improving its performance in facial expression classification tasks. By integrating GSO with deep learning, the proposed approach aims to overcome the challenges of limited computational resources and energy constraints inherent in IoT environments. GSO optimizes the parameters of deep learning models for facial expression recognition, improving accuracy and robustness. The proposed framework provides a comprehensive approach for facial expression analysis in IoT environments, solving challenges such as noise, computational complexity, accuracy, etc. in IoT systems, and opening the way for humans and improved device performance and connectivity. The DCNN-GSO method outperforms the competition with remarkable results: 94% accuracy, 92.3% precision, and 91% recall. With a very low mean absolute error (MAE) of 3.46, it shows itself to be a reliable and accurate solution for practical uses. [ABSTRACT FROM AUTHOR]

Automatic facial expression recognition (FER) plays a valuable role in various fields, including health, road safety, and marketing, where providing feedback on the user's condition is crucial. While significant progress has been made in controlled environments (such as frontal, unconcluded, and well-lit conditions), recognizing facial expressions in unconstrained environments (natural settings) remains challenging. The presence of occlusions poses a particular difficulty as they obscure parts of the facial information captured in the image. To address this issue, researchers have proposed different solutions, broadly categorized into two approaches: those focusing on visible regions of the face and those attempting to reconstruct hidden parts. Currently, most solutions rely on texture or geometry-based methods, with only a few utilizing motion-based approaches. However, incorporating motion appears to be particularly promising in adapting to occlusions due to its unique characteristics, such as close-range propagation and local coherence. In this paper, our focus lies on leveraging motion to overcome the challenges posed by occlusions in FER tasks. [ABSTRACT FROM AUTHOR]

It is known that purely geometric distance metrics cannot reflect the human perception of facial expressions. A novel perceptually based distance metric designed for 3D facial blendshape models is proposed in this paper. To develop this metric, comparative evaluations of facial expressions were collected from a crowdsourcing experiment. Then, the weights of a distance metric, based on descriptive features of the models, were optimized to match the results with crowdsourced data, through a metric learning process. The method incorporates perceptual properties such as curvature and visual saliency. A formal analysis of the results proves the high correlation between the metric output and human perception. The effectiveness and success of the proposed metric were also compared to other distance alternatives. The proposed metric will enable intelligent processing of 3D facial blendshapes data in several ways. It will be possible to generate perceptually valid clustering and visualization of 3D facial blendshapes. It will help reduce storage and computational requirements by removing redundant expressions that are perceptually identical from the overall dataset. It can also be used to assist novice animators while creating plausible and expressive facial animations. [ABSTRACT FROM AUTHOR]

The dramatic impact of the COVID-19 pandemic has resulted in the closure of physical classrooms and teaching methods being shifted to the online medium.To make the online learning environment more interactive, just like traditional offline classrooms, it is essential to ensure the proper engagement of students during online learning sessions.This paper proposes a deep learning-based approach using facial emotions to detect the real-time engagement of online learners. This is done by analysing the students' facial expressions to classify their emotions throughout the online learning session. The facial emotion recognition information is used to calculate the engagement index (EI) to predict two engagement states "Engaged" and "Disengaged". Different deep learning models such as Inception-V3, VGG19 and ResNet-50 are evaluated and compared to get the best predictive classification model for real-time engagement detection. Varied benchmarked datasets such as FER-2013, CK+ and RAF-DB are used to gauge the overall performance and accuracy of the proposed system. Experimental results showed that the proposed system achieves an accuracy of 89.11%, 90.14% and 92.32% for Inception-V3, VGG19 and ResNet-50, respectively, on benchmarked datasets and our own created dataset. ResNet-50 outperforms all others with an accuracy of 92.3% for facial emotions classification in real-time learning scenarios. [ABSTRACT FROM AUTHOR]

Due to the significance of human behavioural intelligence in computing devices, this paper presents the Panjab University Multilingual Audio and Video Facial Expression (PUMAVE) dataset for multimodal emotion recognition in the Indian context. It discusses the characteristics and distinguishing features of the PUMAVE dataset compared with existing datasets. It described the detail procedure followed in the construction of PUMAVE dataset. The validation and reliability analysis are performed on the recorded speech and facial expressions of subjects in six emotions. The effectiveness of the recorded emotion clips is analysed against the responses of annotators. During and after responses, inter-rater responses, and intra-rater responses are analysed to measure the quality of subjective assessment, homogeneity, and consistency in the annotator responses. The PUMAVE dataset is the first multilingual tone-sensitive audio-video facial expression dataset which provides emotional labelled clips of subjects who speak the native language (Punjabi and Hindi) and foreign language (English). [ABSTRACT FROM AUTHOR]

Facial expressions are extremely important in the social interaction as they can display the internal emotions and intentions of an individual. Accurately classifying the facial expressions into various categories is the main task in Automatic Facial Expression Recognition (AFER) systems. The existing local based techniques, at times suffer and generate same feature values for different image portions such as edge, corner and flat regions. To address this issue, Radial Mesh Pattern (RMP), a local texture based approach based on the chess game rules is proposed. With reference to the center pixel in a 5 × 5 neighborhood, the possible positions of Rook, Bishop and Knight are determined and based on these positions, the features are extracted. In this paper, not only binary weights, but also other weights such as fibonacci, prime, natural, squares, odd and even weights have been utilized for feature extraction. To validate the efficiency of the proposed method, RMP is implemented on six 'in the lab' datasets. The performance is measured through recognition accuracy and the results obtained from experiments demonstrate the efficiency of RMP over standard existing methods. [ABSTRACT FROM AUTHOR]

Infinite Gaussian mixture process is a model that computes the Gaussian mixture parameters with order. This process is a probability density distribution with adequate training data that can converge to the input density curve. In this paper, we propose a data mining model namely Beta hierarchical distribution that can solve axial data modeling. A novel hierarchical Two-Hyper-Parameter Poisson stochastic process is developed to solve grouped data modelling. The solution uses data mining techniques to link datum in groups by linking their components. The learning techniques are novel presentations of Gaussian modelling that use prior knowledge of the representation hyper-parameters and approximate them in a closed form. Experiments are performed on axial data modeling of Arabic Script classification and depict the effectiveness of the proposed method using a hand written benchmark dataset which contains complex handwritten Arabic patterns. Experiments are also performed on the application of facial expression recognition and prove the accuracy of the proposed method using a benchmark dataset which contains eight different facial expressions. [ABSTRACT FROM AUTHOR]

Copyright of Revista de Comunicaciôn y Salud is the property of Instituto Internacional de Comunicacion y Salud and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Goal: As an essential human-machine interactive task, emotion recognition has become an emerging area over the decades. Although previous attempts to classify emotions have achieved high performance, several challenges remain open: 1) How to effectively recognize emotions using different modalities remains challenging. 2) Due to the increasing amount of computing power required for deep learning, how to provide real-time detection and improve the robustness of deep neural networks is important. Method: In this paper, we propose a deep learning-based multimodal emotion recognition (MER) called Deep-Emotion, which can adaptively integrate the most discriminating features from facial expressions, speech, and electroencephalogram (EEG) to improve the performance of the MER. Specifically, the proposed Deep-Emotion framework consists of three branches, i.e., the facial branch, speech branch, and EEG branch. Correspondingly, the facial branch uses the improved GhostNet neural network proposed in this paper for feature extraction, which effectively alleviates the overfitting phenomenon in the training process and improves the classification accuracy compared with the original GhostNet network. For work on the speech branch, this paper proposes a lightweight fully convolutional neural network (LFCNN) for the efficient extraction of speech emotion features. Regarding the study of EEG branches, we proposed a tree-like LSTM (tLSTM) model capable of fusing multi-stage features for EEG emotion feature extraction. Finally, we adopted the strategy of decision-level fusion to integrate the recognition results of the above three modes, resulting in more comprehensive and accurate performance. Result and Conclusions: Extensive experiments on the CK+, EMO-DB, and MAHNOB-HCI datasets have demonstrated the advanced nature of the Deep-Emotion method proposed in this paper, as well as the feasibility and superiority of the MER approach.

When driving, the most crucial factor to consider is you and your passengers' safety. Drivers must be kept under observation for any potential harmful act, whether intentional or inadvertent, in order to ensure a safe navigation. As a result, a real-time emotion detection system for a driver has been developed to detect, exploit, and evaluate the driver's emotional state. This paper discusses how to recognize emotions using facial expressions for application in active security systems for drivers. We discuss our research and development of a Convolutional Neural Network-based intelligent system for face image-based expression classification in this paper. [ABSTRACT FROM AUTHOR]

Video facial expression recognition is useful for many applications and received much interest lately. Although some methods give good results in controlled environments (no occlusion), recognition in the presence of partial facial occlusion remains a challenging task. To handle facial occlusions, methods based on the reconstruction of the occluded part of the face have been proposed. These methods are mainly based on the texture or the geometry of the face. However, the similarity of the face movement between different persons doing the same expression seems to be a real asset for the reconstruction. In this paper we exploit this asset and propose a new method based on an auto-encoder with skip connections to reconstruct the occluded part of the face in the optical flow domain. To the best of our knowledge, this is the first work that directly reconstructs the movement for facial expression recognition. We validated our approach in the controlled CK+ datasets on which different occlusions were generated. Our experiments show that the proposed method reduces the gap in the recognition accuracy between occluded and unoccluded situations. We also compare our approach with existing state-of-the-art approaches. In order to lay the basis of a reproducible and fair comparison in the future, we also propose a new experimental protocol that includes occlusion generation and reconstruction evaluation. [ABSTRACT FROM AUTHOR]

Pain is a reliable indicator of health issues; it affects patients' quality of life when not well managed. The current methods in the clinical application undergo biases and errors; moreover, such methods do not facilitate continuous pain monitoring. For this purpose, the recent methodologies in automatic pain assessment were introduced, which demonstrated the possibility for objectively and robustly measuring and monitoring pain when using behavioral cues and physiological signals. This paper focuses on introducing a reliable automatic system for continuous monitoring of pain intensity by analyzing behavioral cues, such as facial expressions and audio, and physiological signals, such as electrocardiogram (ECG), electromyogram (EMG), and electrodermal activity (EDA) from the X-ITE Pain Dataset. Several experiments were conducted with 11 datasets regarding classification and regression; these datasets were obtained from the database to reduce the impact of the imbalanced database problem. With each single modality (Uni-modality) experiment, we used a Random Forest [RF] baseline method, a Long Short-Term Memory (LSTM) method, and a LSTM using a sample weighting method (called LSTM-SW). Further, LSTM and LSTM-SW were used with fused modalities (two modalities = Bi-modality and all modalities = Multi-modality) experiments. Sample weighting was used to downweight misclassified samples during training to improve the performance. The experiments' results confirmed that regression is better than classification with imbalanced datasets, EDA is the best single modality, and fused modalities improved the performance significantly over the single modality in 10 out of 11 datasets. [ABSTRACT FROM AUTHOR]

In e-learning environments, several activities are offered to learners, including learning and assessment activities. During these activities, the learner may encounter difficulties, such as blocking situations or lack of motivation. This paper presents a new approach to detect these difficulties based on the learner's emotional states and recommends solutions to motivate him/her. Our first contribution is recognizing the learner's emotional state using two modules: facial expression recognition using a Gabor filter bank and vocal emotion recognition using MFCC features. Concerning the recommendations in these cases of difficulties, a psychological and/or pedagogical recommendation generator is proposed in this study; it is our second contribution. Three experiments to validate our approach were conducted on two groups of students: test and control. The results indicate that the proposed method improves the learner's emotional state, motivation, and engagement time in the system. [ABSTRACT FROM AUTHOR]

While the recognition of ambiguous emotions is crucial for successful social interactions, previous work has shown that they are perceived differently depending on whether they are viewed on male or female faces. The present paper aims to shed light on this phenomenon by exploring two hypotheses: the confounded signal hypothesis, which posits the existence of perceptual overlaps between emotions and gendered morphotypes, and the social role hypothesis, according to which the observer's responses are biased by stereotypes. Participants were asked to categorise blended faces (i.e. artificial faces made ambiguous by mixing two emotions) in a forced-choice task. Six emotions were used to create each blend (neutral, surprise, sadness, fear, happiness, anger), for a total of 15 expressions. We then applied signal detection theory – considering both the morphotype of the stimuli and the participants' gender – to distinguish participants' perceptual processes from their response biases. The results showed a perceptual advantage for anger on male faces and for sadness on female faces. However, different strategies were deployed when labelling emotions on gendered morphotypes. In particular, a response bias towards angry male faces establishes their special status, as they resulted in both excellent detection and a tendency to be over-reported, especially by women. [ABSTRACT FROM AUTHOR]

Automatic identification of human facial expressions has many potential applications in today's connected world, from mental health monitoring to feedback for onscreen content or shop windows and sign-language prosodic identification. In this work we use visual information as input, namely, a dataset of face points delivered by a Kinect device. The most recent work on facial expression recognition uses Machine Learning techniques, to use a modular data-driven path of development instead of using human-invented ad hoc rules. In this paper, we present a Machine-Learning based method for automatic facial expression recognition that leverages information fusion architecture techniques from our previous work and soft voting. Our approach shows an average prediction performance clearly above the best state-of-the-art results for the dataset considered. These results provide further evidence of the usefulness of information fusion architectures rather than adopting the default ML approach of features aggregation. [ABSTRACT FROM AUTHOR]

Teaching is an activity that requires understanding the class's reaction to evaluate the teaching methodology effectiveness. This operation can be easy to achieve in small classrooms, while it may be challenging to do in classes of 50 or more students. This paper proposes a novel Internet of Things (IoT) system to aid teachers in their work based on the redundant use of non-invasive techniques such as facial expression recognition and physiological data analysis. Facial expression recognition is performed using a Convolutional Neural Network (CNN), while physiological data are obtained via Photoplethysmography (PPG). By recurring to Russel's model, we grouped the most important Ekman's facial expressions recognized by CNN into active and passive. Then, operations such as thresholding and windowing were performed to make it possible to compare and analyze the results from both sources. Using a window size of 100 samples, both sources have detected a level of attention of about 55.5% for the in-presence lectures tests. By comparing results coming from in-presence and pre-recorded remote lectures, it is possible to note that, thanks to validation with physiological data, facial expressions alone seem useful in determining students' level of attention for in-presence lectures. [ABSTRACT FROM AUTHOR]

The main goal of the paper is to examine comprehension of basic emotion and facial expression of children with hearing impairment. The research encompasses a sample of 66 respondents out of which 33 have non-damaged hearing and the remaining 33 respondents have hearing impairment. The age of respondents ranges, in chronological order, from 7 to 15 years. Recognition of emotions and facial expressions is being examined through 4 sets of tasks. The Emotion Recognition Test (ERT) is being used for evaluation, it has been adapted for the purpose of this paper and thereby adjusted for electronic usage via computer and internet. Through statistical processing of given data, following components have been calculated: minimal and maximum values, arithmetic mean, standard deviation, Wilcoxon Signed Ranks Test, Mann-Whitney U test, curvature test and flattening of distribution curve, t-test. Research results concluded that hearing impairment in children affects, with statistical significance, recognition and comprehension of facial expressions and emotions compared to their peers with no hearing impairment. The results were expected given the hearing condition and the consequences which hearing impairment creates. Children with hearing impairment possess poor vocabulary which affects comprehension of emotions. Statistically significant difference occurs among children with hearing impairments and recognition and comprehension of facial expression and emotions within situational context and isolated facial expressions

In the process of communication between people, everyone will have emotions, and different emotions will have different effects on communication. With the help of external performance information accompanied by emotional expression, such as emotional speech signals or facial expressions, people can easily communicate with each other and understand each other. Emotion recognition is an important network of affective computers and research centers for signal processing, pattern detection, artificial intelligence, and human-computer interaction. Emotions convey important information in human communication and communication. Since the end of the last century, people have started the research on emotion recognition, especially how to correctly judge the emotion type has invested a lot of time and energy. In this paper, multi-modal emotion recognition is introduced to recognize facial expressions and speech, and conduct research on adaptive higher education management. Language and expression are the most direct ways for people to express their emotions. After obtaining the framework of the dual-modal emotion recognition system, the BOW model is used to identify the characteristic movement of local areas or key points. The recognition rates of emotion recognition for 1,000 audios of anger, disgust, fear, happiness, sadness and surprise are: 97.3, 83.75, 64.87, 89.87, 84.12, and 86.68%, respectively. [ABSTRACT FROM AUTHOR]

Visible face recognition systems are subjected to failure when recognizing the faces in unconstrained scenarios. So, recognizing faces under variable and low illumination conditions are more important since most of the security breaches happen during night time. Near Infrared (NIR) spectrum enables to acquire high quality images, even without any external source of light and hence it is a good method for solving the problem of illumination. Further, the soft biometric trait, gender classification and non verbal communication, facial expression recognition has also been addressed in the NIR spectrum. In this paper, a method has been proposed to recognize the face along with gender classification and facial expression recognition in NIR spectrum. The proposed method is based on transfer learning and it consists of three core components, i) training with small scale NIR images ii) matching NIR-NIR images (homogeneous) and iii) classification. Training on NIR images produce features using transfer learning which has been pre-trained on large scale VIS face images. Next, matching is performed between NIR-NIR spectrum of both training and testing faces. Then it is classified using three, separate SVM classifiers, one for face recognition, the second one for gender classification and the third one for facial expression recognition. It has been observed that the method gives state-of-the-art accuracy on the publicly available, challenging, benchmark datasets CASIA NIR-VIS 2.0, Oulu-CASIA NIR-VIS, PolyU, CBSR, IIT Kh and HITSZ for face recognition. Further, for gender classification the Oulu-CASIA NIR-VIS, PolyU,and IIT Kh has been analyzed and for facial expression the Oulu-CASIA NIR-VIS dataset has been analyzed. [ABSTRACT FROM AUTHOR]

The optimization of the problems significantly improves the solution of the complex problems. The reduction in the feature dimensionality is enormously salient to reduce the redundant features and improve the system accuracy. In this paper, an amalgamation of different concepts is proposed to optimize the features and improve the system classification. The experiment is performed on the facial expression detection application by proposing the amalgamation of deep neural network models with the variants of the gravitational search algorithm. Facial expressions are the movement of the facial components such as lips, nose, eyes that are considered as the features to classify human emotions into different classes. The initial feature extraction is performed with the local binary pattern. The extracted feature set is optimized with the variants of gravitational search algorithm (GSA) as standard gravitational search algorithm (SGSA), binary gravitational search algorithm (BGSA) and fast discrete gravitational search algorithm (FDGSA). The deep neural network models of deep convolutional neural network (DCNN) and extended deep convolutional neural network (EDCNN) are employed for the classification of emotions from imagery datasets of JAFFE and KDEF. The fixed pose images of both the datasets are acquired and comparison based on average recognition accuracy is performed. The comparative analysis of the mentioned techniques and state-of-the-art techniques illustrates the superior recognition accuracy of the FDGSA with the EDCNN technique. [ABSTRACT FROM AUTHOR]

This paper presents the FACSHuman software program, a tool for creating facial expression materials (pictures and videos) based on the Facial Action Coding System (FACS) developed by Ekman et al. (2002). FACSHuman allows almost all the Action Units (AUs) described in the FACS Manual to be manipulated through a three-dimensional modeling software interface. Four experiments were conducted to evaluate facial expressions of emotion generated by the software and their theoretical efficiency regarding the FACS. The first study (a categorization task of facial emotions such as happiness, anger, etc.) showed that 85% of generated pictures of emotional expressions were correctly categorized. The second study showed that only 82% of the most-used AUs were correctly matched. In the third experiment, two independent FACS coders rated 47 AUs generated by FACSHuman using the standard methodology used in this kind of task (AU identification). Results showed good-to-excellent codification rates (64% and 85%). In the last experiment, 54 combinations of AU were evaluated by the same FACS coders. Results showed good-to-excellent codification rates (68–82%). Results suggested that FACSHuman could be used as experimental material for research into nonverbal communication and emotional expression. [ABSTRACT FROM AUTHOR]

Resting b**ch face (RBF) is described as an unintentional angry facial expression that is evaluated negatively and usually attributed to women. A 2 (smiling/RBF) x 2 (female/male provider) online experiment, guided by expectancy violations theory, investigated whether U.S. adults' perceptions of a healthcare provider, medical care quality, and likelihood to make another appointment would be impacted by the provider's facial expression and sex. Results indicated that RBF was an expectancy violation resulting in decreased liking and perceptions of care quality. The female provider with RBF was evaluated more negatively than the smiling female provider and the male provider with RBF on liking, caring, medical care quality, and likelihood to make a future appointment. Additional findings are further discussed in the paper. Patients may hold biases toward providers based on their facial expressions and biological sex. [ABSTRACT FROM AUTHOR]

Emotion recognition has attracted attention in recent years. It is widely used in healthcare, teaching, human-computer interaction, and other fields. Human emotional features are often used to recognize different emotions. Currently, there is more and more research on multimodal emotion recognition based on the fusion of multiple features. This paper proposes a deep learning model for multimodal emotion recognition based on the fusion of electroencephalogram (EEG) signals and facial expressions to achieve an excellent classification effect. First, a pre-trained convolution neural network (CNN) is used to extract the facial features from the facial expressions. Next, the attention mechanism is introduced to extract more critical facial frame features. Then, we apply CNNs to extract spatial features from original EEG signals, which use a local convolution kernel and a global convolution kernel to learn the features of left and right hemispheres channels and all EEG channels. After feature-level fusion, the fusion features of the facial expression features and EEG features are fed into the classifier for emotion recognition. This paper conducted experiments on the DEAP and MAHNOB-HCI datasets to evaluate the performance of the proposed model. The accuracy of valence dimension classification is 96.63%, and arousal dimension classification is 97.15% on the DEAP dataset, while 96.69% and 96.26% on the MAHNOB-HCI dataset. The experimental results show that the proposed model can effectively recognize emotions.

Objective: Facial expressions have been served as clinical symptoms to convey mental conditions in psychiatry. This paper proposes to recognize patients with schizophrenia (SCZ) using their facial images based on deep learning algorithm, and to investigate objective differences in facial expressions between SCZ patients and healthy controls using deep learning algorithm and statistical analyses. Methods: The study consists of two parts. The first part recruited 106 SCZ patients and 101 healthy controls, and videotaped their facial expressions through a fixed experimental paradigm. The video data were randomly divided into two sets, one for training a convolutional neural network (CNN) with the classification of "healthy control" or "SCZ patient" as output and the other for evaluating the classification result of the trained CNN. In the second part, all facial images of the recruited participants were put into another CNN separately, which was priorly trained with a facial expression database and will output the most likely facial expressions of the recruited participants. Statistical analyses were performed on the obtained facial expressions to find out the objective differences in facial expressions between the two recruited groups. Results: The trained CNN achieved an overall accuracy of 95.18% for classifying "healthy control" or "SCZ patient." Statistical results on the obtained facial expressions demonstrated that the objective differences between the two recruited groups were statistically significant (p <.05). Conclusions: Facial expressions hold great promise as SCZ clues with the help of deep learning algorithm. The proposed approach would be potentially applied to mobile devices for autorecognizing SCZ in the context of clinical and daily life. [ABSTRACT FROM AUTHOR]

• Introduces the novel task of Sexual Facial Expression Recognition (SFER). • Triple-BigGAN; a conditional GAN with a generator, discriminator, and classifier. • SEA-Faces-30k; a new dataset with Erotic, Suggestive-Erotic, and Non-Erotic classes. • Evaluation using FER-2013, ExpW, MNIST, CIFAR-10, SVHN, and SEA-Faces-30k datasets. Automatic recognition of facial images showing erotic expressions can help to understand our social interaction and to detect non-appropriate images even when there is no nakedness present in them. This paper contemplates, for the first time, to exploit facial cues applied to automatic Sexual Facial Expression Recognition (SFER). With this goal, we introduce a new dataset named Sexual Expression and Activity Faces (SEA-Faces-30k) for SFER, which contains 30k manually labeled images under three categories: erotic, suggestive-erotic, and non-erotic. Deep Convolutional Neural Networks require large-scale annotated image datasets with diversity and variations to be properly trained. Unfortunately, gathering such a massive amount of data is not feasible in this area. Therefore, we present a new semi-supervised GAN framework named Triple-BigGAN, which learns a generative model and a classifier simultaneously. It learns both tasks in an end-to-end fashion while using unlabeled or partially labeled data. The Triple-BigGAN framework shows promising classification performance for the SFER task (i.e., 93.59 %) and other five benchmark datasets, i.e., FER-2013, CIFAR-10, Expression in-the-Wild (ExpW), Modified National Institute of Standards and Technology database (MNIST), and Street View House Numbers (SVHN). Next, we evaluated the quality of samples generated by Triple-BigGAN with a resolution of 256 × 256 pixels using Inception Score (IS) and Frechet Inception Distance (FID). Our approach obtained the best FID (i.e., 19.94 %) and IS (i.e., 97.98 %) scores on the SEA-Faces-30k dataset. Further, we empirically demonstrated that synthetic erotic face images generated by Triple-BigGAN could also help in improving the classification performance of deep supervised networks. [ABSTRACT FROM AUTHOR]