Search

Customer care is important for its role in relationship building. This role has traditionally been performed by human customer agents; however, the emergence of interactive generative AI (GenAI) shows potential for using AI for customer care in emotionally charged interactions. Bridging practice and the academic literatures in marketing and computer science, this article develops an AI-enabled customer care journey, from accurate emotion recognition to empathetic response, emotional management support, and, finally, the establishment of an emotional connection. Marketing requirements for each of the stages are derived from in-depth interviews with top managers and a survey of chief marketing officers. By juxtaposing these requirements against the current feeling capabilities of GenAI, the authors highlight the technological challenges engineers must tackle. The article concludes with a set of marketing tenets for implementing and researching the caring machine. These include verifying emotion recognition accuracy using marketing emotion theories through multiple emotion signals and methods, utilizing prompt engineering to enhance GenAI's emotion understanding, employing "response engineering" to personalize emotion management recommendations, and strategically deploying GenAI for emotional connection to simultaneously enhance customer emotional well-being and customer lifetime value. [ABSTRACT FROM AUTHOR]

While prior work has examined the effects of emotions on consumers' attitudes toward ads, little research has explored how the joint presence of positive and negative emotions affects ad-related outcomes and to what extent such ads are common. This research leverages recent advances in emotion recognition technology to identify emotions based on the facial expressions of actors in Super Bowl ads aired over the past five years. We examine how the joint presence of happiness—a basic positive emotion—and sadness, disgust, anger, and fear—basic negative emotions—in these ads predict USA TODAY Ad Meter scores (the premier tracker of public opinion on Super Bowl ads). Results show that the joint presence of happiness and sadness enhances ad liking, the presence of happiness and fear reduces ad liking, while the presence of happiness and anger or happiness and disgust do not affect ad liking. We discuss the theoretical and methodological implications of these findings and future research directions. [ABSTRACT FROM AUTHOR]

IntroductionAlthough frontotemporal dementia (FTD) with right anterior temporal lobe (RATL) predominance has been recognized, a uniform description of the syndrome is still missing. This multicenter study aims to establish a cohesive clinical phenotype.MethodsRetrospective clinical data from 18 centers across 12 countries yielded 360 FTD patients with predominant RATL atrophy through initial neuroimaging assessments.ResultsCommon symptoms included mental rigidity/preoccupations (78%), disinhibition/socially inappropriate behavior (74%), naming/word-finding difficulties (70%), memory deficits (67%), apathy (65%), loss of empathy (65%), and face-recognition deficits (60%). Real-life examples unveiled impairments regarding landmarks, smells, sounds, tastes, and bodily sensations (74%). Cognitive test scores indicated deficits in emotion, people, social interactions, and visual semantics however, lacked objective assessments for mental rigidity and preoccupations.DiscussionThis study cumulates the largest RATL cohort unveiling unique RATL symptoms subdued in prior diagnostic guidelines. Our novel approach, combining real-life examples with cognitive tests, offers clinicians a comprehensive toolkit for managing these patients.HighlightsThis project is the first international collaboration and largest reported cohort. Further efforts are warranted for precise nomenclature reflecting neural mechanisms. Our results will serve as a clinical guideline for early and accurate diagnoses.

BACKGROUND AND HYPOTHESIS: People with serious mental illness (SMI; psychotic and affective disorders with psychosis) are at an increased risk of suicide, yet there is limited research on the correlates of suicide in SMI. Social cognitive impairments are common among people with SMI and several studies have examined social cognition and suicidal ideation (SI) and behavior. This systematic review aims to evaluate the links between various domains of social cognition, SI, and suicidal behavior in SMI. STUDY DESIGN: Electronic databases (PubMed and PsycInfo) were searched through June 2023. Records obtained through this search (N = 618) were screened by 2 independent reviewers according to inclusion criteria. Relevant data were extracted, and study quality was assessed. STUDY RESULTS: Studies (N = 16) from 12 independent samples were included in the systematic review (N = 2631, sample sizes ranged from N = 20 to N = 593). Assessments of social cognition and SI and behavior varied widely between studies. Broadly, effects were mixed. Better emotion recognition of negative affect was linked to SI and a history of suicide attempts, though there is little consistent evidence for the relationship of emotion recognition and SI or behavior. On the other hand, better theory of mind ability was linked to SI and a history of suicide attempts. Furthermore, negative attributional bias was linked to current SI, but not a history of SI or attempt. CONCLUSIONS: This review suggests mixed associations between social cognition, SI, and behavior in SMI. Future research should evaluate additional mediators and moderators of social cognition and suicide, employing prospective designs.

The present study examines the association between people’s interoceptive representation of physical sensations and the recognition of vocal and facial expressions of emotion. We used body maps to study the granularity of the interoceptive conceptualisation of 11 positive emotions (amusement, awe, compassion, contentment, desire, love, joy, interest, pride, relief, and triumph) and a new emotion recognition test (Emotion Expression Understanding Test) to assess the ability to recognise emotions from vocal and facial behaviour. Overall, we found evidence for distinct interoceptive conceptualizations of 11 positive emotions across Asian American, European American, and Latino/a American cultures, as well as the reliable identification of emotion in facial and vocal expressions. Central to new theorising about emotion-related representation, the granularity of physical sensations did not covary with emotion recognition accuracy, suggesting that two kinds of emotion conceptualisation processes might be distinct. [ABSTRACT FROM AUTHOR]

Facial Emotion Recognition (FER) is a very challenging task due to the varying nature of facial expressions, occlusions, illumination, pose variations, cultural and gender differences, and many other aspects that cause a drastic degradation in quality of facial images. In this paper, an anti-aliased deep convolution network (AA-DCN) model has been developed and proposed to explore how anti-aliasing can increase and improve recognition fidelity of facial emotions. The AA-DCN model detects eight distinct emotions from image data. Furthermore, their features have been extracted using the proposed model and numerous classical deep learning algorithms. The proposed AA-DCN model has been applied to three different datasets to evaluate its performance: The Cohn-Kanade Extending (CK+) database has been utilized, achieving an ultimate accuracy of 99.26% in (5 min, 25 s), the Japanese female facial expressions (JAFFE) obtained 98% accuracy in (8 min, 13 s), and on one of the most challenging FER datasets; the Real-world Affective Face (RAF) dataset; reached 82%, in low training time (12 min, 2s). The experimental results demonstrate that the anti-aliased DCN model is significantly increasing emotion recognition while improving the aliasing artifacts caused by the down-sampling layers. [ABSTRACT FROM AUTHOR]

This study presents a Subject-Aware Transformer-based neural network designed for the Electroencephalogram (EEG) Emotion Recognition task (SATEER), which entails the analysis of EEG signals to classify and interpret human emotional states. SATEER processes the EEG waveforms by transforming them into Mel spectrograms, which can be seen as particular cases of images with the number of channels equal to the number of electrodes used during the recording process; this type of data can thus be processed using a Computer Vision pipeline. Distinct from preceding approaches, this model addresses the variability in individual responses to identical stimuli by incorporating a User Embedder module. This module enables the association of individual profiles with their EEGs, thereby enhancing classification accuracy. The efficacy of the model was rigorously evaluated using four publicly available datasets, demonstrating superior performance over existing methods in all conducted benchmarks. For instance, on the AMIGOS dataset (A dataset for Multimodal research of affect, personality traits, and mood on Individuals and GrOupS), SATEER’s accuracy exceeds 99.8% accuracy across all labels and showcases an improvement of 0.47% over the state of the art. Furthermore, an exhaustive ablation study underscores the pivotal role of the User Embedder module and each other component of the presented model in achieving these advancements. [ABSTRACT FROM AUTHOR]

Facial expression recognition (FER) technology has numerous applications in various fields such as health, entertainment and gaming, transportation, advertising and marketing, education, and many more. The recognition system based on seven basic expressions of emotion cannot satisfy the requirement for compound expression recognition as compound expressions have complex features due to a combination of basic emotions. Deep learning-based Recognition of compound emotions has recently emerged as a significant subject of academic research. Employing deep learning models to compound emotion recognition is facing imbalanced images in the dataset. To overcome the problem of imbalanced images in the dataset, the deep convolutional generative adversarial network (DCGAN) architecture has been utilized to balance the imbalanced dataset. The study proposes a novel approach for compound emotion recognition based on deep convolutional generative adversarial network (DCGAN) and convolutional neural network (CNN), which has been validated using the Compound Facial Expression Emotions (CFEE) and Real-world Affective Faces Database (RAFDB) datasets. Through utilizing the proposed approach, it achieved a significantly high accuracy of 72.0% for the CFEE dataset and 65.4% for RAF-DB for compound facial expressions of emotions. The AU-ROC/ AU-PR curves and ablation experiments also validated the performance of the proposed approach. The performance of the proposed model is compared with the state-of-the-art advanced models on the balanced datasets, and for both the CFEE and RAF-DB datasets, the accuracy of compound emotion recognition improved considerably. [ABSTRACT FROM AUTHOR]

Speech emotion recognition (SER) is one of the most important and active areas of. research in speech processing. Numerous approaches have been proposed to address various limitations in this field, but the sheer diversity of speech emotions, as well as their complexity, continue to make SER a tough nut to crack. This paper attempts to conduct a thorough investigation into speech emotion recognition in order to determine the most appropriate feature set and model for SER. A multi-layer perceptron (MLP) and convolutional neural network (CNN) based ensemble model for SER is proposed, which is a simple yet very powerful model for SER that can greatly improve classification accuracy. The model's performance is evaluated based on four benchmark datasets, namely RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song), EmoDB (Emotional Dat0abase), SAVEE (Surrey Audio-Visual Expressed Emotion), and TESS (Toronto Emotional Speech Set). The proposed model dominates over several baseline methods (decision tree (DT), random forest (RF), support vector machine (SVM), k-nearest neighbour (KNN), and the base learners, i.e., MLP and CNN) in terms of various performance metrics for all the datasets. Furthermore, the proposed model outperforms all previous works for RAVDESS (Acc=73.1%), SAVEE (Acc=83.8%), and TESS (Acc=99.9%) datasets. [ABSTRACT FROM AUTHOR]

The affective Brain-Computer Interface (aBCI) systems strive to enhance prediction accuracy for individual subjects by leveraging data from multiple subjects. However, significant differences in EEG (Electroencephalogram) feature patterns among subjects often hinder these systems from achieving the desired outcomes. Although studies have attempted to address this challenge using subject-specific classifier strategies, the scarcity of labeled data remains a major hurdle. In light of this, Domain Adaptation (DA) technology has gradually emerged as a prominent approach in the field of EEG-based emotion recognition, attracting widespread research interest. The crux of DA learning lies in resolving the issue of distribution mismatch between training and testing datasets, which has become a focal point of academic attention. Currently, mainstream DA methods primarily focus on mitigating domain distribution discrepancies by minimizing the Maximum Mean Discrepancy (MMD) or its variants. Nevertheless, the presence of noisy samples in datasets can lead to pronounced shifts in domain means, thereby impairing the adaptive performance of DA methods based on MMD and its variants in practical applications to some extent. Research has revealed that the traditional MMD metric can be transformed into a 1-center clustering problem, and the possibility clustering model is adept at mitigating noise interference during the data clustering process. Consequently, the conventional MMD metric can be further relaxed into a possibilistic clustering model. Therefore, we construct a distributed distance measure with Discriminative Possibilistic Clustering criterion (DPC), which aims to achieve two objectives: (1) ensuring the discriminative effectiveness of domain distribution alignment by finding a shared subspace that minimizes the overall distribution distance between domains while maximizing the semantic distribution distance according to the principle of "sames attract and opposites repel"; and (2) enhancing the robustness of distribution distance measure by introducing a fuzzy entropy regularization term. Theoretical analysis confirms that the proposed DPC is an upper bound of the existing MMD metric under certain conditions. Therefore, the MMD objective can be effectively optimized by minimizing the DPC. Finally, we propose a domain adaptation in Emotion recognition based on DPC (EDPC) that introduces a graph Laplacian matrix to preserve the geometric structural consistency between data within the source and target domains, thereby enhancing label propagation performance. Simultaneously, by maximizing the use of source domain discriminative information to minimize domain discrimination errors, the generalization performance of the DA model is further improved. Comparative experiments on several representative domain adaptation learning methods using multiple EEG datasets (i.e., SEED and SEED-IV) show that, in most cases, the proposed method exhibits better or comparable consistent generalization performance. [ABSTRACT FROM AUTHOR]

In the era of social media, the use of emojis and code-mixed language has become essential in online communication. However, selecting the appropriate emoji that matches a particular sentiment or emotion in the code-mixed text can be difficult. This paper presents a novel task of predicting multiple emojis in English-Hindi code-mixed sentences and proposes a new dataset called SENTIMOJI, which extends the SemEval 2020 Task 9 SentiMix dataset. Our approach is based on exploiting the relationship between emotion, sentiment, and emojis to build an end-to-end framework. We replace the self-attention sublayers in the transformer encoder with simple linear transformations and use the RMS-layer norm instead of the normal layer norm. Moreover, we employ Gated Linear Unit and Fully Connected layers to predict emojis and identify the emotion and sentiment of a tweet. Our experimental results on the SENTIMOJI dataset demonstrate that the proposed multi-task framework outperforms the single-task framework. We also show that emojis are strongly linked to sentiment and emotion and that identifying sentiment and emotion can aid in accurately predicting the most suitable emoji. Our work contributes to the field of natural language processing and can help in the development of more effective tools for sentiment analysis and emotion recognition in code-mixed languages. The codes and data will be available at https://www.iitp.ac.in/~ai-nlp-ml/resources.html#SENTIMOJI to facilitate research. [ABSTRACT FROM AUTHOR]

Background: The impact of social isolation on social cognition is not entirely clear. Objective: The aim of the study is to investigate the association between social isolation and social cognition. Methods: In a population-based sample of 83 individuals aged 50+ years without dementia, we assessed the relationship between social isolation (measured by the Lubben Social Network Scale - LSNS-6) and performance on emotional recognition (measured by the Emotion Recognition Task (ERT)) and on Theory of Mind (ToM) abilities (measured by the Reading the Mind in the Eyes Test (RMET)), two core aspects of social cognition. Results: No significant association was found between social isolation and ToM abilities for both the unadjusted and adjusted models. Similarly, no significant association was observed between social isolation and emotion recognition. Conclusions: Further research is needed to understand the complex correlation between social relationships and cognitive health, particularly in different cognitive domains, adopting a life course perspective. [ABSTRACT FROM AUTHOR]

Humans perceive a range of basic emotional connotations from music, such as joy, sadness, and fear, which can be decoded from structural characteristics of music, such as rhythm, harmony, and timbre. However, despite theory and evidence that music has multiple social functions, little research has examined whether music conveys emotions specifically associated with social status and social connection. This investigation aimed to determine whether the social emotions of dominance and affiliation are perceived in music and whether structural features of music predict social emotions, just as they predict basic emotions. Participants (N = 1513) listened to subsets of 750 music excerpts and provided ratings of energy arousal, tension arousal, valence, dominance, and affiliation. Ratings were modelled based on ten structural features of music. Dominance and affiliation were readily perceived in music and predicted by structural features including rhythm, harmony, dynamics, and timbre. In turn, energy arousal, tension arousal and valence were also predicted by musical structure. We discuss the results in view of current models of music and emotion and propose research to illuminate the significance of social emotions in music. [ABSTRACT FROM AUTHOR]

Human-robot touch interaction plays an essential role in emotional support and human mental health support. The role of therapeutic robots such as Paro in mental support has been widely investigated. However, the impact of humanoid robotic touch on the social regulation of users’ emotions is still unknown. Therefore, a mixed experiment was conducted with the type of touch (grip versus contact) as the between-subjects factor and the presence of touch during movie reception (with versus without touch) as the within-subjects factor. The subjective perception of emotion, ECG, and fNIRS signals were collected during the experiment. The results showed that robot touch regulates subjectively positive emotions, reduces HR, increases HRV, and helps suppress the brain activity on the right DLPFC. No main effect of touch type was found on the regulation effect of subjective emotions, autonomic responses, and central nervous responses. The study provides subjective and neurophysiological evidence for the great potential of humanoid robotics for the social regulation of emotion. [ABSTRACT FROM AUTHOR]

This paper introduces a neural network‐based model designed for classifying emotional states by leveraging multimodal physiological signals. The model utilizes data from the AMIGOS and SEED‐V databases. The AMIGOS database integrates inputs from electroencephalogram (EEG), electrocardiogram (ECG), and galvanic skin response (GSR) to analyze emotional responses, while the SEED‐V database continuously updates EEG signals. We implemented a sequential neural network architecture featuring two hidden layers, which underwent substantial hyperparameter tuning to achieve optimal performance. Our model's effectiveness was tested through binary classification tasks focusing on arousal and valence, as well as a more complex four‐class classification that delineates emotional quadrants for the emotional tags: happy, sad, neutral, and disgust. In these varied scenarios, the model consistently demonstrated accuracy levels ranging from 79% to 86% in the AMIGOS database and up to 97% in SEED‐V. A notable aspect of our approach is the model's ability to accurately recognize emotions without the need for extensive signal preprocessing, a common challenge in multimodal emotion analysis. This feature enhances the practical applicability of our model in real‐world scenarios where rapid and efficient emotion recognition is essential. [ABSTRACT FROM AUTHOR]

The advancement in brain–computer interface technology has facilitated real‐time communication between the brain and external devices, rapidly expanding invasive brain–computer interfaces. Nevertheless, little progress has been made in improving the one‐way, noninvasive brain–computer interface technology using conductive methods, necessitating enhancements to the current paste contact electrodes. In this paper, a captivating spherical electrode is fabricated that exhibits noncytotoxic properties, combining alginate hydrogel with indium tin oxide conductor components. The electrode's remarkable and consistent electrical conductivity enables it to generate stable adaptive deformations that conform to diverse scalp topographies. Most impedance values fall below 10 kΩ, indicating heightened hydrophilicity compared to traditional conductive pastes, as evidenced by contact angle measurements. An empirical electroencephalography data collection study confirms its conductivity and signal transmission reliability. The acquired electroencephalography signals are utilized for emotion recognition utilizing band energy ratio analysis and support vector machine (SVM) algorithms. The operation of this system is both straightforward and convenient, requiring minimal preparation before and after the collection of electroencephalography signals. Utilizing an extensive range of raw materials and straightforward preparatory procedures enhances the potential industrial application of hydrogel electrodes, thereby significantly contributing to advancing civilian electroencephalography equipment and investigating depression‐related pathologies. [ABSTRACT FROM AUTHOR]

• Positive emoji perception was significantly stronger in the upper visual field. • Negative emoji perception was significantly stronger in the lower visual field. • Spatial-emotional inconsistencies in emoji elicited greater LPC amplitudes. Metaphors play a crucial role in language, thinking, and communication. Emoji, as modern metaphors, carry rich meanings and significantly impact emotional expression. While spatial metaphors in emotional lexicons are confirmed, empirical evidence for spatial metaphors in emoji is scarce, and their neural mechanisms are largely unexplored. This study used a spatial Stroop task to investigate emoji spatial metaphors and their effects on emotional perception and neural mechanisms. The results showed that when the positive emoji were in the upper visual field, compared with the middle and lower visual field, the ability to perceive positive emotions was significantly enhanced; Conversely, when the negative emotion emoji was in the lower visual field, the perception of negative emotion was significantly enhanced compared to the middle visual field. EEG data indicate that inconsistencies between Emoji spatial positions and emotional valence lead to increased amplitudes of the Late Positive Component (LPC), revealing heightened neural activity. [ABSTRACT FROM AUTHOR]

One of the main challenges in the process of recognizing emotion through the voice are related to the specific characteristics of an individual's sound spectrum, such as accent and speech rhythm, as well as regionalism and wide variability of spoken phrases. Despite efforts to propose emotion recognition models, providing an increase in accuracy in classifying emotion in a specialized way is an open research question. Faced with these challenges, this work proposes DEEP (DEtection of voice Emotion in Portuguese language), an architecture for detecting voice emotion based on patterns present in the sound spectrum generated by the voice of Brazilian Portuguese speakers. DEEP recognizes each emotion by using a set of specialist Convolutional Neural Networks that receive as input the features extracted from the sound spectrum. With this, DEEP aims to specialize each emotion to increase the rate of correct answers and adapt to different tones and voice conditions that may occur in everyday life. Our results show that DEEP outperforms the emotion recognition measures of other state of art techniques for all evaluated scenarios. [ABSTRACT FROM AUTHOR]

Introduction: Facial expressions have become a common way for interaction among humans. People cannot comprehend and predict the emotions or expressions of individuals through simple vision. Thus, in psychology, detecting facial expressions or emotion analysis demands an assessment and evaluation of decisions for identifying the emotions of a person or any group during communication. With the recent evolution of technology, AI (Artificial Intelligence) has gained significant usage, wherein DL (Deep Learning) based algorithms are employed for detecting facial expressions. Methods: The study proposes a system design that detects facial expressions by extracting relevant features using a Modified ResNet model. The proposed system stacks building-blocks with residual connections and employs an advanced extraction method with quantum computing, which significantly reduces computation time compared to conventional methods. The backbone stem utilizes a quantum convolutional layer comprised of several parameterized quantum-filters. Additionally, the research integrates residual connections in the ResNet-18 model with the Modified up Sampled Bottle Neck Process (MuS-BNP), retaining computational efficacy while benefiting from residual connections. Results: The proposed model demonstrates superior performance by overcoming the issue of maximum similarity within varied facial expressions. The system's ability to accurately detect and differentiate between expressions is measured using performance metrics such as accuracy, F1-score, recall, and precision. Discussion: This performance analysis confirms the efficacy of the proposed system, highlighting the advantages of quantum computing in feature extraction and the integration of residual connections. The model achieves quantum superiority, providing faster and more accurate computations compared to existing methodologies. The results suggest that the proposed approach offers a promising solution for facial expression recognition tasks, significantly improving both speed and accuracy. [ABSTRACT FROM AUTHOR]

Background: Electroencephalogram (EEG) is widely used in emotion recognition due to its precision and reliability. However, the nonstationarity of EEG signals causes significant differences between individuals or sessions, making it challenging to construct a robust model. Recently, domain adaptation (DA) methods have shown excellent results in cross-subject EEG emotion recognition by aligning marginal distributions. Nevertheless, these methods do not consider emotion category labels, which can lead to label confusion during alignment. Our study aims to alleviate this problem by promoting conditional distribution alignment during domain adaptation to improve cross-subject and cross-session emotion recognition performance. Method: This study introduces a multi-source domain adaptation common-branch network for EEG emotion recognition and proposes a novel sample hybridization method. This method enables the introduction of target domain data information by directionally hybridizing source and target domain samples without increasing the overall sample size, thereby enhancing the effectiveness of conditional distribution alignment in domain adaptation. Cross-subject and cross-session experiments were conducted on two publicly available datasets, SEED and SEED-IV, to validate the proposed model. Result: In cross-subject emotion recognition, our method achieved an average accuracy of 90.27% on the SEED dataset, with eight out of 15 subjects attaining a recognition accuracy higher than 90%. For the SEED-IV dataset, the recognition accuracy also reached 73.21%. Additionally, in the cross-session experiment, we sequentially used two out of the three session data as source domains and the remaining session as the target domain for emotion recognition. The proposed model yielded average accuracies of 94.16 and 75.05% on the two datasets, respectively. Conclusion: Our proposed method aims to alleviate the difficulties of emotion recognition from the limited generalization ability of EEG features across subjects and sessions. Though adapting the multi-source domain adaptation and the sample hybridization method, the proposed method can effectively transfer the emotion-related knowledge of known subjects and achieve accurate emotion recognition on unlabeled subjects. [ABSTRACT FROM AUTHOR]

AbstractAs AI technology advances, important questions arise about how the computational power behind these models affects human perception of AI-generated contents. Furthermore, the significant energy consumption of resource-heavy AI models brings growing environmental concerns as the demand for computational resources continues to rise. This research examines the influence of computational power in AI-generated art on human aesthetic perception and emotional response. A group of 100 participants evaluated several artworks either produced by humans or created by an AI operating at three distinct levels of quality, corresponding to varying computational demands. The results showed that participants preferred AI-generated artworks created with greater computational effort, indicating higher likability and more positive emotional reactions. However, the improvements in perception and emotion were modest and exhibited diminishing returns as computational power increased. The study addresses ethical concerns, including the environmental impact of the large computational resources generative AI requires and its implications for sustainable AI development. [ABSTRACT FROM AUTHOR]

A disability is a significant issue that has posed and continues to pose a challenge. Disability is a basis of frustration because it can be observed as a mental, constraint, cognitive, and physical handicap that inhibits the individual’s growth and involvement. Consequently, significant effort has been put into removing these kinds of restrictions. These initiatives address the trouble that disabled people encounter. People with disabilities often need to rely on others to meet their requirements. Machine learning (ML) is excelling in producing smart cities and offering a secure environment for disabled individuals. Emotional detection is an important research domain that can expose many appreciated inputs. Emotion is expressed differently through speech and facial expressions, gestures, and written text. Emotion detection in a text document is fundamentally a content-based classification task, utilizing models from deep learning (DL), complex systems and natural language processing (NLP). This paper presents an Optimal Self-Attention DL-based Recognition of Textual Emotions (OSADL-RTE) technique for Disabled Persons. The presented OSADL-RTE technique focuses on identifying distinct types of emotions in the textual data. As a primary preprocessing step, the OSADL-RTE technique comprises different phases to transform the input in a useful way. For word embedding, the bag of words (BoWs) approach is exploited. The OSADL-RTE technique derives self-attention long short-term memory (SA-LSTM) approach to identify emotions. Lastly, the arithmetic fractals optimization algorithm (AOA) approach correctly tunes the hyperparameter selection of the SA-LSTM approach. The experimental study of the OSADL-RTE approach occurs on the emotion database. The investigational outcome of the OSADL-RTE approach portrayed a superior accuracy outcome of 99.59% over existing methods. [ABSTRACT FROM AUTHOR]

Background: Due to the high heterogeneity of schizophrenia, the factors influencing social cognitive impairment are controversial. The purpose of this study was to investigate the social cognitive dysfunction of deficit schizophrenia (DS), and to explore its clinical impact on the clinical characteristics and neurocognitive function assessment results. Methods: This study involved 100 DS patients, 100 non-deficit schizophrenia (NDS) patients, and 100 healthy controls (HC). Social cognitive functions were assessed using the Eye Complex Emotion Discrimination Task (ECEDT), Game of Dice Task (GDT), and Iowa Gambling Task (IGT), while neurocognitive functions were examined using the Clock Drawing Task (CDT), the Verbal Fluency Task (VFT), Digit Span Test (DST), Stroop Color-word Test (SCWT), and Trail Making Test (TMT). We analyzed the differences in cognitive function among the three groups of patients and the correlation between cognitive function assessment results and Positive and Negative Syndrome Scale (PANSS) scores. Results: Comparison of neurocognitive functions among the three groups through CDT, VFT, DST, SCWT, and TMT revealed that in the values of these tests in the DS group differed significantly from those of the NDS and HC groups. However, the DSB of the NDS group was lower and the TMT results were significantly higher than those of the HC group. In the DS group, ECEDT emotion recognition was positively correlated with stroop colors and stroop interference; the score of gender recognition was positively correlated with VFT, DSF, and SCWT, and TMT-B; the total time spent was positively correlated with TMT; The GDT risky option was negatively correlated with VFT, DST, stroop word, and stroop interference; the negative feedback utilization was negatively correlated with PANSS-Negative; TMT was positively correlated with VFT; IGT was positively correlated with CDT, VFT, DST, and SCWT, but it was negatively correlated with PANSS-Negative and TMT, with statistically significant. Conclusion: There are significant social cognitive impairments in the perception of social information, judgment and resolution of social problems in deficit schizophrenia, which are closely related to negative symptoms and multidimensional neurocognitive dysfunction such as attention, learning, memory, brain information processing speed, cognitive flexibility, and functional executive power. [ABSTRACT FROM AUTHOR]

Revealing human brain activity through machine learning EEG (Electroencephalogram) has become an important scheme to explore the inner emotional state of humans. Because the change of emotion state is dynamic rather than constant, it is difficult to predict the change of emotion state in the field of emotion recognition. This study proposes a label generation framework for instantaneous emotion intensity. A set of supervised labels is generated by having subjects watch videos that stimulate and capture their instantaneous emotional intensity, and combine the supervised labels with EEG features to generate three sets of semi - supervised labels to correspond to the instantaneous emotional state changes of subjects. In this study, EEG features and various machine learning methods are used to analyze the applicability of four groups of labels to emotional state changes. The support vector machine model achieves 80.02%, 54.76% and 56. 14% classification accuracy for two - class, three - class and four - class sentiment intensities on supervised label sets. The experimental results show that the supervised instantaneous emotion intensity labels are more universal for EEG data and emotional state changes across different subjects. [ABSTRACT FROM AUTHOR]

Automatic emotion recognition is a burgeoning field of research and has its roots in psychology and cognitive science. This article comprehensively reviews multimodal emotion recognition, covering various aspects such as emotion theories, discrete and dimensional models, emotional response systems, datasets, and current trends. This article reviewed 179 multimodal emotion recognition literature papers from 2017 to 2023 to reflect on the current trends in multimodal affective computing. This article covers various modalities used in emotion recognition based on the emotional response system under four categories: subjective experience comprising text and self‐report; peripheral physiology comprising electrodermal, cardiovascular, facial muscle, and respiration activity; central physiology comprising EEG, neuroimaging, and EOG; behavior comprising facial, vocal, whole‐body behavior, and observer ratings. This review summarizes the measures and behavior of each modality under various emotional states. This article provides an extensive list of multimodal datasets and their unique characteristics. The recent advances in multimodal emotion recognition are grouped based on the research focus areas such as emotion elicitation strategy, data collection and handling, the impact of culture and modality on multimodal emotion recognition systems, feature extraction, feature selection, alignment of signals across the modalities, and fusion strategies. The recent multimodal fusion strategies are detailed in this article, as extracting shared representations of different modalities, removing redundant features from different modalities, and learning critical features from each modality are crucial for multimodal emotion recognition. This article summarizes the strengths and weaknesses of multimodal emotion recognition based on the review outcome, along with challenges and future work in multimodal emotion recognition. This article aims to serve as a lucid introduction, covering all aspects of multimodal emotion recognition for novices. This article is categorized under:Fundamental Concepts of Data and Knowledge > Human Centricity and User InteractionTechnologies > Cognitive ComputingTechnologies > Artificial Intelligence [ABSTRACT FROM AUTHOR]

Objective: To define the clinical usability of an affect recognition (AR) battery—the Comprehensive Affect Testing System (CATS)—in an Italian sample of patients with amyotrophic lateral sclerosis (ALS). Methods: 96 ALS patients and 116 healthy controls underwent a neuropsychological assessment including the AR subtests of the abbreviated version of the CATS (CATS-A). CATS-A AR subtests and their global score (CATS-A AR Quotient, ARQ) were assessed for their factorial, convergent, and divergent validity. The diagnostic accuracy of each CATS-A AR measure in discriminating ALS patients with cognitive impairment from cognitively normal controls and patients was tested via receiver-operating characteristics analyses. Optimal cut-offs were identified for CATS-A AR measures yielding an acceptable AUC value (≥.70). The ability of CATS-A ARQ to discriminate between different ALS cognitive phenotypes was also tested. Gray-matter (GM) volumes of controls, ALS with normal (ALS-nARQ), and impaired ARQ score (ALS-iARQ) were compared using ANCOVA models. Results: CATS-A AR subtests and ARQ proved to have moderate-to-strong convergent and divergent validity. Almost all considered CATS-A measures reached acceptable accuracy and diagnostic power (AUC range =.79–.83). ARQ showed to be the best diagnostic measure (sensitivity =.80; specificity =.75) and discriminated between different ALS cognitive phenotypes. Compared to ALS-nARQ, ALS-iARQ patients showed reduced GM volumes in the right anterior cingulate, right middle frontal, left inferior temporal, and superior occipital regions. Conclusions: The AR subtests of the CATS-A, and in particular the CATS-A ARQ, are sound measures of AR in ALS. AR deficits may be a valid marker of frontotemporal involvement in these patients. [ABSTRACT FROM AUTHOR]

Compared with discrete emotion space, image emotion analysis based on dimensional emotion space can more accurately represent fine-grained emotion. Meanwhile, this high-precision representation of emotion requires dimensional emotion prediction methods to sense and capture emotional information in images as accurately and richly as possible. However, the existing methods mainly focus on emotion recognition by extracting the emotional regions where salient objects are located while ignoring the joint influence of objects and background on emotion. Furthermore, in the existing literature, when fusing multi-level features, no consideration has been given to the varying contributions of features from different levels to emotional analysis, which makes it difficult to distinguish valuable and useless features and cannot improve the utilization of effective features. This paper proposes an image emotion prediction network named ARMNet. In ARMNet, a unified affective region extraction method that integrates eye fixation detection and attention detection is proposed to enhance the combined influence of objects and backgrounds. Additionally, the multi-level features are fused with the consideration of their different contributions through an improved channel attention mechanism. In comparison to the existing methods, experiments conducted on the CGnA10766 dataset demonstrate that the performance of valence and arousal, as measured by Mean Squared Error (MSE), Mean Absolute Error (MAE), and Coefficient of Determination (R²), has improved by 4.74%, 3.53%, 3.62%, 1.93%, 6.29%, and 7.23%, respectively. Furthermore, the interpretability of the network is enhanced through the visualization of attention weights corresponding to emotional regions within the images. [ABSTRACT FROM AUTHOR]

The article discusses the psychosocial adjustment (PSA) in patients with Amyotrophic Lateral Sclerosis (ALS) and Huntington's disease (HD), comparing their quality of life and level of depressiveness. It also explores cognitive domains in C9orf72 mutation carriers and the impact of an online peer-support group intervention for adult children of ALS patients. Additionally, the study investigates cognitive alterations, apathy, and sexual behavior in ALS patients, highlighting sex differences in cognitive function and the importance of addressing sexual behavior and intimacy in ALS care. [Extracted from the article]

In five experiments, we investigated how Black participants perceive Duchenne and non-Duchenne smiles on Black and White targets. Results consistently demonstrated that when assessing happiness, faces with Duchenne compared to non-Duchenne smiles were rated as happier on both Black and White targets. However, when assessing a more socially evaluative dimension, trustworthiness, perceptions of Black and White targets diverged. Whereas White targets with Duchenne compared to non-Duchenne smiles were rated as more trustworthy, ratings of Black targets with Duchenne and non-Duchenne smiles did not differ, with both appraised as highly trustworthy. Although the degree to which Black participants identified with their race did not moderate these effects, the perceived genuineness of targets did mediate the relationship. One reason why Duchenne compared to non-Duchenne smiles on White but not Black targets were perceived as more trustworthy is because Duchenne compared to non-Duchenne smiles on White but not Black targets were perceived as more genuine. A final study extended these findings by exploring the impact of target race and smile type on partner choice. In accordance with the results related to trustworthiness ratings, Black participants selected White partners with Duchenne compared to non-Duchenne smiles more often but did not differentiate in their choice of Black partners with Duchenne versus non-Duchenne smiles. These findings underscore the importance of investigating not only diverse targets but also diverse perceivers. Our results suggest that Black perceivers use facial cues differently when rating the trustworthiness of Black and White targets and that these perceptions have important downstream consequences. Public Significance Statement: Given the lack of representation in the face-processing literature, researchers have recently been called upon to go beyond White participants and include other racial groups when investigating how people form impressions from facial cues. The primary goal of the present research was, therefore, to examine how Black perceivers are impacted by emotional cues in perceptions of happiness and attributions of trustworthiness to White and Black faces. Our findings indicate that Black perceivers were impacted by even subtle cues related to Duchenne and non-Duchenne smiles on both Black and White targets when assessing happiness. However, when assessing trustworthiness, these participants differed in their ratings of Duchenne and non-Duchenne smiles on White but not Black targets. Furthermore, our findings indicate that these perceptions can have important consequences for our willingness to interact with members of different racial groups. With growing racial tension and intergroup conflict, it is especially important that future research develops a deeper, more thoughtful understanding not only of White peoples' perceptions of key characteristics such as trustworthiness on White faces, as well as other racial faces, but also other racial groups' perceptions of these faces. [ABSTRACT FROM AUTHOR]

Human emotions are easy to identify through facial expressions, body movements, and gestures. Speech carries a lot of emotional cues including variations in pitch, tone, intensity, and rhythm. In recent years, the increasing demand for human–computer interaction has spurred the development of speech recognition methods. Traditional Speech emotion detection methods are less effective in recognizing emotions, considering features like pitch, intensity, and spectral characteristics. To address these issues, this paper proposed a novel method named Dual Kernel Support Vector based Levy Dung Beetle (DKSV-LDB) Algorithm to accurately identify emotions like happiness, anger, sadness, etc. from speech patterns. In this study, the model is designed by combining a Dual Kernel Support Vector Machine (SVM) method with a Dung beetle Optimization algorithm, enriched by the Levy Flight strategy. This work conducted experiments in the datasets namely the CREMA-D, TESS, and EMO-DB (German). The performance evaluation measures such as accuracy, precision, recall, F-measure, and specificity are utilized for the evaluation of the proposed DKSV-LDB method and these results are compared with existing methods. The DKSV-LDB method achieved accuracy, precision, recall, F-measure, and specificity of 98.57%, 97.91%, 97.86%, 97.84%, and 97.78%. The experimental results depict the performance of the developed DKSV-LDB technique for speech emotion identification. [ABSTRACT FROM AUTHOR]

Self-supervised learning models, such as Wav2vec 2.0, extract efficient features for speech processing applications including speech emotion recognition. In this study, we propose a Dimension Reduction Module (DRM) to apply to the output of each transformer block in the Wav2vec 2.0 model. Our DRM consists of an attentive average pooling, a linear layer with a maxout activation function, and a linear layer that reduces the number of dimensions to the number of classes. Subsequently, we propose two methods, classifier combination and feature fusion, to generate the final decision using DRM outputs. In the Classifier Combination method, the outputs of each DRM are fed to a distinct Additive Angular Margin (AAM) softmax loss function. This constructs an individual classifier for each DRM. Then, the outputs of these classifiers are combined using five different statistical methods. In the Feature Fusion method, the outputs of the DRMs are concatenated, and an attention mechanism is applied to them. Then, the attended outputs are fed to an AAM-Softmax loss function, which is used for training all DRMs in addition to the attention mechanism. The proposed models have been evaluated on the EMODB, IEMOCAP, and ShEMO datasets. Our best method, the Attention-based Feature Fusion, has obtained unweighted accuracies of 94.80% on EMODB, 74.00% on IEMOCAP, and 80.60% on ShEMO, which are competitive with the best baseline methods. [ABSTRACT FROM AUTHOR]

Data from multi-modal sensors, such as Red-Green-Blue (RGB) cameras, thermal cameras, microphones, and mmWave radars, have gradually been adopted in various classification problems for better accuracy. Some sensors, like RGB cameras and microphones, however, capture privacy-invasive data, which are less likely to be used in centralized learning. Although the Federated Learning (FL) paradigm frees clients from sharing their sensor data, doing so results in reduced classification accuracy and increased training time. In this article, we introduce a novel Heterogeneous Privacy Federated Learning (HPFL) paradigm to better capitalize on the less privacy-invasive sensor data, such as thermal images and mmWave point clouds, by uploading them to the server for closing the performance gap between FL and centralized learning. HPFL not only allows clients to keep the more privacy-invasive sensor data private, such as RGB images and human voices, but also gives each client total freedom to define the levels of their privacy concern on individual sensor modalities. For example, more sensitive users may prefer to keep their thermal images private, while others do not mind sharing these images. We carry out extensive experiments to evaluate the HPFL paradigm using two representative classification problems: semantic segmentation and emotion recognition. Several key findings demonstrate the merits of HPFL: (i) compared to FedAvg, it improves foreground accuracy by 18.20% in semantic segmentation and boosts the F1-score by 4.20% in emotion recognition, (ii) with heterogeneous privacy concern levels, it achieves an even larger F1-score improvement of 6.17–16.05% in emotion recognition, and (iii) it also outperforms the state-of-the-art FL approaches by 12.04–17.70% in foreground accuracy and 2.54–4.10% in F1-score. [ABSTRACT FROM AUTHOR]

Deepfake detection has become increasingly important in recent years owing to the widespread availability of deepfake generation technologies. Existing deepfake detection methods present two primary limitations; i.e., they are trained on a specific type of deepfake dataset, which renders them vulnerable to unseen deepfakes, and they regard deepfakes as a "black box" with limited explainability, making it difficult for non-AI experts to understand and trust the decisions. Hence, this article proposes a novel neurosymbolic deepfake detection framework that exploits the fact that human emotions cannot be imitated easily owing to their complex nature. We argue that deep fakes typically exhibit inter- or intra-modality inconsistencies in the emotional expressions of the person being manipulated. Thus, the proposed framework performs inter- and intra-modality reasoning on emotions extracted from audio and visual modalities using a psychological and arousal-valence model for deepfake detection. In addition to fake detection, the proposed framework provides textual explanations for its decisions. The results obtained using the Presidential Deepfakes Dataset and World Leaders Dataset of real and manipulated videos demonstrate the effectiveness of our approach in detecting deepfakes and highlight the potential of a neurosymbolic approach for expandability. [ABSTRACT FROM AUTHOR]

Automatic Speech Emotion Recognition (SER) plays a vital role in making human–computer interactions more natural and effective. A significant challenge in SER development is the limited availability of diverse emotional speech datasets, which hinders the application of advanced deep learning models. Transfer learning is a machine learning technique that helps address this issue by utilizing knowledge from pre-trained models to improve performance on a new task in a target domain, even with limited data. This study investigates the use of transfer learning from various pre-trained networks, including speaker embedding models such as d-vector, x-vector, and r-vector, and image classification models like AlexNet, GoogLeNet, SqueezeNet, ResNet-18, and ResNet-50. We also propose enhanced versions of the x-vector and r-vector models incorporating Multi-Head Attention Pooling and Angular Margin Softmax, alongside other architectural improvements. Additionally, reverberation from the Room Impulse Response datasets was added to the speech utterances to diversify and augment the available data. Notably, the enhanced r-vector model achieved classification accuracies of 74.05% Unweighted Accuracy (UA) and 73.68% Weighted Accuracy (WA) on the IEMOCAP dataset, and 80.25% UA and 79.81% WA on the CREMA-D dataset, outperforming the existing state-of-the-art methods. This study shows that using cross-domain transfer learning is beneficial for low-resource emotion recognition. The enhanced models developed in other domains (for non-emotional tasks) can further improve the accuracy of SER. [ABSTRACT FROM AUTHOR]

With the advancement of Artificial Intelligence (AI) and the Internet of Things (IoT), research in the field of emotion detection and recognition has been actively conducted worldwide in modern society. Among this research, speech emotion recognition has gained increasing importance in various areas of application such as personalized services, enhanced security, and the medical field. However, subjective emotional expressions in voice data can be perceived differently by individuals, and issues such as data imbalance and limited datasets fail to provide the diverse situations necessary for model training, thus limiting performance. To overcome these challenges, this paper proposes a novel data augmentation technique using Conditional-DCGAN, which combines CGAN and DCGAN. This study analyzes the temporal signal changes using Mel-spectrograms extracted from the Emo-DB dataset and applies a loss function calculation method borrowed from reinforcement learning to generate data that accurately reflects emotional characteristics. To validate the proposed method, experiments were conducted using a model combining CNN and Bi-LSTM. The results, including augmented data, achieved significant performance improvements, reaching WA 91.46% and UAR 91.61%, compared to using only the original data (WA 79.31%, UAR 78.16%). These results outperform similar previous studies, such as those reporting WA 84.49% and UAR 83.33%, demonstrating the positive effects of the proposed data augmentation technique. This study presents a new data augmentation method that enables effective learning even in situations with limited data, offering a progressive direction for research in speech emotion recognition. [ABSTRACT FROM AUTHOR]

In speech emotion recognition (SER), our research addresses the critical challenges of capturing and evaluating node information and their complex interrelationships within speech data. We introduce Skip Graph Convolutional and Graph Attention Network (SkipGCNGAT), an innovative model that combines the strengths of skip graph convolutional networks (SkipGCNs) and graph attention networks (GATs) to address these challenges. SkipGCN incorporates skip connections, enhancing the flow of information across the network and mitigating issues such as vanishing gradients, while also facilitating deeper representation learning. Meanwhile, the GAT in the model assigns dynamic attention weights to neighboring nodes, allowing SkipGCNGAT to focus on both the most relevant local and global interactions within the speech data. This enables the model to capture subtle and complex dependencies between speech segments, thus facilitating a more accurate interpretation of emotional content. It overcomes the limitations of previous single-layer graph models, which were unable to effectively represent these intricate relationships across time and in different speech contexts. Additionally, by introducing a pre-pooling SkipGCN combination technique, we further enhance the ability of the model to integrate multi-layer information before pooling, improving its capacity to capture both spatial and temporal features in speech. Furthermore, we rigorously evaluated SkipGCNGAT on the IEMOCAP and MSP-IMPROV datasets, two benchmark datasets in SER. The results demonstrated that SkipGCNGAT consistently achieved state-of-the-art performance. These findings highlight the effectiveness of the proposed model in accurately recognizing emotions in speech, offering valuable insights and a solid foundation for future research on capturing complex relationships within speech signals for emotion recognition. [ABSTRACT FROM AUTHOR]

Multimodal sentiment analysis enables machines to interact with people more naturally. The integration of multimodalities can enhance the machines' ability to accurately predict emotions. The main obstacle to multimodal sentiment analysis is integrating information from different modalities. Previous research has used a variety of techniques, including long short-term memory networks (LSTM) and transformers. However, traditional fusion methods cannot better utilize the information from each modality, and some intra- and inter-modal features may be overlooked due to possible differences in feature representations. Therefore, to address this problem, we use a combined transformer that can connect different modal inputs and introduce SS-Trans (Single-Stream Transformer for Multimodal Sentiment Analysis and Emotion Recognition), a single-stream transformer that fuses textual, visual, and speech modalities. The model was pre-trained on the CMU-MOSI and CMU-MOSEI datasets: multi-modal masked image language modeling (MLM) and text–image matching (TIA). Compared to other existing models, SS-Trans improves ACC-2 on these two datasets by 1.06% and 1.33%, and improves F1 values by 1.50% and 1.62%, respectively. The experimental results show that our method achieves the state-of-the-art level. In addition, ablation experiments validate the model and the pre-training task, proving the effectiveness of the proposed model. [ABSTRACT FROM AUTHOR]