Search

The spectral content of macroscopic neural activity evolves throughout development, yet how this maturation relates to underlying brain network formation and dynamics remains unknown. Here, we assess the developmental maturation of electroencephalogram spectra via Bayesian model inversion of the spectral graph model, a parsimonious whole-brain model of spatiospectral neural activity derived from linearized neural field models coupled by the structural connectome. Simulation-based inference was used to estimate age-varying spectral graph model parameter posterior distributions from electroencephalogram spectra spanning the developmental period. This model-fitting approach accurately captures observed developmental electroencephalogram spectral maturation via a neurobiologically consistent progression of key neural parameters: long-range coupling, axonal conduction speed, and excitatory:inhibitory balance. These results suggest that the spectral maturation of macroscopic neural activity observed during typical development is supported by age-dependent functional adaptations in localized neural dynamics and their long-range coupling across the macroscopic structural network.

Depression is a serious mental disorder with complex etiology, exhibiting strong heterogeneity in clinical manifestations such as various subtypes. Research on depression subtypes may deepen the understanding of the disease, contributing to the diagnosis and prognosis. While brain functional network and graph neural networks (GNNs) provide such a means, the task is still challenged by limited feature encoding from the informative fMRI data, ineffective information fusion of brain functional network, and small size of the recruited subjects. Therefore, we propose a hierarchical encoding and fusion framework of brain functions. First, we pre-train a model to extract the features from individual brain regions, which signify nodes in the brain functional network. Then, distinct graphs are constructed to link the nodes within each subject, resulting in multi-view graphs of the brain functional network. We further develop a graph fusion strategy to integrate the multi-view information, by referring to the local encoding of the nodes and their interactions across multiple graph instances. Finally, we attain the classification of depression subtypes based on the fused graph representation. The experimental results demonstrate that our method can superiorly distinguish major depression subtypes and outperform the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Cross-subject EEG emotion recognition suffers a major setback due to high inter-subject variability in emotional responses. Many prior studies have endeavored to alleviate the inter-subject discrepancies of EEG feature distributions, ignoring the variable EEG connectivity and prediction deviation caused by individual differences, which may cause poor generalization to the unseen subject. This article proposes a graph domain disentanglement network (GDDN) to generalize EEG emotion recognition across subjects in terms of EEG connectivity, representation, and prediction. More specifically, a graph domain disentanglement module is proposed to extract common-specific characteristics on both EEG graph connectivity and graph representation, enabling a more comprehensive network transferability to the unseen individual. Meanwhile, to strengthen stable emotion prediction capability, a domain-adaptive classifier aggregation module is developed to facilitate adaptive emotional prediction for the unseen individual conditioned on the domain weights of the input individuals. Finally, an auxiliary supervision module is imposed to alleviate the domain discrepancy and reduce information loss during the disentanglement learning. Extensive experiments on three public EEG emotion datasets, i.e., SEED, SEED-IV, and MPED, validate the superior generalizability of GDDN compared with the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Recent research has focused on the effectiveness of Virtual Reality (VR) in games as a more immersive method of interaction. However, there is a lack of robust analysis of the physiological effects between VR and flatscreen (FS) gaming. This paper introduces the first systematic comparison and analysis of emotional and physiological responses to commercially available games in VR and FS environments. To elicit these responses, we first selected four games through a pilot study of 6 participants to cover all four quadrants of the valence-arousal space. Using these games, we recorded the physiological activity, including Blood Volume Pulse and Electrodermal Activity, and self-reported emotions of 33 participants in a user study. Our data analysis revealed that VR gaming elicited more pronounced emotions, higher arousal, increased cognitive load and stress, and lower dominance than FS gaming. The Virtual Reality and Flat Screen (VRFS) dataset, containing over 15 hours of multimodal data comparing FS and VR gaming across different games, is also made publicly available for research purposes. Our analysis provides valuable insights for further investigations into the physiological and emotional effects of VR and FS gaming. [ABSTRACT FROM AUTHOR]

EEG emotion recognition has been hampered by the clear individual differences in the electroencephalogram (EEG). Nowadays, domain adaptation is a good way to deal with this issue because it aligns the distribution of data across subjects. However, the performance for EEG emotion recognition is limited by the existing research, which mainly focuses on the global alignment between the source domain and the target domain and ignores much fine-grained information. In this study, we propose a method called Graph-based Unsupervised Subdomain Adaptation (Gusa), which simultaneously aligns the distribution between the source and target domains in a fine-grained way from both the channel and emotion subdomains. Gusa employs three modules, such as the Node-wise Domain Constraints Module to align each EEG channel and obtain a domain-variant representation, the Class-level Distribution Constraints Module, and the Emotion-wise Domain Constraints Module, to collect more fine-grained information, create more discriminative representations for each emotion, and lessen the impact of noisy emotion labels. The studies on the SEED, SEED-IV, and MPED datasets demonstrate that Gusa significantly improves the ability of EEG to recognize emotions and can extract more granular and discriminative representations for EEG. [ABSTRACT FROM AUTHOR]

Multi-modal classifiers for emotion recognition have become prominent, as the emotional states of subjects can be more comprehensively inferred from Electroencephalogram (EEG) signals and eye movements. However, existing classifiers experience a decrease in performance due to the distribution shift when applied to new users. Unsupervised domain adaptation (UDA) emerges as a solution to address the distribution shift between subjects by learning a shared latent feature space. Nevertheless, most UDA approaches focus on a single modality, while existing multi-modal approaches do not consider that fine-grained structures should also be explicitly aligned and the learned feature space must be discriminative. In this paper, we propose Coarse and Fine-grained Distribution Alignment with Correlated and Separable Features (CFDA-CSF), which performs a coarse alignment over the global feature space, and a fine-grained alignment between modalities from each domain distribution. At the same time, the model learns intra-domain correlated features, while a separable feature space is encouraged on new subjects. We conduct an extensive experimental study across the available sessions on three public datasets for multi-modal emotion recognition: SEED, SEED-IV, and SEED-V. Our proposal effectively improves the recognition performance in every session, achieving an average accuracy of 93.05%, 85.87% and 91.20% for SEED; 85.72%, 89.60%, and 86.88% for SEED-IV; and 88.49%, 91.37% and 91.57% for SEED-V. [ABSTRACT FROM AUTHOR]

Most existing multi-modal emotion recognition studies are targeted at a classification task that aims to assign a specific emotion category to a combination of several heterogeneous input data, including multimedia signals and physiological signals. A growing number of recent psychological evidence suggests that different discrete emotions may co-exist at the same time, which promotes the development of mixed-emotion recognition to identify a mixture of basic emotions. In this work, we focus on a challenging situation where both positive and negative emotions are presented simultaneously, and propose a multi-modal mixed emotion recognition framework, namely EmotionDict. The key characteristics of our EmotionDict include the following. (1) Inspired by the psychological evidence that such a mixed state can be represented by combinations of basic emotions, we address mixed emotion recognition as a label distribution learning task. An emotion dictionary has been designed to disentangle the mixed emotion representations into a weighted sum of a set of basic emotion elements in a shared latent space and their corresponding weights. (2) We incorporate physiological and overt behavioral multi-modal signals, including electroencephalogram (EEG), peripheral physiological signals, and facial videos, which directly display the subjective emotions. These modalities have diverse characteristics given that they are related to the central or peripheral nervous system, and the motor cortex. (3) We further design auxiliary tasks to learn modality attentions for modality integration. Experiments on two datasets show that our method outperforms existing state-of-the-art approaches on mixed-emotion recognition. [ABSTRACT FROM AUTHOR]

Multi-modal emotion recognition has attracted increasing attention in human-computer interaction, as it extracts complementary information from physiological and behavioral features. Compared to single modal approaches, multi-modal fusion methods are more susceptible to uncertainty in emotion recognition, such as heterogeneity and inconsistent predictions across different modalities. Previous multi-modal approaches ignore systematic modeling of uncertainty in fusion and revelation of dynamic variations in emotion process. In this article, we propose a dynamic confidence-aware fusion network for robust recognition of heterogeneous emotion features, including electroencephalogram (EEG) and facial expression. First, we develop a self-attention based multi-channel LSTM network to preliminarily align the heterogeneous emotion features. Second, we propose a confidence regression network to estimate true class probability (TCP) on each modality, which helps explore the uncertainty at modality level. Then, different modalities are weighted fused according to above two types of uncertainty. Finally, we adopt self-paced learning (SPL) mechanism to further improve the model robustness by alleviating negative effect from the hard learning samples. The experimental results on several multi-modal emotion datasets demonstrate the proposed method outperforms the state-of-the-art methods in emotion recognition performance and explicitly reveals the dynamic variation of emotion with uncertainty estimation. [ABSTRACT FROM AUTHOR]

Emotions play an essential role in affecting the performance of cognitive abilities in continuous cognitive tasks. Most previous studies share a common issue in that the evoked emotions are simply presumed to be real emotions, without taking into account the observation that emotions may be changed when carrying out cognitive activities. This may lead to the inaccurate detection of true emotions, which further adversely affects the investigation of interactions between emotion and cognition. To address this challenging problem, the present work develops an innovative study using EEG measurement to investigate the interaction between dynamic affection and cognitive ability. In particular, a real-time emotion detection model by the use of physiological signals (i.e., EEG) is constructed, to dynamically monitor the current emotional state. Given the observed emotion, the analysis of the interaction between cognitive abilities and dynamic emotions is undertaken from the perspectives of both behavioral performance and brain mechanisms. Research outcomes indicate that emotions are not stable, and are indeed dynamically changed by cognitive performance. Meanwhile, cognitive activities also influence the brain activation pattern revealed under different emotions, which validates the necessity of introducing the dynamic emotion monitoring model. In addition, the best performance has been found when the emotional state is neutral in terms of accuracy and response time. The results of this study provide a potential basis for assessing the cognitive abilities of individuals with different emotions in a variety of applications of cognitive scenarios. [ABSTRACT FROM AUTHOR]

Emotions are important factors in decision-making. With the advent of brain-computer interface (BCI) techniques, researchers developed a strong interest in predicting decisions based on emotions, which is a challenging task. To predict decision-making performance using emotion, we have proposed the Maximizing Mutual Information between Emotion and Decision relevant features (MMI-ED) method, with three modules: (1) Temporal-spatial encoding module captures spatial correlation and temporal dependence from electroencephalogram (EEG) signals; (2) Relevant feature decomposition module extracts emotion-relevant features and decision-relevant features; (3) Relevant feature fusion module maximizes the mutual information to incorporate useful emotion-related feature information during the decision-making prediction process. To construct a dataset that uses emotions to predict decision-making performance, we designed an experiment involving emotion elicitation and decision-making tasks and collected EEG, behavioral, and subjective data. We performed a comparison of our model with several emotion recognition and motion imagery models using our dataset. The results demonstrate that our model achieved state-of-the-art performance, achieving a classification accuracy of 92.96 $\%$ % . This accuracy is 6.83 $\%$ % higher than the best-performing model. Furthermore, we conducted an ablation study to demonstrate the validity of each module and provided explanations for the brain regions associated with the relevant features. [ABSTRACT FROM AUTHOR]

Electroencephalogram (EEG) signals have been widely studied in human emotion recognition. The majority of existing EEG emotion recognition algorithms utilize dozens or hundreds of electrodes covering the whole scalp region (denoted as full-channel EEG devices in this paper). Nowadays, more and more portable and miniature EEG devices with only a few electrodes (denoted as few-channel EEG devices in this paper) are emerging. However, emotion recognition from few-channel EEG data is challenging because the device can only capture EEG signals from a portion of the brain area. Moreover, existing full-channel algorithms cannot be directly adapted to few-channel EEG signals due to the significant inter-variation between full-channel and few-channel EEG devices. To address these challenges, we propose a novel few-channel EEG emotion recognition framework from the perspective of knowledge transfer. We leverage full-channel EEG signals to provide supplementary information, available online, for few-channel signals via a transfer learning-based model CD-EmotionNet, which consists of a base emotion model for efficient emotional feature extraction and a cross-device transfer learning strategy. This strategy helps to enhance emotion recognition performance on few-channel EEG data by utilizing knowledge learned from full-channel EEG data. To evaluate our cross-device EEG emotion transfer learning framework, we construct an emotion dataset containing paired 18-channel and 5-channel EEG signals from 25 subjects, as well as 5-channel EEG signals from 13 other subjects. Extensive experiments show that our framework outperforms state-of-the-art EEG emotion recognition methods by a large margin. [ABSTRACT FROM AUTHOR]

Emotion elicitation using emotional film clips is one of the most common and ecologically valid methods in Affective Computing. However, selecting and validating appropriate materials that evoke a range of emotions is challenging. Here, we present EmoStim: A Database of Emotional Film Clips as a film library with rich and varied content. EmoStim is designed for researchers interested in studying emotions in relation to either discrete or componential models of emotion. To create the database, 139 film clips were selected from literature and then annotated by 638 participants through the CrowdFlower platform. We selected 99 film clips based on the distribution of subjective ratings that effectively distinguished between emotions defined by the discrete model. We show that the selected film clips reliably induce a range of specific emotions according to the discrete model. Further, we describe relationships between emotions, emotion organization in the componential space, and underlying dimensions representing emotional experience. The EmoStim database and participant annotations are freely available for research purposes. The database can be used to enrich our understanding of emotions further and serve as a guide to select or creating additional materials. [ABSTRACT FROM AUTHOR]

Recognizing human emotions from complex, multivariate, and non-stationary electroencephalography (EEG) time series is essential in affective brain-computer interface. However, because continuous labeling of ever-changing emotional states is not feasible in practice, existing methods can only assign a fixed label to all EEG timepoints in a continuous emotion-evoking trial, which overlooks the highly dynamic emotional states and highly non-stationary EEG signals. To solve the problems of high reliance on fixed labels and ignorance of time-changing information, in this paper we propose a time-aware sampling network (TAS-Net) using deep reinforcement learning (DRL) for unsupervised emotion recognition, which is able to detect key emotion fragments and disregard irrelevant and misleading parts. Specifically, we formulate the process of mining key emotion fragments from EEG time series as a Markov decision process and train a time-aware agent through DRL without label information. First, the time-aware agent takes deep features from a feature extractor as input and generates sample-wise importance scores reflecting the emotion-related information each sample contains. Then, based on the obtained sample-wise importance scores, our method preserves top-X continuous EEG fragments with relevant emotion and discards the rest. Finally, we treat these continuous fragments as key emotion fragments and feed them into a hypergraph decoding model for unsupervised clustering. Extensive experiments are conducted on three public datasets (SEED, DEAP, and MAHNOB-HCI) for emotion recognition using leave-one-subject-out cross-validation, and the results demonstrate the superiority of the proposed method against previous unsupervised emotion recognition methods. The proposed TAS-Net has great potential in achieving a more practical and accurate affective brain-computer interface in a dynamic and label-free circumstance. [ABSTRACT FROM AUTHOR]

Affective computing is an important subfield of artificial intelligence, and with the rapid development of brain-computer interface technology, emotion recognition based on EEG signals has received broad attention. It is still a great challenge to effectively explore the multi-dimensional information in the EEG data in spite of a large number of deep learning methods. In this article, we propose a deep learning model called Attention-based Multiple Dimensions EEG Transformer (AMDET), which can leverage the complementarity among the spectral-spatial-temporal features of EEG data by employing the multi-dimensional global attention mechanism. We first transform the original EEG data into 3D temporal-spectral-spatial representations and then the AMDET would use spectral-spatial transformer blocks to extract effective features in the EEG signal and focus on the critical time frame with the temporal attention block. We conduct extensive experiments on the DEAP, SEED, and SEED-IV datasets to evaluate the performance of AMDET and the results outperform the state-of-the-art baseline on three datasets. Accuracy rates of 97.48%, 96.85%, 97.17%, 87.32% were achieved in the DEAP-Arousal, DEAP-Valence, SEED, and SEED-IV datasets, respectively. Based on AMDET, we can achieve over 90% accuracy with only eight channels, significantly improving the possibility of practical applications. [ABSTRACT FROM AUTHOR]

The ability to perceive emotions is an important criterion for judging whether a machine is intelligent. To this end, a large number of emotion recognition algorithms have been developed especially for visual information such as video. Most previous studies are based on hand-crafted features or CNN, in which the former fails to extract expressive features and the latter still faces the undesired affective gap. This drives us to think about what if we could incorporate the human emotional perception capability into CNN. In this paper, we attempt to address this question by exploring alignment between representations of neural networks and human brain activity. In particular, we employ a visually evoked emotional brain activity dataset to conduct a jointly training strategy for CNN. In the training phase, we introduce the representation similarity analysis (RSA) to align the CNN with human brain to obtain more brain-like features. Specifically, representation similarity matrices (RSMs) of multiple convolutional layers are averaged with learnable weights and related to the RSM of human brain. In order to obtain emotion-related brain activity, we conduct voxel selection and denoising with a banded ridge model before computing the RSM. Sufficient experiments on two challenging video emotion recognition datasets and multiple popular CNN architectures suggest that human brain activity is promising to provide an inductive bias for CNN towards better performance of emotion recognition. [ABSTRACT FROM AUTHOR]

An affective brain-computer interface (aBCI) can detect affective states based on brain signals and might assist people in improving their emotion regulation abilities. However, individual differences in emotional brain patterns make cross-subject emotion identification extremely challenging. Traditional supervised single-subject classification schemes require considerable calibration samples from new individuals to train subject-dependent models. Individuals are easily fatigued with long-term EEG collection processes, which may affect performance in subsequent online experiments. In this study, we propose a real-time aBCI system using domain-fusion-based multisource style transfer mapping (DF-MS-STM) to detect positive, neutral, and negative emotional states without the need for additional training sessions. Sixteen subjects participated in our online experiments to test the performance of our aBCI system and an average online prediction accuracy of 72.17±12.25% was obtained for three-class emotion recognition tasks in the last three experimental sessions. Our proposed algorithm significantly outperformed numerous baseline methods in terms of cross-subject emotion classification. In addition, we identified distinct brain patterns in response to different emotional stimuli based on the results of event-related spectral perturbation (ERSP) analyses. These neural patterns might provide new insights for emotional brain mechanistic studies and related aBCIs. [ABSTRACT FROM AUTHOR]

Affective human–vehicle interaction of intelligent cockpits is a key factor affecting the acceptance, trust, and experience for intelligent connected vehicles. Driver emotion detection is the premise of realizing affective human–machine interaction. To achieve accurate and robust driver emotion detection, we propose a novel brain-inspired framework for on-road driver emotion detection using facial expressions. Then, we conduct driver emotion data collection in an on-road context. We develop a data annotation tool, annotate the collected data, and obtain the RoadEmo dataset, a dataset of facial expressions and road scenarios under the driver’s emotional driving. Finally, we validate the detection accuracy of the proposed framework. The experiment results show that our proposed framework achieves excellent detection performance in the on-road driver emotion detection task and outperforms existing frameworks. [ABSTRACT FROM AUTHOR]

In integrated sensing and communication (ISAC) systems, communication signals are supposed to provide high-resolution sensing services. Toward this goal, multiband sensing has recently emerged as a promising technology that jointly utilizes multiple noncontiguous communication bands to improve the sensing performance of ISAC systems. In this article, we first introduce the multiband signal model in the presence of phase-distortion factors, discussing the challenges and benefits of multiband sensing to reveal its intrinsic properties. Then, we investigate the resolution limit and discuss a specific sensing algorithm design of multiband sensing technology, which is designed to leverage multiband gains to achieve high-resolution sensing performance. Finally, a series of open research topics is discussed, which may bring new insights for future research on multiband sensing for ISAC. [ABSTRACT FROM AUTHOR]

ObjectiveGadolinium-based contrast agents (GBCAs) have been widely used to better visualize disease in brain magnetic resonance imaging (MRI). However, gadolinium deposition within the brain and body has raised safety concerns about the use of GBCAs. Therefore, the development of novel approaches that can decrease or even eliminate GBCA exposure while providing similar contrast information would be of significant use clinically.MethodsIn this work, we present a deep learning based approach for contrast-enhanced T1 synthesis on brain tumor patients. A 3D high-resolution fully convolutional network (FCN), which maintains high resolution information through processing and aggregates multi-scale information in parallel, is designed to map pre-contrast MRI sequences to contrast-enhanced MRI sequences. Specifically, three pre-contrast MRI sequences, T1, T2 and apparent diffusion coefficient map (ADC), are utilized as inputs and the post-contrast T1 sequences are utilized as target output. To alleviate the data imbalance problem between normal tissues and the tumor regions, we introduce a local loss to improve the contribution of the tumor regions, which leads to better enhancement results on tumors.ResultsExtensive quantitative and visual assessments are performed, with our proposed model achieving a PSNR of 28.24 dB in the brain and 21.2 dB in tumor regions.Conclusion and significanceOur results suggest the potential of substituting GBCAs with synthetic contrast images generated via deep learning.

Introduction: Transcranial direct current stimulation (tDCS) may effectively preserve and improve cognitive function in patients with mild cognitive impairment (MCI). Research has shown that Individual brain characteristics can influence the effects of tDCS. Computer three-dimensional brain modeling based on magnetic resonance imaging (MRI) has been suggested as an alternative for determining the most accurate tDCS electrode position based on the patients' individual brain characteristics to enhance tDCS effects. Therefore, this study aims to determine the feasibility and safety of applying tDCS treatment using optimized and personalized tDCS electrode positions in patients with Alzheimer's disease (AD)-induced MCI using computer modeling and compare the results with those of a sham group to improve cognitive function. Method: A prospective active-sham group feasibility study was set to recruit 40 participants, who will be randomized into Optimized-tDCS and Sham-tDCS groups. The parameters for tDCS will be 2 mA (disk electrodes R = 1.5 cm) for 30 min during two sets of 15 sessions (2 weeks of resting period in between), using two electrodes in pairs. Using computer modeling, the tDCS electrode positions of each participant will be personalized. Outcome measurements are going to be obtained at three points: baseline, first post-test, and second post-test. The AD assessment scale-cognitive subscale (ADAS-Cog) and the Korean version of Mini-Mental State Examination (K-MMSE), together with other secondary outcomes and safety tests will be used. Discussion: For the present study, we hypothesize that compared to a sham group, the optimized personalized tDCS application would be effective in improving the cognitive function of patients with AD-induced MCI and the participants would tolerate the tDCS intervention without any significant adverse effects. [ABSTRACT FROM AUTHOR]

EEG emotion recognition plays a significant role in various mental health services. Deep learning-based methods perform excellently, but still suffer from interpretability. Although methods such as Gradient-weighted Class Activation Mapping(Grad-CAM) can cope with the above problem, their coarse granularity cannot accurately reveal the mechanism to promote emotional intelligence. In this paper, fine-grained interpretability is proposed, called Concat-aided Grad-CAM. Specifically, the multi-level feature mapping before the fully connected layer is concatenated to obtain the gradients of the target concept so that the discriminant information can be directly located in the high-precision area. Unlike coarse-grained interpretability methods applied in EEG emotion recognition, it can accurately highlight the EEG channels related to emotion rather than an obscure area. In addition, a systematic brain functional network is proposed to reveal the relationship between those channels and to further improve emotion recognition performance. The channels with greater contributions are connected, and those connections are learned by dynamic graph convolutional networks, while the others are independent to eliminate interference. Experiments on four EEG emotion recognition datasets manifest that Concat-aided Grad-CAM can be interpreted by the fine-grained. In addition, it has been shown that the learned brain functional network can improve the performance of the baselines. Significantly, the experiment results achieve state-of-the-art performance in multiple experiments. [ABSTRACT FROM AUTHOR]

Pain is highly subjective, so it is always desirable to develop objective pain assessment methods. Brain imaging techniques, such as functional magnetic resonance imaging (fMRI), have the potential to provide a physiological and quantitative pain assessment tool. However, the ultra-high-dimensional fMRI data and the nonlinear relationship between fMRI and pain greatly degrade the efficiency of fMRI-based pain decoding models. In this article, a novel pain decoding model is proposed based on the whale optimization algorithm (WOA), reinforcement learning (RL), and self-organizing fuzzy logic (SOFL), namely RLWOA-SOFL. The new non-linear WOA method incorporates RL and repository experiences (RE), which is based on a back-propagation neural network (BPNN) to map a set of agents states to appropriate actions, to extract and select features that are highly predictive of pain. More specifically, the proposed RLWOA is self-learning and self-optimizing so it can deal with the high-dimensional and complex fMRI data. On the other hand, to establish a fMRI-based pain decoding model, a novel SOFL method is proposed as a new type of deep fuzzy rule that can learn continuously from new data and identify prototypes to construct fuzzy rules. The proposed RLWOA-SOFL model is applied to real-world pain-evoked fMRI data, and the results show that the new model can decode pain intensity more accurately and can identify pain-related fMRI patterns more reliably. Therefore, the proposed RLWOA-SOFL model has great potential to evaluate the intensity of pain perception in clinical uses. [ABSTRACT FROM AUTHOR]

Stress has been identified as one of major causes of health issues. To detect the stress levels with higher accuracy, fusion of multimodal physiological signals is a promising technique. However, there is an asynchrony between physiological signals observed from different perspectives. Exploring the temporal alignment relationship between modalities is helpful to improve the quality of multimodal fusion. This paper proposes an end-to-end multimodal stress detection model based on Bidirectional Cross- and Self-modal Attention (BCSA) mechanism. Specifically, we first construct different feature extractors based on the characteristics of Blood Volume Pulse (BVP) and Electrodermal Activity (EDA) to complete automated temporal feature extraction. Second, cross-modal attention is used to seek the alignment relationship between the two modalities and fully fuse cross-modal information. The self-modal attention is used to attenuate noise and redundant information, highlight important information and obtain salient stress representations. Finally, the stress representations of the two modalities are processed separately, and the mean square error (MSE) is used to narrow the gap between them. Experimental results on the UBFC-Phys dataset and WESAD dataset show that the proposed model can effectively improve the accuracy of stress recognition, and outperforms several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Affective brain-computer interface based on electroencephalography (EEG) is an important branch in the field of affective computing. However, the individual differences in EEG emotional data and the noisy labeling problem in the subjective feedback seriously limit the effectiveness and generalizability of existing models. To tackle these two critical issues, we propose a novel transfer learning framework with Prototypical Representation based Pairwise Learning (PR-PL). The discriminative and generalized EEG features are learned for emotion revealing across individuals and the emotion recognition task is formulated as pairwise learning for improving the model tolerance to the noisy labels. More specifically, a prototypical learning is developed to encode the inherent emotion-related semantic structure of EEG data and align the individuals’ EEG features to a shared common feature space under consideration of the feature separability of both source and target domains. Based on the aligned feature representations, pairwise learning with an adaptive pseudo labeling method is introduced to encode the proximity relationships among samples and alleviate the label noises effect on modeling. Extensive results on two benchmark databases (SEED and SEED-IV) under four different cross-validation evaluation protocols validate the model reliability and stability across subjects and sessions. Compared to the literature, the average enhancement of emotion recognition across four different evaluation protocols is 2.04% (SEED) and 2.58% (SEED-IV). [ABSTRACT FROM AUTHOR]

How can we reliably transfer affect models trained in controlled laboratory conditions (in-vitro) to uncontrolled real-world settings (in-vivo)? The information gap between in-vitro and in-vivo applications defines a core challenge of affective computing. This gap is caused by limitations related to affect sensing including intrusiveness, hardware malfunctions and availability of sensors. As a response to these limitations, we introduce the concept of privileged information for operating affect models in real-world scenarios (in the wild). Privileged information enables affect models to be trained across multiple modalities available in a lab, and ignore, without significant performance drops, those modalities that are not available when they operate in the wild. Our approach is tested in two multimodal affect databases one of which is designed for testing models of affect in the wild. By training our affect models using all modalities and then using solely raw footage frames for testing the models, we reach the performance of models that fuse all available modalities for both training and testing. The results are robust across both classification and regression affect modeling tasks which are dominant paradigms in affective computing. Our findings make a decisive step towards realizing affect interaction in the wild. [ABSTRACT FROM AUTHOR]

Learning is a key function in the brain to be able to achieve the activity patterns required to perform various activities. While specific behaviors are determined by activity in localized regions, the interconnections throughout the entire brain play a key role in enabling its ability to exhibit desired activity. To mimic this setup, this paper examines the use of reservoir computing to control a linear-threshold network brain model to a desired trajectory. We first formally design open- and closed-loop controllers that achieve reference tracking under suitable conditions on the synaptic connectivity. Given the impracticality of evaluating closed-form control signals, particularly with growing network complexity, we provide a framework where a reservoir of a larger size than the network is trained to drive the activity to the desired pattern. We illustrate the versatility of this setup in two applications: selective recruitment and inhibition of neuronal populations for goal-driven selective attention, and network intervention for the prevention of epileptic seizures.

Brain-heart interactions (BHI) are critical for generating and processing emotions, including anxiety. Understanding specific neural correlates would be instrumental for greater comprehension and potential therapeutic interventions of anxiety disorders. While prior work has implicated the pontine structure as a central processor in cardiac regulation in anxiety, the distributed nature of anxiety processing across the cortex remains elusive. To address this, we performed a whole-brain-heart analysis using the full frequency directed transfer function to study resting-state spectral differences in BHI between high and low anxiety groups undergoing fMRI scans. Our findings revealed a hemispheric asymmetry in low-frequency interplay (0.05 Hz - 0.15 Hz) characterized by ascending BHI to the left insula and descending BHI from the right insula. Furthermore, we provide evidence supporting the “pacemaker hypothesis”, highlighting the pons’ function in regulating cardiac activity. Higher frequency interplay (0.2 Hz - 0.4Hz) demonstrate a preference for ascending interactions, particularly towards ventral prefrontal cortical activity in high anxiety groups, suggesting the heart’s role in triggering a cognitive response to regulate anxiety. These findings highlight the impact of anxiety on BHI, contributing to a better understanding of its effect on the resting-state fMRI signal, with further implications for potential therapeutic interventions in treating anxiety disorders.

The high failure rate of central nervous system (CNS) drugs is partly associated with an insufficient understanding of target site exposure. Blood–brain barrier (BBB) permeability evaluation tools are needed to explore drugs' ability to access the CNS. An outstanding aspect of physiologically based pharmacokinetic (PBPK) models is the integration of knowledge on drug-specific and system-specific characteristics, allowing the identification of the relevant factors involved in target site distribution. We aimed to qualify a PBPK platform model to be used as a tool to predict CNS concentrations when significant transporter activity is absent and human data are sparse or unavailable. Data from the literature on the plasma and CNS of rats and humans regarding acetaminophen, oxycodone, lacosamide, ibuprofen, and levetiracetam were collected. Human BBB permeability values were extrapolated from rats using inter-species differences in BBB surface area. The percentage of predicted AUC and Cmax within the 1.25-fold criterion was 85% and 100% for rats and humans, respectively, with an overall GMFE of <1.25 in all cases. This work demonstrated the successful application of the PBPK platform for predicting human CNS concentrations of drugs passively crossing the BBB. Future applications include the selection of promising CNS drug candidates and the evaluation of new posologies for existing drugs. [ABSTRACT FROM AUTHOR]

Affective annotation refers to the process of labeling media content based on the emotions they evoke. Since such experiences are inherently subjective and depend on individual differences, the central challenge is associating digital content with its affective, interindividual experience. Here, we present a first-of-its-kind methodology for affective annotation directly from brain signals by monitoring the affective experience of a crowd of individuals via functional near-infrared spectroscopy (fNIRS). An experiment is reported in which fNIRS was recorded from 31 participants to develop a brain-computer interface (BCI) for affective annotation. Brain signals evoked by images were used to draw predictions about the affective dimensions that characterize the stimuli. By combining annotations, the results show that monitoring crowd responses can draw accurate affective annotations, with performance improving significantly with increases in crowd size. Our methodology demonstrates a proof-of-concept to source affective annotations from a crowd of BCI users without requiring any auxiliary mental or physical interaction. [ABSTRACT FROM AUTHOR]

Electroencephalogram (EEG) variability poses a great challenge to the affective brain-computer interface (aBCI) for practical applications. Most aBCI frameworks have been demonstrated successfully but deliberated on single-day data, which can be realistically susceptible to psychophysiological changes and further hinder the exploration of inter-individual EEG commonality. This study proposes a multiple-day scenario that learns exclusively from diverse EEG correlates of emotional responses on different days (i.e., enriched data diversity) by using a unified independent components analysis framework. Given an eight-day dataset of 10 subjects (i.e., 80 sessions), the results demonstrated that the multiple-day scenario intensified the inter-subject emotion commonality (i.e., the percentage of subjects with the same signature) to a certain extent when considering sufficient cross-day sessions, whereas the most commonly adopted single-day analysis (i.e., diversity-confined) led to session-dependent inferior outcomes. Given the best case, the emotional valence dimension was associated with relatively reproducible frontal beta, central midline gamma, and occipital beta modulations with 30%–40% subject commonality, whereas the arousal counterpart suffered more substantially from EEG variability and barely returned representative signatures. These results suggest that EEG signature representation may be substantially compromised by limited data diversity, impeding the efficacy and generalizability of the aBCI model in real-life settings. [ABSTRACT FROM AUTHOR]

The integration of machine/deep learning and sensing technologies is transforming healthcare and medical practice. However, inherent limitations in healthcare data, namely scarcity, quality, and heterogeneity, hinder the effectiveness of supervised learning techniques which are mainly based on pure statistical fitting between data and labels. In this article, we first identify the challenges present in machine learning for pervasive healthcare and we then review the current trends beyond fully supervised learning that are developed to address these three issues. Rooted in the inherent drawbacks of empirical risk minimization that underpins pure fully supervised learning, this survey summarizes seven key lines of learning strategies, to promote the generalization performance for real-world deployment. In addition, we point out several directions that are emerging and promising in this area, to develop data-efficient, scalable, and trustworthy computational models, and to leverage multi-modality and multi-source sensing informatics, for pervasive healthcare. [ABSTRACT FROM AUTHOR]

Elucidating the intricate neural mechanisms underlying brain functions requires integrative brain dynamics modeling. To facilitate this process, it is crucial to develop a general-purpose programming framework that allows users to freely define neural models across multiple scales, efficiently simulate, train, and analyze model dynamics, and conveniently incorporate new modeling approaches. In response to this need, we present BrainPy. BrainPy leverages the advanced just-in-time (JIT) compilation capabilities of JAX and XLA to provide a powerful infrastructure tailored for brain dynamics programming. It offers an integrated platform for building, simulating, training, and analyzing brain dynamics models. Models defined in BrainPy can be JIT compiled into binary instructions for various devices, including Central Processing Unit, Graphics Processing Unit, and Tensor Processing Unit, which ensures high-running performance comparable to native C or CUDA. Additionally, BrainPy features an extensible architecture that allows for easy expansion of new infrastructure, utilities, and machine-learning approaches. This flexibility enables researchers to incorporate cutting-edge techniques and adapt the framework to their specific needs.

IntroductionTranscranial direct current stimulation (tDCS) has shown positive but inconsistent results in stroke rehabilitation. This could be attributed to inter-individual variations in brain characteristics and stroke lesions, which limit the use of a single tDCS protocol for all post-stroke patients. Optimizing the electrode location in tDCS for each individual using magnetic resonance imaging (MRI) to generate three-dimensional computer models and calculate the electric field (E-field) induced by tDCS at a specific target point in the primary motor cortex may help reduce these inconsistencies. In stroke rehabilitation, locating the optimal position that generates a high E-field in a target area can influence motor recovery. Therefore, this study was designed to determine the effect of personalized tDCS electrode positions on hand-knob activation in post-stroke patients.MethodThis is a crossover study with a sample size of 50 participants, who will be randomly assigned to one of six groups and will receive one session of either optimized-active, conventional-active, or sham tDCS, with 24 h between sessions. The tDCS parameters will be 1 mA (5 × 5 cm electrodes) for 20 min. The motor-evoked potential (MEP) will be recorded before and after each session over the target area (motor cortex hand-knob) and the MEP hotspot. The MEP amplitude at the target location will be the primary outcome.DiscussionWe hypothesize that the optimized-active tDCS session would show a greater increase in MEP amplitude over the target area in patients with subacute and chronic stroke than conventional and sham tDCS sessions.Clinical trial registration:https://cris.nih.go.kr, identifier KCT0007536.

Depression is a critical problem in modern society that affects an estimated 350 million people worldwide, causing feelings of sadness and a lack of interest and pleasure. Emotional disorders are gaining interest and are closely entwined with depression, because one contributes to an understanding of the other. Despite the achievements in the two separate tasks of emotion recognition and depression detection, there has not been much prior effort to build a unified model that can connect these two tasks with different modalities, including multimedia (text, audio, and video) and unobtrusive physiological signals (e.g., electroencephalography). We propose a novel temporal convolutional transformer with knowledge embedding to address the joint task of depression detection and emotion recognition. This approach not only learns multimodal embeddings across domains via the temporal convolutional transformer but also exploits special-domain knowledge from medical knowledge graphs to improve the performance of detection and recognition. It is essential that the features learned by our method can be perceived as a priori and are suitable for increasing the performance of other related tasks. Our method illustrates the case of “two birds with one stone” in the sense that two or more tasks can be efficiently handled with our unique model, which captures effective features. Experimental results on ten real-world datasets show that the proposed approach significantly outperforms other state-of-the-art approaches. On the other hand, experiments in which our methodology is applied to other reasoning tasks show that our approach effectively supports model reasoning related to emotion and improves its performance. [ABSTRACT FROM AUTHOR]

Emotional information plays an important role in various multimedia applications. Movies, as a widely available form of multimedia content, can induce multiple positive emotions and stimulate people's pursuit of a better life. Different from negative emotions, positive emotions are highly correlated and difficult to distinguish in the emotional space. Since different positive emotions are often induced simultaneously by movies, traditional single-target or multi-class methods are not suitable for the classification of movie-induced positive emotions. In this paper, we propose TransEEG, a model for multi-label positive emotion classification from a viewer's brain activities when watching emotional movies. The key features of TransEEG include (1) explicitly modeling the spatial correlation and temporal dependencies of multi-channel EEG signals using the Transformer structure based model, which effectively addresses long-distance dependencies, (2) exploiting the label-label correlations to guide the discriminative EEG representation learning, for that we design an Inter-Emotion Mask for guiding the Multi-Head Attention to learn the inter-emotion correlations, and (3) constructing an attention score vector from the representation-label correlation matrix to refine emotion-relevant EEG features. To evaluate the ability of our model for multi-label positive emotion classification, we demonstrate our model on a state-of-the-art positive emotion database CPED. Extensive experimental results show that our proposed method achieves superior performance over the competitive approaches. [ABSTRACT FROM AUTHOR]

We investigate inferring individual preferences and the contradiction of individual preferences with group preferences through direct measurement of the brain. We report an experiment where brain activity collected from 31 participants produced in response to viewing images is associated with their self-reported preferences. First, we show that brain responses present a graded response to preferences, and that brain responses alone can be used to train classifiers that reliably estimate preferences. Second, we show that brain responses reveal additional preference information that correlates with group preference, even when participants self-reported having no such preference. Our analysis of brain responses carries significant implications for researchers in general, as it suggests an individual's explicit preferences are not always aligned with the preferences inferred from their brain responses. These findings call into question the reliability of explicit and behavioral signals. They also imply that additional, multimodal sources of information may be necessary to infer reliable preference information. [ABSTRACT FROM AUTHOR]

Automatic emotion recognition based on electroencephalogram (EEG) has attracted rapidly increasing interests. Due to large inter-subject variabilities, subject-independent emotion recognition faces great challenges. Recently, domain adaptation methods have been successfully applied in this field due to their ability to align features from different subjects. However, since EEG signals corresponding to some emotions have similar oscillation patterns, they are often confused and aligned to the wrong categories, which limits the generalization ability of the model across subjects. Besides, almost all methods only support offline applications, which require collecting a large number of samples of new subjects. To achieve online recognition, a simpler model is needed. In this paper, a novel Gated Recurrent Unit-Minimum Class Confusion (GRU-MCC) model is proposed. Specifically, a simple feature extractor based on gated recurrent unit (GRU) is firstly applied to model the spatial dependence of multiple electrodes and obtain high-level discriminative features. Then, during training, minimum class confusion (MCC) loss is introduced to reduce the confusion between the correct and ambiguous classes for the target subject and increase the transfer gains. We conduct both offline and online experiments on two public datasets: SEED and MPED. The results indicate that our method can obtain the superior performance. [ABSTRACT FROM AUTHOR]

The second most common cause of stroke, accounting for 10% of hospital admissions, is intracerebral hemorrhage (ICH), and risk factors include diabetes, smoking, and hypertension. People with intracerebral bleeding experience symptoms that are related to the functions that are managed by the affected part of the brain. Having obtained 15 computed tomography (CT) scans from five patients with ICH, we decided to use three-dimensional (3D) modeling technology to estimate the bleeding volume. CT was performed on admission to hospital, and after one week and two weeks of treatment. We segmented the brain, ventricles, and hemorrhage using semi-automatic algorithms in Slicer 3D, then improved the obtained models in Blender. Moreover, the accuracy of the models was checked by comparing corresponding CT scans with 3D brain model cross-sections. The goal of the research was to examine the possibility of using 3D modeling technology to visualize intracerebral hemorrhage and assess its treatment. [ABSTRACT FROM AUTHOR]

Understanding adolescent decision-making is significant for informing basic models of neurodevelopment as well as for the domains of public health and criminal justice. System-based theories posit that adolescent decision-making is guided by activity amongst reward and control processes. While successful at explaining behavior, system-based theories have received inconsistent support at the neural level, perhaps because of methodological limitations. Here, we used two complementary approaches to overcome said limitations and rigorously evaluate system-based models. Using decision-level modeling of fMRI data from a risk-taking task in a sample of 2000+ decisions across 51 human adolescents (25 females, mean age = 15.00 years), we find support for system-based theories of decision-making. Neural activity in lateral prefrontal cortex and a multivariate pattern of cognitive control both predicted a reduced likelihood of risk-taking, whereas increased activity in the nucleus accumbens predicted a greater likelihood of risk-taking. Interactions between decision-level brain activity and age were not observed. These results garner support for system-based accounts of adolescent decision-making behavior.SIGNIFICANCE STATEMENT:Adolescent decision-making behavior is of great import for basic science, and carries equally consequential implications for public health and criminal justice. While dominant psychological theories seeking to explain adolescent decision-making have found empirical support, their neuroscientific implementations have received inconsistent support. This may be partly due to statistical approaches employed by prior neuroimaging studies of system-based theories. We used brain modeling-an approach that predicts behavior from brain activity-of univariate and multivariate neural activity metrics to better understand how neural components of psychological systems guide decision behavior in adolescents. We found broad support for system-based theories such that neural systems involved in cognitive control predicted a reduced likelihood to make risky decisions, whereas value-based systems predicted greater risk-taking propensity.

The multiple-network poroelasticity (MPET) equations describe deformation and pressures in an elastic medium permeated by interacting fluid networks. In this paper, we (i) place these equations in the theoretical context of coupled elliptic–parabolic problems, (ii) use this context to derive residual-based a posteriori error estimates and indicators for fully discrete MPET solutions and (iii) evaluate the performance of these error estimators in adaptive algorithms for a set of test cases: ranging from synthetic scenarios to physiologically realistic simulations of brain mechanics. [ABSTRACT FROM AUTHOR]

Alzheimer’s disease (AD) is a grand challenge. It’s common, affecting about 10% of the population; it’s expensive, costing the nation nearly ${\$}$ 350 billion in 2023; and it often causes years of suffering for patients and their loved ones. Other age-related neurodegenerative conditions, such as vascular dementia and Parkinson’s disease, also continue to pose problems, both for patients and society as a whole. [ABSTRACT FROM AUTHOR]

Emotion analysis based on peripheral physiological signals has attracted increasing attention recently in affective computing. Previous works usually predict emotional states using a single emotion label for each discrete time. However, in real-world scenarios, it is not sufficient due to the fact that the real-world emotional state is usually a mixture of basic emotions. In this paper, we formulate the emotion analysis as an emotion distribution learning (EDL) problem and make two contributions. First, we establish a standardised dataset containing four negative emotions (anger, disgust, sadness, fear) and three positive emotions (tenderness, joy, amusement), which could be a useful benchmark for the EDL task. Second, we propose an emotion distribution prediction system which has the following distinct characteristics: (1) after processing raw peripheral physiological signals, we compute totally 89 representative features from four channels, i.e., GSR, SKT, ECG and HR, (2) an adaptive feature selection strategy based on recursive feature elimination (RFE) is used to select the most significant features in our EDL task, and (3) we design a dedicated EDL model based on convolution neural networks that takes information from both the feature correlation and the time domain into consideration. Experiments were conducted to validate our proposed system, and the results indicated that (1) the proposed feature selection strategy effectively selects significant features and improves algorithmic performance, and (2) the proposed EDL model can obtain good results in terms of six evaluation measures and outperform existing methods. [ABSTRACT FROM AUTHOR]

The detection of mild depression is conducive to the early intervention and treatment of depression. This study explored the fusion of electroencephalography (EEG) and pupil area signals to build an effective and convenient mild depression recognition model. We proposed Mutual Information Based Fusion Model (MIBFM), which innovatively used pupil area signals to select EEG electrodes based on mutual information. Then we extracted features from EEG and pupil area signals in different bands, and fused bimodal features using the denoising autoencoder. Experimental results showed that MIBFM could obtain the highest accuracy of 87.03%. And MIBFM exhibited better performance than other existing methods. Our findings validate the effectiveness of the use of pupil area as signals, which makes eye movement signals can be easily obtained using high resolution camera, and the EEG electrode selection scheme based on mutual information is also proved to be an applicable solution for data dimension reduction and multimodal complementary information screening. This study casts a new light for mild depression recognition using multimodal data of EEG and pupil area signals, and provides a theoretical basis for the development of portable and universal application systems. [ABSTRACT FROM AUTHOR]

Previous electroencephalogram (EEG) emotion recognition relies on single-task learning, which may lead to overfitting and learned emotion features lacking generalization. In this paper, a graph-based multi-task self-supervised learning model (GMSS) for EEG emotion recognition is proposed. GMSS has the ability to learn more general representations by integrating multiple self-supervised tasks, including spatial and frequency jigsaw puzzle tasks, and contrastive learning tasks. By learning from multiple tasks simultaneously, GMSS can find a representation that captures all of the tasks thereby decreasing the chance of overfitting on the original task, i.e., emotion recognition task. In particular, the spatial jigsaw puzzle task aims to capture the intrinsic spatial relationships of different brain regions. Considering the importance of frequency information in EEG emotional signals, the goal of the frequency jigsaw puzzle task is to explore the crucial frequency bands for EEG emotion recognition. To further regularize the learned features and encourage the network to learn inherent representations, contrastive learning task is adopted in this work by mapping the transformed data into a common feature space. The performance of the proposed GMSS is compared with several popular unsupervised and supervised methods. Experiments on SEED, SEED-IV, and MPED datasets show that the proposed model has remarkable advantages in learning more discriminative and general features for EEG emotional signals. [ABSTRACT FROM AUTHOR]

Electroencephalography (EEG)-based affective computing has a scarcity problem. As a result, it is difficult to build effective, highly accurate and stable models using machine learning algorithms, especially deep learning models. Data augmentation has recently shown performance improvements in deep learning models with increased accuracy, stability and reduced overfitting. In this paper, we propose a novel data augmentation framework, named the generative adversarial network-based self-supervised data augmentation (GANSER). As the first to combine adversarial training with self-supervised learning for EEG-based emotion recognition, the proposed framework generates high-quality and high-diversity simulated EEG samples. In particular, we utilize adversarial training to learn an EEG generator and force the generated EEG signals to approximate the distribution of real samples, ensuring the quality of the augmented samples. A transformation operation is employed to mask parts of the EEG signals and force the generator to synthesize potential EEG signals based on the unmasked parts to produce a wide variety of samples. A masking possibility during transformation is introduced as prior knowledge to generalize the classifier for the augmented sample space. Finally, numerous experiments demonstrate that our proposed method can improve emotion recognition with an increase in performance and achieve state-of-the-art results. [ABSTRACT FROM AUTHOR]

EEG signals have been reported to be informative and reliable for emotion recognition in recent years. However, the inter-subject variability of emotion-related EEG signals still poses a great challenge for the practical applications of EEG-based emotion recognition. Inspired by recent neuroscience studies on inter-subject correlation, we proposed a Contrastive Learning method for Inter-Subject Alignment (CLISA) to tackle the cross-subject emotion recognition problem. Contrastive learning was employed to minimize the inter-subject differences by maximizing the similarity in EEG signkal representations across subjects when they received the same emotional stimuli in contrast to different ones. Specifically, a convolutional neural network was applied to learn inter-subject aligned spatiotemporal representations from EEG time series in contrastive learning. The aligned representations were subsequently used to extract differential entropy features for emotion classification. CLISA achieved state-of-the-art cross-subject emotion recognition performance on our THU-EP dataset with 80 subjects and the publicly available SEED dataset with 15 subjects. It could generalize to unseen subjects or unseen emotional stimuli in testing. Furthermore, the spatiotemporal representations learned by CLISA could provide insights into the neural mechanisms of human emotion processing. [ABSTRACT FROM AUTHOR]

Emotion recognition based on electroencephalography (EEG) has attracted significant attention due to its wide range of applications, especially in Human-Computer Interaction(HCI). Previous research treats different segments of EEG signals uniformly, ignoring the fact that emotions are unstable and discrete during an extended period. In this paper, we propose a novel two-step spatial-temporal emotion recognition framework. First, considering that the human emotion has not only ”short-term continuity” but also ”long-term similarity”, we propose a hierarchical self-attention network to jointly model local and global temporal information, so as to localize most related segments and reduce the influence of noise at the temporal level. Second, in order to extract discriminative features at the spatial level to enhance the emotion recognition performance, we further employ the squeeze-and-excitation module (SE module) along with the channel correlation loss (CC-Loss) to select the most task-related channels. We also define a new task called emotion localization, which aims to localize fragments with stronger emotions. We evaluate the proposed method on the proposed emotion localization task and typical emotion recognition task with three publicly available datasets, i.e., SEED, DEAP, and MAHNOB-HCI. The experimental results demonstrate that the proposed approach outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

It was observed that accuracy of the Subject-Dependent emotion recognition model was much higher than that of the Subject-Independent model in the field of electroencephalogram (EEG) based affective computing. This phenomenon is mainly caused by the individual difference of EEG, which is the key issue to be solved for the application of emotion recognition. In this work, 14 subjects from the SEED were selected for individual difference analysis. Through individual aggregation features evaluation, sample space visualization, and correlation analysis, we proposed four quantification indicators to analyze individual difference phenomenon. Finally, we presented the Personal-Zscore (PZ) feature processing method, and it was found that the data set processed with PZ method could represent emotion better than the original data set, and the conventional model with the PZ method was more robust. The accuracies of emotion recognition models trained with PZ processing have been improved to some extent, which showed that the PZ method could effectively eliminate the individual aggregation of feature space and improve the emotional representation ability of data sets. Hence, our findings may provide a new insight into the foundation for universal implementation of EEG-based application, and the Personal-Zscore feature processing method is of great significance for the development of effective emotion recognition system. [ABSTRACT FROM AUTHOR]