Emotion classification with multi‐modal physiological signals using multi‐attention‐based neural network

Abstract The ability to effectively classify human emotion states is critically important for human‐computer or human‐robot interactions. However, emotion classification with physiological signals is still a challenging problem due to the diversity of emotion expression and the characteristic differences in different modal signals. A novel learning‐based network architecture is presented that can exploit four‐modal physiological signals, electrocardiogram, electrodermal activity, electromyography, and blood volume pulse, and make a classification of emotion states. It features two kinds of attention modules, feature‐level, and semantic‐level, which drive the network to focus on the information‐rich features by mimicking the human attention mechanism. The feature‐level attention module encodes the rich information of each physiological signal. While the semantic‐level attention module captures the semantic dependencies among modals. The performance of the designed network is evaluated with the open‐source Wearable Stress and Affect Detection dataset. The developed emotion classification system achieves an accuracy of 83.88%. Results demonstrated that the proposed network could effectively process four‐modal physiological signals and achieve high accuracy of emotion classification.