Search

Mood disorders, including unipolar depression (UD) and bipolar disorder (BD) [1] , are reported to be one of the most common mental illnesses in recent years. In diagnostic evaluation on the outpatients with mood disorder, a large portion of BD patients are initially misdiagnosed as having UD [2] . As most previous research focused on long-term monitoring of mood disorders, short-term detection which could be used in early detection and intervention is thus desirable. This work proposes an approach to short-term detection of mood disorder based on the patterns in emotion of elicited speech responses. To the best of our knowledge, there is no database for short-term detection on the discrimination between BD and UD currently. This work collected two databases containing an emotional database (MHMC-EM) collected by the Multimedia Human Machine Communication (MHMC) lab and a mood disorder database (CHI-MEI) collected by the CHI-MEI Medical Center, Taiwan. As the collected CHI-MEI mood disorder database is quite small and emotion annotation is difficult, the MHMC-EM emotional database is selected as a reference database for data adaptation. For the CHI-MEI mood disorder data collection, six eliciting emotional videos are selected and used to elicit the participants’ emotions. After watching each of the six eliciting emotional video clips, the participants answer the questions raised by the clinician. The speech responses are then used to construct the CHI-MEI mood disorder database. Hierarchical spectral clustering is used to adapt the collected MHMC-EM emotional database to fit the CHI-MEI mood disorder database for dealing with the data bias problem. The adapted MHMC-EM emotional data are then fed to a denoising autoencoder for bottleneck feature extraction. The bottleneck features are used to construct a long short term memory (LSTM)-based emotion detector for generation of emotion profiles from each speech response. The emotion profiles are then clustered into emotion codewords using the K-means algorithm. Finally, a class-specific latent affective structure model (LASM) is proposed to model the structural relationships among the emotion codewords with respect to six emotional videos for mood disorder detection. Leave-one-group-out cross validation scheme was employed for the evaluation of the proposed class-specific LASM-based approaches. Experimental results show that the proposed class-specific LASM-based method achieved an accuracy of 73.33 percent for mood disorder detection, outperforming the classifiers based on SVM and LSTM.

This article presents an approach to response selection and message-response (MR) database expansion from the unstructured data on the psychological consultation websites for a retrieval-based question answering (QA) system in a constrained domain for emotional support and comforting. First, we manually construct an initial MR database based on the articles collected from the psychological consultation websites. The Chinese Knowledge and Information Processing probabilistic context-free grammar is adopted to obtain the semantic dependency graphs (SDGs) of all the messages and responses in the initial MR database. For each sentence in the MR database, all the semantic dependencies, each composed of two words and their semantic relation, are extracted from the SDG of the sentence to form a semantic dependency set. Finally, a matrix with the element representing the correlation between the semantic dependencies of the messages and their corresponding responses is constructed as a semantic dependency pair model (SDPM) for response selection. Moreover, as the number of MR pairs in the psychological consultation websites is increasing day by day, the MR database in the QA system should be expanded to meet the needs of the users. For MR database expansion, the unstructured data from the message board are automatically collected. For the collected data, the supervised latent Dirichlet allocation is adopted for event detection and then the event-based delta Bayesian Information Criterion is used for message and response article segmentation. Each extracted message segment is then fed to the constructed retrieval-based QA system to find the best matched response segment and the matching score is also estimated to verify if the new MR pair is suitable to be included in the expanded MR database. Fivefold cross validation was employed to evaluate the performance of the proposed retrieval-based QA system over the expanded MR database based on SDPM. Compared to the vector space model-based method, the Okapi BM25 model, and the deep learning-based sequence-to-sequence with attention model, the proposed approach achieved a more favorable performance according to a statistical significance test. The retrieval accuracy based on MR expansion was also evaluated and a satisfactory result was obtained confirming the effectiveness of the expanded MR database. In addition, the user's satisfaction score of the proposed system was evaluated using the Cronbach's alpha value and the satisfaction score of the proposed SDPM was higher than those of the methods for comparison.

Speech emotion recognition is becoming increasingly important for many applications. In real-life communication, non-verbal sounds within an utterance also play an important role for people to recognize emotion. In current studies, only few emotion recognition systems considered nonverbal sounds, such as laughter, cries or other emotion interjection, which naturally exists in our daily conversation. In this work, both verbal and nonverbal sounds within an utterance were thus considered for emotion recognition of real-life conversations. Firstly, an SVM-based verbal/nonverbal sound detector was developed. A Prosodic Phrase (PPh) auto-tagger was further employed to extract the verbal/nonverbal segments. For each segment, the emotion and sound features were respectively extracted based on convolutional neural networks (CNNs) and then concatenated to form a CNN-based generic feature vector. Finally, a sequence of CNN-based feature vectors for an entire dialog turn was fed to an attentive long short-term memory (LSTM)-based sequence-to-sequence model to output an emotional sequence as recognition result. Experimental results on the recognition of seven emotional states in the NNIME (The NTHU-NTUA Chinese interactive multimodal emotion corpus) showed that the proposed method achieved a detection accuracy of 52.00% outperforming the traditional methods.

In early stages, patients with bipolar disorder are often diagnosed as having unipolar depression in mood disorder diagnosis. Because the long-term monitoring is limited by the delayed detection of mood disorder, an accurate and one-time diagnosis is desirable to avoid delay in appropriate treatment due to misdiagnosis. In this paper, an elicitation-based approach is proposed for realizing a one-time diagnosis by using responses elicited from patients by having them watch six emotion-eliciting videos. After watching each video clip, the conversations, including patient facial expressions and speech responses, between the participant and the clinician conducting the interview were recorded. Next, the hierarchical spectral clustering algorithm was employed to adapt the facial expression and speech response features by using the extended Cohn–Kanade and eNTERFACE databases. A denoizing autoencoder was further applied to extract the bottleneck features of the adapted data. Then, the facial and speech bottleneck features were input into support vector machines to obtain speech emotion profiles (EPs) and the modulation spectrum (MS) of the facial action unit sequence for each elicited response. Finally, a cell-coupled long short-term memory (LSTM) network with an $L$ -skip fusion mechanism was proposed to model the temporal information of all elicited responses and to loosely fuse the EPs and the MS for conducting mood disorder detection. The experimental results revealed that the cell-coupled LSTM with the $L$ -skip fusion mechanism has promising advantages and efficacy for mood disorder detection.

Mood disorders encompass a wide array of mood issues, including unipolar depression (UD) and bipolar disorder (BD). In diagnostic evaluation on the outpatients with mood disorder, a high percentage of BD patients are initially misdiagnosed as having UD. It is crucial to establish an accurate distinction between BD and UD to make a correct and early diagnosis, leading to improvements in treatment and course of illness. In this study, eliciting emotional videos are firstly used to elicit the patients’ emotions. After watching each video clips, their facial expressions and speech responses are collected when they are interviewing with a clinician. In mood disorder detection, the facial action unit (AU) profiles and speech emotion profiles (EPs) are obtained, respectively, by using the support vector machines (SVMs) which are built via facial features and speech features adapted from two selected databases using a denoising autoencoder-based method. Finally, a Coupled Hidden Markov Model (CHMM)-based fusion method is proposed to characterize the temporal information. The CHMM is modified to fuse the AUs and the EPs with respect to six emotional videos. Experimental results show the promising advantage and efficacy of the CHMM-based fusion approach for mood disorder detection.

A complete emotional expression typically contains a complex temporal course in a natural conversation. Related research on utterance-level, segment-level and multi-level processing lacks understanding of the underlying relation of emotional speech. In this work, a convolutional neural network (CNN) with audio word-based embedding is proposed for emotion modeling. In this study, vector quantization is first applied to convert the low level features of each speech frame into audio words using k-means algorithm. Word2vec is adopted to convert an input speech utterance into the corresponding audio word vector sequence. Finally, the audio word vector sequences of the training emotional speech data with emotion annotation are used to construct the CNN- based emotion model. The NCKU-ES database, containing seven emotion categories: happiness, boredom, anger, anxiety, sadness, surprise and disgust, was collected and five-fold cross validation was used to evaluate the performance of the proposed CNN-based method for speech emotion recognition. Experimental results show that the proposed method achieved an emotion recognition accuracy of 82.34%, improving by 8.7% compared to the Long Short Term Memory (LSTM)- based method, which faced the challenging issue of long input sequence. Comparing with raw features, the audio word-based embedding achieved an improvement of 3.4% for speech emotion recognition.

In clinical diagnosis of mood disorder, a large portion of bipolar disorder patients (BDs) are misdiagnosed as unipolar depression (UDs). Clinicians have confirmed that BDs generally show ``reduced affect’’ during clinical treatment. Thus, it is expected to build an objective and one-time diagnosis system for diagnosis assistance by using machine-learning techniques. In this study, facial expressions of BD, UD and control group (C) elicited by emotional video clips are collected for exploring temporal fluctuation characteristics of intensities of facial muscles expression among the three groups. The differences of facial expressions among mood disorders are investigated by observing macroscopic fluctuations. To deal with these problems, the corresponding methods for feature extraction and modeling are proposed. From the viewpoint of macroscopic facial expression, action unit (AU) is applied for describing the temporal transformation of muscles. Then, modulation spectrum is used for extracting short-term variation of AU. The multilayer perceptron (MLP)-based disorder prediction model is then applied to obtain the prediction results. For evaluation of the proposed method, 12 subjects for three group are included in the K-fold (K=12) cross validation experiments. The experiment results reached 61.1\% classification accuracy, and outperformed the other baseline methods.

This study proposes a long-short term memory (LSTM)-based approach to text emotion recognition based on semantic word vector and emotional word vector of the input text. For each word in an input text, the semantic word vector is extracted from the word 2vec model. Besides, each lexical word is projected to all the emotional words defined in an affective lexicon to derive an emotional word vector. An autoencoder is then adopted to obtain the bottleneck features from the emotional word vector for dimensionality reduction. The autoencoder bottleneck features are then concatenated with the features in the semantic word vector to form the final textual features for emotion recognition. Finally, given the textual feature sequence of the entire sentence, the LSTM is used for emotion recognition by modeling the contextual emotion evolution of the input text. For evaluation, the NLPCC-MHMC-TE database containing seven emotion categories: anger, boredom, disgust, anxiety, happiness, sadness, and surprise was constructed and used. Five-fold cross-validation was employed to evaluate the performance of the proposed method. Experimental results show that the proposed LSTM-based method achieved a recognition accuracy of 70.66%, improving 5.33% compared with the CNN-based method. Besides, the proposed method based on integration of the semantic word vector and emotional word vector of the input text outperformed that using the individual feature vector.

This study proposes an approach to dialog state tracking (DST) in a conversational interview coaching system. For the interview coaching task, the semantic slots, used mostly in traditional dialog systems, are difficult to define manually. This study adopts the topic profile of the response from the interviewee as the dialog state representation. In addition, as the response generally consists of several sentences, the summary vector obtained from a long short-term memory neural network (LSTM) is likely to contain noisy information from many irrelevant sentences. This study proposes a sentence attention mechanism combining the sentence attention weights from a convolutional neural tensor network (CNTN) and the topic profile by selectively focusing on significant sentences for attention-based dialog state tracking. This study collected 260 interview dialogs consisting of 3,016 dialog turns for performance evaluation. A five-fold cross validation scheme was employed and the results show that the proposed method outperformed the semantic slot-based baseline method.

Prolonging independence of seniors improves their quality of life and reduces caring costs. Monitoring health conditions and events increases the safety of the senior and is helpful for independent living. With the maturing of speech technology, it is now possible to monitor activities in daily living space via acoustic signals. The advantages include fewer disturbances and more privacy. This study aims to present an acoustic activated recognition framework to model the daily activity of seniors and provide quantitative evidence of physical functions and social interactivity for living support and the health-related quality of life assessment. Acoustic streams were recorded from designed scenarios within a living space. Fast acoustic pre-segmentation and transcription was implemented using the delta Bayesian information criterion. Hidden Markov model with a developed behavior grammar network was adopted to automatically recognize acoustic events. A Gaussian mixture model combined with multidimensional sca...

According to demographic changes, the services designed for the elderly are becoming more needed than before and increasingly important. In previous work, social media or community-based question-answer data were generally used to build the chatbot. In this study, we collected the MHMC chitchat dataset from daily conversations with the elderly. Since people are free to say anything to the system, the collected sentences are converted into patterns in the preprocessing part to cover the variability of conversational sentences. Then, an LSTM-based multi-layer embedding model is used to extract the semantic information between words and sentences in a single turn with multiple sentences when chatting with the elderly. Finally, the Euclidean distance is employed to select a proper question pattern, which is further used to select the corresponding answer to respond to the elderly. For performance evaluation, five-fold cross-validation scheme was employed for training and evaluation. Experimental results show that the proposed method achieved an accuracy of 79.96% for top-1 response selection, which outperformed the traditional Okapi model.

In the diagnosis of mental health disorder, a large portion of the Bipolar Disorder (BD) patients is likely to be misdiagnosed as Unipolar Depression (UD) on initial presentation. As speech is the most natural way to express emotion, this work focuses on tracking emotion profile of elicited speech for short-term mood disorder identification. In this work, the Deep Scattering Spectrum (DSS) and Low Level Descriptors (LLDs) of the elicited speech signals are extracted as the speech features. The hierarchical spectral clustering (HSC) algorithm is employed to adapt the emotion database to the mood disorder database to alleviate the data bias problem. The denoising autoencoder is then used to extract the bottleneck features of DSS and LLDs for better representation. Based on the bottleneck features, a long short term memory (LSTM) is applied to generate the time-varying emotion profile sequence. Finally, given the emotion profile sequence, the HMM-based identification and verification model is used to determine mood disorder. This work collected the elicited emotional speech data from 15 BDs, 15 UDs and 15 healthy controls for system training and evaluation. Five-fold cross validation was employed for evaluation. Experimental results show that the system using the bottleneck feature achieved an identification accuracy of 73.33%, improving by 8.89%, compared to that without bottleneck features. Furthermore, the system with verification mechanism, improving by 4.44%, outperformed that without verification.

In clinical diagnosis of mood disorder, depression is one of the most common psychiatric disorders. There are two major types of mood disorders: major depressive disorder (MDD) and bipolar disorder (BPD). A large portion of BPD are misdiagnosed as MDD in the diagnostic of mood disorders. Short-term detection which could be used in early detection and intervention is thus desirable. This study investigates microscopic facial expression changes for the subjects with MDD, BPD and control group (CG), when elicited by emotional video clips. This study uses eight basic orientations of motion vector (MV) to characterize the subtle changes in microscopic facial expression. Then, wavelet decomposition is applied to extract entropy and energy of different frequency bands. Next, an autoencoder neural network is adopted to extract the bottleneck features for dimensionality reduction. Finally, the long short term memory (LSTM) is employed for modeling the long-term variation among different mood disorders types. For evaluation of the proposed method, the elicited data from 36 subjects (12 for each of MDD, BPD and CG) were considered in the K-fold (K=12) cross validation experiments, and the performance for distinguishing among MDD, BPD and CG achieved 67.7% accuracy.

This study presents an approach to personality trait (PT) perception from speech signals using wavelet-based multiresolution analysis and convolutional neural networks (CNNs). In this study, first, wavelet transform is employed to decompose the speech signals into the signals at different levels of resolution. Then, the acoustic features of the speech signals at each resolution are extracted. Given the acoustic features, the CNN is adopted to generate the profiles of the Big Five Inventory-10 (BFI- 10), which provide a quantitative measure for expressing the degree of the presence or absence of a set of 10 basic BFI items. The BFI-10 profiles are further fed into five artificial neural networks (ANN), each for one of the five personality dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism for PT perception. To evaluate the performance of the proposed method, experiments were conducted over the SSPNet Speaker Personality Corpus (SPC), including 640 clips randomly extracted from the French news bulletins in the INTERSPEECH 2012 speaker trait sub-challenge. From the experimental results, an average PT perception accuracy of 71.97% was obtained, outperforming the ANN-based method and the Baseline method in the INTERSPEECH 2012 speaker trait sub-challenge.

A complete emotional expression contains a complex temporal course in a conversation. Related research on utterance and segment-level processing lacks considering subtle differences in characteristics and historical information. In this work, as Deep Scattering Spectrum (DSS) can obtain more detailed energy distributions in frequency domain than the Low Level Descriptors (LLDs), this work combines LLDs and DSS as the speech features. Autoencoder neural network is then applied to extract the bottleneck features for dimensionality reduction. Finally, the long-short term memory (LSTM) is employed to characterize temporal variation of speech emotion for emotion recognition. For evaluation, the MHMC emotion database was collected and used for performance evaluation. Experimental results show that the proposed method using the bottleneck features from the combination of the LLDs and DSS achieved an emotion recognition accuracy of 98.1%, outperforming the systems using LLDs or DSS individually.

The best way to prepare for an interview is to review the different types of possible interview questions you will be asked during an interview and practice responding to questions. An interview coaching system tries to simulate an interviewer to provide mock interview practice simulation sessions for the users. The traditional interview coaching systems provide some feedbacks, including facial preference, head nodding, response time, speaking rate, and volume, to let users know their own performance in the mock interview. But most of these systems are trained with insufficient dialog data and provide the pre-designed interview questions. In this study, we propose an approach to dialog state tracking and action selection based on deep learning methods. First, the interview corpus in this study is collected from 12 participants, and is annotated with dialog states and actions. Next, a long-short term memory and an artificial neural network are employed to predict dialog states and the Deep RL is adopted to learn the relation between dialog states and actions. Finally, the selected action is used to generate the interview question for interview practice. To evaluate the proposed method in action selection, an interview coaching system is constructed. Experimental results show the effectiveness of the proposed method for dialog state tracking and action selection.

This study presents an approach to dialog state tracking (DST) in an interview conversation by using the long short-term memory (LSTM) and artificial neural network (ANN). First, the techniques of word embedding are employed for word representation by using the word2vec model. Then, each input sentence is represented by a sentence hidden vector using the LSTM-based sentence model. The sentence hidden vectors for each sentence are fed to the LSTM-based answer model to map the interviewee's answer to an answer hidden vector. For dialog state detection, the answer hidden vector is finally used to detect the dialog state using an ANN-based dialog state detection model. To evaluate the proposed method, an interview conversation system was constructed, and an average accuracy of 89.93% was obtained for dialog state detection.

In mood disorder diagnosis, bipolar disorder (BD) patients are often misdiagnosed as unipolar depression (UD) on initial presentation. It is crucial to establish an accurate distinction between BD and UD to make a correct and early diagnosis, leading to improvements in treatment and course of illness. To deal with this misdiagnosis problem, in this study, we experimented on eliciting subjects' emotions by watching six eliciting emotional video clips. After watching each video clips, their speech responses were collected when they were interviewing with a clinician. In mood disorder detection, speech emotions play an import role to detect manic or depressive symptoms. Therefore, speech emotion profiles (EP) are obtained by using the support vector machine (SVM) which are built via speech features adapted from selected databases using a denoising autoencoder-based method. Finally, a Long Short-Term Memory (LSTM) recurrent neural network is employed to characterize the temporal information of the EPs with respect to six emotional videos. Comparative experiments clearly show the promising advantage and efficacy of the LSTM-based approach for mood disorder detection.

Mood disorders include unipolar disorder (UD) and bipolar disorder (BD). In this work, database collection of facial expressions and speech responses based on eliciting emotional videos for mood disorder detection is proposed. First, the chi-squared test was used to evaluate the agreement among manual selection of eliciting emotional videos from 11 candidate emotion videos. Six eliciting emotional videos were selected and then applied to elicit speech responses and facial expressions of the patients. In this paper, the collection procedure of the elicited data is described and the statistics of the database is provided. The collected mood database can be used in mood disorder detection and studies on emotions from elicitation.

Early detection of changes in patient's condition and making an accurate nursing diagnosis are crucial in clinical practice. Current protocol in nursing diagnosis highly depends on caregiver's experience. Demand on processing a large number of nursing documents and transferring expertise has become an important issue. This paper applies computational linguistics approach to extract knowledge and structural model of diagnosis for assisted assessment and change detection of affective expression. A corpus with peer-reviewed case reports and nursing documents were collected for analysis. The evaluation results show that narrative texts with inherent structure and emotional information can be systematically extracted.

More and more efforts have been made to assist interdependent people in aging society today, particularly to the elderly living alone at home. This paper proposes an activity recognition system based on detection of acoustic events caused by residents within the living space. In the data collection process, an activity sound recording system with auto gain controlled and linearly amplified capability is equipped in a ward-like environment. The microphones are installed along the path from bedroom to lavatory. An activity sound database containing 105 designed scenarios is simulated and transcript in terms of 21 acoustic events such as footfall, fall down, and calling for help to form the acoustic event database. 21 Hidden Markov models (HMM) corresponding to acoustic events are trained by the acoustic event database. A keyword spotting recognition technology is adopted to recognize acoustic events and its ordered sequences for modeling activities. Acoustic features are computed by 39-dimensional mel-scaled frequency cepstral coefficients (MFCCs) with their first and second derivatives. The preliminary results show the average recognition rate of acoustic events is 91% for inside testing and 72% for outside testing respectively. The proposed approach shows an encouraging potential on the demand of modeling the regular living patterns of every variety.