Search

For longitudinal behavior analysis, task type is an inevitable and important variable. In this article, we propose an event-based behavior modeling approach and employ non-invasive wearable sensing modalities (eye activity, speech and head movement) to recognize task load level under four different task load types. The novelty lies in converting physiological and behavioral signals into meaningful events and utilizing their sequence across multiple modalities to distinguish load levels and types. We evaluated this approach on head-worn sensor data from 24 participants completing four different tasks for recognizing (i) low and high load level for a given task load type, (ii) low and high load level regardless of load type, and (iii) both load level and load type. Findings show that the recognition rate is reasonable in (i), close to chance level in (ii), and well above chance level in (iii) for 8 classes using participant-dependent and -independent schemes. Further, a fusion of the proposed event-based features and conventional continuous features achieved the best or similar performance in most cases. These results suggest that task type needs to be considered when using continuous features and that the proposed event-based modeling paradigm is promising for longitudinal behavior analysis.

Currently, there is an increasing global need for COVID-19 screening to help reduce the rate of infection and at-risk patient workload at hospitals. Smartphone-based screening for COVID-19 along with other respiratory illnesses offers excellent potential due to its rapid-rollout remote platform, user convenience, symptom tracking, comparatively low cost, and prompt result processing timeframe. In particular, speech-based analysis embedded in smartphone app technology can measure physiological effects relevant to COVID-19 screening that are not yet digitally available at scale in the healthcare field. Using a selection of the Sonde Health COVID-19 2020 dataset, this study examines the speech of COVID-19-negative participants exhibiting mild and moderate COVID-19-like symptoms as well as that of COVID-19-positive participants with mild to moderate symptoms. Our study investigates the classification potential of acoustic features (e.g., glottal, prosodic, spectral) from short-duration speech segments (e.g., held vowel, pataka phrase, nasal phrase) for automatic COVID-19 classification using machine learning. Experimental results indicate that certain feature-task combinations can produce COVID-19 classification accuracy of up to 80% as compared with using the all-acoustic feature baseline (68%). Further, with brute-forced n-best feature selection and speech task fusion, automatic COVID-19 classification accuracy of upwards of 82–86% was achieved, depending on whether the COVID-19-negative participant had mild or moderate COVID-19-like symptom severity.

Eye activity-based within-task cognitive load measurement (CLM) is currently not feasible in everyday situations. One important issue to be addressed to move such CLM beyond controlled laboratory environments is determining practical methods for mitigating the pupillary light reflex (PLR) effect in CLM. In this paper, four approaches to dealing with the PLR effect within a modified verbal digit span task are investigated: ignore the PLR, exclude PLR data, compensate for PLR and use PLR features for measurement. During experimental work, cognitive load and the PLR were induced with a modified verbal digit span task and changes in brightness of a large monitor, respectively. The “exclude PLR,” “compensate for PLR,” and “use PLR features” methods were found to improve classification performance by up to 18.5% relative to the “ignore PLR” method, which yielded the worst classification accuracy of 58% using an average pupil diameter feature. Features derived from the transient properties of the PLR response associated with cognitive load were found to yield the superior classification accuracy of 70%, which is an improvement compared with previously published approaches which treated the PLR responses as interference. The findings from this paper suggest that the PLR cannot be easily ignored or normalized, and clearly demonstrate the importance of PLR-aware feature extraction for the design of future eyewear-based always-on CLM in conditions that are more realistic than a darkened, controlled laboratory.

The processing of speech as an explicit sequence of events is common in automatic speech recognition (linguistic events), but has received relatively little attention in paralinguistic speech classification despite its potential for characterizing broad acoustic event sequences. This paper proposes a framework for analyzing speech as a sequence of acoustic events, and investigates its application to depression detection. In this framework, acoustic space regions are tokenized to ‘words’ representing speech events at fixed or irregular intervals. This tokenization allows the exploitation of acoustic word features using proven natural language processing methods. A key advantage of this framework is its ability to accommodate heterogeneous event types: herein we combine acoustic words and speech landmarks, which are articulation-related speech events. Another advantage is the option to fuse such heterogeneous events at various levels, including the embedding level. Evaluation of the proposed framework on both controlled laboratory-grade supervised audio recordings as well as unsupervised self-administered smartphone recordings highlight the merits of the proposed framework across both datasets, with the proposed landmark-dependent acoustic words achieving improvements in F1(depressed) of up to 15% and 13% for SH2-FS and DAIC-WOZ respectively, relative to acoustic speech baseline approaches.

There are many benefits to facilitating “always-on” pupillary light reflex (PLR)-aware pupil size measurement in eyewear, including improving the reliability of pupil-based cognitive and affective load monitoring and enabling PLR-based diagnosis of cognitive and eye-related diseases which have neurological symptoms manifested in the form of aberrant PLR responses. However, the detection of PLR responses for application in eyewear devices for everyday usage, beyond PLR measurement in confined clinical sessions, has not been investigated. To this end, a means of characterizing PLRs in less controlled environmental settings is investigated and subsequently a method of PLR detection is developed and evaluated. A low-cost head-mounted Web camera was used to record near-field eye video sequences which were processed with the self-tuning threshold algorithm for pupil diameter estimation and blink detection. PLR was induced by luminance change of a monitor and brightness change of the displayed image on a monitor. A transient model-based PLR detection algorithm which utilizes the general correlation of PLR amplitude and velocity was developed and evaluated on the data sets in terms of false alarm and false rejection rates. The findings from this research suggest that the PLR can be detected reliably using low-cost wearable pupil-measurement systems without using a separate sensor for detecting the luminance conditions. The correlation between pupil diameter amplitude and maximum velocity of PLR was shown to be sufficiently consistent for PLR detection.

Here we present AusKidTalk [1], an audio-visual (AV) corpus of Australian children’s speech collected to facilitate the development of speech based technological solutions for children. It builds upon the technology and expertise developed through the collection of an earlier corpus of Australian adult speech, AusTalk [2,3]. This multi-site initiative was established to remedy the dire shortage of children’s speech corpora in Australia and around the world that are sufficiently sized to train accurate automated speech processing tools for children. We are collecting ~600 hours of speech from children aged 3–12 years that includes single word and sentence productions as well as narrative and emotional speech. In this paper, we discuss the key requirements for AusKidTalk and how we designed the recording setup and protocol to meet them. We also discuss key findings from our feasibility study of the recording protocol, recording tools, and user interface.

Physical activity recognition using wearable sensors has achieved good performance in discriminating heterogeneous activities for health monitoring, but there has been less investigation of sedentary activities, e.g., desk work, which is often physically homogenous, to improve health in office environments. In this study, we explored head movement as a new sensing modality for physical and mental activity analysis. A new algorithm which segments gyroscope signals into atomic head movement events is proposed. Instead of recognizing activities in terms of predefined categories, we recognized four dimensions of task load: cognitive, perceptual, communicative, and physical, analogous to current manual workload assessment methods like NASA-TLX. We collected head movement data from 24 participants who wore a tri-axial inertial sensor at head while performing multiple tasks with varying load levels at office. An average of 70% accuracy was achieved for recognizing cognitive load levels, and more than 80% for the other three load types. The proposed event features outperformed a set of 181 features from previous physical activity recognition studies. We also demonstrated that these atomic event features are diagnostic of different load types in cross-load type classification, showing the promise of physical and mental load monitoring for health.

The INTERSPEECH 2014 Computational Paralinguistics Challenge provides for the first time a unified test-bed for the automatic recognition of speakers’ cognitive and physical load in speech. In this paper, we describe these two Sub-Challenges, their conditions, baseline results and experimental procedures, as well as the COMPARE baseline features generated with the openSMILE toolkit and provided to the participants in the Challenge.

The eyelid contour, pupil contour and blink event are important features of eye activity, and their estimation is a crucial research area for emerging wearable camera-based eyewear in a wide range of applications e.g. mental state estimation. Current approaches often estimate a single eye activity, such as blink or pupil center, from far-field and non-infrared (IR) eye images, and often depend on the knowledge of other eye components. This paper presents a unified approach to simultaneously estimate the landmarks for the eyelids, the iris and the pupil, and detect blink from near-field IR eye images based on a statistically learned deformable shape model and local appearance. Unlike the facial landmark estimation problem, by comparison, different shape models are applied to all eye states – closed eye, open eye with iris visible, and open eye with iris and pupil visible – to deal with the self-occluding interactions among the eye components. The most likely eye state is determined based on the learned local appearance. Evaluation on three different realistic datasets demonstrates that the proposed three-state deformable shape model achieves state-of-the-art performance for the open eye with iris and pupil state, where the normalized error was lower than 0.04. Blink detection can be as high as 90% in recall performance, without direct use of pupil detection. Cross-corpus evaluation results show that the proposed method improves on the state-of-the-art eyelid detection algorithm. This unified approach greatly facilitates eye activity analysis for research and practice when different types of eye activity are required rather than employ different techniques for each type. Our work is the first study proposing a unified approach for eye activity estimation from near-field IR eye images and achieved the state-of-the-art eyelid estimation and blink detection performance.

Speech emotion recognition is a challenging task and heavily depends on hand-engineered acoustic features, which are typically crafted to echo human perception of speech signals. However, a filter bank that is designed from perceptual evidence is not always guaranteed to be the best in a statistical modelling framework where the end goal is for example emotion classification. This has fuelled the emerging trend of learning representations from raw speech especially using deep learning neural networks. In particular, a combination of Convolution Neural Networks (CNNs) and Long Short Term Memory (LSTM) have gained great traction for the intrinsic property of LSTM in learning contextual information crucial for emotion recognition; and CNNs been used for its ability to overcome the scalability problem of regular neural networks. In this paper, we show that there are still opportunities to improve the performance of emotion recognition from the raw speech by exploiting the properties of CNN in modelling contextual information. We propose the use of parallel convolutional layers to harness multiple temporal resolutions in the feature extraction block that is jointly trained with the LSTM based classification network for the emotion recognition task. Our results suggest that the proposed model can reach the performance of CNN trained with hand-engineered features from both IEMOCAP and MSP-IMPROV datasets., Comment: INTERSPEECH 2019

Developments in head-mounted displays and wearable eyewear devices provide an opportunity to recognize expressions from near-field images, however this is not an area that has received significant research attention to date. In this study, we explore the identification of seven basic upper facial action units by analyzing just part of the face, i.e. infrared eye image, and applying a deformable eye model to obtain detailed information from the eye region. Based on this eye model, a novel feature extraction method is proposed that is a hybrid of geometric and appearance features extracted from around the edge of the eye. Evaluation on a novel database of near-field infrared facial expressions from 16 participants shows that 7-class upper face action unit classification can achieve around 78.8% accuracy using the proposed method, which is promising for wearable applications of automatic facial expression analysis.

Inspite the emerging importance of Speech Emotion Recognition (SER), the state-of-the-art accuracy is quite low and needs improvement to make commercial applications of SER viable. A key underlying reason for the low accuracy is the scarcity of emotion datasets, which is a challenge for developing any robust machine learning model in general. In this paper, we propose a solution to this problem: a multi-task learning framework that uses auxiliary tasks for which data is abundantly available. We show that utilisation of this additional data can improve the primary task of SER for which only limited labelled data is available. In particular, we use gender identifications and speaker recognition as auxiliary tasks, which allow the use of very large datasets, e.g., speaker classification datasets. To maximise the benefit of multi-task learning, we further use an adversarial autoencoder (AAE) within our framework, which has a strong capability to learn powerful and discriminative features. Furthermore, the unsupervised AAE in combination with the supervised classification networks enables semi-supervised learning which incorporates a discriminative component in the AAE unsupervised training pipeline. This semi-supervised learning essentially helps to improve generalisation of our framework and thus leads to improvements in SER performance. The proposed model is rigorously evaluated for categorical and dimensional emotion, and cross-corpus scenarios. Experimental results demonstrate that the proposed model achieves state-of-the-art performance on two publicly available datasets., Accepted in IEEE Transactions on Affective Computing

Cognitive workload is an important indicator of mental activity that has implications for human–computer interaction, biomedical and task analysis applications. Previously, subjective rating (self-assessment) has often been a preferred measure, due to its ease of use and relative sensitivity to the cognitive load variations. However, it can only be feasibly measured in a post-hoc manner with the user’s cooperation, and is not available as an online, continuous measurement during the progress of the cognitive task. In this paper, we used a cognitive task inducing seven different levels of workload to investigate workload discrimination using electroencephalography (EEG) signals. The entropy, energy, and standard deviation of the wavelet coefficients extracted from the segmented EEGs were found to change very consistently in accordance with the induced load, yielding strong significance in statistical tests of ranking accuracy. High accuracy for subject-independent multichannel classification among seven load levels was achieved, across the twelve subjects studied. We compare these results with alternative measures such as performance, subjective ratings, and reaction time (response time) of the subjects and compare their reliability with the EEG-based method introduced. We also investigate test/re-test reliability of the recorded EEG signals to evaluate their stability over time. These findings bring the use of passive brain-computer interfaces (BCI) for continuous memory load measurement closer to reality, and suggest EEG as the preferred measure of working memory load.

Learning the latent representation of data in unsupervised fashion is a very interesting process that provides relevant features for enhancing the performance of a classifier. For speech emotion recognition tasks, generating effective features is crucial. Currently, handcrafted features are mostly used for speech emotion recognition, however, features learned automatically using deep learning have shown strong success in many problems, especially in image processing. In particular, deep generative models such as Variational Autoencoders (VAEs) have gained enormous success for generating features for natural images. Inspired by this, we propose VAEs for deriving the latent representation of speech signals and use this representation to classify emotions. To the best of our knowledge, we are the first to propose VAEs for speech emotion classification. Evaluations on the IEMOCAP dataset demonstrate that features learned by VAEs can produce state-of-the-art results for speech emotion classification., Comment: Proc. Interspeech 2018

The fact that emotions are dynamic in nature and evolve across time has been explored relatively less often in automatic emotion recognition systems to date. Although within-utterance information about emotion changes recently has received some attention, there remain open questions unresolved, such as how to approach delta emotion ground truth, how to predict the extent of emotion change from speech, and how well change can be predicted relative to absolute emotion ratings. In this article, we investigate speech-based automatic systems for continuous prediction of the extent of emotion changes in arousal/valence. We propose the use of regression (smoothed) deltas as ground truth for emotion change, which yielded considerably higher inter-rater reliability than first-order deltas, a commonly used approach in previous research, and represent a more appropriate approach to derive annotations for emotion change research, findings which are applicable beyond speech-based systems. In addition, the first system design for continuous emotion change prediction from speech is explored. Experimental results under the Output-Associative Relevance Vector Machine framework interestingly show that changes in emotion ratings may be better predicted than absolute emotion ratings on the RECOLA database, achieving 0.74 vs. 0.71 for arousal and 0.41 vs. 0.37 for valence in concordance correlation coefficients. However, further work is needed to achieve effective emotion change prediction performances on the SEMAINE database, due to the large number of non-change frames in the absolute emotion ratings.

Predicting emotion intensity and severity of depression are both challenging and important problems within the broader field of affective computing. As part of the AVEC 2017, we developed a number of systems to accomplish these tasks. In particular, word affect features, which derive human affect ratings (e.g. arousal and valence) from transcripts, were investigated for predicting depression severity and liking, showing great promise. A simple system based on the word affect features achieved an RMSE of 6.02 on the test set, yielding a relative improvement of 13.6% over the baseline. For the emotion prediction sub-challenge, we investigated multimodal fusion, which incorporated a measure of uncertainty associated with each prediction within an Output-Associative fusion framework for arousal and valence prediction, whilst liking prediction systems mainly focused on text-based features. Our best emotion prediction systems provided significant relative improvements over the baseline on the test set of 39.5%, 17.6%, and 29.3% for arousal, valence, and liking. Of particular note is that consistent improvements were observed when incorporating prediction uncertainty across various system configurations for predicting arousal and valence, suggesting the importance of taking into consideration prediction uncertainty for fusion and more broadly the advantages of probabilistic predictions.

Continuous prediction of dimensional emotions (e.g. arousal and valence) has attracted increasing research interest recently. When processing emotional speech signals, phonetic features have been rarely used due to the assumption that phonetic variability is a confounding factor that degrades emotion recognition/prediction performance. In this paper, instead of eliminating phonetic variability, we investigated whether Phone Log-Likelihood Ratio (PLLR) features could be used to index arousal and valence in a pairwise low/high framework. A multi-stage staircase regression (SR) framework which enables fusion at three different stages is also investigated. Results on the RECOLA database show that PLLR outperforms EGEMAPS features for arousal and valence. Interestingly, long-term averaged PLLR proved to be more robust and emotionally informative than local frame-level PLLR, which contains more phoneme-specific information. Within the multistage SR framework, PLLR yielded an 8.2% and 11.6% relative improvement in CCC for arousal and valence respectively, showing great promise for including phonetic features in emotion prediction systems.

This paper introduces a novel speech-based depression score prediction paradigm, the 2-stage ranking prediction framework, and highlights the benefits it brings to depression prediction. Conventional regression approaches aim to discern a single functional relationship between speech features and depression scores, making an implicit assumption about the existence of a single fixed relationship between the features and scores. However, as the relationship between severity of depression and the clinical score may vary over the range of the assessment scale, this style of analysis may not be suited to depression prediction. The proposed framework on the other hand, imposes a series of partitions on the feature space, with each partition corresponding to a distinct predefined range of depression scores, and predicts the score based on measures of membership to each partition. This approach provides additional flexibility by allowing different rankings to be learnt for different depression scores, and relaxes assumptions made by conventional regression approaches. Results demonstrate the framework's suitability for depression score prediction: different 2-stage implementations, based on heterogeneous feature extraction and modelling approaches, produce state-of-the-art results on the AVEC-2013 dataset. It is also demonstrated that, unlike fusion of conventional regression systems, the fusion of two-stage systems consistently improves prediction performance.

Depression is a severe mental health disorder with high societal costs. Current clinical practice depends almost exclusively on self-report and clinical opinion, risking a range of subjective biases. The long-term goal of our research is to develop assistive technologies to support clinicians and sufferers in the diagnosis and monitoring of treatment progress in a timely and easily accessible format. In the first phase, we aim to develop a diagnostic aid using affective sensing approaches. This paper describes the progress to date and proposes a novel multimodal framework comprising of audio-video fusion for depression diagnosis. We exploit the proposition that the auditory and visual human communication complement each other, which is well-known in auditory-visual speech processing; we investigate this hypothesis for depression analysis. For the video data analysis, intra-facial muscle movements and the movements of the head and shoulders are analysed by computing spatio-temporal interest points. In addition, various audio features (fundamental frequency f0, loudness, intensity and mel-frequency cepstral coefficients) are computed. Next, a bag of visual features and a bag of audio features are generated separately. In this study, we compare fusion methods at feature level, score level and decision level. Experiments are performed on an age and gender matched clinical dataset of 30 patients and 30 healthy controls. The results from the multimodal experiments show the proposed framework’s effectiveness in depression analysis.

Highlights? Tutorial style introduction to sparse representation classification (SRC) and its applications. ? Investigation on SRC for speaker verification using i-vector from the total variability model. ? Comparative study of SRC systems with different sparseness methods and dictionary composition. ? Development of SRC background dataset selection based on column vector frequency was proposed. ? Results show i-SRC achieves the best performance on NIST 2010 SRE. Sparse representation-based methods have very lately shown promise for speaker recognition systems. This paper investigates and develops an i-vector based sparse representation classification (SRC) as an alternative classifier to support vector machine (SVM) and Cosine Distance Scoring (CDS) classifier, producing an approach we term i-vector-sparse representation classification (i-SRC). Unlike SVM which fixes the support vector for each target example, SRC allows the supports, which we term sparse coefficient vectors, to be adapted to the test signal being characterized. Furthermore, similarly to CDS, SRC does not require a training phase. We also analyze different types of sparseness methods and dictionary composition to determine the best configuration for speaker recognition. We observe that including an identity matrix in the dictionary helps to remove sensitivity to outliers and that sparseness methods based on ?1 and ?2 norm offer the best performance. A combination of both techniques achieves a 18% relative reduction in EER over a SRC system based on ?1 norm and without identity matrix. Experimental results on NIST 2010 SRE show that the i-SRC consistently outperforms i-SVM and i-CDS in EER by 0.14-0.81%, and the fusion of i-CDS and i-SRC achieves a relative EER reduction of 8-19% over i-SRC alone.

High cognitive load arises from complex time and safety-critical tasks, for example, mapping out flight paths, monitoring traffic, or even managing nuclear reactors, causing stress, errors, and lowered performance. Over the last five years, our research has focused on using the multimodal interaction paradigm to detect fluctuations in cognitive load in user behavior during system interaction. Cognitive load variations have been found to impact interactive behavior: by monitoring variations in specific modal input features executed in tasks of varying complexity, we gain an understanding of the communicative changes that occur when cognitive load is high. So far, we have identified specific changes in: speech, namely acoustic, prosodic, and linguistic changes; interactive gesture; and digital pen input, both interactive and freeform. As ground-truth measurements, galvanic skin response, subjective, and performance ratings have been used to verify task complexity. The data suggest that it is feasible to use features extracted from behavioral changes in multiple modal inputs as indices of cognitive load. The speech-based indicators of load, based on data collected from user studies in a variety of domains, have shown considerable promise. Scenarios include single-user and team-based tasks; think-aloud and interactive speech; and single-word, reading, and conversational speech, among others. Pen-based cognitive load indices have also been tested with some success, specifically with pen-gesture, handwriting, and freeform pen input, including diagraming. After examining some of the properties of these measurements, we present a multimodal fusion model, which is illustrated with quantitative examples from a case study. The feasibility of employing user input and behavior patterns as indices of cognitive load is supported by experimental evidence. Moreover, symptomatic cues of cognitive load derived from user behavior such as acoustic speech signals, transcribed text, digital pen trajectories of handwriting, and shapes pen, can be supported by well-established theoretical frameworks, including O'Donnell and Eggemeier's workload measurement [1986] Sweller's Cognitive Load Theory [Chandler and Sweller 1991], and Baddeley's model of modal working memory [1992] as well as McKinstry et al.'s [2008] and Rosenbaum's [2005] action dynamics work. The benefit of using this approach to determine the user's cognitive load in real time is that the data can be collected implicitly that is, during day-to-day use of intelligent interactive systems, thus overcomes problems of intrusiveness and increases applicability in real-world environments, while adapting information selection and presentation in a dynamic computer interface with reference to load.

Within the field of affective computing, human emotion and disorder/disease recognition have progressively attracted more interest in multimodal analysis. This submission to the Depression Classification and Continuous Emotion Prediction challenges for AVEC2016 investigates both, with a focus on audio subsystems. For depression classification, we investigate token word selection, vocal tract coordination parameters computed from spectral centroid features, and gender-dependent classification systems. Token word selection performed very well on the development set. For emotion prediction, we investigate emotionally salient data selection based on emotion change, an output-associative regression approach based on the probabilistic outputs of relevance vector machine classifiers operating on low-high class pairs (OA RVM-SR), and gender-dependent systems. Experimental results from both the development and test sets show that the RVM-SR method under the OA framework can improve on OA RVM, which performed very well in the AV+EC2015 challenge.

Analysis of movement using accelerometers mounted on the torso or limbs has shown good potential for the recognition of physical activities. However many contemporary lifestyle tasks are sedentary, and less is known about how these can be automatically characterized using movement signals. This paper proposes possibly the first system that employs head movement for recognizing different levels of mental activity and for discriminating between various kinds of sedentary and non-sedentary tasks. Results from analysis of a 20-participant database show that head movement is surprisingly indicative of cognitive load and discriminative between different task types, as well as exhibiting some sensitivity to the instant of task change. Given the possibility for wearing hats or glasses with embedded inertial measurement units, this suggests a range of interesting future applications, including monitoring of sedentary daily activities, and developing rough estimates of mental exertion.

Speech is a promising modality for the convenient measurement of cognitive load, and recent years have seen the development of several cognitive load classification systems. Many of these systems have utilised mel frequency cepstral coefficients (MFCC) and prosodic features like pitch and intensity to discriminate between different cognitive load levels. However, the accuracies obtained by these systems are still not high enough to allow for their use outside of laboratory environments. One reason for this might be the imperfect acoustic description of speech provided by MFCCs. Since these features do not characterise the distribution of the spectral energy within subbands, in this paper, we investigate the use of spectral centroid frequency (SCF) and spectral centroid amplitude (SCA) features, applying them to the problem of automatic cognitive load classification. The effect of varying the number of filters and the frequency scale used is also evaluated, in terms of the effectiveness of the resultant spectral centroid features in discriminating between cognitive loads. The results of classification experiments show that the spectral centroid features consistently and significantly outperform a baseline system employing MFCC, pitch, and intensity features. Experimental results reported in this paper indicate that the fusion of an SCF based system with an SCA based system results in a relative reduction in error rate of 39% and 29% for two different cognitive load databases.

Classification of multichannel EEG recordings during motor imagination has been exploited successfully for brain-computer interfaces (BCI). In this paper, we consider EEG signals as the outputs of a networked dynamical system (the cortex), and exploit synchronization features from the dynamical system for classification. Herein, we also propose a new framework for learning optimal filters automatically from the data, by employing a Fisher ratio criterion. Experimental evaluations comparing the proposed dynamical system features with the CSP and the AR features reveal their competitive performance during classification. Results also show the benefits of employing the spatial and the temporal filters optimized using the proposed learning approach.

In the space of affect detection in multimedia, there is a strong demand for more tagged data in order to better understand human emotions, the way they are expressed, and approaches for detecting them automatically. Unfortunately, emotion datasets are typically small due to the manual process of annotating them with emotional labels. In response, we present for the first time the application of automatically tagged Twitter data to the problem of speech emotion recognition (SER). SER has been shown to benefit from the combination of acoustic and linguistic features, albeit when the linguistic training data is from the same database as the test data. Using the presence of emoticons for automatic tagging, we compile a corpus of over 800,000 tweets that is totally independent from our evaluation database. By supplementing an acoustic classifier with linguistic information, we classify the spontaneous content within the USC-IEMOCAP corpus on valence and activation descriptors. With comparison to prior literature, we demonstrate performance improvements for valence of 2% and 6% over an acoustic-only system, using linguistic training data from Twitter and IEMOCAP respectively.

Continuous emotion dimension prediction has increased in popularity over the last few years, as the shift away from discrete classification based tasks has introduced more realism in emotion modeling. However, many questions remain including how best to combine information from several modalities (e.g. audio, video, etc). As part of the AV+EC 2015 Challenge, we investigate annotation delay compensation and propose a range of multimodal systems based on an output-associative fusion framework. The performance of the proposed systems are significantly higher than the challenge baseline, with the strongest performing system yielding 66.7% and 53.9% relative increases in prediction accuracy over the AV+EC 2015 test set arousal and valence baselines respectively. Results also demonstrate the importance of annotation delay compensation for continuous emotion analysis. Of particular interest was the output-associative based fusion framework, which performed very well in a number of significantly different configurations, highlighting that incorporating both affective dimensional dependencies and temporal information is a promising research direction for predicting emotion dimensions.

Monitoring pupil and blink dynamics has applications in cognitive load measurement during human-machine interaction. However, accurate, efficient, and robust pupil size and blink estimation pose significant challenges to the efficacy of real-time applications due to the variability of eye images, hence to date, require manual intervention for fine tuning of parameters. In this paper, a novel self-tuning threshold method, which is applicable to any infrared-illuminated eye images without a tuning parameter, is proposed for segmenting the pupil from the background images recorded by a low cost webcam placed near the eye. A convex hull and a dual-ellipse fitting method are also proposed to select pupil boundary points and to detect the eyelid occlusion state. Experimental results on a realistic video dataset show that the measurement accuracy using the proposed methods is higher than that of widely used manually tuned parameter methods or fixed parameter methods. Importantly, it demonstrates convenience and robustness for an accurate and fast estimate of eye activity in the presence of variations due to different users, task types, load, and environments. Cognitive load measurement in human-machine interaction can benefit from this computationally efficient implementation without requiring a threshold calibration beforehand. Thus, one can envisage a mini IR camera embedded in a lightweight glasses frame, like Google Glass, for convenient applications of real-time adaptive aiding and task management in the future.

Variations in the acoustic space due to changes in speaker mental state are potentially overshadowed by variability due to speaker identity and phonetic content. Using the Audio/Visual Emotion Challenge and Workshop 2013 Depression Dataset we explore the suitability of i-vectors for reducing these latter sources of variability for distinguishing between low or high levels of speaker depression. In addition we investigate whether supervised variability compensation methods such as Linear Discriminant Analysis (LDA), and Within Class Covariance Normalisation (WCCN), applied in the i-vector domain, could be used to compensate for speaker and phonetic variability. Classification results show that i-vectors formed using an over-sampling methodology outperform a baseline set by KL-means supervectors. However the effect of these two compensation methods does not appear to improve system accuracy. Visualisations afforded by the t-Distributed Stochastic Neighbour Embedding (t-SNE) technique suggest that despite the application of these techniques, speaker variability is still a strong confounding effect.

In automatic forensic speaker recognition research two frameworks, namely single-stage and two-stage modelling, are used. Although both have their own strengths and limitations, performance is an important attribute that needs to be considered when selecting these methods for forensic research. This paper compares a calibrated single-stage system with a calibrated two-stage system using a common database in terms of different performance metrics. Neither of the systems provides a consistent advantage over the other in terms of all performance measures, raising the question of whether the use of a two-stage system, which requires additional data and effort, is warranted.

Automatic forensic voice comparison (FVC) studies have often employed Gaussian Mixture Model - Universal Background Model (GMM-UBM) modeling based on mean-only maximum a posteriori (MAP) adaptation or full MAP adaptation with little consideration of other variants of MAP adapted configurations. Our study indicates that an FVC system improvement in log-likelihood ratio cost (Cllr) of up to 6.8% can be achieved via fusion of other MAP adapted configurations such as variance-only and weight-only adaptations. We also demonstrate a novel adaptation and fusion strategy named Separate MAP (SMAP) that yielded a substantial FVC performance improvement in Cllr, up to 24.2%, and which is more robust under limited data conditions compared with the conventional mean-only or full MAP adaptation. The fusion strategy involves fusing multiple MAP adapted GMM sub-configurations where in each of these GMM sub-configurations only a small subset of the total number of GMM parameters are MAP adapted separately based on the same speaker-specific training data.

Mental workload is an important factor during writing, which may affect the writing efficiency and user experience. This paper aims at a method to classify the mental workload levels during writing process, via examination of online writing features in a two-stage algorithm structure. At the first stage, a curvature tracking method is applied to the handwriting script, to examine the curvature for individual writing points. Then a selection process allocates writing points into subsets, each corresponding to one curvature span. The second stage extracts velocity features, used to characterize mental workload, from points in each curvature span. A Parzen-window classifier is applied on velocity features from each curvature span. The classification decisions from individual classifiers are fused with a selective voting scheme for the overall mental workload classification decision. This paper finally discusses the classification accuracy for three mental workload levels and compares it with previous work.

Using the task-evoked pupillary response (TEPR) to index cognitive load can contribute significantly to the assessment of memory function and cognitive skills in patients. However, the measurement of pupillary response is currently limited to a well-controlled lab environment due to light reflex and also relies heavily on expensive video-based eye trackers. Furthermore, commercial eye trackers are usually dedicated to gaze direction measurement, and their calibration procedure and computing resource are largely redundant for pupil-based cognitive load measurement (PCLM). In this study, we investigate the validity of cognitive load measurement with (i) pupil light reflex in a less controlled luminance background; (ii) a low-cost infrared (IR) webcam for the TEPR in a controlled luminance background. ANOVA results show that with an appropriate baseline selection and subtraction, the light reflex is significantly reduced, suggesting the possibility of less constrained practical applications of PCLM. Compared with the TEPR from a commercial remote eye tracker, a low-cost IR webcam achieved a similar TEPR pattern and no significant difference was found between the two devices in terms of cognitive load measurement across five induced load levels.

Accurate detection of depression from spontaneous speech could lead to an objective diagnostic aid to assist clinicians to better diagnose depression. Little thought has been given so far to which classifier performs best for this task. In this study, using a 60-subject real-world clinically validated dataset, we compare three popular classifiers from the affective computing literature - Gaussian Mixture Models (GMM), Support Vector Machines (SVM) and Multilayer Perceptron neural networks (MLP) - as well as the recently proposed Hierarchical Fuzzy Signature (HFS) classifier. Among these, a hybrid classifier using GMM models and SVM gave the best overall classification results. Comparing feature, score, and decision fusion, score fusion performed better for GMM, HFS and MLP, while decision fusion worked best for SVM (both for raw data and GMM models). Feature fusion performed worse than other fusion methods in this study. We found that loudness, root mean square, and intensity were the voice features that performed best to detect depression in this dataset.

Quantifying behavioural changes in depression using affective computing techniques is the first step in developing an objective diagnostic aid, with clinical utility, for clinical depression. As part of the AVEC 2013 Challenge, we present a multimodal approach for the Depression Sub-Challenge using a GMM-UBM system with three different kernels for the audio subsystem and Space Time Interest Points in a Bag-of-Words approach for the vision subsystem. These are then fused at the feature level to form the combined AV system. Key results include the strong performance of acoustic audio features and the bag-of-words visual features in predicting an individual's level of depression using regression. Interestingly, in the context of the small amount of literature on the subject, is that our feature level multimodal fusion technique is able to outperform both the audio and visual challenge baselines.

While period detection in biological sequence data has received considerable attention, it is unclear which methods may be best suited to the problem of exploratory period estimation, where the objective is to compare the relative strengths of many periods on a linear-period scale. This paper compares several promising methods for period estimation on an integer-period scale in terms of attributes such as correct identification of dominant periods, period resolution and computational complexity, using synthetic sequences. Different methods reveal very different periodicity profiles, however the exactly periodic subspace decomposition and hybrid autocorrelation-IPDFT methods seem to provide good performance with respect to the above attributes. Finally, the methods are compared for a challenging DNA sequence fragment, from P.falciparum.

We propose the use of entropy-based features; spectral and approximate entropies, of recorded EEG signals to characterize mental load when performing a cognitive task. It is demonstrated on a seven load-level task that the spectral entropy is a good discriminator of mental load level and decreases consistently in accordance with the increased load. The extracted approximate entropy quantifies the irregularity of the EEGs, indicating a decrease in irregularity as the load increases. We also perform EEG source estimation to choose a smaller subset of EEG channels which make the most contribution in the load level discrimination. We conclude that the entropy-based features are capable of measuring the imposed mental load from the selected channels in two brain regions. This may demonstrate that the brain behaves in a more regular or focused manner when dealing with higher task loads. The efficacy of entropy-based features across frequency sub-bands is also investigated.

Measuring cognitive load changes can contribute to better treatment of patients, can help design effective strategies to reduce medical errors among clinicians and can facilitate user evaluation of health care information systems. This paper proposes an eye-based automatic cognitive load measurement (CLM) system toward realizing these prospects. Three types of eye activity are investigated: pupillary response, blink and eye movement (fixation and saccade). Eye activity features are investigated in the presence of emotion interference, which is a source of undesirable variability, to determine the susceptibility of CLM systems to other factors. Results from an experiment combining arithmetic-based tasks and affective image stimuli demonstrate that arousal effects are dominated by cognitive load during task execution. To minimize the arousal effect on CLM, the choice of segments for eye-based features is examined. We then propose a feature set and classify three levels of cognitive load. The performance of cognitive load level prediction was found to be close to that of a reaction time measure, showing the feasibility of eye activity features for near-real time CLM.

We investigate the use of wavelet-based complexity measures of electroencephalogram (EEG) signals to evaluate changes in working memory load during the performance of a cognitive task with varying difficulty/load levels. Extracted wavelet-complexity measures associated with four entropic measures; that is Shannon, Tsallis, Escort-Tsallis and Renyi entropies demonstrate good discrimination among seven load levels imposed on the working memory with a classification rate of up to 96% using signals recorded from the frontal lobe of the brain. The extracted measures' values show a consistent decrease in the selected channels in two frontal and occipital lobes, as the memory load increases, indicating the EEGs disorder declines while the complexity grows. This illustrates that the brain behaves in a more organized manner characterized by more order and maximal complexity when dealing with higher load levels. The growing complexity can also reflect the higher activation of neural networks involved, as the task load increases.

This study was undertaken to investigate spectral features derived from EEG signals for measuring cognitive load. Measurements of this kind have important commercial and clinical applications for optimizing the performance of users working under high mental load conditions, or as cognitive tests. Based on EEG recordings for a reading task in which three different levels of cognitive load were induced, it is shown that a set of spectral features — the spectral entropy, weighted mean frequency and its bandwidth, and spectral edge frequency — are all able to discriminate the three load levels effectively. An interesting result is that spectral entropy, which reflects the distribution of spectral energy rather than its magnitude, provides very good discrimination between cognitive load levels. We also report those EEG channels for which statistical significance between load levels was achieved. The effect of frequency bands on the spectral features is also investigated here. The results indicate that the choice of optimal frequency band can be dependent on the spectral feature extracted.

Recent results seem to cast some doubt over the assumption that improvements in fused recognition accuracy for speaker recognition systems based on different acoustic features are due mainly to the different origins of the features (e.g. magnitude, phase, modulation information). In this study, we utilize clustering comparison measures to investigate acoustic and speaker modelling aspects of the speaker recognition task separately and demonstrate that front-end diversity can be achieved purely through different ‘partitioning’ of the acoustic space. Further, features that exhibit good ‘stability’ with respect to repeated clustering are shown to also give good EER performance in speaker recognition. This has implications for feature choice, fusion of systems employing different features, and for UBM data selection. A method for the latter problem is presented that gives up to an 11% relative reduction in EER using only 20–30% of the usual UBM training data set.

Previous work in speech-based cognitive load classification has shown that the glottal source contains important information for cognitive load discrimination. However, the reliability of glottal flow features depends on the accuracy of the glottal flow estimation, which is a non-trivial process. In this paper, we propose the use of acoustic voice source features extracted directly from the speech spectrum (or cepstrum) for cognitive load classification. We also propose pre-and post-processing techniques to improve the estimation of the cepstral peak prominence (CPP). 3-class classification results on two databases showed CPP as a promising cognitive load classification feature that outperforms glottal flow features. Score-level fusion of the CPP-based classification system with a formant frequency-based system yielded a final improved accuracy of 62.7%, suggesting that CPP contains useful voice source information that complements the information captured by vocal tract features.

Sparse representations of signals have received a great deal of attention in recent years, and the sparse representation classifier has very lately appeared in a speaker recognition system. This approach represents the (sparse) GMM mean supervector of an unknown speaker as a linear combination of an over-complete dictionary of GMM supervectors of many speaker models, and l 1 -norm minimization results in a non-zero coefficient corresponding to the unknown speaker class index. Here this approach is tested on large databases, introducing channel-/session-variability compensation, and fused with a GMM-SVM system. Evaluations on the NIST 2001 SRE and NIST 2006 SRE database show that when the outputs of the MFCC UBM-GMM based classifier (for NIST 2001 SRE) or MFCC GMM-SVM based classifier (for NIST 2006 SRE) are fused with the MFCC GMM-Sparse Representation Classifier (GMM-SRC) based classifier, an absolute gain of 1.27% and 0.25% in EER can be achieved respectively.