Search

A Compound Expression Recognition (CER) as a subfield of affective computing is a novel task in intelligent human-computer interaction and multimodal user interfaces. We propose a novel audio-visual method for CER. Our method relies on emotion recognition models that fuse modalities at the emotion probability level while decisions regarding the prediction of compound expressions are based on the pair-wise sum of weighted emotion probability distributions. Notably our method does not use any training data specific to the target task. Thus the problem is a zero-shot classification task. The method is evaluated in multi-corpus training and cross-corpus validation setups. We achieved F1 scores of 32.15% and 25.56% for the AffWild2 and C-EXPR-DB test subsets without training on target corpus and target task respectively. Therefore our method is on par with methods developed training target corpus or target task. The source code is publicly available from https://elenaryumina.github.io/AVCER.

Automatic emotion recognition has gained significant attention over the past two decades due to the central role that emotions play in human communication. While multi-modal systems demonstrate high performances on laboratory-controlled data their validity on non-lab-controlled namely 'in-the-wild' data remains a challenge. This work investigates audio-visual deep learning approaches for emotion recognition in-the-wild with a particular focus on the effectiveness of architectures based on fine-tuned Convolutional Neural Networks (CNN) and Public Dimensional Emotion Model (PDEM) for video and audio modality respectively. We explore and compare various temporal modeling techniques (e.g. transformer architectures) and fusion strategies by leveraging the embeddings from developed multi-stage trained modality-specific Deep Neural Networks (DNN). The results are reported on the AffWild2 dataset following the Affective Behavior Analysis in-the-Wild 2024 (ABAW'24) challenge protocol. Our investigation highlights the complexities of robust multi-modal emotion recognition in an unconstrained environment providing insights into the usage of various deep learning architectures for tackling this challenging task.

Gegevensbescherming en privacy worden steeds crucialer naarmate gegevens waardevoller en potentieel krachtiger worden. Bovendien neemt de druk van regelgeving en de samenleving toe op ontwikkelaars van Artificial Intelligence (AI) modellen om ervoor te zorgen dat hun modellen niet discriminerend zijn. Ten slotte is er behoefte aan interpreteerbare, transparante AI modellen voor taken met grote belangen. Over het algemeen vereist het meten van de fairness van ieder AI model de gevoelige kenmerken van de individuen in de dataset, waardoor privacy dus in het gedrang komt. In dit werk worden de afwegingen tussen fairness, privacy en interpreteerbaarheid verder onderzocht. We onderzoeken specifiek de Statistical Parity (SP) van beslisbomen door middel van Differentiële Privacy (DP). Dit zijn alle drie populaire methodes in hun respectievelijke onderzoeksvelden. We stellen een methode voor, genaamd Privacy-Aware Fairness Estimation of Rules (PAFER), die SP kan schatten terwijl rekening wordt gehouden met DP voor op beslisbomen. DP, waarbij gebruik wordt gemaakt van een derde partij die veilig omgaat met deze gevoelige gegevens, garandeert privacy door ruis toe te voegen aan de gevoelige gegevens. Verschillende ruis mechanismen voor DP worden empirisch vergeleken. We laten experimenteel zien dat met behulp van het Laplace-mechanisme de methode in staat is om SP met een lage fout te schatten, terwijl de privacy van de individuen in de dataset met grote zekerheid wordt gegarandeerd. We tonen ook experimenteel en theoretisch aan dat de methode beter presteert voor beslisbomen die beter interpreteerbaar zijn.

Depression is a prevalent mental health disorder affecting both patients and society. The ability to identify at-risk individuals early, accurately, and without human intervention can be considered an important task as it enables timely and appropriate intervention and treatment. In recent years, numerous models have shown to be successful in detecting depression based on audiovisual cues. However, the growing use of machine learning (ML) systems for this task has raised concerns about potential biases within these systems. This thesis explores gender fairness in multimodal depression detection using the D-Vlog dataset, which comprises vlogs derived from social media (YouTube). This study addresses the gender bias observed in previous models, particularly the performance disparity between genders. While previous studies have effectively used textual data to detect depression from social media, no research has yet applied this approach to the D-Vlog dataset. This study integrates the textual modality, experiments with various fusion strategies, and evaluates multiple bias mitigation techniques, aiming to improve both the fairness and performance of depression detection models developed using the D-Vlog dataset. The methodology involves extracting the textual modality from the vlogs in the form of transcripts, followed by preprocessing steps to obtain both word and sentence embeddings and to prevent potential data leakage. A modality-based approach analyses the impact of the textual modality on performance and fairness, where uni- and multimodal models are trained using different modalities and fusion approaches. After applying various bias mitigation methods, the study assesses their effects on fairness and performance. Experimental results reveal that incorporating the textual modality boosts the performance of both uni- and multimodal depression detection models, though a trade-off between performance and fairness is observed. Moreover, it was found that the choice

In recent times, significant advancements have been witnessed in the field of language models, particularly with the emergence of Large Language Models (LLMs) that are trained on vast amounts of data extracted from internet archives. These LLMs, such as ChatGPT, have become widely accessible, allowing users to generate text for various purposes including articles, essays, jokes, and poetry. Given that LLMs are trained on a diverse range of text sources, encompassing platforms like Reddit and Twitter, it is foreseeable that future training datasets will also incorporate text generated by previous iterations of the models themselves. In light of this development, our research aims to investigate the influence of artificial text in the pre-training phase of language models. Specifically, we conducted a comparative analysis between a language model, RoBERTa, pre-trained using CNN/DailyMail news articles, and ChatGPT, which employed the same articles for its training and evaluated their performance on three downstream tasks as well as their potential gender bias, using sentiment analysis as a metric. Through a series of experiments, we demonstrate that the utilization of artificial text during pre-training does not have a significant impact on either the performance of the models in downstream tasks or their gender bias. In conclusion, our findings suggest that the inclusion of text generated by LLMs in their own pre-training process does not yield substantial effects on the subsequent performance of the models in downstream tasks or their potential gender bias

Word embeddings, renowned for their role as superior semantic feature vector representation in diverse NLP tasks, can exhibit an undesired bias for stereotypical categories. The bias arises from the statistical and societal biases within the datasets used for training. In this study, we analyze the gender bias in four different pre-trained word embeddings for a range of affective computing tasks in the mental health domain including the detection of psychiatric disorders such as depression, and alcohol/substance abuse. We incorporate both contextual and non-contextual embeddings, which are trained not just on general domain data but also on data specific to the clinical domain. Our findings indicate that the bias in embeddings is towards different gender groups, depending on the type of embeddings and the training dataset. Furthermore, we highlight how these existing associations transfer to subsequent tasks and might even be amplified during supervised training for patient phenotyping. We also show that a simple method of data augmentation- swapping gender words - noticeably reduces bias in these subsequent tasks. The scripts to reproduce the results are available at: https://github.com/gizemsogancioglu/gender-bias-mental-health.

Machine learning (ML) algorithms are increasingly used in high-stake domains like healthcare. While ML systems frequently outperform humans in specific tasks, ensuring safety and transparency is critical in these domains. Interpretability, therefore, plays a crucial role in understanding the decision-making process, auditing and correction of ML models and establishing trust. Furthermore, there is a growing demand for automated machine learning (AutoML) to facilitate model development without expert intervention. However, the combination of interpretability and AutoML has received limited attention thus far. In this study, we propose two objective model-agnostic measures of interpretability to quantify model compactness and explanation stability, embedded within an automated interpretable ML pipeline. We experiment with a set of interpretable models on medical classification tasks reporting the proposed measures along with the predictive performances. We further conduct a user study with domain experts to evaluate the correlation between these measures and the subjective concept of interpretability. Our findings demonstrate the effectiveness of the proposed measures, affirming their success and validating their utility in creating an interpretable automated pipeline.

This thesis examines the performance of Fisher Vector representations in classifying personality traits from audio. The Chalearn LAP First Impression dataset is used, which is a multimodal dataset. The audio modality of the dataset is focused on, and different audio feature extraction methods, including wav2vec 2.0, openSMILE, and public dimensional emotion model (PDEM), are studied for their performance on the classification task. Different encoding approaches, such as Fisher Vector, are also studied to see how they affect the performance of the classifier. The results of this thesis suggest that Fisher Vector representations are not the best choice for classifying personality traits from audio for the certain dataset. However, other feature extraction methods, such as openSMILE LLDs and PDEM, can achieve good performance on this task. The thesis also provides some insights into the selection of parameters for feature engineering and the interpretability of Fisher Vector representations.

In recent times, significant advancements have been witnessed in the field of language models, particularly with the emergence of Large Language Models (LLMs) that are trained on vast amounts of data extracted from internet archives. These LLMs, such as ChatGPT, have become widely accessible, allowing users to generate text for various purposes including articles, essays, jokes, and poetry. Given that LLMs are trained on a diverse range of text sources, encompassing platforms like Reddit and Twitter, it is foreseeable that future training datasets will also incorporate text generated by previous iterations of the models themselves. In light of this development, our research aims to investigate the influence of artificial text in the pre-training phase of language models. Specifically, we conducted a comparative analysis between a language model, RoBERTa, pre-trained using CNN/DailyMail news articles, and ChatGPT, which employed the same articles for its training and evaluated their performance on three downstream tasks as well as their potential gender bias, using sentiment analysis as a metric. Through a series of experiments, we demonstrate that the utilization of artificial text during pre-training does not have a significant impact on either the performance of the models in downstream tasks or their gender bias. In conclusion, our findings suggest that the inclusion of text generated by LLMs in their own pre-training process does not yield substantial effects on the subsequent performance of the models in downstream tasks or their potential gender bias

This study presents an exploration of using multi-output forecasting models in predicting Origin Destination (OD) time series that represent the travel behavior of travelers using the railway system facilitated by de Nederlandse Spoorwegen (NS). DeepAR was used as a comparative baseline system. We present an intelligible and effective handcrafted feature set. Using this feature set, we found that multi-output models using LightGBM significantly outperform the DeepAR forecasting model. When coupled with TreeSHAP, we show that the generated explanations are understandable and relatively sparse. Furthermore, we propose an interpretable Linear Regression (LR) model with L1 regularization that matches the DeepAR forecasting performance while having an average of 12 nonzero coeficients. The study also establishes that dimensionality reduction of past time series history via beta-VAE and SVD methods is less effective compared to utilizing the uncompressed feature vector.

In recent years, Artificial Intelligence (AI) has seen a rapid development where it can aid, and sometimes completely replace, humans with decision making due to its superior computation and information processing skills. However, using AI in decision making tasks has not been without flaws, as researchers warn that blindly trusting AI could prove to have major adverse effects for humans. This is due to bias in AI, described as a prejudice of favoritism toward certain subjects, even if they are rationally unjustified. Examples range from university rankings to recidivism tests, where using AI resulted in damaging effects and perpetuation of bias. As a consequence, legislators over the world have introduced new acts to ensure transparency, explainability and fairness in AI, like the recent EU AI act and GDPR. To support this, the field of Responsible AI has set out to make AI more transparent and fair. However, we saw a gap in the current state of the art in fairness assessment toolkits. Researchers urgently called for the creation of a methodology in assisting users with fairness due to its complexity and context dependency. That is why we created this toolkit, in which users are automatically guided by interactive questions on selecting the most suitable model and fairness criteria for a given task, all openly and freely available in JASP. This methodology was created by identifying characteristics of fairness measures, creating a decision tree whose internal nodes composed of interactive questions, mutating this tree and generating candidate trees and subsequently evaluating the trees to select the best one. With this toolkit we hope the help the effort on making AI more transparent and fair.