Search

Post-partum hemorrhaging is a medical emergency that occurs during childbirth and, in extreme cases, can be life-threatening. It is the number one cause of maternal mortality worldwide. High-quality training of medical staff can contribute to early diagnosis and work towards preventing escalation towards more serious cases. Healthcare education uses manikin-based simulators to train obstetricians for various childbirth scenarios before training on real patients. However, these medical simulators lack certain key features portraying important symptoms and are incapable of communicating with the trainees. The authors present a digital embodiment agent that can improve the current state of the art by providing a specification of the requirements as well as an extensive design and development approach. This digital embodiment allows educators to respond and role-play as the patient in real time and can easily be integrated with existing training procedures. This research was performed in collaboration with medical experts, making a new contribution to medical training by bringing digital humans and the representation of affective interfaces to the field of healthcare.

Objective: We investigate experienced joy of use (JoU) and usability using a multimodal methods approach by systematically varying mobile phone interactions. Methods: We combined subjective and objective measures to investigate whether positive emotional experiences and moments of joy during the interaction can be distinguished from neutral and negative emotional experiences. In a study with 30 participants, electroencephalography (EEG), electrocardiography (ECG), electrodermal activity (EDA), facial emotion recognition, and questionnaires were used. Results: There were greater positive experiences in interactions designed to elicit JoU, even under bad usability. We did not observe a difference between the conditions in the EEG indices. However, a higher heart rate and components in the EDA phasic response as well as facial muscle activity associated with anger were linked to good usability combined with no JoU. Conclusion: The multimodal methods approach reveals great potential to investigate JoU and usability in naturalistic scenarios. Application: The developed framework provides a groundwork to evaluate and improve interactions with technology. Thereby, users and their emotional experiences are placed at the centre when designing user interfaces. By detecting moments of joy, this approach can support a better understanding of how technology can be purposefully designed for joyful experiences.

In what ways could the future of emotional bonds between humans and conversational AI change us? To explore this question in a multi-faceted manner, designers, engineers, philosophers as separate focus groups were given a design fiction probe— a story of a chatbot’s disappearance from a person’s life. Though articulated in discipline-specific ways, participants expressed similar concerns and hopes: 1) caring for a machine could teach people to emotionally care for themselves and others, 2) the boundary between human and non-human emotions may become blurred when people project their own emotions onto AI, e.g., a bot’s ”breakdown” as one’s own, and 3) people may then intertwine their identities with AI through emotions. We consider ethical ramifications of socially constructed emotions between humans and conversational agents.

A variety of chatbots were developed to elicit people’s emotional responses and manage the emotions of people. These chatbots often use text-based emotional sentences. However, there are few studies that chatbots use extremely simplified communications to induce some emotional responses. We investigated the effect of mood contagion and tried to induce positive or negative words from people on our web-based ”word chain game” system as the most simplified conversational style. We also defined the system’s emotion as making the system use words that have polarity, which means each word’s strength of positive or negative emotion. The result shows that users of the system using positive words were slightly influenced to use more positive words than the neutral or negative system.

Human decision making is central in many functions across a broad spectrum of fields such as marketing, investment, smart contract formulations, political campaigns, and organizational strategic management. Behavioral economics seeks to study the psychological, cultural, and social factors contributing to decision making along reasoning. It should be highlighted here that behavioral economics do not negate classical economic theory but rather extend it in two distinct directions. First, a finer granularity can be obtained by studying the decision making process not of massive populations but instead of individuals and groups with signal estimation or deep learning techniques based on a wide array of attributes ranging from social media posts to physiological signs. Second, time becomes a critical parameter and changes to the disposition towards alternative decisions can be tracked with input-output or state space models. The primary findings so far are concepts like bounded rationality and perceived risk, while results include optimal strategies for various levels of information awareness and action strategies based on perceived loss aversion principles. From the above it follows that behavioral economics relies on deep learning, signal processing, control theory, social media analysis, affective computing, natural language processing, and gamification to name only a few fields. Therefore, it is directly tied to computer science in many ways. THECOG will be a central meeting point for researchers of various backgrounds in order to generate new interdisciplinary and groundbreaking results.

We introduce the MuSe-Toolbox - a Python-based open-source toolkit for creating a variety of continuous and discrete emotion gold standards. In a single framework, we unify a wide range of fusion methods and propose the novel Rater Aligned Annotation Weighting (RAAW), which aligns the annotations in a translation-invariant way before weighting and fusing them based on the inter-rater agreements between the annotations. Furthermore, discrete categories tend to be easier for humans to interpret than continuous signals. With this in mind, the MuSe-Toolbox provides the functionality to run exhaustive searches for meaningful class clusters in the continuous gold standards. To our knowledge, this is the first toolkit that provides a wide selection of state-of-the-art emotional gold standard methods and their transformation to discrete classes. Experimental results indicate that MuSe-Toolbox can provide promising and novel class formations which can be better predicted than hard-coded classes boundaries with minimal human intervention. The implementation is out-of-the-box available with all dependencies using a Docker container.

Affect detection in the "real" wild - where people go about their daily routines in their homes and workplaces - is arguably a different problem than affect detection in the lab or in the "quasi" wild (e.g., YouTube videos). How will our affect detection systems hold up when put to the test in the real wild? Some in the Affective Computing community had an opportunity to address this question as part of the MOSAIC (Multimodal Objective Sensing to Assess Individuals with Context [1]) program which ran from 2017 to 2020. Results were sobering, but informative. I'll discuss those efforts with an emphasis on performance achieved, insights gleaned, challenges faced, and lessons learned.

Emotional intelligence is a fundamental component towards a complete and natural interaction between human and machine. Towards this goal several emotion theories have been exploited in the affective computing domain. Along with the studies developed in the theories of emotion, there are two major approaches to characterize emotional models: categorical models and dimensional models. Whereas, categorical models indicate there are a few basic emotions that are independent on the race (e.g. Ekman's model), dimensional approaches suggest that emotions are not independent, but related to one another in a systematic manner (e.g. Circumplex of Affect). Although these models have been dominating in the affective computing research, recent studies in emotion theories have shown that these models only capture a small fraction of the variance of what people perceive. In this talk, I will present the new directions in emotion theory that can better capture the emotional behavior of individuals. First, I will discuss the statistical analysis behind key emotions that are conveyed in human vocalizations, speech prosody, and facial expressions, and how these relate to conventional categorical and dimensional models. Based on these new emotional models, I will describe new datasets we have collected at Hume AI, and show the different patterns captured when training deep neural network models.

Emotion is an inherently subjective psychophysiological human-state and to produce an agreed-upon representation (gold standard) for continuous emotion requires a time-consuming and costly training procedure of multiple human annotators. There is strong evidence in the literature that physiological signals are sufficient objective markers for states of emotion, particularly arousal. In this contribution, we utilise a dataset which includes continuous emotion and physiological signals - Heartbeats per Minute (BPM), Electrodermal Activity (EDA), and Respiration-rate - captured during a stress inducing scenario (Trier Social Stress Test). We utilise a Long Short-Term Memory, Recurrent Neural Network to explore the benefit of fusing these physiological signals with arousal as the target, learning from various audio, video, and textual based features. We utilise the state-of-the-art MuSe-Toolbox to consider both annotation delay and inter-rater agreement weighting when fusing the target signals. An improvement in Concordance Correlation Coefficient (CCC) is seen across features sets when fusing EDA with arousal, compared to the arousal only gold standard results. Additionally, BERT-based textual features' results improved for arousal plus all physiological signals, obtaining up to .3344 CCC compared to .2118 CCC for arousal only. Multimodal fusion also improves overall CCC with audio plus video features obtaining up to .6157 CCC to recognize arousal plus EDA and BPM.

The term “aesthetic experience” corresponds to inner states of individuals exposed to art. Investigating form, content, and aesthetic values of artistic objects, indoor and outdoor spaces, urban areas, and modern interactive technology is essential to improve social behaviour, quality of life, and health of humans in the long term. Quantifying and interpreting the aesthetic experience of art receivers in different contexts can contribute towards (a) creating art and (b) better understanding humans’ affective reactions to aesthetic stimuli. Focusing on different types of artistic content, such as movies, music, urban art, ancient artwork, and modern interactive technology, the goal of the Second International Workshop on Multimodal Affect and Aesthetic Experience is to enhance the interdisciplinary collaboration among researchers from the following domains: affective computing, aesthetics, human-robot interaction, and digital archaeology and art.

Affective social multimedia computing is an emergent research topic for both affective computing and multimedia research communities. Social multimedia is fundamentally changing how we communicate, interact, and collaborate with other people in our daily lives. Comparing with well-organized broadcast news and professionally made videos such as commercials, TV shows, and movies, social multimedia media computing imposes great challenges to research communities. Social multimedia contains much affective information. Effective extraction of affective information from social multimedia can greatly help social multimedia computing (e.g., processing, index, retrieval, and understanding). Although much progress have been made in traditional multimedia research on multimedia content analysis, indexing, and retrieval based on subjective concepts such as emotion, aesthetics, and preference, affective social multimedia computing is a new research area. The affective social multimedia computing aims to proceed affective information from social multi-media. For massive and heterogeneous social media data, the research requires multidisciplinary understanding of content and perceptual cues from social multimedia. This workshop served as a successful step towards this goal and attracted contributions from different research disciplines on the analysis of affective signals in interaction (multimodal analyses enabling artificial agents in Human-Machine Interaction, social Interaction with artificial agents) and social multimedia (e.g., twitter, wechat, weibo, youtube, facebook, etc). This paper provides a summary of the activities of the workshop and the accepted papers and abstracts.

This workshop discusses how interactive, multimodal technology such as virtual agents can be used in social skills training for measuring and training social-affective interactions. Sensing technology now enables analyzing user’s behaviors and physiological signals. Various signal processing and machine learning methods can be used for such prediction tasks. Such social signal processing and tools can be applied to measure and reduce social stress in everyday situations, including public speaking at schools and workplaces.

We present a generative adversarial network to synthesize 3D pose sequences of co-speech upper-body gestures with appropriate affective expressions. Our network consists of two components: a generator to synthesize gestures from a joint embedding space of features encoded from the input speech and the seed poses, and a discriminator to distinguish between the synthesized pose sequences and real 3D pose sequences. We leverage the Mel-frequency cepstral coefficients and the text transcript computed from the input speech in separate encoders in our generator to learn the desired sentiments and the associated affective cues. We design an affective encoder using multi-scale spatial-temporal graph convolutions to transform 3D pose sequences into latent, pose-based affective features. We use our affective encoder in both our generator, where it learns affective features from the seed poses to guide the gesture synthesis, and our discriminator, where it enforces the synthesized gestures to contain the appropriate affective expressions. We perform extensive evaluations on two benchmark datasets for gesture synthesis from the speech, the TED Gesture Dataset and the GENEA Challenge 2020 Dataset. Compared to the best baselines, we improve the mean absolute joint error by 10--33%, the mean acceleration difference by 8--58%, and the Fr\'echet Gesture Distance by 21--34%. We also conduct a user study and observe that compared to the best current baselines, around 15.28% of participants indicated our synthesized gestures appear more plausible, and around 16.32% of participants felt the gestures had more appropriate affective expressions aligned with the speech., Comment: 11 pages, 4 figures, 2 tables. Proceedings of the 29th ACM International Conference on Multimedia, October 20-24, 2021, Virtual Event, China

The research on human emotion under multimedia stimulation based on physiological signals is an emerging field, and important progress has been achieved for emotion recognition based on multi-modal signals. However, it is challenging to make full use of the complementarity among spatial-spectral-temporal domain features for emotion recognition, as well as model the heterogeneity and correlation among multi-modal signals. In this paper, we propose a novel two-stream heterogeneous graph recurrent neural network, named HetEmotionNet, fusing multi-modal physiological signals for emotion recognition. Specifically, HetEmotionNet consists of the spatial-temporal stream and the spatial-spectral stream, which can fuse spatial-spectral-temporal domain features in a unified framework. Each stream is composed of the graph transformer network for modeling the heterogeneity, the graph convolutional network for modeling the correlation, and the gated recurrent unit for capturing the temporal domain or spectral domain dependency. Extensive experiments on two real-world datasets demonstrate that our proposed model achieves better performance than state-of-the-art baselines., Accepted by ACM MM 2021. The SOLE copyright holder is ACM Multimedia, all rights reserved

In this art project, we create Affective Color Fields: an interactive artifact that takes in a user's narrative of their emotional experiences and dynamically transforms it into Rothkoesque color fields through emotion classification. Inspired by Mark Rothko's abstract depiction of human emotions and Merleau-Ponty's phenomenological inquiry, we wish to establish an intimate relationship between interactive art and the subject by employing user's own interpretation and framing of life events. Through the performative and improvisational art-making process, users can playfully appropriate our artifact for a rich and personal aesthetic experience.

Among all solutions of emotion recognition tasks, electroencephalogram (EEG) is a very effective tool and has received broad attention from researchers. In addition, information across multimedia in EEG often provides a more complete picture of emotions. However, few of the existing studies concurrently incorporate EEG information from temporal domain, frequency domain and functional brain connectivity. In this paper, we propose a Multi-Domain Adaptive Graph Convolutional Network (MD-AGCN), fusing the knowledge of both the frequency domain and the temporal domain to fully utilize the complementary information of EEG signals. MD-AGCN also considers the topology of EEG channels by combining the inter-channel correlations with the intra-channel information, from which the functional brain connectivity can be learned in an adaptive manner. Extensive experimental results demonstrate that our model exceeds state-of-the-art methods in most experimental settings. At the same time, the results show that MD-AGCN could extract complementary domain information and exploit channel relationships for EEG-based emotion recognition effectively.

Facial action unit (AU) recognition has attracted increasing attention due to its indispensable role in affective computing, especially in the field of affective human-computer interaction. Due to the subtle and transient nature of AU, it is challenging to capture the delicate and ambiguous motions in local facial regions among consecutive frames. Considering that context is essential to resolve ambiguity in human visual system, modeling context within or among facial images emerges as a promising approach for AU recognition task. To this end, we propose CaFGraph, a novel context-aware facial multi-graph that can model both morphological & muscular-based region-level local context and region-level temporal context. CaFGraph is the first work to construct a universal facial multi-graph structure that is independent of both task settings and dataset statistics for almost all fine-grained facial behavior analysis tasks, including but not limited to AU recognition. To make full use of the context, we then present CaFNet that learns context-aware facial graph representations via CaFGraph from facial images for multi-label AU recognition. Experiments on two widely used benchmark datasets, BP4D and DISFA, demonstrate the superiority of our CaFNet over the state-of-the-art methods.

Recognizing human emotions from videos has attracted significant attention in numerous computer vision and multimedia applications, such as human-computer interaction and health care. It aims to understand the emotional response of humans, where candidate emotion categories are generally defined by specific psychological theories. However, with the development of psychological theories, emotion categories become increasingly diverse and fine-grained, samples are also increasingly difficult to collect. In this paper, we investigate a new task of zero-shot video emotion recognition, which aims to recognize rare unseen emotions. Specifically, we propose a novel multimodal protagonist-aware transformer network, which is composed of two branches: one is equipped with a novel dynamic emotional attention mechanism and a visual transformer to learn better visual representations; the other is an acoustic transformer for learning discriminative acoustic representations. We manage to align the visual and acoustic representations with semantic embeddings of fine-grained emotion labels through jointly mapping them into a common space under a noise contrastive estimation objective. Extensive experimental results on three datasets demonstrate the effectiveness of the proposed method.

Menti’s Journey is a singleplayer browser game for elementary school children aged 6-10. As the main character Menti, the player gradually tries to regain their lost emotions and reveals the power associated with them. Accompanied by Super-Menti, the journey leads across several islands where one of the emotions is out of control. Through a camera-controlled emotion recognition software, the children can test their newly discovered emotion powers in mini-games and through interactions with objects in the world to finally regain all lost emotions.

Recognizing emotions through pervasive technologies has been the objective of several research studies. Still, the inherent complexities of emotion recognition result in applications that support the process rather than fully address it. Existing methods for data collection combine multiple sensors and devices. The analysis and classification of users’ and environmental data face challenges to reach high accuracy, responsiveness, and reliability. Because emotion recognition is user-dependent, it still remains as an open challenge to be resolved. This paper provides a unified view about emotion recognition, presenting a multidimensional design space that includes multiple approaches. We discuss the practicalities involved with the implementation of each approach and the deployment of emotion recognition systems. We conclude with a discussion about their advantages, limitations, and open questions.

The paper presents the emteqPROtm system, which uses a face-mounted multi-sensor mask to measure the facial physiological responses, facial muscle activations, and motions from the user. These responses are then analyzed by machine learning algorithms to recognize and better understand the user's affective state and context, i.e., emotions, arousal, valence, stress response, activities, etc. The system can work by itself, as an open mask, or can be combined with a commercial Virtual Reality head mounted display. It comprises 3 sensor modalities: a 7-contact f-EMG sensor, a PPG sensor, and a 3-axis IMU, enabling it to measure the affective state of the user in real time. We will demonstrate how the system is used in practice in a Virtual Reality environment. This newly developed technology has the potential to significantly improve the way we collect data, design experiences, and interact within Virtual, Mixed and Augmented Realities.

Emotion detection research has gained significant attention across different scientific disciplines, including Human-Computer Interaction and Ubiquitous Computing. In this work, we provide an overview of challenges that researchers and designers face when designing and deploying emotion detection systems. We discuss these challenges in the context of an online video conferencing tool we designed to detect emotions from facial expressions. Finally, we propose ideas on how to better design emotion inference systems and how to provide visual feedback on emotions.

Conversational agents are getting increasingly popular and find applications in health and customer services. Conversations in these fields are often emotionally charged. It is, therefore, necessary to handle the conversation with some degree of empathy to be effective. In this work, we leverage advances in the field of natural language processing to create a dialogue system that can convincingly generate empathic responses to text-based messages. To improve the system's ability to converse with empathy, we train the language model on empathic conversations and inject additional emotional information in the response generation. We propose two chatbots: a benchmark bot and an empathic bot. Additionally, we implement an emotion classifier that allows us to predict the emotional state of text-based messages. We evaluate both chatbots in quantitative studies and compare them with human responses in qualitative studies involving human judges. Our evaluation shows that our empathic chatbot outperforms the benchmark bot and even the human-generated responses in terms of perceived empathy. Additionally, we achieve state-of-the-art results in terms of response quality using transformer-based language models. Finally we report that we can double the initial performance of the emotion classifier using undersampling techniques, yielding a final F1-score of 0.81 in six basic emotions.

Every year, in Europe alone, hundreds of workers die by falling from high height. This number could be greatly reduced by means of better training and quick detection of individuals with issues toward work at height. Workers proving to be less suited for the job can be subject to more intensive training or recruited for different positions. Unfortunately, the early detection of workers unsuited for working at height involves specialized personnel and expensive equipment to recreate a stressful environment. In this paper we propose a methodology to predict fear of heights by means of a virtual reality environment. We demonstrate that a 3D virtual environment is feasible for the prediction and give guidelines about meaningful physiological parameters useful for detection.

In this study, we focused on the fact that the color information of an image has a significant effect on the emotion recalled, and we created a dataset with discrete and continuous emotion labels for color and grayscale image pairs, which are not present in existing emotion image datasets. As a result of the analysis of the continuous-valued emotion labels, we found that the color images evoked a wider range of positive and negative emotions. We also conducted experiments to transform the color of the image to the target emotion using the emotion label and the image as input and succeeded in changing the color of the entire image according to the input emotion value. Another experiment is conducted to convert the emotion category and we confirmed that the image was generated in such a way that the evoked emotion was changed compared to the original image.

Traces of searcher behaviour, such as query reformulation or clicks, are commonly used to evaluate a running search engine. The underlying expectation is that these behaviours are proxies for something more important, such as relevance, utility, or satisfaction. Affective computing technology gives us the tools to help confirm some of these expectations, by examining visceral expressive responses during search sessions. However, work to date has only studied small populations in laboratory settings and with a limited number of contrived search tasks. In this study, we analysed longitudinal, in-situ, search behaviours of 152 information workers, over the course of several weeks while simultaneously tracking their facial expressions. Results from over 20,000 search sessions and 45,000 queries allow us to observe that indeed affective expressions are consistent with, and complementary to, existing "click-based'' metrics. On a query-level, searches that result in a short dwell time are associated with a decrease in smiles (expressions of "happiness'') and that if a query is reformulated the results of the reformulation are associated with an increase in smiling---suggesting a positive outcome as people converge on the information they need. On a session-level, sessions that feature reformulations are more commonly associated with fewer smiles and more furrowed brows (expressions of "anger/frustration''). Similarly, sessions with short-dwell clicks are also associated with fewer smiles. These data provide an insight into visceral aspects of search experience and present a new dimension for evaluating engine performance.

This paper presents a real-time emotion-annotation tool using a personal mobile device. To this end, an application based on the Valence-Arousal model was developed, following two different approaches (”Two-step Sequential Annotation” and ”One-step Matrix Annotation”). The application was tested through, an experiment where users performed annotations with each version of the app and then filled a feedback questionnaire. This questionnaire contained statements used to assess the usability and mental workload. A total of 16 (9 female) participants aged between 21 and 34 years old engaged in this experiment. Overall the results were quite encouraging in both versions, taking into account that this is still a work in progress. The ”Single-step Matrix Annotation” was considered the preferred version.

Artificial intelligence has been applied to different sectors in the government from data analysis to predictive analytics, as well as, policing, combating the COVID-19 pandemic, and political election campaigns. In the academic field, the exploitation and understanding of data generated in social media have mostly focused on unimodal sentiment analysis, based on one-dimensional sentiment analysis. This research proposes an analysis of the emotional charge for the U.S. presidential elections in 2020, based on a hybrid approach that combines affective computing and classic statistical analysis. We analyze the multi-emotional charge of candidates and voters, as well as the potential relationship of the candidates' emotions on the voters. Through this analysis is possible to determine the degree of agreement between candidates and voters. Future research is proposed for the area of affective computing in political campaigns.

Emotion recognition, especially facial expression recognition (FER), has played a vital role in understanding human cognition. The current work focuses on the classification, learning and analysis of the six basic emotions (happy, sadness, fear, disgust, anger, and surprise) and other in-depth research fields. However, from the perspective of psychology, human emotions are subjective and complex, and the definition of emotion categories is also controversial, which has an important impact on the accuracy of the analysis results. This paper focuses on the basic issues of emotion classification, presets the position of complex emotions, and uses the improved k-means clustering algorithm to reclassify emotion categories with different emotions based on the subjective voting results of the FER+ face emotion data set. The recognition accuracy is used as objective data to classify the subjective emotion categories, and finally, the recognition accuracy of the emotion classification categories is used as a verification method to prove that the reclassified emotion categories can significantly improve the results of its classification, learning and analysis.

Recognizing human emotions and responding appropriately has the potential to radically change the way we interact with technology. However, to train machines to sensibly detect and recognize human emotions, we need valid emotion ground truths. A fundamental challenge here is the momentary emotion elicitation and capture (MEEC) from individuals continuously and in real-time, without adversely affecting user experience nor breaching ethical standards. In this virtual half-day CHI 2021 workshop, we will (1) have participant talks and an inspirational keynote presentation (2) ideate elicitation, sensing, and annotation techniques (3) create mappings of when to apply an elicitation method.

Interactive computing systems are able to receive, as inputs, activity generated by the user’s physiology (e.g., skin conductance, heart rate, brain potentials, and so forth). Besides health-related applications, this type of physiological sensing enables systems to infer users’ states (e.g., task engagement, anxiety, workload, and so forth). More recently, a number of techniques emerged that can also stimulate physiological activity (e.g., electrical muscle stimulation, galvanic vestibular stimulation, transcranial stimulation). These can serve as outputs of an interactive system to induce desired behavior in the user. Taken together, we envision systems that will close the loop between physiological input and output—interactive systems able to read and influence the user’s body. To realize this, we propose a Special Interest Group on Physiological I/O that will consolidate successful practices and identify research challenges to address as a community.

We conducted a daytime naturalistic driving study that involved the same 19 km town itinerary under similar light traffic and fair-weather conditions. We applied a real-time unobtrusive design that could serve as template in future driving studies. In this design, driving parameters and drivers’ arousal levels were captured via a vehicle data acquisition and thermal imaging system, respectively. Analyzing the data, we found that about half of the n = 11 healthy participants exhibited significantly larger arousal reactions to acceleration with respect to the rest of the sample. Acceleration events were of the mundane type, such as entering a highway from an entrance ramp or starting from a red light. The results suggest an underlying grouping of normal drivers with respect to the loading induced by commonplace acceleration. The finding carries implications for certain professions and the design of semi-autonomous vehicles.

Smart devices have pervaded every aspect of humans’ daily lives. Though single device UX products are relatively successful, the experience of cross-device interaction is still far from satisfactory and can be a source of frustration. Inconsistent UI styles, unclear coordination, varying fidelity, pairwise interactions, lack of understanding intent, limited data sharing and security, and other problems typically degrade the experience in a multi-device ecosystem. Redesigning the UX, tailored to multi-device ecosystems to enhance the user experience, turns out to be challenging but at the same time affording many new opportunities. This workshop brings together researchers, practitioners and developers with different backgrounds, including from fields such as computationally design, affective computing, and multimodal interaction to exchange views, share ideas, and explore future directions on UX for distributed scenarios, especially for those heterogeneous cross-device ecosystems. The topics cover but are not limited to distributed UX design, accessibility, cross-device HCI, human factors in distributed scenarios, user-centric interfaces, and multi-device ecosystems.

Touch plays an essential role in communicating emotions and intensifying interpersonal communication[4]. A lot of research focuses on how to create or improve haptic interfaces looking into challenges and possibilities that the haptic technology can offer[30]. The objective of this research is to investigate whether people can share subjective feelings through simple vibrotactile feedback. In an initial experiment, we used the TECHTILE toolkit[19] to record 28 vibration sample sets for 4 different emotions (joy, anger, sadness, relaxation). We then replayed the vibrations to test how well they could be recognized. The results support the hypothesis that people can use vibration feedback as a medium for expressing specific subjective feelings. It also indicates some universalities in affective vibrotactile stimuli that even strangers with little to no knowledge about the senders could recognize the emotional meanings.

Music is a universal medium that can elicit strong emotion, and can significantly help us in gaining focus while doing specific tasks. However, it is unclear what types of music can help to improve focus while doing other activities. In this paper, we investigate the effects of six different music stimuli on participants’ verbal and physiological responses while identifying genuine and acted emotions from video clips. Initial analysis was conducted on the comments participants made on the different stimuli in order to identify emerging patterns. Then, participants’ verbal and EEG responses were collected, processed and analyzed to classify two types of emotion. Empirical analysis of the results show that binaural beats, which are believed to increase focus on tasks, can often cause discomfort and therefore hinder focus. On the other hand, music containing a sombre tone, or familiar popular music with high level valence can help improve focus. Identifying which music stimuli can improve focus can be highly beneficial in managing day-to-day tasks and activities. This study will also be useful in broadening the range of music stimuli used in affective computing studies.

Drones are rapidly populating human spaces, yet little is known about how these flying robots are perceived and understood by humans. Recent works suggested that their acceptance is predicated upon their sociability. This paper explores the use of facial expressions to represent emotions on social drones. We leveraged design practices from ground robotics and created a set of rendered robotic faces that convey basic emotions. We evaluated individuals’ response to these emotional facial expressions on drones in two empirical studies (N = 98, N = 98). Our results demonstrate that individuals accurately recognize five drone emotional expressions, as well as make sense of intensities within emotion categories. We describe how participants were emotionally affected by the drone, showed empathy towards it, and created narratives to interpret its emotions. As a consequence, we formulate design recommendations for social drones and discuss methodological insights on the use of static versus dynamic stimuli in affective robotics studies.

Expressivity is frequently recurring as a term in HCI, but it is often approached from different perspectives. Affective computing prompted research into emotional expressivity, and with technology becoming more ubiquitous and tangible, the opportunities for expressive behaviors towards systems as well as the expressivity of systems increases. By analyzing exemplar research-through-design cases and a literature survey on the use of expressivity in interaction, we discuss how different perspectives and concepts contribute to understand expressivity in interaction. We integrate these perspectives and make them operational for interaction design by creating a framework including design considerations such as freedom of interaction, action-perception loops, multimodality, subtlety, ambiguity, skill development and temporal form. The framework is a result of a mixed-method approach including a review of existing definitions and scholarly artefacts, and a systematic literature review to identify design cases including an analysis of these design cases. We finally illustrate how the framework has been used to inform the design of a shape-changing soft-robotic interface. As a result, we contribute an integrated framework on how to design for expressivity in interaction.

Emotional regulation plays an important role in the process of affective computing and interactive system. Although studies have confirmed that music can be used as an effective medium for regulating emotions, it does not explain the effects of regulation from different dimensions of emotions. In this study, we collected electroencephalogram (EEG) data of 40 participants while regulating negative emotions by positive music. A valence-arousal model was trained on EEG features through machine learning. The model provided a binary prediction of valence (high or low) of 78.759.48% and 73.985.54% for arousal. Our results confirmed the function of positive music in emotion improvement. More attention should be paid to the value of music in emotional regulation arousal in clinical practice.

In this paper, we develop a taxonomy of conceptual models and proxy data used for digital analysis of human emotional expression and outline how the combinations and permutations of these models and data impact their incorporation into artificial intelligence (AI) systems. We argue we should not take computer scientists at their word that the paradigms for human emotions they have developed internally and adapted from other disciplines can produce ground truth about human emotions; instead, we ask how different conceptualizations of what emotions are, and how they can be sensed, measured and transformed into data, shape the ethical and social implications of these AI systems.

“Footsie” is an interactive kinetic art installation. It inspires to connect people through the counter-intuitive physical interaction that is mediated by flirtatious machines. It consists of a table and four robotic chairs; the chair legs carry the motion of “footsie” and arms caress the back of the participant. While it tackles the controversial culture and social norms on touching others body or being touched, an experimental bodily experience explores new aesthetic interaction between lonely human and a mischievous digital entity. The interaction is provocative in a slightly uncomfortable way, and the paradoxical feelings and stimulating sensations open simultaneous interpretations on the meaning of Human-Machine-Interaction. Through nature inspired design like human spine, flexible muscle of mollusca and a caterpillar’s undulating wave motion, these pliable structures were fabricated with 3D printed meta-materials to create the sensual gestures.

Wearables equipped with pervasive sensors enable us to monitor physiological and behavioral signals in our everyday life. We propose the WellAff system able to recognize affective states for wellbeing support. It also includes health care scenarios, in particular patients with chronic kidney disease (CKD) suffering from bipolar disorders. For the need of a large-scale field study, we revised over 50 off-the-shelf devices in terms of usefulness for emotion, stress, meditation, sleep, and physical activity recognition and analysis. Their usability directly comes from the types of sensors they possess as well as the quality and availability of raw signals. We found there is no versatile device suitable for all purposes. Using Empatica E4 and Samsung Galaxy Watch, we have recorded physiological signals from 11 participants over many weeks. The gathered data enabled us to train a classifier that accurately recognizes strong affective states., Accepted to the International Workshop on Artificial Intelligence for Mobile and Ubiquitous Communication System, EAI MobiQuitous 2020

With the development of online education, the problem of emotional deficiency has gradually emerged. In recent years, the research about emotional interaction in online education, and the application of Affective Computing in online education came into being. This paper introduces the impact of Affective Computing on online education, including the use of expressions, speech, eye movement, and physiological signals, as well as text and questionnaires and other modals, or multi-modal to infer students' emotional state and learning state in the learning process, and then to realize personalized learning support. The emerging technologies such as Artificial Intelligence, Big Data and Brain Computer Interface, can be applied to online education combined with Affective Computing technologies to improve the interaction between learners and the on-line learning system. This paper proposes a framework of online learning and experiment system based on Affective Computing supported by Artificial Intelligence, Big Data and Brain Computer Interface. The system can provide personalized education for students and help them to carry out experiments in a virtual reality environment. The framework can reduce the impact on students in the process of data collection, so that it can obtain more real data, and the framework uses emerging technologies to provide support and reference for emotion analysis and recognition, which will make the analysis and response more timely and emotion recognition more accurate.

Pediatric chronic illnesses continue to rise in the U.S. Prepubescent children often develop delays in learning about their health that can lead to unhealthy habits, mental health problems, and feelings that managing their disease is almost impossible. With health and emotional support tools often being inaccessible to families, the need for an accessible, mainstream, and fun emphatic health and wellness learning tool (EHWLT) has increased. This work-in-progress describes a web-based children's tool known as Sydney that is designed to teach young children how to manage their chronic illness symptoms using playful stories.

The automatic and real-time recognition of the user's emotional state is a feature that can provide benefits for different areas of Human-Computer Interaction. The scientific literature presents several techniques that can be used to recognize the user's emotional state. However, many techniques involve the use of sensors that can result in financial costs and cause discomfort to the user. In this scenario, the recognition of the emotional state through the analysis of facial expressions presents itself as a useful and practical approach, since it does not involve the use of sensors attached to the user's body and executed in different types of devices. Despite these advantages, software that allow the analysis of facial expressions for free are still incipient, and performance evaluation of this type of software usually is not available. In order to contribute to this context and assist researchers who need this type of software, this study presents a comparative analysis of two open-source emotion recognition software ("CLMTrackr" and "Face-api.js") simulating different environmental conditions related to lighting and distance. Considering images from two datasets, we generate 8675 videos simulating 25 different environmental conditions. Our results indicate that the environmental conditions did not cause major impacts on the accuracy of the software, and CLMTrackr and Face-api.js, presented, respectively, 28% and 64% of average accuracy.

Extensive couples' literature shows that how couples feel after a conflict is predicted by certain emotional aspects of that conversation. Understanding the emotions of couples leads to a better understanding of partners? mental well-being and consequently their relationships. Hence, automatic emotion recognition among couples could potentially guide interventions to help couples improve their emotional well-being and their relationships. It has been shown that people's global emotional judgment after an experience is strongly influenced by the emotional extremes and ending of that experience, known as the peak-end rule. In this work, we leveraged this theory and used machine learning to investigate, which audio segments can be used to best predict the end-of-conversation emotions of couples. We used speech data collected from 101 Dutch-speaking couples in Belgium who engaged in 10-minute long conversations in the lab. We extracted acoustic features from (1) the audio segments with the most extreme positive and negative ratings, and (2) the ending of the audio. We used transfer learning in which we extracted these acoustic features with a pre-trained convolutional neural network (YAMNet). We then used these features to train machine learning models - support vector machines - to predict the end-of-conversation valence ratings (positive vs negative) of each partner. The results of this work could inform how to best recognize the emotions of couples after conversation-sessions and eventually, lead to a better understanding of couples? relationships either in therapy or in everyday life., ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction, ISBN:978-1-4503-8002-7

Emotions are subjective experiences involving perceptual and con-textual factors [4]. There is no objective tool for precise measurement of emotions. However, we can anticipate an emotion's emergence through the knowledge of common responses to events in similar situations. We can also measure proxies of emotions by recognizing emotional expressions [3]. Studying emotional response to multimedia allows identifying expected emotions in users consuming the content. For example,abrupt loud voices are novel and unsettling which result in surprise and higher experience of arousal [2,6]. For a particular type of con-tent such as music, mid-level attributes such as rhythmic stability or melodiousness have strong association with expected emotions[1]. Given that such mid-level attributes are more related to the con-tent, their machine-perception is more straightforward. Moreover,their perception in combination with user models enables building person-specific emotion anticipation models.In addition to studying expected emotions, we can also observe users emotional reactions to understand emotion in multimedia.Typical methods of emotion recognition include recognizing emotions from facial or vocal expressions. Recognition of emotional expressions requires large amount of labeled data, expensive to produce. Hence, the most recent advances in machine-based emotion perception include methods that can leverage unlabeled data through self-supervised and semi-supervised learning [3, 5]. In this talk, I review the field and showcase methods for automatic modeling and recognition of emotions and sentiment indifferent contexts [3,8]. I show how we can identify underlying factors contributing to the construction of subjective experience of emotions [1,7]. Identification of these factors allows us to use them as mid-level attributes to build machine learning models for emotion and sentiment understanding. I also show how emotions and sentiment can be recognized from expressions with the goal of building empathetic autonomous agents [8].

Recognizing the emotions of the elderly is important as it could give an insight into their mental health. Emotion recognition systems that work well on the elderly could be used to assess their emotions in places such as nursing homes and could inform the development of various activities and interventions to improve their mental health. However, several emotion recognition systems are developed using data from younger adults. In this work, we train machine learning models to recognize the emotions of elderly individuals via performing a 3-class classification of valence and arousal as part of the INTERSPEECH 2020 Computational Paralinguistics Challenge (COMPARE). We used speech data from 87 participants who gave spontaneous personal narratives. We leveraged a transfer learning approach in which we used pretrained CNN and BERT models to extract acoustic and linguistic features respectively and fed them into separate machine learning models. Also, we fused these two modalities in a multimodal approach. Our best model used a linguistic approach and outperformed the official competition of unweighted average recall (UAR) baseline for valence by 8.8% and the mean of valence and arousal by 3.2%. We also showed that feature engineering is not necessary as transfer learning without fine-tuning performs as well or better and could be leveraged for the task of recognizing the emotions of elderly individuals. This work is a step towards better recognition of the emotions of the elderly which could eventually inform the development of interventions to manage their mental health., ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction, ISBN:978-1-4503-8002-7

The emotions we experience shape our perception, and our emotion is shaped by our perceptions. Taste perception is also influenced by emotions. Positive and negative emotions alter sweetness, sourness, and bitterness perception. However, most previous studies mainly explored valence changes. The effect of arousal on taste perception is less studied. In this study, we asked volunteers to watch positive affect inducing videos with high arousal and low arousal. Our results showed a successful induction of high and low arousal levels as confirmed by self-report and electrophysiological signals. Moreover, self-report affective ratings did not show a significant effect on self-reported taste ratings. However, we found a negative correlation between smile occurrence and sweetness ratings. In addition, EDA scores were positively correlated with saltiness. This suggests that even if the self-reported affective state is not granular enough, looking at more fine-grained affective cues can inform ratings of taste.