Search

Validation of the EmojiGrid self-report tool for the affective appraisal of odorants

Addtional data files with: A. Toet, & J.B.F. van Erp. 2020. Affective rating of audio and video clips using the EmojiGrid. F1000Research, 9 (970) https://doi.org/10.12688/f1000research.25088.1. The file sound_results.xlsx contains the SAM and EmojiGrid valence and arousal ratings for selected stimuli from the IADS-E database by Yang et al. (2018). The file video_results.xlsx contains the SAM and EmojiGrid valence and arousal ratings for video clips from Aguado et al. (2018)

Validation of the EmojiGrid as an intuitive, immersive and efficient tool to assess emotions evoked by immersive VR experiences

Results of studies performed to validate the EmojiGrid as a graphical self-report tool to assess experienced end perceived valence and arousal.

Background: In this study we measured the affective appraisal of sounds and video clips using a newly developed graphical self-report tool: the EmojiGrid. The EmojiGrid is a square grid, labeled with emoji that express different degrees of valence and arousal. Users rate the valence and arousal of a given stimulus by simply clicking on the grid. Methods: In Experiment I, observers (N=150, 74 males, mean age=25.2±3.5) used the EmojiGrid to rate their affective appraisal of 77 validated sound clips from nine different semantic categories, covering a large area of the affective space. In Experiment II, observers (N=60, 32 males, mean age=24.5±3.3) used the EmojiGrid to rate their affective appraisal of 50 validated film fragments varying in positive and negative affect (20 positive, 20 negative, 10 neutral). Results: The results of this study show that for both sound and video, the agreement between the mean ratings obtained with the EmojiGrid and those obtained with an alternative and validated affective rating tool in previous studies in the literature, is excellent for valence and good for arousal. Our results also show the typical universal U-shaped relation between mean valence and arousal that is commonly observed for affective sensory stimuli, both for sound and video. Conclusions: We conclude that the EmojiGrid can be used as an affective self-report tool for the assessment of sound and video-evoked emotions.

This study investigates the relation between several factors that are known to influence the affective appraisal of odors through probabilistic network analysis.

This site provides the images needed to implement the EmojiGrid affective self-report tool.

Questionnaire to measure the quality of mediated social communication

Recent improvements in Extended Reality (XR) technology created an increase in XR products and solutions in the industry, while raising new requirements for new or improved architectural concepts. This need can be particularly complex as XR applications often relate both to 3D geometric rendering and multimedia paradigms. This paper outlines the main concepts relevant to XR, both from a game engineering and multimedia streaming system perspective. XR requires new metadata and media/game orchestration to allow complex interaction between users, objects, and (volumetric) multimedia content, which also results in new requirements on synchronisation (i.e., for global object state and positioning). Furthermore, the paper presents the functional blocks needed in new XR system architectures and how they will glue both (game and media) spaces together. The discussion of functional components and architecture relates to the ongoing activities in relevant standardisation bodies like Khronos, MPEG, and 3GPP. To make XR successful on the long term, the industry needs to agree on interoperable solutions and how to merge both game and media paradigms to allow complex multi-user XR applications.

We recently developed the Holistic Social Presence Questionnaire (HSPQ) to measure the quality of mediated social communication experiences. Initial re-search confirmed the content and face validity of the HSPQ. This study investi-gates the convergent validity and sensitivity of the HSPQ. Participants completed a decision-making task in groups using either Microsoft Teams or Mibo to com-municate with each other. Upon completing the task, participants rated items from the HSPQ and the validated Networked Minds Questionnaire (NMQ). We ex-pected that Mibo would induce a stronger sense of social presence than MS Teams, since it involves self-movement, orientation, group-forming and spatial audio. The HSPQ showed convergent validity with the NMQ: the ratings on both questionnaires were significantly correlated. However, both the NMQ and HSPQ indicated that participants experienced no significant difference in social presence between both conditions. Hence, further research involving a more immersive communication tool that induces a stronger sense of social presence is needed to assess the sensitivity of the HSPQ.

In our digital age, human social communication is increasingly mediated. However, mediated social communication (MSC) systems will only become a viable alternative for in-person social interactions when they reliably and intuitively convey all relevant social and spatial cues needed by communication partners to establish effective communication, collaboration, mutual understanding, and trust. A lot of research has been done on mediated communication in general, and on the effect of social cues in particular. However, this research typically focuses on particular social cues, investigating effects of implementations in particular situations. In addition, the research often focuses on answering psychological or cognitive questions. The translation to design questions such as functional and technical requirements is often left to the developers of the technical systems. This leaves a gap between psychological research results and a translation towards practical guidelines for designers and developers. This paper aims to make a first step towards guidelines for the implementation of social cues for social XR implementations.

We investigated how different graphical representations convey the underlying uncertainty distribution in ensemble predictions. In ensemble predictions, a set of forecasts is produced, indicating the range of possible future states. Adopting a use case from life sciences, we asked non-expert participants to compare ensemble predictions of the growth distribution of individual children to that of the normal population. For each individual child, the historical growth data of a set of 20 of its best matching peers was adopted as the ensemble prediction of the child’s growth curve. The ensemble growth predictions were plotted in seven different graphical formats (an ensemble plot, depicting all 20 forecasts and six summary representations, depicting the peer group mean and standard deviation). These graphs were plotted on a population chart with a given mean and variance. For comparison, we included a representation showing only the initial part of the growth curve without any future predictions. For 3 months old children that were measured at four occasions since birth, participants predicted their length at the age of 2 years. They compared their prediction to either (1) the population mean or to (2) a “normal” population range (the mean ± 2(standard deviation)). Our results show that the interpretation of a given uncertainty visualization depends on its visual characteristics, on the type of estimate required and on the user’s numeracy. Numeracy correlates negatively with bias (mean response error) magnitude (i.e. people with lower numeracy show larger response bias). Compared to the summary plots that yield a substantial overestimation of probabilities, and the No-prediction representation that results in quite variable predictions, the Ensemble representation consistently shows a lower probability estimation, resulting in the smallest overall response bias. The current results suggest that an Ensemble or “spaghetti plot” representation may be the best choice for communicating the uncertainty in ensemble predictions to non-expert users.

Immersive 360° VR systems are increasingly used in entertainment, marketing and design and development processes. Many of these applications involve emotional experiences. Since momentary emotions significantly determine a user’s response and decisions, it is essential to understand the influence of media content and technological factors on the user’s emotional response. To measure the emotional responses to immersive VR experiences, efficient and validated instruments are required. Most tools currently used to measure the emotional response of users compromise the immersive experience since they are cognitively demanding, time consuming, and their application requires the user to leave the VR. We investigated the validity of an immersive, efficient and intuitive EmojiGrid graphical self-report tool for the assessment of emotions evoked by 360º VR videos. Using the EmojiGrid, 40 participants rated 62 360º VR videos from a validated public database. The resulting mean valence and arousal values agree with the corresponding values provided with the database (obtained with an alternative validated rating tool). We conclude that the EmojiGrid is a valid self-report tool for the assessment of VR-evoked emotions.

Introduction: The main purpose of this study was to investigate the overall relation between the mean (at the nomothetic or group level) subjective valence and arousal ratings for odors. Although well established in other sensory modalities (e.g., visual, auditory, gustatory, tactile), this relation has not previously been investigated for odors covering a large range of the valence dimension. In addition, we evaluated the EmojiGrid (a recently introduced intuitive graphical affective self-report tool) for the affective appraisal of odors. Methods: Young and healthy participants (N = 56, 32 females) used the EmojiGrid to rate the perceived valence and arousal for 40 different and randomly presented odors, ranging in valence from unpleasant to pleasant. Results: The overall relation between mean valence and arousal can be described by a U-shaped (quadratic) form; odors scoring near neutral on mean valence have the lowest mean arousal ratings, while odors scoring either high (pleasant) or low (unpleasant) on mean valence show higher mean arousal ratings. The results for odors that were also used in previous studies in the literature agree with their earlier reported values. Conclusion: Mean arousal ratings increase with (positive or negative) emotional valence. Participants intuitively used the EmojiGrid to report their affective appraisal of odors without any verbal labels or written instructions. Implications: The current findings are relevant for various applications and environments (e.g., public, retail, entertainment) where odors are used to induce desired emotional states (e.g., relaxation, arousal) and behaviors. The EmojiGrid can efficiently be applied to assess whether specific odors evoke the desired subjective affective experiences.

We evaluated the convergent validity of the new language-independent EmojiGrid rating tool for the affective appraisal of perceived touch events. The EmojiGrid is a rectangular response grid, labeled with facial icons (emoji) that express different degrees of valence and arousal. We previously showed that participants can intuitively and reliably report their affective appraisal of different sensory stimuli (e.g., images, sounds, smells) by clicking on the EmojiGrid, without additional verbal instructions. However, because touch events can be bidirectional and are a dynamic expression of action, we cannot generalize previous results to the touch domain. In this study, participants (N = 65) used the EmojiGrid to report their own emotions when looking at video clips showing different touch events. The video clips were part of a validated database that provided corresponding normative ratings (obtained with a 9-point SAM scale) for each clip. The affective ratings for inter-human touch obtained with the EmojiGrid show excellent agreement with the data provided in the literature (intraclass correlations of .99 for valence and .79 for arousal). For object touch events, these values are .81 and .18, respectively. This may indicate that the EmojiGrid is more sensitive to perspective (sender versus receiver) than classic tools. Also, the relation between valence and arousal shows the classic U-shape at the group level. Thus, the EmojiGrid appears to be a valid graphical self-report instrument for the affective appraisal of perceived touch events, especially for inter-human touch.

Immersive 360º VR systems are increasingly used in entertainment, marketing and design and development processes. Many of these applications involve emotional experiences. Since momentary emotions significantly determine a user’s response and decisions, it is essential to understand the influence of media content and technological factors on the user’s emotional response. To measure the emotional responses to immersive VR experiences, efficient and validated instruments are required. Most tools currently used to measure the emotional response of users compromise the immersive experience since they are cognitively demanding, time consuming, and their application requires the user to leave the VR. We investigated the validity of an immersive, efficient and intuitive EmojiGrid graphical self-report tool for the assessment of emotions evoked by 360º VR videos. Using the EmojiGrid, 40 participants rated 62 360º VR videos from a validated public database. The resulting mean valence and arousal values agree with the corresponding values provided with the database (obtained with an alternative validated rating tool). We conclude that the EmojiGrid is a valid self-report tool for the assessment of VR-evoked emotions

Immersive 360º VR systems are increasingly used in entertainment, marketing and design and development processes. Many of these applications involve emotional experiences. Since momentary emotions significantly determine a user’s response and decisions, it is essential to understand the influence of media content and technological factors on the user’s emotional response. To measure the emotional responses to immersive VR experiences, efficient and validated instruments are required. Most tools currently used to measure the emotional response of users compromise the immersive experience since they are cognitively demanding, time consuming, and their application requires the user to leave the VR. We investigated the validity of an immersive, efficient and intuitive EmojiGrid graphical self-report tool for the assessment of emotions evoked by 360º VR videos. Using the EmojiGrid, 40 participants rated 62 360º VR videos from a validated public database. The resulting mean valence and arousal values agree with the corresponding values provided with the database (obtained with an alternative validated rating tool). We conclude that the EmojiGrid is a valid self-report tool for the assessment of VR-evoked emotions

We recently introduced the EmojiGrid as an intuitive graphical self-report tool to measure food-evoked valence and arousal. The EmojiGrid is Cartesian grid, labeled with facial icons (emoji) expressing different degrees of valence and arousal. The lack of verbal labels makes it a valuable, language-independent tool for cross-cultural research. Users can efficiently report their subjective ratings of both valence and arousal with a single click on the location of the grid that best represents their affective state after perceiving a given stimulus. The EmojiGrid has previously been validated for the assessment of emotions evoked by food images. In this study we validated the EmojiGrid for the affective appraisal of odors. Observers (N=56, 24 males, mean age=24.3±4.6) smelled 40 randomly presented odors (27 food and 13 non-food smells), ranging from very unpleasant and arousing (e.g., feces, fish), via pleasant and calming (e.g., clove, cinnamon) to very pleasant and stimulating (e.g., peach, caramel). The odor samples consisted of felt pens, with tips that were impregnated with 4 mL of fluid odorant substance. Each pen was presented once, for about 5 seconds and about 2 cm below both nostrils. The participants sniffed following a verbal command. Immediately after sniffing the pen was removed, and participants were given at least 30 s to smell fresh air. The participants reported their affective appraisal of each odor using the EmojiGrid. The resulting mean valence and arousal ratings closely agree with those from previous studies in the literature that were obtained with alternative rating tools. In addition, we find that the EmojiGrid yields the typical universal U-shaped relation between mean valence and arousal that is commonly observed for a wide range of affective sensory (visual, auditory, tactile, gustatory) stimuli. We conclude that the EmojiGrid is also a valid affective self-report tool for the assessment of odor evoked emotions.

Background. The information displayed on a Head Mounted Display (HMD) can only be read by making eye movements, since head movements have no effect on the ocular image position. Aniseikonia (a common visual deficit) is expected to cause eye strain and limit the readability of large FoV binocular HMDs. As the FoV increases, the screen layout needs to optimize overall display readability by preventing clutter while taking common optometric conditions into account. Methods. We measured the ability to quickly determine the orientation of a target T (Т vs Ʇ) surrounded by 4 randomly oriented (up, down, left, right) flanker T’s as a function of target-flanker spacing and eccentricity, in conditions where the target had either the same or opposite opposite luminance polarity as the flankers. All 12 subjects scored normal on relevant optometric tests (stereopsis, visual acuity, Awaya aniseikonia test, phoria). An aniseikonic lens placed in front of one eye optically enlarged the image by 2½%, simulating a common optometric condition. The additional delay caused by the presence of the four flankers is adopted as the ‘Crowding component’ of the reaction time. Results. Compared to the Same polarity condition, Opposite polarity reduced the Crowding time by a factor of 2.3 (p< 0.001). The Crowding times can be described as an extension of Fitts’ law. Unexpectedly, the mild aniseikonia condition doubled the Crowding time (p< 0.001) and caused the highest level of eye strain (p< 0.001). Conclusion. For all eccentricities and target-flanker spacings, the Crowding time more than halved in the opposite polarity condition, while it doubled due to the addition of just 2½% aniseikonia. Practical implications. Even users with mild aniseikonia are likely to experience problems while reading a large FOV HMD. ‘Polarity decluttering’ can significantly enhance symbology legibility.

This is a poster presentation in Pangborn 2019.

CROCUFID is a CRoss-CUltural Food Image Database that currently contains 840 images, consisting of 675 food images and 165 images of non-food items. For the first 479 food images CROCUFID also includes detailed metadata and observer ratings of valence, arousal, desire-to-eat, perceived healthiness, and familiarity. CROCUFID is intended to facilitate comparability across cross-cultural studies and advance experimental research on the determinants of food-elicited emotions (e.g., in online experiments or neuro-physiological studies).

Supplemental material, Supplemental_material for Graphical uncertainty representations for ensemble predictions by Alexander Toet, Jan BF van Erp, Erik M Boertjes and Stef van Buuren in Information Visualization

We developed active tangibles (Sensators) that can be used in combination with multitouch tabletops and that can provide multisensory (visual, auditory, and vibrotactile) feedback. For spatial alignment and rotation tasks we measured subjective user experience and objective performance with these Sensators. We found that active feedback increased accuracy in both tasks, for all feedback modalities. Active visual feedback yielded the highest overall subjective user experience and preference scores. Our contribution is that active feedback improves subjectively perceived performance and reduces perceived mental workload. Additionally, our findings indicate that users prefer to be guided by visual signs over auditory and vibrotactile signs.

Human evaluation of facial expressions is significantly affected by the emotional context of the visual background [Koji and Fernandes, Can. J. Exp. Psychol., 2010, 64(2), 107-116]. We recently found that dynamic visual textures elicit a wide range of emotional responses, with dominance (strength or conspicuity) being one of the principal affective dimensions [Toet et al., i-Perception, 2012, 2(9), 969-991]. In the current study we investigate whether dynamic textured backgrounds also affect the judgement of human facial expressions. Participants rated the dominance of 12 (neutral ) male faces. In the first experiment we validated the neutrality of these faces by placing them on a neutral (black) background. Results show that none of the faces resulted in a non-zero dominance score. In the second experiment the faces were overlaid (opacity 80%) on 12 different natural dynamic background textures, six of which were very strong/conspicuous and six which were very weak/inconspicuous. The results show that the (neutral) faces were rated significantly more dominant on strong/conspicuous backgrounds than on neutral backgrounds. There is no significant difference between ratings obtained with weak/inconspicuous backgrounds and with neutral backgrounds. We conclude that natural dynamic backgrounds (typically not perceived as emotional) can significantly affect the evaluation of facial expressions.

Although Tangible User Interfaces are considered an intuitive means of human-computer interaction, they oftentimes lack the option to provide active feedback. We developed ‘Sensators’: generic shaped active tangibles to be used on a multi-touch table. Sensators can represent digital information by means of ‘Sensicons’: multimodal messages consisting of visual, auditory, and vibro- tactile cues. In our demonstration, we will present Sensators as suitable tools for research on multimodal perception in different tangible HCI tasks.

We present data on the human sensitivity to optic acceleration i.e. temporal modulations of the speed and direction of moving objects. Modulation thresholds are measured as a function of modulation frequency and speed for different periodical velocity vector modulation functions using a localized target. Evidence is presented that human detection of velocity vector modulations is not directly based on the acceleration signal (the temporal derivative of the velocity vector modulation). Instead, modulation detection is accurately described by a two-stage model: a low-pass temporal filter transformation of the true velocity vector modulation followed by a variance detection stage. A functional description of the first stage is a second order low-pass temporal filter having a characteristic time constant of 40 msec. In effect, the temporal low-pass filter is an integration of the velocity vector modulation within a temporal window of 100–140 msec. A non-trivial link of this low-pass filter stage to the temporal characteristics of standard motion detection mechanisms will be discussed. Velocity vector modulations are detected in the second-stage, whenever the variance of the filtered velocity vector exceeds a certain threshold variance in either the speed or direction dimension. The threshold standard deviations for this variance detection stage are estimated to be 17% for speed modulations and 9% for motion direction modulations.

This paper highlights work-in-progress towards the conceptualization, simulation, fabrication and initial testing of a silicon-germanium (SiGe) integrated CMOS-MEMS high-g accelerometer for military, munition, fuze and shock measurement applications. Developed on IMEC’s SiGe MEMS platform, the MEMS offers a dynamic range of 5,000 g and a bandwidth of 12 kHz. The low noise readout circuit adopts a chopper-stabilization technique implementing the CMOS through the TSMC 0.18 µm process. The device structure employs a fully differential split comb-drive set up with two sets of stators and a rotor all driven separately. Dummy structures acting as protective over-range stops were designed to protect the active components when under impacts well above the designed dynamic range. Published version