Search

Automatically detecting mental state such as stress from video images of the face could support evaluating stress responses in applicants for high risk jobs or contribute to timely stress detection in challenging operational settings (e.g., aircrew, command center operators). Challenges in automatically estimating mental state include the generalization of models across contexts and across participants. We here aim to create robust models by training them using data from different contexts and including physiological features. Fifty-one participants were exposed to different types of stressors (cognitive, social evaluative and startle) and baseline variants of the stressors. Video, electrocardiogram (ECG), electrodermal activity (EDA) and self-reports (arousal and valence) were recorded. Logistic regression models aimed to classify between high and low arousal and valence across participants, where “high” and “low” were defined relative to the center of the rating scale. Accuracy scores of different models were evaluated: models trained and tested within a specific context (either a baseline or stressor variant of a task), intermediate context (baseline and stressor variant of a task), or general context (all conditions together). Furthermore, for these different model variants, only the video data was included, only the physiological data, or both video and physiological data. We found that all (video, physiological and video-physio) models could successfully distinguish between high- and low-rated arousal and valence, though performance tended to be better for (1) arousal than valence, (2) specific context than intermediate and general contexts, (3) video-physio data than video or physiological data alone. Automatic feature selection resulted in inclusion of 3–20 features, where the models based on video-physio data usually included features from video, ECG and EDA. Still, performance of video-only models approached the performance of video-physio models. Arous

Not only is visual attention shifted to objects in the external world, attention can also be directed to objects in memory. We have recently shown that pupil size indexes how strongly items are attended externally, which was reflected in more precise encoding into visual working memory. Using a retro-cuing paradigm, we here replicated this finding by showing that stronger pupil constrictions during encoding were reflective of the depth of encoding. Importantly, we extend this previous work by showing that pupil size also revealed the intensity of internal attention toward content stored in visual working memory. Specifically, pupil dilation during the prioritization of one among multiple internally stored representations predicted the precision of the prioritized item. Furthermore, the dynamics of the pupillary responses revealed that the intensity of internal and external attention independently determined the precision of internalized visual representations. Our results show that both internal and external attention are not all-or-none processes, but should rather be thought of as continuous resources that can be deployed at varying intensities. The employed pupillometric approach allows to unravel the intricate interplay between internal and external attention and their effects on visual working memory.

Pupillary dynamics reflect effects of distinct and important operations of visual working memory: encoding, maintenance, and prioritization. Here, we review how pupil size predicts memory performance and how it provides novel insights into the mechanisms of each operation. Visual information must first be encoded into working memory with sufficient precision. The depth of this encoding process couples to arousal-linked baseline pupil size as well as a pupil constriction response before and after stimulus onset, respectively. Subsequently, the encoded information is maintained over time to ensure it is not lost. Pupil dilation reflects the effortful maintenance of information, wherein storing more items is accompanied by larger dilations. Lastly, the most task-relevant information is prioritized to guide upcoming behavior, which is reflected in yet another dilatory component. Moreover, activated content in memory can be pupillometrically probed directly by tagging visual information with distinct luminance levels. Through this luminance-tagging mechanism, pupil light responses reveal whether dark or bright items receive more attention during encoding and prioritization. Together, conceptualizing pupil responses as a sum of distinct components over time reveals insights into operations of visual working memory. From this viewpoint, pupillometry is a promising avenue to study the most vital operations through which visual working memory works. This article is categorized under: Psychology > Attention Psychology > Memory Psychology > Theory and Methods.

The pupil of the eye constricts when moving focus from an object further away to an object closer by. This is called the pupil near response, which typically occurs together with accommodation and vergence responses. When immersed in virtual reality mediated through a head-mounted display, this triad is disrupted by the vergence-accommodation conflict. However, it is not yet clear if the disruption also affects the pupil near response. Two experiments were performed to assess this. The first experiment had participants follow a target that first appeared at a far position and then moved to either a near position (far-to-near; FN) or to another far position (far-to-far; FF). The second experiment had participants follow a target that jumped between five positions, which was repeated at several distances. Experiment 1 showed a greater pupil constriction amplitude for FN trials, compared to FF trials, suggesting that the pupil near response is intact in head-mounted display mediated virtual reality.

Purpose: We improve pupillary responses and diagnostic performance of flicker pupil perimetry through alterations in global and local color contrast and luminance contrast in adult patients suffering from visual field defects due to cerebral visual impairment (CVI). Methods: Two experiments were conducted on patients with CVI (Experiment 1: 19 subjects, age M and SD 57.9 ± 14.0; Experiment 2: 16 subjects, age M and SD 57.3 ± 14.7) suffering from absolute homonymous visual field (VF) defects. We altered global color contrast (stimuli consisted of white, yellow, cyan and yellow-equiluminant-to-cyan colored wedges) in Experiment 1, and we manipulated luminance and local color contrast with bright and dark yellow and multicolor wedges in a 2-by-2 design in Experiment 2. Stimuli consecutively flickered across 44 stimulus locations within the inner 60 degrees of the VF and were offset to a contrasting (opponency colored) dark background. Pupil perimetry results were compared to standard automated perimetry (SAP) to assess diagnostic accuracy. Results: A bright stimulus with global color contrast using yellow (p= 0.009) or white (p= 0.006) evoked strongest pupillary responses as opposed to stimuli containing local color contrast and lower brightness. Diagnostic accuracy, however, was similar across global color contrast conditions in Experiment 1 (p= 0.27) and decreased when local color contrast and less luminance contrast was introduced in Experiment 2 (p= 0.02). The bright yellow condition resulted in highest performance (AUC M = 0.85 ± 0.10, Mdn = 0.85). Conclusion: Pupillary responses and pupil perimetry’s diagnostic accuracy both benefit from high luminance contrast and global but not local color contrast.

For human interaction, it is important to understand what emotional state others are in. Especially the observation of faces aids us in putting behaviours into context and gives insight into emotions and mental states of others. Detecting whether someone is nervous, a form of state anxiety, is such an example as it reveals a person’s familiarity and contentment with the circumstances. With recent developments in computer vision we developed behavioural nervousness models to show which time-varying facial cues reveal whether someone is nervous in an interview setting. The facial changes, reflecting a state of anxiety, led to more visual exposure and less chemosensory (taste and olfaction) exposure. However, experienced observers had difficulty picking up these changes and failed to detect nervousness levels accurately therewith. This study highlights humans’ limited capacity in determining complex emotional states but at the same time provides an automated model that can assist us in achieving fair assessments of so far unexplored emotional states.

Visuospatial neglect is a frequent and disabling disorder, mostly after stroke, that presents in impaired awareness to stimuli on one side of space. Neglect causes disability and functional dependence, even long after the injury. Improving measurements of the core attentional deficit might hold the key for better understanding of the condition and development of treatment. We present a rapid, pupillometry-based method that assesses automatic biases in (covert) attention, without requiring behavioral responses. We exploit the phenomenon that pupil light responses scale with the degree of covert attention to stimuli, and thereby reveal what draws (no) attention. Participants with left-sided neglect after right-sided lesions following stroke (n = 5), participants with hemianopia/quadrantanopia following stroke (n = 11), and controls (n = 22) were presented with two vertical bars, one of which was white and one of which was black, while fixating the center. We varied which brightness was left and right, respectively across trials. In line with the hypotheses, participants with neglect demonstrated biased pupil light responses to the brightness on the right side. Participants with hemianopia showed similar biases to intact parts of the visual field, whilst controls exhibited no bias. Together, this demonstrates that the pupil light response can reveal not only visual, but also attentional deficits. Strikingly, our pupillometry-based bias estimates were not in agreement with neuropsychological paper-and-pencil assessments conducted on the same day, but were with those administered in an earlier phase post-stroke. Potentially, we pick up on persistent biases in the covert attentional system that participants increasingly compensate for in classical neuropsychological tasks and everyday life. The here proposed method may not only find clinical application, but also advance theory and aid the development of successful restoration therapies by introducing a precise, longit

What determines how much one encodes into visual working memory? Traditionally, encoding depth is considered to be indexed by spatiotemporal properties of gaze, such as gaze position and dwell time. Although these properties inform about where and how long one looks, they do not necessarily inform about the current arousal state or how strongly attention is deployed to facilitate encoding. Here, we found that two types of pupillary dynamics predict how much information is encoded during a copy task. The task involved encoding a spatial pattern of multiple items for later reproduction. Results showed that smaller baseline pupil sizes preceding and stronger pupil orienting responses during encoding predicted that more information was encoded into visual working memory. Additionally, we show that pupil size reflects not only how much but also how precisely material is encoded. We argue that a smaller pupil size preceding encoding is related to increased exploitation, whereas larger pupil constrictions signal stronger attentional (re)orienting to the to-be-encoded pattern. Our findings support the notion that the depth of visual working memory encoding is the integrative outcome of differential aspects of attention: how alert one is, how much attention one deploys, and how long it is deployed. Together, these factors determine how much information is encoded into visual working memory.

Attention can be shifted with or without an accompanying saccade (i.e., overtly or covertly, respectively). Thus far, it is unknown how cognitively costly these shifts are, yet such quantification is necessary to understand how and when attention is deployed overtly or covertly. In our first experiment (N = 24 adults), we used pupillometry to show that shifting attention overtly is more costly than shifting attention covertly, likely because planning saccades is more complex. We pose that these differential costs will, in part, determine whether attention is shifted overtly or covertly in a given context. A subsequent experiment (N = 24 adults) showed that relatively complex oblique saccades are more costly than relatively simple saccades in horizontal or vertical directions. This provides a possible explanation for the cardinal-direction bias of saccades. The utility of a cost perspective as presented here is vital to furthering our understanding of the multitude of decisions involved in processing and interacting with the external world efficiently.

Purpose: A scene consisting of a white stimulus on a black background incorporates strong luminance contrast. When both stimulus and background receive different colors, luminance contrast decreases but color contrast increases. Here, we sought to charac-terize the pattern of stimulus salience across varying trade-offs of color and luminance contrasts by using the pupil light response. Methods: Three experiments were conducted with 17, 16, and 17 healthy adults. For all experiments, a flickering stimulus (2 Hz; alternating color to black) was presented super-imposed on a background with a complementary color to the stimulus (i.e., opponency colors in human color perception: blue and yellow for Experiment 1, red and green for Experiment 2, and equiluminant red and green for Experiment 3). Background luminance varied between 0% and 45% to trade off luminance and color contrast with the stimulus. By comparing the locus of the optimal trade-off between color and luminance across different color axes, we explored the generality of the trade-off. Results: The strongest pupil responses were found when a substantial amount of color contrast was present (at the expense of luminance contrast). Pupil response ampli-tudes increased by 15% to 30% after the addition of color contrast. An optimal pupillary responsiveness was reached at a background luminance setting of 20% to 35% color contrast across several color axes. Conclusions: These findings suggest that a substantial component of pupil light responses incorporates color processing. More sensitive pupil responses and more salient stimulus designs can be achieved by adding subtle levels of color contrast between stimulus and background. Translational Relevance: More robust pupil responses will enhance tests of the visual field with pupil perimetry.

The eye's retinotopic exposure to an adapter typically produces an after-image. For example, an observer who fixates a red adapter on a gray background will see an illusory cyan after-image after removing the adapter. The after-image's content, like its color or intensity, gives insight into mechanisms responsible for adaptation and processing of a specific feature. To facilitate adaptation, vision scientists traditionally present stable, unchanging adapters for prolonged durations. How adaptation affects perception when features (e.g., color) dynamically change over time is not understood. To investigate adaptation to a dynamically changing feature, participants viewed a colored patch that changed from a color to gray, following either a direct or curved path through the (roughly) equiluminant color plane of CIE LAB space. We varied the speed and curvature of color changes across trials and experiments. Results showed that dynamic adapters produce after-images, vivid enough to be reported by the majority of participants. An after-image consisted of a color complementary to the average of the adapter's colors with a small bias towards more recent rather than initial adapter colors. The modelling of the reported after-image colors further confirmed that adaptation rapidly instigates and gradually dissipates. A second experiment replicated these results and further showed that the probability of observing an after-image diminishes only slightly when the adapter displays transient (stepwise, abrupt) color transitions. We conclude from the results that the visual system can adapt to dynamic colors, to a degree that is robust to the potential interference of transient changes in adapter content.

The inversion of a picture of a face hampers the accuracy and speed at which observers can perceptually process it. Event-related potentials and pupillary responses, successfully used as biomarkers of face inversion in the past, suggest that the perception of visual features, that are organized in an unfamiliar manner, recruits demanding additional processes. However, it remains unclear whether such inversion effects generalize beyond face stimuli and whether indeed more mental effort is needed to process inverted images. Here we aimed to study the effects of natural scene inversion on visual evoked potentials and pupil dilations. We simultaneously measured responses of 47 human participants to presentations of images showing upright or inverted natural scenes. For inverted scenes, we observed relatively stronger occipito-temporo-parietal N1 peak amplitudes and larger pupil dilations (on top of an initial orienting response) than for upright scenes. This study revealed neural and physiological markers of natural scene inversion that are in line with inversion effects of other stimulus types and demonstrates the robustness and generalizability of the phenomenon that unfamiliar configurations of visual content require increased processing effort.

Brendan L Portengen,1,2 Giorgio L Porro,1,* Douwe Bergsma,3 Evert J Veldman,4 Saskia M Imhof,1 Marnix Naber2,* 1Department of Ophthalmology, University Medical Center Utrecht, Utrecht, the Netherlands; 2Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, the Netherlands; 3HemianopsieStichting, Amersfoort, the Netherlands; 4Bartiméus Foundation, Zeist, the Netherlands*These authors contributed equally to this workCorrespondence: Brendan L Portengen, Department of Ophthalmology, University Medical Center Utrecht, PO Box 85500, Room E 03.136, Utrecht, 3508 GA, the Netherlands, Email b.l.portengen-2@umcutrecht.nlPurpose: We improve pupillary responses and diagnostic performance of flicker pupil perimetry through alterations in global and local color contrast and luminance contrast in adult patients suffering from visual field defects due to cerebral visual impairment (CVI).Methods: Two experiments were conducted on patients with CVI (Experiment 1: 19 subjects, age M and SD 57.9 ± 14.0; Experiment 2: 16 subjects, age M and SD 57.3 ± 14.7) suffering from absolute homonymous visual field (VF) defects. We altered global color contrast (stimuli consisted of white, yellow, cyan and yellow-equiluminant-to-cyan colored wedges) in Experiment 1, and we manipulated luminance and local color contrast with bright and dark yellow and multicolor wedges in a 2-by-2 design in Experiment 2. Stimuli consecutively flickered across 44 stimulus locations within the inner 60 degrees of the VF and were offset to a contrasting (opponency colored) dark background. Pupil perimetry results were compared to standard automated perimetry (SAP) to assess diagnostic accuracy.Results: A bright stimulus with global color contrast using yellow (p= 0.009) or white (p= 0.006) evoked strongest pupillary responses as opposed to stimuli containing local color contrast and lower brightness. Diagnostic accuracy, however, was similar across global color contra