Search

Situation awareness (SA) is a widely used cognitive construct in human factors, often theoretically posited to be a critical causal factor and/or construct for performance. However, there are conce...

Object perception occurs within a dynamic world, where the environment and the observer (both body and eyes) are continually moving, shifting and changing. We seek to characterize and quant...

The U.S. Army is currently working to integrate artificial intelligence, or AI-enabled systems, into military working teams in the form of both embodied (i.e., robotic) and embedded (i.e., computer or software) intelligent agents with the express purpose of improving performance during all phases of the mission. However, this is largely uncharted territory, making it unclear how to do this integration effectively for human-AI teams. This chapter provides an overview of the Combat Capabilities Development Command (DEVCOM) Army Research Laboratory’s effort to address the human as a critical gap with associated implications on effective teaming. This chapter articulates four major research thrusts critical to integrating AI-enabled systems into military operations, giving examples within these broader thrusts that are currently addressing specific research gaps. The four major research thrusts include: (1) Enabling Soldiers to predict AI; (2) Quantifying Soldier understanding for AI; (3) Soldier-guided AI adaptation; and (4) Characterizing Soldier-AI performance. These research thrusts are the organizing basis for explaining a path toward integration and effective human-autonomy teaming at the tactical edge.

Technology advances in artificial intelligence (AI) and augmented or mixed reality (AR) offer the potential for advanced performance capabilities in the military and commercial sectors. However, the impact of utilizing such technology on users’ normative perceptual, attentional, and higher-level cognitive processes is not well understood. To take full advantage of current and future technological advancements, systems designers for the Army must better understand how human visual cognition changes in the face of the novel visual stimuli provided by these technologies. Here, we present an approach anchored in foundational cognitive research to derive principles for how humans understand, interact with, and are cognitively altered by the interactions between AI and AR. We will discuss our approach in the context of a specific application, Aided Target Recognition (AiTR). We will discuss a series of planned research efforts, the foundational findings supporting these efforts, and their potential implications for AiTR development.

Commercial augmented reality (AR) and mixed reality (MR) displays are designed for indoor nearfield gaming tasks. Conversely, outdoor warfighter tasks have a different set of needs for optimal performance, e.g. for aided target recognition (AiTR). The information display needs to be visible across a wide luminance range, from mesopic to photopic (0.001 to 100,000 cd/m2), including maxto- min luminance ratio exceeding 10,000-to-1 within a single scene (high dynamic range luminance, HDR). The information display also should not distract from other tasks, a difficult requirement because the saliency of the information display depends on its relationship to the HDR background texture. We suggest that a transparency-adjustable divisive display AR (ddAR) could achieve these luminance and saliency needs, with potentially less complexity and processing power than current additive displays. We report measurements of acuity to predict how such ddAR might affect low contrast visibility under gaze shifts, which often result in 10- or 100-fold changes in luminance. We developed an HDR display projection system with up to 100,000-to-1 luminance contrast ratio and assessed how luminance dynamics affect acuity to semi-transparent letters against a uniform background. Immediately following a luminance flash, visual acuity is unaffected at 20% letter contrast, and it is only weakly affected at 10% letter contrast (+0.10 and +0.12 LogMAR for flashes of 25× and 100× luminance). The resilience of low contrast letter acuity across luminance changes suggests that soft highlighting and ddAR could effectively convey information, to improve AiTR under real world luminance.

Augmented-reality (AR) displays render virtual content overlaid on the real world. AR offers great potential for Soldiers to immediately perceive relevant tactical information without the need to consult a physical map or other reference material, or divert their eyes away from the immediate environment. However, realizing the potential benefits of using AR requires that the conformal subset of virtual content is perceived in the correct physical location. Conformal virtual content must be correctly associated with the physical location, object, or person intended to be augmented. In this report, we provide emerging recommendations aimed at improving spatial perception of virtual content.

We investigated whether anxiety influences perceptual-motor calibration in a braking to avoid a collision task. Participants performed either a discrete braking task (Experiment 1) or a continuous braking task (Experiment 2), with the goal of stopping before colliding with a stop sign. Half of participants performed the braking task after an anxiety induction. We investigated whether anxiety reduced the frequency of crashing and if it influenced the calibration of perception (visual information) and action (brake pressure) dynamically between-trials in Experiment 1 and within-trials in Experiment 2. In the discrete braking task, anxious participants crashed less often and made larger corrective adjustments trial-to-trial after crashing, suggesting that the influence of anxiety on behavior did not occur uniformly, but rather dynamically with anxiety amplifying the reaction to previous crashes. However, when performing continuous braking, anxious participants crashed more often, and their within-trial adjustments of deceleration were less related to visual information compared to controls. Taken together, these findings suggest that the timescale and nature of the task mediates the influence of anxiety on the performance of goal-directed actions.

In light of the Army’s intent to leverage advances in Artificial Intelligence (AI) for augmenting dismounted Soldier Lethality through the development of in-scope and Heads-Up Display (HUD)-based Automatic Target Recognition (ATR) systems, the Combat Capabilities Development Command U. S. Army Research Laboratory’s Human Research and Engineering Directorate (CCDC-ARL/HRED) has identified several critical gaps that must be addressed in order to effectively team the Soldier with ATR for the desired augmented Lethality. One of these areas pertains to the way in which ATR is displayed and requires a thorough understanding and leveraging of relevant cognitive processes that will enable use of this technology. Additionally, insufficient consideration of perceptual, attentional, and cognitive capabilities increases the risk of burdening the Soldier with excessive, unnecessary, or distracting representations of information, which may impede Lethality rather than augment it. HRED’s planned and ongoing research is intended to develop novel mechanisms through which Soldiers teamed with ATR will perform more adaptively and effectively than either the Soldier or the intelligent system could accomplish individually. Based on HRED’s significant expertise in the cognitive sciences, coupled with familiarity with the military-relevant domain spaces, the following initial recommendations for ATR information display requirements are made:1.ATR highlighting should leverage a non-binary display schema to continuously encode threat information (e.g., target class/identity, uncertainty, and prioritization).2.ATR highlighting should be integrated with the target itself instead of functioning as a discrete feature of the display (i.e., highlight the target rather than highlighting a region with the target inside).3.Information about threat certainty or classification confidence (which can also include priority) should be embedded into ATR highlighting.4.Yellow highlights may offer advantages for display5.Changing information (e.g., target certainty) should be accomplished through formation or modification of highlight gradients rather than sudden changes in the display.6.Human performance evaluations of ATR should consider the incorporation of changing threat states and contexts into scenarios for more relevant findings.7.Human performance evaluations of ATR should consider the incorporation of uncued (non-highlighted) targets and miscued targets (false identifications; e.g. ATR identifies non-threat as threat) for more relevant findings.

The Army plans to integrate artificial intelligence (AI)/machine learning (ML) and other intelligent decision-making aids into future dismounted Warfighter systems to augment situational awareness and target acquisition capabilities. However, due to the unique constraints of dismounted operations, successful implementation of intelligent decisionmaking aids in dismounted systems necessitates a human-in-the-loop approach, which includes the ability for the Warfighter to provide feedback to the autonomous system. Human-in-the-loop feedback can augment current machine learning techniques by reducing the size of datasets needed to train algorithms and allow algorithms to be flexible and adaptive to changing battlespace conditions. As such, research is required to define the bidirectional interactions between man and machine in this context, to optimize human-intelligent agent teaming for the dismounted Warfighter. In this paper, we focus on a specific application of dismounted Human-AI interaction to weapon mounted target acquisition (small arms fire control systems) and discuss issues pertaining to an important component of this optimization: how intelligent information is communicated to the end user. We consider how the intelligent information is presented to the Warfighter, and what underlying cognitive and perceptual processes can be leveraged to optimize teamed decision making. Such factors are critical to the successful implementation of human-in-the-loop AI in dismounted applications and ultimately the effectiveness of intelligent decision-making aids.

It is unknown whether teams communicate more effectively in virtual spaces than when using more traditional technologies due to a lack of objective measures of team performance. Social psychology research shows that teams struggle to share information and suggests the absence of certain social cues in virtual reality (VR) would making sharing information more difficult. We developed a methodology to assess objective performance of communication during an information-gathering task. In this study, using a within-subjects design, 24 pairs of participants examined maps of cities both in VR, using a head-mounted display, and on desktop monitors. For each map, pairs worked together to read the maps, identify patterns amongst variables, and make predictions beyond the given data. In VR, the only cues from one’s partner were speech and the ability to point out places on the map with a laser. In contrast, when working on the desktop monitors, partners were also able to view one another. Because groups struggle to share unique information, we anticipated a task providing partners with unique variables and requiring pairs to exchange information in order to solve problems would create a sufficiently challenging situation and be an effective way to measure team performance. Overall, performance in VR did not suffer, as pairs were just as fast and equally accurate at identifying patterns and predicting future events. Pairs performed better at reading the maps when in VR but did have better recall of the information viewed on the desktop monitors.

The superordinate mechanism view of emotions predicts that fear should influence perception to carry out the evolved function of overcoming immediate threats. Previous work demonstrates that fear does adaptively influence visual perception. However, there are recurring situations in which auditory perception is used for overcoming immediate threats (e.g., avoiding predators after dark). Some research suggests that the auditory system, independent of fear, is adaptively biased to hear approaching sounds as closer than equidistant receding sounds (a.k.a. the looming bias). The present study investigated whether fear, as a superordinate mechanism, influences auditory perception such that sounds are perceived to be closer, ultimately providing an advantage when avoiding immediate threats. Participants judged whether or not they could reach to an aurally or visually perceived target while either in a fearful or neutral state. The results demonstrated that while in a fearful state, participants judged targets to be closer to them, but only when the target was perceived aurally. This finding extends previous work on adaptive biases in auditory perception to include the influence of fear.

The conscious representation of our physical appearance is important for many aspects of everyday life. Here, we asked whether different visual experiences of our bodies influence body width estimates. In Experiment 1, width estimates of three body parts (foot, hips, and shoulders) without any visual access were compared to estimates with visual feedback available in a mirror or from a first-person perspective. In the no visual access and mirror condition, participants additionally estimated their head width. There was no influence of viewing condition on body part width estimates. Consistent with previous research, all body part widths were overestimated with greater overestimation of hip and head width. In Experiment 2, participants estimated the size of unfamiliar noncorporeal objects to test whether this overestimation was partially due to the metric body size estimation method or our experimental conditions. Object width was overestimated with visual feedback in a mirror available as compared to when directly looking at the object, but only for objects placed at shoulder and head height. We conclude that at least some of the overestimation of body part width seems to be body specific and occurs regardless of the visual information provided about the own body.

Recent research shows that the body is used to scale environmental extents. We question whether the body is used to scale heights as measured by real actions (Experiments 1 and 2) or by judgments about action and extent made from a single viewpoint (Experiments 3 and 4). First, participants walked under barriers naturally, when wearing shoes, or when wearing a helmet. Participants required a larger margin of safety (they ducked at shorter heights) when they were made taller. In follow-up experiments, participants visually matched barrier heights and judged whether they could walk under them when wearing shoes or a helmet. Only the helmet decreased visually matched estimates; action judgments were no different when participants' eye height increased. The final experiment suggested that the change in matched estimates may have been due to lack of experience wearing the helmet. Overall, the results suggest that perceived height is scaled to the body and that when body height is altered, experience may moderate the rescaling of height.

Previous research has demonstrated an influence of one’s emotional state on estimates of spatial layout. For example, estimates of heights are larger when the viewer is someone typically afraid of heights (trait fear) or someone who, in the moment, is experiencing elevated levels of fear (state fear). Embodied perception theories have suggested that such a change in perception occurs in order to alter future actions in a manner that reduces the likelihood of injury. However, other work has argued that when acting, it is important to have access to an accurate perception of space and that a change in conscious perception does not necessitate a change in action. No one has yet investigated emotional state, perceptual estimates, and action performance in a single paradigm. The goal of the current paper was to investigate whether fear influences perceptual estimates and action measures similarly or in a dissociable manner. In the current work, participants either estimated gap widths (Experiment 1) or were asked to step over gaps (Experiment 2) in a virtual environment. To induce fear, the gaps were placed at various heights up to 15 meters. Results showed an increase in gap width estimates as participants indicated experiencing more fear. The increase in gap estimates was mirrored in participants’ stepping behavior in Experiment 2; participants stepped over fewer gaps when experiencing higher state and trait fear and, when participants actually stepped, they stepped farther over gap widths when experiencing more fear. The magnitude of the influence of fear on both perception and action were also remarkably similar (5.3 and 3.9 cm, respectively). These results lend support to embodied perception claims by demonstrating an influence on action of a similar magnitude as seen on estimates of gap widths.

Objective: Our goal was to evaluate the degree to which display technologies influence the perception of size in an image. Background: Research suggests that factors such as whether an image is displayed stereoscopically, whether a user’s viewpoint is tracked, and the field of view of a given display can affect users’ perception of scale in the displayed image. Method: Participants directly estimated the size of a gap by matching the distance between their hands to the gap width and judged their ability to pass unimpeded through the gap in one of five common implementations of three display technologies (two head-mounted displays [HMD] and a back-projection screen). Results: Both measures of gap width were similar for the two HMD conditions and the back projection with stereo and tracking. For the displays without tracking, stereo and monocular conditions differed from each other, with monocular viewing showing underestimation of size. Conclusions: Display technologies that are capable of stereoscopic display and tracking of the user’s viewpoint are beneficial as perceived size does not differ from real-world estimates. Evaluations of different display technologies are necessary as display conditions vary and the availability of different display technologies continues to grow. Applications: The findings are important to those using display technologies for research, commercial, and training purposes when it is important for the displayed image to be perceived at an intended scale.

Judgments of affordances, the potential actions that an observer can carry out within an environment, require observers to relate information about their body to information in the environment. Although humans can accurately judge affordances for others, it is unknown whether other people's capability to act influences one's own affordance judgments. Based on theoretical accounts and recent empirical evidence highlighting the importance of social information in perception and action, we hypothesized that the action capabilities of another person would influence one's own affordance judgments. Participants judged their own and another's ability to pass through an aperture in 3 experiments that varied the differences in body sizes between the participant and another agent using naturally occurring body size differences or an artificial large body suit. RESULTS showed an influence of the other's body size on self-affordance judgments only when the participant and the other agent remained in their natural body size (Experiment 3), but not when the body size differences between the participant and the other agent were extreme because of the body suit (Experiments 1 and 2). (PsycINFO Database Record Language: en

Accurate perception of the size of objects in computer-generated imagery is important for a growing number of applications that rely on absolute scale, such as medical visualization and architecture. Addressing this problem requires both the development of effective evaluation methods and an understanding of what visual information might contribute to differences between virtual displays and the real world. In the current study, we use 2 affordance judgments--perceived graspability of an object or reaching through an aperture--to compare size perception in high-fidelity graphical models presented on a large screen display to the real world. Our goals were to establish the use of perceived affordances within spaces near to the observer for evaluating computer graphics and to assess whether the graphical displays were perceived similarly to the real world. We varied the nature of the affordance task and whether or not the display enabled stereo presentation. We found that judgments of grasping and reaching through can be made effectively with screen-based displays. The affordance judgments revealed that sizes were perceived as smaller than in the real world. However, this difference was reduced when stereo viewing was enabled or when the virtual display was viewed before the real world.

Given that observing one's body is ubiquitous in experience, it is natural to assume that people accurately perceive the relative sizes of their body parts. This assumption is mistaken. In a series of studies, we show that there are dramatic systematic distortions in the perception of bodily proportions, as assessed by visual estimation tasks, where participants were asked to compare the lengths of two body parts. These distortions are not evident when participants estimate the extent of a body part relative to a noncorporeal object or when asked to estimate noncorporal objects that are the same length as their body parts. Our results reveal a radical asymmetry in the perception of corporeal and noncorporeal relative size estimates. Our findings also suggest that people visually perceive the relative size of their body parts as a function of each part's relative tactile sensitivity and physical size.

According to dance theory, dancers are uniquely aware of the relationship between the environment and their body, making them a type of spatial expert. Inherent in their practice are the abilities to assess the location of other people, objects and the environment relative to their body, and to use movement imagery. The current study tests if dancers perceive the world differently than non-dancers. To test this, dancers performed a battery of spatial tasks (egocentric and exocentric distance estimates, a height judgment, and affordance judgments, i.e., perceived vertical and horizontal passability, stepability, and jumpability) and completed paper-and-pencil spatial ability tests. Dancers differed from non-dancers in their movement imagery ability but superior imagery ability did not result in greater accuracy in imagined distance estimates. Height judgments were overestimated in both groups but less so in dancers. Dancers were found to be, on average, less conservative in their affordance judgments than non-dancers.

Accurate perception of the size of 3D objects depicted on 2D desktop displays is important for many applications. Whether users perceive objects depicted on a display to be the same size as comparable real world objects is not well understood. We propose using affordances judgments as a way of measuring the perceived size of objects depicted in desktop virtual environments and the real world. The methodology involves indicating whether or not a particular action can be performed in a given environment, making it a flexible measure that can be used across different display technologies. In two studies, we test users' perceptions of size by asking them to make affordance judgments in both the real world and a geometrically matched desktop virtual environment. In the first study, users judge whether they can grasp an object and in the second study, they judge whether they can fit their hand through an opening. In both experiments we show that users perceive the size of objects in the desktop virtual environment to be smaller than in the real world.

Three experiments examined perceived absolute distance in a head-mounted display virtual environment (HMD-VE) and a matched real-world environment, as a function of the type and orientation of the distance viewed. In Experiment 1, participants turned and walked, without vision, a distance to match the viewed interval for both egocentric (viewer-to-target) and exocentric (target-to-target) extents. Egocentric distances were underestimated in the HMD-VE while exocentric distances were estimated similarly across environments. Since egocentric distances were displayed in the depth plane and exocentric distances in the frontal plane, the pattern of results could have been related to the orientation of the distance or to the type of distance. Experiments 2 and 3 tested these alternatives. Participants estimated exocentric distances presented along the depth or frontal plane either by turning and walking (Experiment 2) or by turning and throwing a beanbag to indicate the perceived extent (Experiment 3). For both Experiments 2 and 3, depth intervals were underestimated in the HMD-VE compared to the real world. However, frontal intervals were estimated similarly across environments. The findings suggest anisotropy in HMD-VE distance perception such that distance underestimation in the HMD-VE generalizes to intervals in the depth plane, but not to intervals in the frontal plane.

Previous research has demonstrated that manipulating vision influences balance. Here, we question whether manipulating balance can influence vision and how it may influence vision--specifically, the perception of width. In Experiment 1, participants estimated the width of beams while balanced and unbalanced. When unbalanced, participants judged the widths to be smaller. One possible explanation is that unbalanced participants did not view the stimulus as long as when balanced because they were focused on remaining balanced. In Experiment 2, we tested this notion by limiting viewing time. Experiment 2 replicated the findings of Experiment 1, but viewing time had no effect on width judgments. In Experiment 3, participants' level of arousal was manipulated, because the balancing task likely produced arousal. While jogging, participants judged the beams to be smaller. In Experiment 4, participants completed another arousing task (counting backward by sevens) that did not involve movement. Again, participants judged the beams to be smaller when aroused. Experiment 5A raised participants' level of arousal before estimating the board widths (to control for potential dual-task effects) and showed that heightened arousal still influenced perceived width of the boards. Collectively, heightened levels of arousal, caused by multiple manipulations (including balance), influenced perceived width.

Perception of an accurate sense of the scale depicted in computer graphics is important for many applications. How to best characterize the accuracy of space perception in computer graphics is a question that does not have a simple answer. This paper describes the use of perceived affordances as a way of measuring the perceptual fidelity of virtual environments with respect to how well they convey information about geometric scale. The methodology involves a verbal indication that a particular action can or cannot be performed in a viewed environment. By varying the spatial structure of the environment, these affordance judgments can be used to probe how accurately viewers are able to perceive action-relevant spatial information. The result is a measure relevant to action, less subject to bias than verbal reports of more primitive properties such as size or distance, and applicable to non-virtual-environment display systems in which the actual action cannot be performed. We demonstrate the approach in an experiment comparing one type of affordance judgment, perceived passability, with judgments of size and distance in matched real world and virtual world environments.

Previous research has shown that changes to the body can influence the perception of distances in near space (Witt et al, 2005 Journal of Experimental Psychology: Human Perception and Performance31 880–888). In this paper, we question whether changes to the body can also influence the perception of extents in extrapersonal space, namely the perception of aperture widths. In experiment 1, broad-shouldered participants visually estimated the size of apertures to be smaller than narrow-shouldered participants. In experiment 2, we questioned whether changes to the body, which included holding a large object, wearing a large object, or simply holding out the arms would influence perceived width. Surprisingly, we found that only when participants' hands were widened was extrapersonal space rescaled. In experiment 3, we explored the boundaries of the effect observed in experiment 2 by asking participants to hold their arms at four different positions in order to determine the arm width at which apertures appeared smaller. We found that arm positions that were larger than the shoulder width made apertures appear smaller. The results suggest that dimensions of the body play a role in the scaling of environmental parameters in extrapersonal space.