Search

Explainable AI provides insights to users into the why for model predictions, offering potential for users to better understand and trust a model, and to recognize and correct AI predictions that are incorrect. Prior research on human and explainable AI interactions has focused on measures such as interpretability, trust, and usability of the explanation. There are mixed findings whether explainable AI can improve actual human decision-making and the ability to identify the problems with the underlying model. Using real datasets, we compare objective human decision accuracy without AI (control), with an AI prediction (no explanation), and AI prediction with explanation. We find providing any kind of AI prediction tends to improve user decision accuracy, but no conclusive evidence that explainable AI has a meaningful impact. Moreover, we observed the strongest predictor for human decision accuracy was AI accuracy and that users were somewhat able to detect when the AI was correct vs. incorrect, but this was not significantly affected by including an explanation. Our results indicate that, at least in some situations, the why information provided in explainable AI may not enhance user decision-making, and further research may be needed to understand how to integrate explainable AI into real systems.

Situation awareness (SA) is a widely used cognitive construct in human factors, often theoretically posited to be a critical causal factor and/or construct for performance. However, there are conce...

Augmented Reality (AR) and Mixed Reality (MR) have been two of the most explosive research topics in the last few years. Head-Mounted Devices (HMDs) are essential intermediums for using AR and MR technology, playing an important role in the research progress in these two areas. Behavioral research with users is one way of evaluating the technical progress and effectiveness of HMDs. In addition, AR and MR technology is dependent upon virtual interactions with the real environment. Thus, conditions in real environments can be a significant factor for AR and MR measurements with users. In this paper, we survey 87 environmental-related HMD papers with measurements from users, spanning over 32 years. We provide a thorough review of AR- and MR-related user experiments with HMDs under different environmental factors. Then, we summarize trends in this literature over time using a new classification method with four environmental factors, the presence or absence of user feedback in behavioral experiments, and ten main categories to subdivide these papers (e.g., domain and method of user assessment). We also categorize characteristics of the behavioral experiments, showing similarities and differences among papers.

Successful navigation generally requires an accurate spatial representation of the environment. However, under guided navigation (i.e. route directions are provided by a GPS system or car passenger) no detailed representation of space is necessary because one only needs to follow directions. The first experiment separated components of navigation, spatial decision-making (planning and selection of routes) and navigational control (executing complex actions such as stopping and steering) for environmental learning. Spatial knowledge was assessed using multiple measures (spatial updating using immersive, head-mounted virtual reality and map construction). More accurate environmental knowledge was acquired when spatial decision-making was present at learning, ps < .001, ds > .81. No difference in spatial knowledge was found between active navigation and spatial decision-making by itself, ps > .83, ds < .07. This finding indicates that the weakened spatial knowledge with control alone (guided navigation) can be explained by the absence of spatial decision-making. Experiments 2 and 3 investigated the use of aids for ameliorating the loss of spatial knowledge for guided navigation with auditory directions. In Experiment 2, guided navigation was augmented using visual aids (primarily egocentric) which reinforced the locations of landmarks. In Experiment 3, guided navigation was augmented by adding cardinal directions (e.g. north), which are exocentric, to the auditory guidance, presented either with monaural sound or stereo sound which corresponded to environmental headings. Spatial knowledge in Experiments 2 and 3 was assessed using route replication, novel route execution, and map construction. Contrary to the hypotheses, the aids did not improve all measures of environmental knowledge. Instead, there was a match between the particular reference frame reinforced by the aid and the relevant frame of reference for the measure of spatial knowledge. In Experiment 2, the visual aids improved accuracy for replicating routes, p = .01, d = .75, this measure is egocentric. In Experiment 3, the stereo and monaural cardinal directions improved accuracy for executing a novel route, a directional measure of survey knowledge, ps < .06, ds > .52, which is egocentric and exocentric.

Augmented-reality (AR) displays render virtual content overlaid on the real world. AR offers great potential for Soldiers to immediately perceive relevant tactical information without the need to consult a physical map or other reference material, or divert their eyes away from the immediate environment. However, realizing the potential benefits of using AR requires that the conformal subset of virtual content is perceived in the correct physical location. Conformal virtual content must be correctly associated with the physical location, object, or person intended to be augmented. In this report, we provide emerging recommendations aimed at improving spatial perception of virtual content.

SA is a widely used cognitive construct in human factors, summarized as “knowing what is going on.” Generally, SA is theoretically posited to be a critical causal factor and/or construct for performance. However, some researchers have raised concerns that SA may be circular and also that SA may lack the appropriate psychological mechanisms relevant to performance. We address these conflicting perspectives using meta-analysis to evaluate the specific and general patterns of associations among SA-performance effect sizes. Specifically, we focus on the validity of SA for performance—the degree to which SA represents or captures the relevant psychological processes and mechanisms related to task performance. From the empirical literature, we coded associations of eight unique measures of SA with (task) performance: 492 effects from 38 papers met the systematic review inclusion criteria. In contrast to SA’s broadly theorized fundamental link with performance, the magnitude of most meta-analytic mean effect sizes for SA measures was limited to medium or lower effects. Although there was a significant overall mean effect, its magnitude was also limited (r = 0.24). In addition, there was high unexplained systematic variation with an enormous plausible range for individual effects (r = -0.20 to 0.60). The meta-analytic results are inconsistent with theories postulating SA is fundamental to performance. Instead, SA’s validity for performance tends to be, on average, weak with large variations among effects. Therefore, theories may need to be revised. Furthermore, even presuming SA is causally linked to performance as generally theorized, improvements in SA (such as SA-based design and training) may not correspond to meaningful increases in task performance.

In light of the Army’s intent to leverage advances in Artificial Intelligence (AI) for augmenting dismounted Soldier Lethality through the development of in-scope and Heads-Up Display (HUD)-based Automatic Target Recognition (ATR) systems, the Combat Capabilities Development Command U. S. Army Research Laboratory’s Human Research and Engineering Directorate (CCDC-ARL/HRED) has identified several critical gaps that must be addressed in order to effectively team the Soldier with ATR for the desired augmented Lethality. One of these areas pertains to the way in which ATR is displayed and requires a thorough understanding and leveraging of relevant cognitive processes that will enable use of this technology. Additionally, insufficient consideration of perceptual, attentional, and cognitive capabilities increases the risk of burdening the Soldier with excessive, unnecessary, or distracting representations of information, which may impede Lethality rather than augment it. HRED’s planned and ongoing research is intended to develop novel mechanisms through which Soldiers teamed with ATR will perform more adaptively and effectively than either the Soldier or the intelligent system could accomplish individually. Based on HRED’s significant expertise in the cognitive sciences, coupled with familiarity with the military-relevant domain spaces, the following initial recommendations for ATR information display requirements are made:1.ATR highlighting should leverage a non-binary display schema to continuously encode threat information (e.g., target class/identity, uncertainty, and prioritization).2.ATR highlighting should be integrated with the target itself instead of functioning as a discrete feature of the display (i.e., highlight the target rather than highlighting a region with the target inside).3.Information about threat certainty or classification confidence (which can also include priority) should be embedded into ATR highlighting.4.Yellow highlights may offer advantages for display5.Changing information (e.g., target certainty) should be accomplished through formation or modification of highlight gradients rather than sudden changes in the display.6.Human performance evaluations of ATR should consider the incorporation of changing threat states and contexts into scenarios for more relevant findings.7.Human performance evaluations of ATR should consider the incorporation of uncued (non-highlighted) targets and miscued targets (false identifications; e.g. ATR identifies non-threat as threat) for more relevant findings.

In this paper we provide a critical analysis with met- rics that will inform guidelines for designing distributed systems for Collective Situational Understanding (CSU). CSU requires both collective insight—i.e., accurate and deep understanding of a situation derived from uncertain and often sparse data and collective foresight—i.e., the ability to predict what will happen in the future. When it comes to complex scenarios, the need for a distributed CSU naturally emerges, as a single monolithic approach not only is unfeasible: it is also undesirable. We therefore propose a principled, critical analysis of AI techniques that can support specific tasks for CSU to derive guidelines for designing distributed systems for CSU.

Social sensing techniques were designed for analyzing unreliable data [1], but not explicitly built for adversarial generated and manipulated data. The adversarial use of social media to spread deceptive or misleading information poses a social, economic, and political threat [2]. Deceptive information spreads quickly and inexpensively online relative to traditional methods of dissemination (e.g., print, radio, and television). For example, bots (i.e., dedicated software for sharing text information [3]) can distribute information faster than humans. Such deceptive information is commonly referred to as fake (fabricated) news, which can be a form of propaganda (i.e., manipulation to advance a particular view or agenda). Information spread is particularly effective if the content resonates with the preconceptions and biases of social groups or communities because the spread will be reinforced by implied trust in information coming from other members (echo chambers and filter bubbles) [4]. We conjecture that the future of online deception, including fake news, will extend beyond text to high-quality, massproduced machine-generated and manipulated images, video, and audio [5].

Due to copyright restrictions, the access to the full text of this article is only available via subscription. In this paper we present a methodology to exploit human-machine coalitions for situational understanding. Situational understanding refers to the ability to relate relevant information and form logical conclusions, as well as identify gaps in information. This process for comprehension of the meaning information requires the ability to reason inductively, for which we will exploit the machines’ ability to ‘learn’ from data. However, important phenomena are often rare in occurrence with high degrees of uncertainty, thus severely limiting the availability of instance data for training, and hence the applicability of many machine learning approaches. Therefore, we present the benefits of Subjective Bayesian Networks—i.e., Bayesian Networks with imprecise probabilities—for situational understanding, and the role of conversational interfaces for supporting decision makers in the evolution of situational understanding. Army Research Laboratory ; Ministry of Defence

Objective: We investigated how increases in task-relevant information affect human decision-making performance, situation awareness (SA), and trust in a simulated command-and-control (C2) environment. Background: Increased information is often associated with an improvement of SA and decision-making performance in networked organizations. However, previous research suggests that increasing information without considering the task relevance and the presentation can impair performance. Method: We used a simulated C2 task across two experiments. Experiment 1 varied the information volume provided to individual participants and measured the speed and accuracy of decision making for task performance. Experiment 2 varied information volume and information reliability provided to two participants acting in different roles and assessed decision-making performance, SA, and trust between the paired participants. Results: In both experiments, increased task-relevant information volume did not improve task performance. In Experiment 2, increased task-relevant information volume reduced self-reported SA and trust, and incorrect source reliability information led to poorer task performance and SA. Conclusion: These results indicate that increasing the volume of information, even when it is accurate and task relevant, is not necessarily beneficial to decision-making performance. Moreover, it may even be detrimental to SA and trust among team members. Application: Given the high volume of available and shared information and the safety-critical and time-sensitive nature of many decisions, these results have implications for training and system design in C2 domains. To avoid decrements to SA, interpersonal trust, and decision-making performance, information presentation within C2 systems must reflect human cognitive processing limits and capabilities.

Peer-to-peer lending is hypothesized to help equalize economic opportunities for the world's poor. We empirically investigate the "flat-world" hypothesis, the idea that globalization eventually leads to economic equality, using crowdfinancing data for over 660,000 loans in 220 nations and territories made between 2005 and 2013. Contrary to the flat-world hypothesis, we find that peer-to-peer lending networks are moving away from flatness. Furthermore, decreasing flatness is strongly associated with multiple variables: relatively stable patterns in the difference in the per capita GDP between borrowing and lending nations, ongoing migration flows from borrowing to lending nations worldwide, and the existence of a tie as a historic colonial. Our regression analysis also indicates a spatial preference in lending for geographically proximal borrowers. To estimate the robustness for these patterns for future changes, we construct a network of borrower and lending nations based on the observed data. Then, to perturb the network, we stochastically simulate policy and event shocks (e.g., erecting walls) or regulatory shocks (e.g., Brexit). The simulations project a drift towards rather than away from flatness. However, levels of flatness persist only for randomly distributed shocks. By contrast, loss of the top borrowing nations produces more flatness, not less, indicating how the welfare of the overall system is tied to a few distinctive and critical country-pair relationships.

The spread of opinions in social networks is dependent on structural properties of the network and the individual characteristics of its nodes. To capture this dependence, several abstract models of such spread were proposed. First, we model the difference between the dynamics of opinion spread in communities with a static social network versus a dynamic social network. Here we use the theoretical model of spread of opinions called the Binary Agreement Model based on the naming game. Using this model, we study potential mechanisms for the dependencies observed in the data by matching model generated evolution of opinion with the empirically observed evolution in the data. Second, we examine the unique set of behavioral network data (based on electronic logs of dyadic contact via smartphones) collected at the University of Notre Dame. The participants are a sample of members of the entering class of freshmen in the fall of 2011 whose opinions on a wide variety of political and social issues have been regularly recorded—at the beginning and end of each semester—for the last three years. Using this data set, we measure the evolution of participants’ opinions and ascertain how much this evolution depends on the cultural traits of individuals and the structural properties of social networks that they form. Our analysis of our empirical dataset shows that ties among people who are more likely to share opinions (e.g. same race, gender, or socioeconomic class) decay at a slower rate than ties among persons who are likely to have different opinions. The analysis also indicates that the partner selection of individuals is associated sharing a (political) opinion. These results offer an assessment of the level of impact of culture and social network dynamics on the evolution of opinions in multi-cultural social networks.

—Controlled Natural Language (CNL) has great potential to support human-machine interaction (HMI) because it provides an information representation that is both human readable and machine processable.We investigated the effectiveness of a CNL-based conversational interface for HMI in a behavioural experiment called Simple Human Experiment Regarding Locally Observed Collective Knowledge (SHERLOCK). In SHERLOCK, individuals acted in groups to discover and report information to the machine using natural language (NL), which the machine then processed into CNL. The machine fused responses from different users to form a common operating picture, a dashboard showing the level of agreement for distinct information. To obtain information to add to this dashboard, users explored the real world in a simulated crowd-sourced sensing scenario. This scenario represented a simplified, controlled analogue for tactical intelligence (i.e., direct intelligence of the environment), which is key for rapidly planning military, law enforcement, and emergency operations. Overall, despite close to zero training, 74% of the users inputted NL that was machine interpretable and addressed the assigned tasks. An experimental manipulation aimed to increase user-machine interaction, however, did not improve performance as hypothesised. Nevertheless, results indicate the conversational interface may be effective in assisting humans with collection and fusion of information in a crowd-sourcing context.

Most risk analysis models systematically underestimate the probability and impact of catastrophic events (e.g., economic crises, natural disasters, and terrorism) by not taking into account interconnectivity and interdependence of risks. To address this weakness, we propose the Cascading Alternating Renewal Process (CARP) to forecast interconnected global risks. However, assessments of the model's prediction precision are limited by lack of sufficient ground truth data. Here, we establish prediction precision as a function of input data size by using alternative long ground truth data generated by simulations of the CARP model with known parameters. We illustrate the approach on a model of fires in artificial cities assembled from basic city blocks with diverse housing. The results confirm that parameter recovery variance exhibits power law decay as a function of the length of available ground truth data. Using CARP, we also demonstrate estimation using a disparate dataset that also has dependencies: real-world prediction precision for the global risk model based on the World Economic Forum Global Risk Report. We conclude that the CARP model is an efficient method for predicting catastrophic cascading events with potential applications to emerging local and global interconnected risks., 12 pages 6 figures

Repeated measures correlation (rmcorr) is a statistical technique for determining the common within-individual association for paired measures assessed on two or more occasions for multiple individuals (Bland & Altman, 1995a; Bland & Altman, 1995b). Simple regression/correlation is often applied to non-independent observations or aggregated data; this may produce biased, specious results due to violation of independence and/or differing patterns between-participants versus within-participants. Unlike simple regression/correlation, rmcorr does not violate the assumption of independence of observations. Also, rmcorr tends to have much greater statistical power because neither averaging nor aggregation is necessary for an intra-individual research question. Rmcorr estimates the common regression slope, the association shared among individuals. To make rmcorr accessible, we provide background information for its assumptions and equations, visualization, power, and types of modeling pooling (variance): Including tradeoffs with rmcorr compared to multi-level modeling. We introduce the R package (rmcorr) and demonstrate its use for inferential statistics and visualization with two example datasets. The examples are used to illustrate research questions at different levels of analysis, intra-individual and inter-individual. Rmcorr is well-suited for research questions regarding the common linear association in paired repeated measures data. All results are fully reproducible.

We investigate automation bias for confirming\ud erroneous information with a conversational interface.\ud Participants in our studies used a conversational interface to\ud report information in a simulated intelligence, surveillance, and\ud reconnaissance (ISR) task. In the task, for flexibility and ease of\ud use, participants reported information to the conversational\ud agent in natural language. Then, the conversational agent\ud interpreted the user’s reports in a human- and machine-readable\ud language. Next, participants could accept or reject the agent’s\ud interpretation. Misparses occur when the agent incorrectly\ud interprets the report and the user erroneously accepts it. We\ud hypothesize that the misparses naturally occur in the experiment\ud due to automation bias and complacency because the agent\ud interpretation was generally correct (92%). These errors indicate\ud some users were unable to maintain situation awareness using\ud the conversational interface. Our results illustrate concerns for\ud implementing a flexible conversational interface in safety critical\ud environments (e.g., military, emergency operations).

In the wake of rapid maturing of Internet of Things (IoT) approaches and technologies in the commercial sector,\ud the IoT is increasingly seen as a key ‘disruptive’ technology in military environments. Future operational environments\ud are expected to be characterized by a lower proportion of human participants and a higher proportion of\ud autonomous and semi-autonomous devices. This view is reflected in both US ‘third offset’ and UK ‘information\ud age’ thinking and is likely to have a profound effect on how multinational coalition operations are conducted\ud in the future. Much of the initial consideration of IoT adoption in the military domain has rightly focused\ud on security concerns, reflecting similar cautions in the early era of electronic commerce. As IoT approaches\ud mature, this initial technical focus is likely to shift to considerations of interactivity and policy. In this paper,\ud rather than considering the broader range of IoT applications in the military context, we focus on roles for IoT\ud concepts and devices in future intelligence, surveillance and reconnaissance (ISR) tasks, drawing on experience\ud in sensor-mission resourcing and human-computer collaboration (HCC) for ISR. We highlight the importance of\ud low training overheads in the adoption of IoT approaches, and the need to balance proactivity and interactivity\ud (push vs pull modes). As with sensing systems over the last decade, we emphasize that, to be valuable in ISR\ud tasks, IoT devices will need a degree of mission-awareness in addition to an ability to self-manage their limited\ud resources (power, memory, bandwidth, computation, etc). In coalition operations, the management and potential\ud sharing of IoT devices and systems among partners (e.g., in cross-coalition tactical-edge ISR teams) becomes a\ud key issue due heterogeneous factors such as language, policy, procedure and doctrine. Finally, we briefly outline a\ud platform that we have developed in order to experiment with human-IoT teaming on ISR tasks, in both physical\ud and virtual settings.

Previous research has suggested that perceived distances are scaled by the action capabilities of the body. The present studies showed that when "reachability" is constrained due to a difficult grasp required to pick up an object, perceived distance to the object increases. Participants estimated the distances to tools with handle orientations that made them either easy or difficult to grasp with their dominant and nondominant hands. Right-handed participants perceived tools that were more difficult to grasp to be farther away than tools that were easier to grasp. However, perceived distance did not differ in left-handed participants. These studies suggest that, when reaching toward a target, the distance to that target is scaled in terms of how far one can effectively reach, given the type of reaching posture that is executed. Furthermore, this effect is modulated by handedness.

Spatial perception is sensitive to the energetic costs required to perform intended actions. For example, hills look steeper to people who are fatigued or burdened by a heavy load. Similarly, perceived distance is also influenced by the energy required to walk or throw to a target. Such experiments demonstrate that perception is a function, not just of optical information, but also of the perceiver’s potential to act and the energetic costs associated with the intended action. In the current paper, we expand on the notion of “cost” by examining perceived distance in patients diagnosed with chronic pain, a multifactorial disease, which is experienced while walking. We found that chronic pain patients perceive target distances to be farther away compared with a control group. These results indicate the physical, and perhaps emotional, costs of chronic pain affect spatial perceptions.

A considerable amount of research has been conducted on the role interactivity, active versus passive navigation, for learning the spatial layout of a virtual environment (VE). However, active navigation is not unitary. It has two distinct components: decision-making and control. In the present work we investigated which main component of active navigation was critical for acquiring spatial knowledge of a virtual city. We found that spatial knowledge was comparable when the VE was learned with active navigation or decision-making in the absence of control, but was much worse when only control was present. These results suggest decision-making, not control, is the critical component for learning a VE.

A correction to this article has been published and is linked from the HTML version of this paper. The error has not been fixed in the paper.

Recent developments in sensing technologies, mobile devices and context-aware user interfaces have made it possible to represent information fusion and situational awareness as a conversational process among actors - human and machine agents - at or near the tactical edges of a network. Motivated by use cases in the domain of security, policing and emergency response, this paper presents an approach to information collection, fusion and sense-making based on the use of natural language (NL) and controlled natural language (CNL) to support richer forms of human-machine interaction. The approach uses a conversational protocol to facilitate a flow of collaborative messages from NL to CNL and back again in support of interactions such as: turning eyewitness reports from human observers into actionable information (from both trained and untrained sources); fusing information from humans and physical sensors (with associated quality metadata); and assisting human analysts to make the best use of available sensing assets in an area of interest (governed by management and security policies). CNL is used as a common formal knowledge representation for both machine and human agents to support reasoning, semantic information fusion and generation of rationale for inferences, in ways that remain transparent to human users. Examples are provided of various alternative styles for user feedback, including NL, CNL and graphical feedback. A pilot experiment with human subjects shows that a prototype conversational agent is able to gather usable CNL information from untrained human subjects.

Perception of one's body is related not only to the physical appearance of the body, but also to the neural representation of the body. The brain contains many body maps that systematically differ between right- and left-handed people. In general, the cortical representations of the right arm and right hand tend to be of greater area in the left hemisphere than in the right hemisphere for right-handed people, whereas these cortical representations tend to be symmetrical across hemispheres for left-handers. We took advantage of these naturally occurring differences and examined perceived arm length in right- and left-handed people. When looking at each arm and hand individually, right-handed participants perceived their right arms and right hands to be longer than their left arms and left hands, whereas left-handed participants perceived both arms accurately. These experiments reveal a possible relationship between implicit body maps in the brain and conscious perception of the body.

When people are engaged in a skilled behavior, such as occurs in sports, their perceptions relate optical information to their performance. In the present research, we demonstrate the effects of performance on size perception in golfers. We found that golfers who played better judged the hole to be bigger than did golfers who did not play as well. In follow-up laboratory experiments, participants putted on a golf mat from a location near or far from the hole and then judged the size of the hole. Participants who putted from the near location perceived the hole to be bigger than did participants who putted from the far location. Our results demonstrate that perception is influenced by the perceiver's current ability to act effectively in the environment.

This study examined the effect of physical display size on spatial knowledge of a virtual environment (VE). Previous research by Tan (2004) found that performance for a path integration task in a VE was superior on a large display compared to a small one, even with visual angle held constant. This may be because large displays evoke a more egocentric frame of reference than small displays. To test whether the advantages of large displays extended to a navigation transfer task, the present studies examined the effect of display size on transferring knowledge about the spatial layout of a desktop VE to virtual reality (VR). Participants used a joystick to control their movement through a virtual city environment, finding and learning the locations of five targets separated by buildings. After 20 minutes of learning, participants’ spatial knowledge was tested in a fully immersive environment by using virtual reality. Participants stood at each target location and pointed to each of the other, unseen targets. In the first study participants learned the VE on either an 18” LCD display or a 112” projected display. Visual angle was not equated. In the second study participants learned the VE on either a 25” or 72” projected display and visual angle was held constant. In both studies we found the average angular pointing error was significantly lower on the large display. Our results suggest that the advantage of a large desktop display for a virtual environment transfers to VR and is not dependent on visual angle. Bigger is Better: Large Visual Displays Improve Spatial Knowledge of a Virtual Environment