Search

While non-player opponents in commercial video games often rely on simple artificial intelligence techniques, machine learning techniques that capture human strategies could make them more engaging. Cognitive Shadow is a prototype tool that combines several artificial intelligence techniques to continuously model human decision-making patterns during tasks that require categorical decision-making. The present study aims to assess the potential of Cognitive Shadow to create learning opponents that will counter the player's decisions in a strategy game, making it more challenging and engaging. The game developed to this end is a more complex version of rock-paper-scissors, set within the context of a wizards’ duel. Each participant (Player 1) took part in three game sessions of 12 battles (each including five rounds), only being told that they would face a non-player opponent. During Session 1, Cognitive Shadow was in learning mode, thus the non-player opponent (Player 2) chose its plays at random. During Session 2, Cognitive Shadow was active and helped counter participants’ decisions without their knowledge. Before Session 3, participants were informed that their opponent was using machine learning to anticipate and counter their strategy. The results showed that Player 2 was more effective with the help of Cognitive Shadow, having won significantly more battles in Sessions 2 and 3 than in Session 1. In addition, the level of engagement reported by human players increased significantly in Session 3. These results indicate that cognitive shadowing can be used in a strategy game to increase engagement when players are aware of the learning behavior.

Decision making can be modeled in various ways for the design of decision-support systems. One strategy privileged for this purpose is policy capturing, i.e. using statistical techniques (and more recently machine learning) to model judgement policies. The Cognitive Shadow is a prototype tool suited for frugal learning that automatically learns a user’s decision pattern in real time based on an ensemble of seven supervised learning algorithms. This tool can provide advisory warnings when the user decision is inconsistent with the predicted outcome. Evolutionary computation methods could reinforce the system’s efficiency because of their ability to deal with computational complexity via evolution-inspired optimization mechanisms. The goal of this study was to assess the potential of evolutionary algorithms for frugal learning in an online policy capturing context. To do so, we tested three evolutionary algorithms on three different datasets (each split in three sizes), and compared both their prediction performance and training time with that of the other modeling techniques already implemented in the Cognitive Shadow system. Although all three evolutionary models were generally outperformed by non-evolutionary learning algorithms, one genetic programming method showed good prediction performance for the more complex use cases with the smaller datasets.

Security surveillance represents a cognitively challenging task. To support surveillance operators, Thales Research and Technology Canada developed the Scantracker, which relies on real-time oculometric data to trigger notifications that mitigate camera negligence, attentional tunneling and vigilance decrement. Recently, this tool was adapted to augmented reality (AR), thus opening opportunities for design adjustments. This paper documents and demonstrates the utility of adopting a design thinking approach to improve the Scantracker and to adapt AR implementation to users’ needs. Surveillance operators took part in two design thinking workshops to evaluate and suggest improvements to the Scantraker from a user perspective. Then, the tool was adjusted and tested in a security surveillance simulation. Results of the design thinking workshop and the surveillance simulation are presented and discussed to support this user-centered approach for the development of cognitive-support tools for surveillance.

With the burgeoning of wearable devices and passive body/brain-computer interfaces (B/BCIs), automated stress monitoring in everyday settings has gained significant attention recently, with applications ranging from serious games to clinical monitoring. With mobile users, however, challenges arise due to other overlapping (and potentially confounding) physiological responses (e.g., due to physical activity) that may mask the effects of stress, as well as movement artifacts that can be introduced in the measured signals. For example, the classical increase in heart rate can no longer be attributed solely to stress and could be caused by the activity itself. This makes the development of mobile passive B/BCIs challenging. In this paper, we introduce PASS, a multimodal database ofPhysicalActivity andStresScollected from 48 participants. Participants performed tasks of varying stress levels at three different activity levels and provided quantitative ratings of their perceived stress and fatigue levels. To manipulate stress, two video games (i.e., a calm exploration game and a survival game) were used. Peripheral physical activity (electrocardiography, electrodermal activity, breathing, skin temperature) as well as cerebral activity (electroencephalography) were measured throughout the experiment. A complete description of the experimental protocol is provided and preliminary analyses are performed to investigate the physiological reactions to stress in the presence of physical activity. The PASS database, including raw data and subjective ratings has been made available to the research community athttp://musaelab.ca/pass-database/. It is hoped that this database will help advance mobile passive B/BCIs for use in everyday settings.

Mental workload assessment is of great importance for safety critical applications, especially in situations that involve physical demands, such as with first responders (e.g., paramedics, firefighters, or police officers). Advancements in physiological signal monitoring with wearable sensors have made way for real-time mental workload assessment using physiological signals. However, these models have typically been conducted in controlled laboratory settings and rely on a single physiological modality. As a result, such models often experience a drop in performance due to movement artifacts introduced in real-life conditions. In this paper, we demonstrate that a multi-modal mental workload model not only improves measurement accuracy, but can also increase robustness against physical activity artifacts. To this end, an experiment was conducted where mental workload and physical activity levels were modulated simultaneously while physiological data was collected from 48 participants using off-the-shelf wearable devices. Results show improved mental workload assessment with multi-modal fusion under varying physical activity conditions.

Providing decision support to operators in command and control contexts requires careful assessment of its impacts on task performance. Here we describe a human-in-the-loop experiment using a naval air defense testbed to compare three conditions: 1) a baseline interface (DSSBASE; 2) one that displays the temporal proximity of radar aircraft (DSSTEMP); and 3) one which adds a change history panel (DSSCHEX). Threatevaluation accuracy and cognitive models of participants’ judgments did not significantly differ across conditions. Eye-tracking data showed that DSSTEMP lead to a reduced verification of track attributes. Confusion matrices also differed across conditions: DSSTEMP lead to more errors that are “two-categoriesaway” when erroneously classifying hostile aircraft as non-hostile or uncertain. We conclude that in addition to looking at standard metrics of task accuracy and response times, critical insights can be obtained by assessing how different designs for decision support may alter strategies and cognitive processes.

Thales Research and Technology Canada is developing a decision support system consisting in multiple classification models trained simultaneously online to capture experts’ decision policies based on their previous decisions. The system learns decision patterns from examples annotated by a human expert during a training phase of knowledge capture. Because of the small volume of labeled data, we investigated a machine learning technique called active learning that copes with the dilemma of learning with minimal resources and aims at requesting the most informative samples in a pool given the current models. The current study evaluates the impact of using active learning over an uninformed strategy (e.g., random sampling) in the context of policy capturing to reduce the annotation cost during the knowledge capture phase. This work shows that active learning has potential over random sampling for capturing human decision policies with minimal amount of examples and for reducing annotation cost significantly.

Assessment of mental workload is crucial in safety-critical applications. Often, such applications require the user to be ambulant, such as first responders (e.g., paramedics, firefighters, or police officers). Typically, mental workload models have relied on electroencephalography (EEG) signals. EEGs, however, are known to be highly sensitive to movement artefacts, thus limited applications exist for ambulant users and studies have mostly occurred in controlled laboratory settings. In this paper, we explore the robustness of new non-linear features against movement artefacts and test their effectiveness in monitoring mental workload for ambulant users with the end goal of developing mitigation measures based on mental state of operators. To this end, an EEG experiment was conducted where mental workload and physical activity levels were modulated simultaneously and data was collected from 48 participants. Classical EEG features used for workload assessment, such as spectral power and amplitude/phase coherence, were used as benchmarks and compared against the proposed non-linear multi-scale permutation entropy features. Experimental results show the proposed features consistently outperforming the benchmark ones, thus high-lighting their robustness to movement artefacts.

Stress is well known to have negative effects on health and workplace performance. Physiological sensing using wearables shows in turn great potential for realtime stress monitoring. While some off-the-shelf consumer products (e.g. smartwatches) already feature stress detection, there is still a pressing need to improve the robustness of these models in ecological settings where physical activity can hamper detection accuracy. In this paper, we show that using a multimodal physiological stress model can not only improve model accuracy, but can increase robustness to physical activity inference. To do so, we propose a video game based method to elicit emotional responses. More specifically, 48 participants played video games in which psychological stress and physical activity were jointly modulated. Physiological features showing robustness to stress are analyzed in order to guide further research.

Mental workload assessment is crucial in many real life applications which require constant attention and where imbalance of mental workload resources may cause safety hazards. As such, mental workload and its relationship with heart rate variability (HRV) have been well studied in the literature. However, the majority of the developed models have assumed individuals are not ambulant, thus bypassing the issue of movement-related electrocardiography (ECG) artifacts and changing heart beat dynamics due to physical activity. In this work, multi-scale features for mental workload assessment of ambulatory users is explored. ECG data was sampled from users while they performed different types and levels of physical activity while performing the multi-attribute test battery (MATB-II) task at varying difficulty levels. Proposed features are shown to outperform benchmark ones and further exhibit complementarity when used in combination. Indeed, results show gains over the benchmark HRV measures of 24.41 % in accuracy and of 27.97 % in F1 score can be achieved even at high activity levels.

Policy capturing is a judgment analysis method that typically uses linear statistical modeling to estimate expert judgments. A variant to this technique is to capture decision policies using data-mining algorithms designed to handle nonlinear decision rules, missing attributes, and noisy data. In the current study, we tested the effectiveness of a decision-tree induction algorithm and an instance-based classification method for policy capturing in comparison to the standard linear approach. Decision trees are relevant in naturalistic decision-making contexts since they can be used to represent “fast-and-frugal” judgment heuristics, which are well suited to describe human cognition under time pressure. We examined human classification behavior using a simulated naval air defense task in order to empirically compare the C4.5 decision-tree algorithm, the k-nearest neighbors algorithm, and linear regression on their ability to capture individual decision policies. Results show that C4.5 outperformed the other methods in terms of goodness of fit and cross-validation accuracy. Decision-tree models of individuals’ judgment policies actually classified contacts more accurately than their human counterparts, resulting in a threefold reduction in error rates. We conclude that a decision-tree induction algorithm can yield useful models for training and decision support applications, and we discuss the application of judgmental bootstrapping in real time in dynamic environments.

Research on adaptive systems based on human psychophysiological assessments has been growing rapidly over the last decade. One fundamental component of such a system is human state assessment, on which the adaptation depends, at least partly. This is critical as the confidence of operators in such system will be determined to some extent by the accuracy of the models responsible for the assessment. This chapter presents work carried out in order to better understand the relationship between bio-behavioral data, psychological state, and operational performance of the operators. Modeling physiological parameters and performance was performed through a manipulation of three factors. The first factor was the size of smoothing window for performance. The second factor was the performance decrement threshold for labelling functional and sub-functional states. Finally, the third factor was the mode of classification being either prospective or descriptive. We used two types of classifiers, a linear and a non-linear classifier, and compared performance. Insights emerging from this work support that the use of multiple sources of bio-behavioral data, combined in a non-linear fashion, increases the psychometric qualities of state classifiers. This suggests that if such systems are to be used in safety-critical systems, they should be implemented using a wide variety of sensors to increase classification accuracies of performance.

Assessment of mental workload is crucial for applications which require constant attention and where conditions such as mental fatigue and drowsiness must be avoided. As such, electroencephalography (EEG) based mental workload models have been developed in the past. The majority of these models, however, have assumed individuals are not ambulant, thus bypassing the issue of movement-related EEG artefacts. While such models may be useful for a number of applications (e.g., operators are sitting), they may not apply in situations in which operators are performing their task under different physical activity levels. Representative examples can include first responders, such as paramedics, firefighters, or police officers. In this work, we take the first steps towards overcoming this limitation and present results of an experiment simultaneously eliciting increasing mental workload states at varying physical activity levels. EEG data from forty-seven participants was collected while they performed the NASA Revised Multi-Attribute Task Battery II (MATB-II) under three different activity level conditions (no, medium, high). In this study, we report the effects of activity on the noise-robustness and distribution of several spectral, amplitude/phase coherence, and amplitude modulation features, with the ultimate goal of deriving a feature set tailored towards automated workload assessment during physical activity. Preliminary results show spectral features acquired from the frontal area of the cortex as the most promising and that activity aware mental workload models should be developed.

Policy capturing is a decision analysis method that typically uses linear statistical modeling to estimate the basis of expert judgments. Using more flexible data mining algorithms may yield more accurate models or instead result in poor functional estimations. The objective of this study is to test the effectiveness of a decision tree induction algorithm for policy capturing in comparison to the standard linear approach. We examined human classification behavior using a simulated naval air-defense task in order to empirically compare the C4.5 decision tree algorithm to linear regression on their ability to capture individual decision policies. The pattern of results shows that C4.5 outperformed linear regression in terms of goodness-of-fit and cross-validation accuracy. Results also show that the decision tree models of individuals’ judgment policies actually classified contacts more accurately than their human counterparts. We conclude that non-linear policy capturing can yield useful models for training and decision support applications.

The present work investigates a cumulative part-task training method that builds up task complexity adaptively based on individual learner states. A research-oriented game entitled “Space Fortress” was used to evaluate two training conditions in a between-group design prior to a third condition involving an adaptive cumulative part task training method. The latter detects when the learner is ready to progress and dynamically adjusts training progression. Here we report the results of the first two conditions. First was the full task condition, where the learner was exposed to the entire task throughout the training session. The second condition followed a cumulative part-task training approach, where sub-tasks were added at fixed progression points. Results showed no statistically significant gain nor loss in terms of learning outcomes between the full task and the non-adaptive cumulative part task condition, adding evidence to previous mixed findings. A trigger rule needed for the adaptive cumulative part task training condition was developed based on short-term patterns of change in performance and mental workload to be used as a dynamic criterion for adaptation. Furthermore, bio-behavioral measures were evaluated as potential proxies for performance and workload with the aim of applying this adaptive method in contexts where performance and workload cannot be directly measured at regular intervals.

Automated mental workload measurement is particularly important in safety-critical settings, such as in nuclear plants, aviation, air traffic control, shipping, and transportation, to name a few. As an example, recent statistics have suggested that 90% of the accidents in the transport industry are due to human factors. In this paper, we explore the potential of off-the-shelf wearable technologies in monitoring mental workload in real-time, thus potentially reducing the number of accidents due to human errors. Wearable technologies, while providing the user with ease-of-use, comfort, and portability, have several limitations, such as lower quality signal readings (e.g., due to dry electrodes) and smaller number of recording sites. Such limitations place a burden on the accuracy of existing mental workload models. To overcome this limitation, we propose the use of phase-amplitude and amplitude-amplitude coupling features computed from a portable commercial electroencephalography (EEG) device. Experiments with three different tasks, namely N-back, mental rotation and visual search, show the proposed features being significantly correlated with multiple dimensions of the widely-used NASA task load index test and providing complementary information to other conventional features.

The assessment of an operator's functional state (i.e., the multidimensional pattern of human psychophysiological conditions that mediates performance) has great potential for increasing safety and reliability of critical systems. However, live monitoring of functional state using physiological and behavioral data still faces several challenges before achieving the level of precision required in many operational contexts. One open question is the level of granularity of the models. Is a general model sufficient or should subject-specific models be trained to ensure high accuracy? Another challenge concerns the formalization of a valid ground truth for training classifiers. This is critical in order to train models that are operationally relevant. This paper introduces the Decontextualized Dynamic Performance (DDP) metric which allows models to be trained simultaneously on different tasks using machine learning algorithms. This paper reports the performance of various classification algorithms at different levels of granularity. We compare a general model, task-specific models, and subject-specific models. Results show that the classification methods do not lead to statistically different performance, and that the predictive accuracy of subject-specific and task-specific models was actually comparable to a general model. We also compared various time-window sizes for the new DDP metric and found that results were degrading with a larger time window size.

The authors report two experiments studying the requirements for effective decision making in a complex environment. The focus lies on three components of individual cognitive readiness: situation awareness (SA), problem solving, and decision making. Participants performed a simulated society management task in which they could allocate resources to stabilize a national crisis involving multiple interrelated factors (political, economic, environmental, and social). A striking aspect of this simulation is that even though information about the causes and effects within the system is available, most individuals fail to bring the system to the targeted state because of unintended consequences of their decisions. The experiments test the impact of two cognitive support tools designed to improve anticipation of future outcomes. Results show that supporting short-term anticipation (with perfectly accurate projections) was insufficient to improve effectiveness, but supporting long-term anticipation (with approximate projections) successfully improved performance in this complex environment. We conclude with a review of requirements that training and technological support should address to augment individual cognitive readiness for operations in complex environments and propose an extension to SA theory by conceptualizing a Level 4 SA (long-term projection) that may be particularly important to overcome the “wall of complexity.”

Omnipresent in everyday multitasking environments task interruptions are usually detrimental to individual performance. Here, we examined whether teaming up renders an individual less vulnerable to interruptions in complex and dynamic situations. We employed a microworld to simulate command and control in a crisis management situation and to examine the relative impact of interruptions on operators working in a functional dyad versus operators working alone. While task interruption was detrimental to efficacy in supervisory control of both single and team interrupted operators, the latter were less vulnerable than the former. However, teaming up did not translate into faster resumption time, a consequence of the overhead attributable to coordination and communication requirements of collaborative work. These findings suggest that in complex and dynamic environments working in a small team confers more resistance to task interruption than working alone and speed of interruption recovery is no guarantee of quality of recovery.

Research in cognitive systems engineering (CSE) and decision support requires an understanding of the psychological processes involved in a given task. The purpose of the present study is to investigate how policy capturing and process tracing may help understand the decision mechanisms involved in a naval air-defense task and characterize how human decision making effectiveness can be improved. We report results from a study in which participants performed a threat evaluation and weapons assignment task within a naval air-defense microworld. Policy capturing and process tracing techniques provide both converging perspectives and complementary insights into complex decision making processes.

Temporal awareness is key to successful decision making in a wide range of command and control situations, yet little explicit support to maintaining temporal awareness is provided by Decision Support Systems (DSS) for time-critical decisions. In the context of simulated weapon-target scheduling, the present study compared the decision support gained from two display formats: typical geospatial display and temporal display. The results demonstrated that the temporal display facilitates scheduling performance though its beneficial impact seems to require greater familiarization.

Feldman [(2000). Minimization of Boolean complexity in human concept learning. Nature, 407, 630–633] has provided evidence that the complexity (subjective difficulty) of a Boolean concept is related to the length of its minimum algebraic description. The catalog of Boolean concepts [Feldman, J. (2003a). A catalog of Boolean concepts. Journal of Mathematical Psychology, 47, 98–112] provides an extensive list of basic categorization problems along with their minimal algebraic descriptions and complexity levels. In the present study, we demonstrated that the minimum algebraic description is very useful for describing the most economical categorization rule that can be used, but does not always correspond to the category definition used by a participant in a concept-learning task. We developed a new minimization procedure, reanalyzed Feldman's catalog and revised the complexity of 148 out of 253 basic concept types. Among these cases, two new solutions that correspond to concept types used in the classic study on categorization by Shepard, Hovland, and Jenkins [(1961). Learning and memorization of classifications. Psychological Monographs: General and Applied, 75, 1–42] predicted a new difficulty rank ordering (for Types I–VI) that did not agree with experimental results reported in the literature. New experimental results reported here show that participants used similar non-minimal classification rules that can be explained by the hypothesis-testing sequence in the RULEX model [Nosofsky, R. M., Palmeri, T. J., & McKinley, S. C. (1994). Rule-plus-exception model of classification learning. Psychological Review, 101, 53–79].

Intelligent video surveillance systems (IVSS) that can track people, identify unattended objects, and recognize activity are constantly being developed, updated, and tested against standard test stimuli. The development of these systems, however, seems to focus on technological advancements and largely ignores how automation will affect the human closed-circuit television (CCTV) operator. This paper outlines the advantages of employing a cognitive system engineering approach to the design of IVSS, reviews the literature that has examined human factors issues related to CCTV automation, and introduces a testbed for investigating the effects of automated surveillance technology on human CCTV operators.

The present work describes the application of a multi-criteria assessment tool called MYRIAD to the evaluation of a methodology that was developed to improve team understanding of a complex situation. The methodology aims to support a multi-disciplinary team working collaboratively on the development of a mission plan during an expeditionary stability operation. We conducted a case study assessing collaborative understanding and team performance while subject matter experts employed the methodology as a complement to the standard planning process. Results were analyzed using MYRIAD, a preference modeling system that allows combining disparate measures into a coherent assessment capturing several key logical relationships between metrics – ones that may not be modeled using the traditional weighted sum approach. MYRIAD was also used to perform a sensitivity analysis in order to derive from the preference model which aspects of the planning methodology would lead to the greatest overall improvement. Results helped identifying priority areas for future development.

Objective: This study examined whether teaming up mitigates individual vulnerability to task interruptions in complex dynamic situations. Background: Omnipresent in everyday multitasking environments, task interruptions are usually detrimental to individual performance. This is particularly crucial in dynamic command and control (C2) safety-critical contexts because of the additional challenge imposed by the continually evolving situation during the interruption. Method: We employed a firefighting microworld to simulate C2 in the context of supervisory control to examine the relative impact of interruptions on participants working in a functional dyad versus operators working alone. Results: Although task interruption was detrimental to participants’ efficacy of monitoring resources, the negative impact of interruption was reduced for those working in teams. Teaming up translated into faster resumption time, but only if both teammates were interrupted simultaneously. Interrupting only one team member was associated with increased postinterruption communications and slower resumption time. Conclusion: These findings suggest that in complex dynamic situations working in a small team confers more resistance to task interruption than working alone by virtue of the reduced individual workload typical of teamwork. The benefit of collaborative work seems nevertheless mediated by the coordination and communication overhead associated with teamwork. Application: The present findings have practical implications for operators dealing with unexpected events such as task interruptions in C2 environments.

Interactive learning environments are increasingly used to help people better deal with complex situations. One way to improve decision making effectiveness is to train systems thinking skills using interactive simulations in order to reduce the occurrence of “unintended side-effects” of interventions and catastrophic failures. We present a prototype training procedure intended for military officers and civilian personnel engaged in “full spectrum” operations. The Complex Decision Making simulation environment (CODEM) is the core of the proposed training procedure. CODEM aims to improve systems thinking skills, adaptability and other abilities associated with the integrative concept of cognitive readiness. Four training scenarios are designed to reproduce key properties of complex decision making situations. An intelligent tutor providing corrective feedback on decision making behaviors is integrated into each scenario. The tutor interventions help avoid tunnel vision (i.e., the opposite of systems thinking) by discouraging the use of overly simple heuristics. Nine behavioral metrics are monitored by the intelligent tutor—five are related to information seeking behaviors, and four are related to specific decision patterns. The effectiveness of this prototype training procedure is currently being assessed experimentally.

The complex decision making experimental platform (CODEM) is intended as a shareable research tool to stimulate multidisciplinary research on complex dynamic situation management and as an environment for training and testing cognitive readiness. The experimenter can set general parameters, configure the interface, specify the model, insert events and define the resources and capabilities of each player using the scenario development tool. No programming skills are required. Task complexity can be varied by introducing feedback loops, delayed effects, time pressure, situational uncertainty, adjusting model transparency and changing the relationships between system elements. CODEM creates detailed logs of events and actions essential for cognitive process tracing and the evaluation of decision making effectiveness. The first task designed with CODEM is a counter-insurgency scenario in which a coalition force seeks to stabilize a failing state. A genetic algorithm is used to estimate the best strategy in that scenario for comparison with human results. An adversarial version also allows insurgents to be controlled by a human opponent (or a red team) rather than an artificial agent. CODEM can be used as a cognitive engineering testbed and as a training environment for improving decision making and adaptation skills in complex situations.

We investigated the effects of a decision aid designed to support the development of effective long term strategies when seeking to influence a complex adaptive system. The experiment compared a test condition with feedforward support (i.e., outcome projection) against a baseline condition both in terms of cognitive processing and goal attainment (i.e., strategy efficiency). Results show that the decision aid significantly improved the development of effective long term strategies, yet detracted participants from engaging in information acquisition behaviors. In both conditions, participants displayed a poor (unaided) ability to anticipate the system's near term behavior, which was significantly lower in the decision aid condition. Goal attainment was positively related to frequency and comprehensiveness of information acquisition, Level 3 situation awareness (near term projection), and the degree to which the participants' strategies were proactive. The decision aid therefore clearly helped overcome a serious limitation in human processing abilities in terms of outcome projection (thus helping participants develop better strategies). However, results show that decision aids may also come with certain drawbacks that need to be systematically identified and mitigated to obtain the full benefits of cognitive support technology.