Search

The Cognitive Shadow is a prototype decision support tool that can notify users when they deviate from their usual judgment pattern. Expert decision policies are learned automatically online while performing one’s task using a combination of machine learning algorithms. This study investigated whether combining this system with the use of a process tracing technique could improve its ability to model human decision policies. Participants played the role of anti-submarine warfare commanders and rated the likelihood of detecting a submarine in different ocean areas based on their environmental characteristics. In the process tracing condition, participants were asked to reveal only the information deemed necessary, and only that information was sent to the system for model training. In the control condition, all the available information was sent to the system with each decision. Results showed that process tracing data improved the model’s ability to predict human decisions compared to the control condition.

Security surveillance entails many cognitive challenges (e.g., task interruption, vigilance decrements, cognitive overload). To help surveillance operators overcome these difficulties and perform more efficient visual search, gaze-based intelligent systems can be developed. The present study aimed at testing the impact of the Scantracker system—which pinpointed neglected cameras while detecting and correcting attentional tunneling and vigilance decrease—on human scanning behavior and surveillance performance. Participants took part in a surveillance simulation, monitoring cameras and searching for ongoing incidents, and half of them was supported by the Scantracker. Although behavioral surveillance performance was not improved, participants supported by the Scantracker showed more efficient gaze-based measures of surveillance. Moreover, some of these measures were associated with performance, suggesting that scan pattern improvements might lead indirectly to more efficient incident detection. Overall, these results speak to the potential of using gaze- aware intelligent systems to support surveillance operators.

The Cognitive Shadow is a prototype tool intended to support decision making by autonomously modeling human operators’ response pattern and providing online notifications to the operators about the decision they are expected to make in new situations. Since the system can be configured either in a reactive “shadowing” or a proactive “recommendation” mode, this study aimed to determine its most effective mode in terms of human and model accuracy, workload, and trust. Subjects participated in an aircraft threat evaluation simulation without decision support or while using either mode of the Cognitive Shadow. Whereas the recommendation mode had no advantage over the control condition, the shadowing mode led to higher human and model accuracy. These benefits were maintained even when the tool was unexpectedly removed. Neither mode influenced workload, and the initial lower trust rating in the shadowing mode faded quickly, making it the best overall configuration for the cognitive assistant.

While non-player opponents in commercial video games often rely on simple artificial intelligence techniques, machine learning techniques that capture human strategies could make them more engaging. Cognitive Shadow is a prototype tool that combines several artificial intelligence techniques to continuously model human decision-making patterns during tasks that require categorical decision-making. The present study aims to assess the potential of Cognitive Shadow to create learning opponents that will counter the player's decisions in a strategy game, making it more challenging and engaging. The game developed to this end is a more complex version of rock-paper-scissors, set within the context of a wizards’ duel. Each participant (Player 1) took part in three game sessions of 12 battles (each including five rounds), only being told that they would face a non-player opponent. During Session 1, Cognitive Shadow was in learning mode, thus the non-player opponent (Player 2) chose its plays at random. During Session 2, Cognitive Shadow was active and helped counter participants’ decisions without their knowledge. Before Session 3, participants were informed that their opponent was using machine learning to anticipate and counter their strategy. The results showed that Player 2 was more effective with the help of Cognitive Shadow, having won significantly more battles in Sessions 2 and 3 than in Session 1. In addition, the level of engagement reported by human players increased significantly in Session 3. These results indicate that cognitive shadowing can be used in a strategy game to increase engagement when players are aware of the learning behavior.

Operators monitoring closed-circuit television systems face many error-prone situations given the cognitive challenges they experience. The Scantracker is a prototype tool developed to support surveillance, relying on live oculometric data to trigger notifications that mitigate camera negligence, attention tunneling and vigilance decrement. To be robust and flexible, this tool must however collect valid gaze measures at a sufficient sampling rate and allow the display of surveillance-supportive visual notifications. This paper documents the integration of the Scantracker with the Microsoft HoloLens 2 augmented reality (AR) system embarked with eye tracking and reports a validity study performed in a high-fidelity surveillance simulation. The AR solution led to superior gaze validity than previous mobile or fixed oculometer integration. Implications of the solution integration are discussed and compared with the other implementations.

Decision making can be modeled in various ways for the design of decision-support systems. One strategy privileged for this purpose is policy capturing, i.e. using statistical techniques (and more recently machine learning) to model judgement policies. The Cognitive Shadow is a prototype tool suited for frugal learning that automatically learns a user’s decision pattern in real time based on an ensemble of seven supervised learning algorithms. This tool can provide advisory warnings when the user decision is inconsistent with the predicted outcome. Evolutionary computation methods could reinforce the system’s efficiency because of their ability to deal with computational complexity via evolution-inspired optimization mechanisms. The goal of this study was to assess the potential of evolutionary algorithms for frugal learning in an online policy capturing context. To do so, we tested three evolutionary algorithms on three different datasets (each split in three sizes), and compared both their prediction performance and training time with that of the other modeling techniques already implemented in the Cognitive Shadow system. Although all three evolutionary models were generally outperformed by non-evolutionary learning algorithms, one genetic programming method showed good prediction performance for the more complex use cases with the smaller datasets.

Security surveillance represents a cognitively challenging task. To support surveillance operators, Thales Research and Technology Canada developed the Scantracker, which relies on real-time oculometric data to trigger notifications that mitigate camera negligence, attentional tunneling and vigilance decrement. Recently, this tool was adapted to augmented reality (AR), thus opening opportunities for design adjustments. This paper documents and demonstrates the utility of adopting a design thinking approach to improve the Scantracker and to adapt AR implementation to users’ needs. Surveillance operators took part in two design thinking workshops to evaluate and suggest improvements to the Scantraker from a user perspective. Then, the tool was adjusted and tested in a security surveillance simulation. Results of the design thinking workshop and the surveillance simulation are presented and discussed to support this user-centered approach for the development of cognitive-support tools for surveillance.

Resource-constrained project scheduling for ship refit and maintenance is a major challenge for planners. A smart scheduling solution proposed herein relies on a combination of optimization methods including constraint programming and mixed-integer linear programming. The method employs model-based AI, heuristic methods and discrete-event simulation to efficiently schedule project tasks while handling precedence constraints, resource constraints (labor, equipment) and capacity constraints. The present study investigated the key challenge of managing geospatial constraints. A new solution is presented that captures in a generic fashion the geospatial relationships of work areas and that analyses schedules to detect proximity and path-based conflicts occurring over the course of the project plan. A visualization support tool was designed that generates an abstract 3D model to help intuitively understand and contextualize detected conflicts. Solution effectiveness was assessed and validated using a test scenario. The resulting method is deemed applicable to a broad range of domains.

With the burgeoning of wearable devices and passive body/brain-computer interfaces (B/BCIs), automated stress monitoring in everyday settings has gained significant attention recently, with applications ranging from serious games to clinical monitoring. With mobile users, however, challenges arise due to other overlapping (and potentially confounding) physiological responses (e.g., due to physical activity) that may mask the effects of stress, as well as movement artifacts that can be introduced in the measured signals. For example, the classical increase in heart rate can no longer be attributed solely to stress and could be caused by the activity itself. This makes the development of mobile passive B/BCIs challenging. In this paper, we introduce PASS, a multimodal database ofPhysicalActivity andStresScollected from 48 participants. Participants performed tasks of varying stress levels at three different activity levels and provided quantitative ratings of their perceived stress and fatigue levels. To manipulate stress, two video games (i.e., a calm exploration game and a survival game) were used. Peripheral physical activity (electrocardiography, electrodermal activity, breathing, skin temperature) as well as cerebral activity (electroencephalography) were measured throughout the experiment. A complete description of the experimental protocol is provided and preliminary analyses are performed to investigate the physiological reactions to stress in the presence of physical activity. The PASS database, including raw data and subjective ratings has been made available to the research community athttp://musaelab.ca/pass-database/. It is hoped that this database will help advance mobile passive B/BCIs for use in everyday settings.

Assessment of mental workload is crucial for applications that require sustained attention and where conditions such as mental fatigue and drowsiness must be avoided. Previous work that attempted to devise objective methods to model mental workload were mainly based on neurological or physiological data collected when the participants performed tasks that did not involve physical activity. While such models may be useful for scenarios that involve static operators, they may not apply in real-world situations where operators are performing tasks under varying levels of physical activity, such as those faced by first responders, firefighters, and police officers. Here, we describe WAUC, a multimodal database of mental Workload Assessment Under physical aCtivity. The study involved 48 participants who performed the NASA Revised Multi-Attribute Task Battery II under three different activity level conditions. Physical activity was manipulated by changing the speed of a stationary bike or a treadmill. During data collection, six neural and physiological modalities were recorded, namely: electroencephalography, electrocardiography, breathing rate, skin temperature, galvanic skin response, and blood volume pulse, in addition to 3-axis accelerometry. Moreover, participants were asked to answer the NASA Task Load Index questionnaire after each experimental section, as well as rate their physical fatigue level on the Borg fatigue scale. In order to bring our experimental setup closer to real-world situations, all signals were monitored using wearable, off-the-shelf devices. In this paper, we describe the adopted experimental protocol, as well as validate the subjective, neural, and physiological data collected. The WAUC database, including the raw data and features, subjective ratings, and scripts to reproduce the experiments reported herein will be made available at:http://musaelab.ca/resources/.

Mental workload assessment is of great importance for safety critical applications, especially in situations that involve physical demands, such as with first responders (e.g., paramedics, firefighters, or police officers). Advancements in physiological signal monitoring with wearable sensors have made way for real-time mental workload assessment using physiological signals. However, these models have typically been conducted in controlled laboratory settings and rely on a single physiological modality. As a result, such models often experience a drop in performance due to movement artifacts introduced in real-life conditions. In this paper, we demonstrate that a multi-modal mental workload model not only improves measurement accuracy, but can also increase robustness against physical activity artifacts. To this end, an experiment was conducted where mental workload and physical activity levels were modulated simultaneously while physiological data was collected from 48 participants using off-the-shelf wearable devices. Results show improved mental workload assessment with multi-modal fusion under varying physical activity conditions.

Heart rate variability (HRV) has been studied in the context of human behavior analysis and many features have been extracted from the inter-beat interval (RR) time series and tested as correlates of constructs such as mental workload, stress and anxiety. Such constructs are crucial in assessing quality-of-life of individuals, as well as their overall performance when doing critical tasks. Most studies, however, have been conducted in controlled laboratory environments with artificially-induced psychological responses. While this assures that high quality data are collected, the amount of data is limited and the transferability of the findings to more ecologically-appropriate settings remains unknown. Additionally, it is desirable for such mental state monitoring systems to have high temporal resolution, thus allowing for quick feedback and adaptive decision making. In this article, we explore the use of features computed from time windows much shorter than typically reported in the literature. More specifically, we evaluate the potential of HRV and breathing features computed over so-called ultra-short-term segments (i.e., < 5 minutes) for stress and mental workload prediction. Experiments with 27 police academy trainees show that short time windows as low as 60 seconds can provide useful insights, in particular for mental workload assessment. Moreover, the fusion of HRV and breathing features showed to be an important aspect for reliable behavioural assessment in highly ecological settings.

Providing decision support to operators in command and control contexts requires careful assessment of its impacts on task performance. Here we describe a human-in-the-loop experiment using a naval air defense testbed to compare three conditions: 1) a baseline interface (DSSBASE; 2) one that displays the temporal proximity of radar aircraft (DSSTEMP); and 3) one which adds a change history panel (DSSCHEX). Threatevaluation accuracy and cognitive models of participants’ judgments did not significantly differ across conditions. Eye-tracking data showed that DSSTEMP lead to a reduced verification of track attributes. Confusion matrices also differed across conditions: DSSTEMP lead to more errors that are “two-categoriesaway” when erroneously classifying hostile aircraft as non-hostile or uncertain. We conclude that in addition to looking at standard metrics of task accuracy and response times, critical insights can be obtained by assessing how different designs for decision support may alter strategies and cognitive processes.

The training curriculum of air force cadets is currently identical for all, not taking trainees’ individual differences in skill acquisition into account. A model of physiological arousal conceptualized as “ease in flight” is proposed as an objective metric for individualization. Considering that a significant part of air force cadets training takes place on a flight simulator, the metrics used to provide cursus recommendation should be valid both in flight and in a simulator. This work concerns the validation of “ease in flight” as a metric for training individualization in a simulated task environment. Eight participants performed two consecutive flights on a low fidelity aircraft simulator, whilst wearing a chest strap to measure the electrical activity of the heart and respiratory activity. Results show that declared ease in flight and declared stress are strongly negatively correlated. In addition, measured ease in flight increased significantly from first to second flight. Together, these results suggest that the ease in flight model previously defined using data from experts in-flight generalizes to simulated flight, both from a perceived and objective point of view. Finally, the potential of the model for providing adaptive cursus recommendation through the individualized analysis of measured ease in flight across different required skills is discussed.

The validity of two commercial-off-the-shelf eye trackers, including a fixed system and a mobile device, was compared in the context of a gaze-based assistant tool implemented to aid security surveillance. Participants performed a security surveillance simulation that required them to detect incidents across multiple video feeds. Aid from the Scantracker system—pinpointing neglected cameras while detecting and correcting attentional tunneling and vigilance decrease—was provided using gaze measures. Despite the lower amount of valid data for the mobile system, both devices computed similar measures of surveillance performance. Findings speak to the feasibility of using mobile robust-to-movement decision-support systems based on real-time gaze data to help improve incident detection in security surveillance environments.

Thales Research and Technology Canada is developing a decision support system consisting in multiple classification models trained simultaneously online to capture experts’ decision policies based on their previous decisions. The system learns decision patterns from examples annotated by a human expert during a training phase of knowledge capture. Because of the small volume of labeled data, we investigated a machine learning technique called active learning that copes with the dilemma of learning with minimal resources and aims at requesting the most informative samples in a pool given the current models. The current study evaluates the impact of using active learning over an uninformed strategy (e.g., random sampling) in the context of policy capturing to reduce the annotation cost during the knowledge capture phase. This work shows that active learning has potential over random sampling for capturing human decision policies with minimal amount of examples and for reducing annotation cost significantly.

Assessment of mental workload is crucial in safety-critical applications. Often, such applications require the user to be ambulant, such as first responders (e.g., paramedics, firefighters, or police officers). Typically, mental workload models have relied on electroencephalography (EEG) signals. EEGs, however, are known to be highly sensitive to movement artefacts, thus limited applications exist for ambulant users and studies have mostly occurred in controlled laboratory settings. In this paper, we explore the robustness of new non-linear features against movement artefacts and test their effectiveness in monitoring mental workload for ambulant users with the end goal of developing mitigation measures based on mental state of operators. To this end, an EEG experiment was conducted where mental workload and physical activity levels were modulated simultaneously and data was collected from 48 participants. Classical EEG features used for workload assessment, such as spectral power and amplitude/phase coherence, were used as benchmarks and compared against the proposed non-linear multi-scale permutation entropy features. Experimental results show the proposed features consistently outperforming the benchmark ones, thus high-lighting their robustness to movement artefacts.

Stress is well known to have negative effects on health and workplace performance. Physiological sensing using wearables shows in turn great potential for realtime stress monitoring. While some off-the-shelf consumer products (e.g. smartwatches) already feature stress detection, there is still a pressing need to improve the robustness of these models in ecological settings where physical activity can hamper detection accuracy. In this paper, we show that using a multimodal physiological stress model can not only improve model accuracy, but can increase robustness to physical activity inference. To do so, we propose a video game based method to elicit emotional responses. More specifically, 48 participants played video games in which psychological stress and physical activity were jointly modulated. Physiological features showing robustness to stress are analyzed in order to guide further research.

Interest in stress recognition has notably increased over the past few years. In this work, we focus on recognizing stress from speech. We propose the use of modulation spectral features as input to a convolutional neural network (CNN) for classifying stress. As benchmark, the OpenSMILE features used in the INTERSPEECH 2010 Paralinguistic Challenge is adopted and evaluated with a support vector machine (SVM) and a deep neural network (DNN) based backends. Experiments are performed with the well-known Speech Under Simulated and Actual Stress (SUSAS) database. Performances are investigated considering 2-class, 4-class and 9-class classification problems. Results show that the proposed approach outperforms the benchmark on a challenging 9-class classification task with accuracy as high as 70% representing gains of roughly 18% over the benchmark.

Mental workload assessment is crucial in many real life applications which require constant attention and where imbalance of mental workload resources may cause safety hazards. As such, mental workload and its relationship with heart rate variability (HRV) have been well studied in the literature. However, the majority of the developed models have assumed individuals are not ambulant, thus bypassing the issue of movement-related electrocardiography (ECG) artifacts and changing heart beat dynamics due to physical activity. In this work, multi-scale features for mental workload assessment of ambulatory users is explored. ECG data was sampled from users while they performed different types and levels of physical activity while performing the multi-attribute test battery (MATB-II) task at varying difficulty levels. Proposed features are shown to outperform benchmark ones and further exhibit complementarity when used in combination. Indeed, results show gains over the benchmark HRV measures of 24.41 % in accuracy and of 27.97 % in F1 score can be achieved even at high activity levels.

The present work introduces a prototype intelligent cognitive assistant that continuously learns the decision pattern of the user online, instantly recognizes deviations from that pattern as potential errors and then alerts the user accordingly. We investigated the potential of this prototype system using a human-in-the-loop experiment designed to assess impacts on decision making performance, workload and trust in the decision support capability. Study participants interacted with a naval air-defence testbed to classify radar contacts as friendly, uncertain or hostile based on track parameters displayed on screen. The between-group experimental design included a control condition and two decision support conditions (with system reliability provided either offline or online). Results showed that both decision support conditions significantly improved task accuracy compared to the control condition. The advisory system was successful at improving human performance without burdening the user with excessive additional workload, even when providing reliability information in real time.

Background Future long-duration space exploration missions, such as traveling to Mars, will create an increase in communication time delays and disruptions and remove the viability of emergency returns to Earth for timely medical treatment. Thus, higher levels of medical autonomy are necessary. Crew selection is proposed as the first line of defense to minimize medical risk for future missions; however, the second proposed line of defense is medical preparedness and crew member autonomy. In an effort to develop a decision support system, the Canadian Space Agency mandated a team of scientists from Thales Research and Technology Canada (Québec, QC) and Université Laval (Québec, QC) to create an evidence-based medical condition database linking mission-critical human conditions with key causal factors, diagnostic and treatment information, and probable outcomes. Objective To complement this database, we are currently conducting a scoping review to better understand the depth and breadth of evidence about managing medical conditions in space. Methods This scoping review will adhere to quality standards for scoping reviews, employing Levac, Colquhoun, and O’Brien's 6-stage methodology; the reported results will follow the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) extension for scoping reviews. In stage 1, we identified the research question in collaboration with the Canadian Space Agency (CSA), the main knowledge user. We prioritized 10 medical conditions: (1) acute coronary syndrome, (2) atrial fibrillation, (3) eye penetration, (4) herniated disk, (5) nephrolithiasis, (6) pulmonary embolism, (7) retinal detachment, (8) sepsis, (9) stroke, and (10) spaceflight associated neuro-ocular syndrome. In stage 2, with the help of an information specialist from Cochrane Canada Francophone, papers were identified through searches of the following databases: ARC, Embase, IeeeXplore, Medline Ovid, PsychINFO, and Web of Science. In stage 3, studies will be selected and assessed using a 3-step process and emerging, refined exclusion criteria. In stage 4, the data will be charted in a table based on parameters required by the CSA and developed using Google spreadsheets for shared access. In stage 5, evidence-based descriptive summaries will be produced for each condition, as well as descriptive analyses of collected data. Finally, in stage 6, the findings will be shared with the CSA to guide the completion of this project. Results This study was planned in December 2018. Stage 1 has been completed. The initial database search strategy with all target conditions combined identified a total of 10,403 citations to review through title and abstract screening and after duplicate removal. We plan to complete stages 2-6 by the beginning of 2021. Conclusions This scoping review will map the literature on the management of 10 priority medical conditions in space. It will also enable us to identify knowledge gaps that must be addressed in future research, ensuring successful and medically safe future missions as humankind embarks upon new frontiers of space exploration. International Registered Report Identifier (IRRID) DERR1-10.2196/24323

While simple heuristics can be ecologically rational and effective in naturalistic decision making contexts, complex situations require analytical decision making strategies, hypothesis-testing and learning. Sub-optimal decision strategies – using simplified as opposed to analytic decision rules – have been reported in domains such as healthcare, military operational planning, and government policy making. We investigate the potential of a computational toolkit called “IMAGE” to improve decision-making by developing structural knowledge and increasing understanding of complex situations. IMAGE is tested within the context of a complex military convoy management task through (a) interactive simulations, and (b) visualization and knowledge representation capabilities. We assess the usefulness of two versions of IMAGE (desktop and immersive) compared to a baseline. Results suggest that the prosthesis helped analysts in making better decisions, but failed to increase their structural knowledge about the situation once the cognitive prosthesis is removed.

Policy capturing is a judgment analysis method that typically uses linear statistical modeling to estimate expert judgments. A variant to this technique is to capture decision policies using data-mining algorithms designed to handle nonlinear decision rules, missing attributes, and noisy data. In the current study, we tested the effectiveness of a decision-tree induction algorithm and an instance-based classification method for policy capturing in comparison to the standard linear approach. Decision trees are relevant in naturalistic decision-making contexts since they can be used to represent “fast-and-frugal” judgment heuristics, which are well suited to describe human cognition under time pressure. We examined human classification behavior using a simulated naval air defense task in order to empirically compare the C4.5 decision-tree algorithm, the k-nearest neighbors algorithm, and linear regression on their ability to capture individual decision policies. Results show that C4.5 outperformed the other methods in terms of goodness of fit and cross-validation accuracy. Decision-tree models of individuals’ judgment policies actually classified contacts more accurately than their human counterparts, resulting in a threefold reduction in error rates. We conclude that a decision-tree induction algorithm can yield useful models for training and decision support applications, and we discuss the application of judgmental bootstrapping in real time in dynamic environments.

Research on adaptive systems based on human psychophysiological assessments has been growing rapidly over the last decade. One fundamental component of such a system is human state assessment, on which the adaptation depends, at least partly. This is critical as the confidence of operators in such system will be determined to some extent by the accuracy of the models responsible for the assessment. This chapter presents work carried out in order to better understand the relationship between bio-behavioral data, psychological state, and operational performance of the operators. Modeling physiological parameters and performance was performed through a manipulation of three factors. The first factor was the size of smoothing window for performance. The second factor was the performance decrement threshold for labelling functional and sub-functional states. Finally, the third factor was the mode of classification being either prospective or descriptive. We used two types of classifiers, a linear and a non-linear classifier, and compared performance. Insights emerging from this work support that the use of multiple sources of bio-behavioral data, combined in a non-linear fashion, increases the psychometric qualities of state classifiers. This suggests that if such systems are to be used in safety-critical systems, they should be implemented using a wide variety of sensors to increase classification accuracies of performance.

Assessment of mental workload is crucial for applications which require constant attention and where conditions such as mental fatigue and drowsiness must be avoided. As such, electroencephalography (EEG) based mental workload models have been developed in the past. The majority of these models, however, have assumed individuals are not ambulant, thus bypassing the issue of movement-related EEG artefacts. While such models may be useful for a number of applications (e.g., operators are sitting), they may not apply in situations in which operators are performing their task under different physical activity levels. Representative examples can include first responders, such as paramedics, firefighters, or police officers. In this work, we take the first steps towards overcoming this limitation and present results of an experiment simultaneously eliciting increasing mental workload states at varying physical activity levels. EEG data from forty-seven participants was collected while they performed the NASA Revised Multi-Attribute Task Battery II (MATB-II) under three different activity level conditions (no, medium, high). In this study, we report the effects of activity on the noise-robustness and distribution of several spectral, amplitude/phase coherence, and amplitude modulation features, with the ultimate goal of deriving a feature set tailored towards automated workload assessment during physical activity. Preliminary results show spectral features acquired from the frontal area of the cortex as the most promising and that activity aware mental workload models should be developed.

Policy capturing is a decision analysis method that typically uses linear statistical modeling to estimate the basis of expert judgments. Using more flexible data mining algorithms may yield more accurate models or instead result in poor functional estimations. The objective of this study is to test the effectiveness of a decision tree induction algorithm for policy capturing in comparison to the standard linear approach. We examined human classification behavior using a simulated naval air-defense task in order to empirically compare the C4.5 decision tree algorithm to linear regression on their ability to capture individual decision policies. The pattern of results shows that C4.5 outperformed linear regression in terms of goodness-of-fit and cross-validation accuracy. Results also show that the decision tree models of individuals’ judgment policies actually classified contacts more accurately than their human counterparts. We conclude that non-linear policy capturing can yield useful models for training and decision support applications.

The present work investigates a cumulative part-task training method that builds up task complexity adaptively based on individual learner states. A research-oriented game entitled “Space Fortress” was used to evaluate two training conditions in a between-group design prior to a third condition involving an adaptive cumulative part task training method. The latter detects when the learner is ready to progress and dynamically adjusts training progression. Here we report the results of the first two conditions. First was the full task condition, where the learner was exposed to the entire task throughout the training session. The second condition followed a cumulative part-task training approach, where sub-tasks were added at fixed progression points. Results showed no statistically significant gain nor loss in terms of learning outcomes between the full task and the non-adaptive cumulative part task condition, adding evidence to previous mixed findings. A trigger rule needed for the adaptive cumulative part task training condition was developed based on short-term patterns of change in performance and mental workload to be used as a dynamic criterion for adaptation. Furthermore, bio-behavioral measures were evaluated as potential proxies for performance and workload with the aim of applying this adaptive method in contexts where performance and workload cannot be directly measured at regular intervals.

There has been substantial investment in the field of closed-circuit television (CCTV) technology to increase the level of surveillance efficiency and system performance for the security of citizens and the protection of public infrastructures. In control rooms, operators have to actively monitor a number of camera feeds that generally surpass the number of displays available. Operators tend to prioritize some visual scenes to cope with the sheer amount of visual content to monitor. The objective of the present study is to assess the impact on surveillance performance of different monitoring strategies adopted by (nonexpert) operators using a highly realistic CCTV operation simulation.

Surveillance operators must scan multiple camera feeds to ensure timely detection of incidents; however, variability in scanning behavior can lead to untimely/failed detection of critical information in feeds that were neglected for a long period. Using an eye tracker to monitor screen fixations we can calculate (in real-time) the time elapsed since the last scan of each particular feed, allowing the setting-up of targeted countermeasures contingent on operator oculomotor behavior. One avenue is to provide operators with timely alerts to modulate the scan pattern to avoid attentional tunneling and inattentional blindness. We test such an adaptive solution within a major event surveillance simulation and preliminary results show that operator scan behavior can be modulated, although further investigation is required to determine warning frequency and modality to optimize the balance between saliency and workload increase. Future work will focus on adding a real-time vigilance detection and countermeasure capability.

Automated mental workload measurement is particularly important in safety-critical settings, such as in nuclear plants, aviation, air traffic control, shipping, and transportation, to name a few. As an example, recent statistics have suggested that 90% of the accidents in the transport industry are due to human factors. In this paper, we explore the potential of off-the-shelf wearable technologies in monitoring mental workload in real-time, thus potentially reducing the number of accidents due to human errors. Wearable technologies, while providing the user with ease-of-use, comfort, and portability, have several limitations, such as lower quality signal readings (e.g., due to dry electrodes) and smaller number of recording sites. Such limitations place a burden on the accuracy of existing mental workload models. To overcome this limitation, we propose the use of phase-amplitude and amplitude-amplitude coupling features computed from a portable commercial electroencephalography (EEG) device. Experiments with three different tasks, namely N-back, mental rotation and visual search, show the proposed features being significantly correlated with multiple dimensions of the widely-used NASA task load index test and providing complementary information to other conventional features.

The assessment of an operator's functional state (i.e., the multidimensional pattern of human psychophysiological conditions that mediates performance) has great potential for increasing safety and reliability of critical systems. However, live monitoring of functional state using physiological and behavioral data still faces several challenges before achieving the level of precision required in many operational contexts. One open question is the level of granularity of the models. Is a general model sufficient or should subject-specific models be trained to ensure high accuracy? Another challenge concerns the formalization of a valid ground truth for training classifiers. This is critical in order to train models that are operationally relevant. This paper introduces the Decontextualized Dynamic Performance (DDP) metric which allows models to be trained simultaneously on different tasks using machine learning algorithms. This paper reports the performance of various classification algorithms at different levels of granularity. We compare a general model, task-specific models, and subject-specific models. Results show that the classification methods do not lead to statistically different performance, and that the predictive accuracy of subject-specific and task-specific models was actually comparable to a general model. We also compared various time-window sizes for the new DDP metric and found that results were degrading with a larger time window size.

Temporal awareness is key to successful decision making in a wide range of command and control situations, yet little explicit support to maintaining temporal awareness is provided by Decision Support Systems (DSS) for time-critical decisions. In the context of simulated weapon-target scheduling, the present study compared the decision support gained from two display formats: typical geospatial display and temporal display. The results demonstrated that the temporal display facilitates scheduling performance though its beneficial impact seems to require greater familiarization.

Feldman [(2000). Minimization of Boolean complexity in human concept learning. Nature, 407, 630–633] has provided evidence that the complexity (subjective difficulty) of a Boolean concept is related to the length of its minimum algebraic description. The catalog of Boolean concepts [Feldman, J. (2003a). A catalog of Boolean concepts. Journal of Mathematical Psychology, 47, 98–112] provides an extensive list of basic categorization problems along with their minimal algebraic descriptions and complexity levels. In the present study, we demonstrated that the minimum algebraic description is very useful for describing the most economical categorization rule that can be used, but does not always correspond to the category definition used by a participant in a concept-learning task. We developed a new minimization procedure, reanalyzed Feldman's catalog and revised the complexity of 148 out of 253 basic concept types. Among these cases, two new solutions that correspond to concept types used in the classic study on categorization by Shepard, Hovland, and Jenkins [(1961). Learning and memorization of classifications. Psychological Monographs: General and Applied, 75, 1–42] predicted a new difficulty rank ordering (for Types I–VI) that did not agree with experimental results reported in the literature. New experimental results reported here show that participants used similar non-minimal classification rules that can be explained by the hypothesis-testing sequence in the RULEX model [Nosofsky, R. M., Palmeri, T. J., & McKinley, S. C. (1994). Rule-plus-exception model of classification learning. Psychological Review, 101, 53–79].

Intelligent video surveillance systems (IVSS) that can track people, identify unattended objects, and recognize activity are constantly being developed, updated, and tested against standard test stimuli. The development of these systems, however, seems to focus on technological advancements and largely ignores how automation will affect the human closed-circuit television (CCTV) operator. This paper outlines the advantages of employing a cognitive system engineering approach to the design of IVSS, reviews the literature that has examined human factors issues related to CCTV automation, and introduces a testbed for investigating the effects of automated surveillance technology on human CCTV operators.

Policy capturing is an approach to decision analysis using statistical models such as multiple linear regression or machine learning algorithms to approximate the mental models of decision makers. The present work seeks to apply a robust policy capturing technique to functionally mirror expert mental models and create individually-tailored cognitive assistants. The “cognitive shadow” method aims to improve decision quality by recognizing probable errors in cases where the decision maker is diverging from his usual judgmental patterns. The tool actually shadows the decision maker by un-intrusively monitoring the situation and comparing its own decisions to those of the human decision maker, and then provides advisory warnings in case of a mismatch. The support methodology is designed to be minimally intrusive to avoid an increase in cognitive load, either in real-time or off-line dynamic decision making situations. Importantly, user trust is likely to be a key asset since the cognitive shadow is derived from one's own judgments. A use case of the cognitive shadow is described within the context of a maritime threat classification task, using the classic CART decision tree induction algorithm for policy capturing. This approach is deemed applicable to a large variety of domains such as supervisory control, intelligence analysis and surveillance in defence and security, and of particular relevance in high-reliability organizations with low tolerance for error.

We report a study that investigates the role of situation awareness acquisition behaviors and decisional heuristics on mission success in a complex strategic decision making simulation. Thirty-two subjects completed a turn-based counter-insurgency scenario against a simulated adversary. Subjects played the role of a commander whose goal was to stabilize a failing state by allocating resources in various spheres of intervention like “Infrastructure development” and “Security operations”. We measured goal attainment, subjects' behaviors in terms of situation awareness, and decisional heuristics. Results show that overall performance was far from optimal. Two behaviors were positively associated with performance: system exploration behaviors at the first turn of the simulation and outcome assessment behaviors for subsequent turns. Other results suggest that the nature of the information that is sought, and the nature of the decision heuristics used, played a critical role in goal attainment. Results are discussed in terms of training and cognitive support requirements.

Interactive learning environments are increasingly used to help people better deal with complex situations. One way to improve decision making effectiveness is to train systems thinking skills using interactive simulations in order to reduce the occurrence of “unintended side-effects” of interventions and catastrophic failures. We present a prototype training procedure intended for military officers and civilian personnel engaged in “full spectrum” operations. The Complex Decision Making simulation environment (CODEM) is the core of the proposed training procedure. CODEM aims to improve systems thinking skills, adaptability and other abilities associated with the integrative concept of cognitive readiness. Four training scenarios are designed to reproduce key properties of complex decision making situations. An intelligent tutor providing corrective feedback on decision making behaviors is integrated into each scenario. The tutor interventions help avoid tunnel vision (i.e., the opposite of systems thinking) by discouraging the use of overly simple heuristics. Nine behavioral metrics are monitored by the intelligent tutor—five are related to information seeking behaviors, and four are related to specific decision patterns. The effectiveness of this prototype training procedure is currently being assessed experimentally.

The complex decision making experimental platform (CODEM) is intended as a shareable research tool to stimulate multidisciplinary research on complex dynamic situation management and as an environment for training and testing cognitive readiness. The experimenter can set general parameters, configure the interface, specify the model, insert events and define the resources and capabilities of each player using the scenario development tool. No programming skills are required. Task complexity can be varied by introducing feedback loops, delayed effects, time pressure, situational uncertainty, adjusting model transparency and changing the relationships between system elements. CODEM creates detailed logs of events and actions essential for cognitive process tracing and the evaluation of decision making effectiveness. The first task designed with CODEM is a counter-insurgency scenario in which a coalition force seeks to stabilize a failing state. A genetic algorithm is used to estimate the best strategy in that scenario for comparison with human results. An adversarial version also allows insurgents to be controlled by a human opponent (or a red team) rather than an artificial agent. CODEM can be used as a cognitive engineering testbed and as a training environment for improving decision making and adaptation skills in complex situations.

The prevalence of surveillance and information collection technologies provides decision-makers with greater volume and complexity of information to monitor and on which to base decisions than ever before. In this ever increasing dynamic and information rich environment, the role for decision support systems (DSS) to augment cognition and situation awareness (SA) is becoming crucial. However, unless a better understanding is gained of the factors that promote SA without interfering with other critical cognitive functions, the design and development of such technology may serve only to exacerbate rather than enhance the desired effect. The present study investigates how DSS designed to support particular aspects of SA may affect task performance. In the context of a functional computer-controlled simulation of single ship naval anti-air warfare, a baseline condition was compared to two conditions in which a DSS was integrated to the original interface to support different facets of SA. Participants in the two DSS conditions showed an increased SA level, as measured by the QUASA technique, compared to those in the control condition. Despite this benefit, the two DSS actually lead to a reduced performance, as indexed by defense effectiveness. These findings suggest that the benefits of DSS in terms of SA may be accompanied by an overhead with adverse effects on task performance, particularly in situations of high cognitive load and time constraints. This calls for more holistic evaluations of the cognitive impacts of decision support technologies and the development of methods to simultaneously address competing constraints when designing DSS.

We investigated the effects of a decision aid designed to support the development of effective long term strategies when seeking to influence a complex adaptive system. The experiment compared a test condition with feedforward support (i.e., outcome projection) against a baseline condition both in terms of cognitive processing and goal attainment (i.e., strategy efficiency). Results show that the decision aid significantly improved the development of effective long term strategies, yet detracted participants from engaging in information acquisition behaviors. In both conditions, participants displayed a poor (unaided) ability to anticipate the system's near term behavior, which was significantly lower in the decision aid condition. Goal attainment was positively related to frequency and comprehensiveness of information acquisition, Level 3 situation awareness (near term projection), and the degree to which the participants' strategies were proactive. The decision aid therefore clearly helped overcome a serious limitation in human processing abilities in terms of outcome projection (thus helping participants develop better strategies). However, results show that decision aids may also come with certain drawbacks that need to be systematically identified and mitigated to obtain the full benefits of cognitive support technology.

Motivation - Computational models are increasingly used to gain insights into complex phenomena. However, better tools are needed to analyze and understand them. Research approach - We describe the IMAGE technology and an ongoing empirical evaluation of IMAGE compared to a baseline condition. Findings/Design - IMAGE provides interactive visualizations to facilitate the management and comparison of multiple simulations, the exploration of data, and the explicit representation of knowledge. Research limitations/Implications - While it may not be possible to fully understand nor predict the behaviour of complex systems, analysts and decision makers would benefit from technologies designed to facilitate their analysis. Originality/Value - IMAGE is a set of advanced technologies working in synergy that seeks to augment comprehension of complex systems. Take away message - IMAGE provides powerful innovative tools for simulation, exploration, and representation--three functions that the human mind cannot perform effectively in complex settings without the support of cognitive artifacts.