Search

International audience; There has been a considerable amount of research to conceptualize how cognition handle multitasking situations. Despite these efforts, it is still not clear how task parameters shape attentionnal resources allocation. For instance, many research have suggested that difficulty levels could explain these conflicting observations and very few have considered other factors such as task importance. In the present study, twenty participants had to carry out two N-Back tasks simultaneously, each subtask having distinct difficulty (0,1 or 2-Back) and importance (1 or 3 points) levels. Participants's cumulative dwell time were collected to assess their attentional strategies. Results showed that depending on the global level of difficulty (combination of the two levels of difficulty), attentional resources of people were driven either by the subtask difficulty (under low-global-difficulty) or the subtask importance (under high-global-difficulty), in a non-compensatory way. We discussed these results in terms of decision-making heuristics and metacognition.

Recent technological progress has allowed the development of low-cost and highly portable brain sensors such as pre-amplified dry-electrodes to measure cognitive activity out of the laboratory. This technology opens promising perspectives to monitor the &ldquo, brain at work&rdquo, in complex real-life situations such as while operating aircraft. However, there is a need to benchmark these sensors in real operational conditions. We therefore designed a scenario in which twenty-two pilots equipped with a six-dry-electrode EEG system had to perform one low load and one high load traffic pattern along with a passive auditory oddball. In the low load condition, the participants were monitoring the flight handled by a flight instructor, whereas they were flying the aircraft in the high load condition. At the group level, statistical analyses disclosed higher P300 amplitude for the auditory target (Pz, P4 and Oz electrodes) along with higher alpha band power (Pz electrode), and higher theta band power (Oz electrode) in the low load condition as compared to the high load one. Single trial classification accuracy using both event-related potentials and event-related frequency features at the same time did not exceed chance level to discriminate the two load conditions. However, when considering only the frequency features computed over the continuous signal, classification accuracy reached around 70% on average. This study demonstrates the potential of dry-EEG to monitor cognition in a highly ecological and noisy environment, but also reveals that hardware improvement is still needed before it can be used for everyday flight operations.

International audience; Inattentional deafness can have deleterious consequences in complex real-life situations (e.g. healthcare, aviation) leading to miss critical auditory signals. Such failure of auditory attention is thought to rely on top-down biasing mechanisms at the central executive level. A complementary approach to account for this phenomenon is to consider the existence of visual dominance over hearing that could be implemented via direct visual-to-auditory pathways. To investigate this phenomenon, thirteen aircraft pilots, equipped with a 32-channel EEG system, faced a low and high workload scenarii along with an auditory oddball task in a motion flight simulator. Prior to the flying task, the pilots were screened to assess their working memory span and visual dominance susceptibility. The behavioral results disclosed that the volunteers missed 57.7% of the auditory alarms in the difficult condition. Among all evaluated capabilities, only the visual dominance index was predictive of the miss rate in the difficult scenario. These findings provide behavioral evidences that other early cross-modal competitive process than top down modulation process could account for inattentional deafness. The electrophysiological analyses showed that the miss over the hit alarms led to a significant amplitude reduction of early perceptual (N100) and late attentional (P3a and P3b) event-related potentials components. Eventually, we implemented an EEG-based processing pipeline to perform single-trial classification of inattentional deafness. The results indicate that this processing chain could be used in an ecological setting as it led to 72.2% mean accuracy to discriminate missed from hit auditory alarms.

This study was concerned with the development and testing of online cognitive-load monitoring methods by means of a working-memory experiment using electrocardiogram (ECG) analyses for future applications in mixed-initiative human-machine interaction (HMI). To this end, we first identified potentially reliable cognitive-workload-related cardiac metrics and algorithms for online processing. We then compared our online results to those conventionally obtained with state-of-the-art offline methods. Finally, we evaluated the possibility of classifying low versus high working-memory load using different classification algorithms. Our results show that both offline and online methods reliably estimate the workload associated with a multi-level working-memory task at the group level, whether it is with the heart rhythm or the heart rate variation (standard deviation of the RR interval). Moreover, we found significant working-memory load classification accuracy using both two-dimensional linear discriminant analyses (LDA) or a support vector machine (SVM). We hence argue that our online algorithm is reliable enough to provide online electrocardiographic metrics as a tool for real-life workload evaluation and can be a valuable feature for mixed-initiative systems.

Objective: The purpose of the present study was to find psychophysiological proxies that are straightforward to use and could be implemented in actual flight conditions to accurately discriminate pilots’ workload levels. Background: Piloting an aircraft is a complex activity where cognitive limitations may jeopardize flight safety. There is a need to implement solutions to monitor pilots’ workload level to improve flight safety. There has been recent interest in combining psychophysiological measurements. Most of these studies were conducted in flight simulators at the group level, limiting the interpretation of the results. Methods: We conducted an experiment with 11 pilots performing two standard traffic patterns in a light aircraft. Five metrics were derived from their ocular and cardiac activities and were evaluated through three flight phases: takeoff, downwind, and landing. Results: Statistical analyses showed that the saccadic rate was the most efficient metric to distinguish between the three flight phases. In addition, a classifier trained on the ocular data collected from the first run predicted the flight phase within a second run with an accuracy of 75%. No gain in the classifier accuracy has been found by combining cardiac and ocular metrics. Conclusions: Ocular-based metrics may be more suitable than cardiac ones to provide relevant information on pilots’ flying activity in operational settings. Applications: Electrocardiographic and eye-tracking devices could be implemented in future cockpits as additional flight data for accident analysis, an objective pilot’s state evaluation for training, and proxies for human-machine interactions to improve flight safety.

International audience

Pupil diameter is a widely-studied cognitive load measure, which, despite its convenience for non-intrusive operator state monitoring in complex environments, is still not available for in situ measurements because of numerous methodological limitations. The most important of these limitations is the influence of pupillary light reflex. Hence, there is the need of providing a pupil-based cognitive load measure that is independent of light conditions. In this paper, we present a promising technique of pupillary signal analysis resulting in luminance-independent measure of mental effort that could be used in real-time without a priori on luminous conditions. Twenty-two participants performed a short-term memory task under different screen luminance conditions. Our results showed that the amplitude of pupillary dilation due to load on memory was luminance-dependent with higher amplitude corresponding to lower-luminance condition. Furthermore, our experimentation showed that load on memory and luminance factors express themselves differently according to frequency. Therefore, as our statistical analysis revealed, the ratio between low (0-1.6 Hz) and high frequency (1.6-4 Hz) bands (LF/HF ratio) of power spectral densities of pupillary signal is sensitive to the cognitive load but not to luminance. Our results are promising for the measurement of load on memory in ecological settings.

The present study aims to further investigate the prioritization process that sustains executive control in complex multitasking. It intends to provide a description of task-attributes processing, in which several of these attributes present opposite polarities. The general aim of this work is to propose a heuristic decision-making account of prioritization.

According to the literature, mental workload (MW) is defined as the interaction of mental demands imposed on operators by tasks they attend to and the effort they put into achieving these tasks. An interesting prospect to assess MW is to consider the use of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), which allow nonintrusive probing of a user's state. Several features can be extracted from the signal collected by their respective sensors, such as the power spectral density in different frequency bands for the EEG, and the level of oxygenated hemoglobin (HbO) for the fNIRS. These features are classically found to vary depending on the MW imposed on the participants. Additional features can be derived from the EEG and fNIRS signals such as connectivity features. Recently, Charbonnier and collaborators showed that connectivity features computed between EEG electrodes and the Frobenius distance between the matrices of connectivity features could be used to estimate quite accurately their mental state.

Missing auditory alarms is a critical safety issue in many domains such as aviation. Previous work has proposed that the underlying transitory cognitive impairments of auditory neglect could lie on early sensorial gating or late attentional processes. To investigate this phenomenon and its neural correlates, we analyzed auditory Event-Related Potentials (ERPs) in a double-task paradigm that comprised an active oddball task performed in a motion flight simulator, along with a flying scenario of low or high difficulty depending on the experiment block. We found that both early sensorial and late attentional processes may account for the inattentional deafness phenomenon.

International audience; The employment of physiological measurements to perform on-line assessment of operators' mental states is crucial in the field of human-robot interaction (HRI) researches and is still an open topic to the best of our knowledge. In order to progress towards systems that would dynamically adapt to operators' mental states, a first step is to determine an adequate protocol that elicits variations in engagement. To this purpose, this work focuses on analyzing operator's physiological data streams recorded during a human-robot mission executed in a original virtual environment [Drougard17]. In detail, the conceived mission consists in a mutual cooperation of a firefighter robot and its human operator to extinguish fires. A high level of complexity is obtained by the number and the random nature of events to be handled during the mission. As an example, guiding the robot, managing its water tank level and taking care of its electric charge are all tasks to be accomplished simultaneously, which can randomly be assigned to autonomous or manually operative mode. In addition to that, an extra task consisting in keeping an adequate level of an external water tank to allow robot refill is assigned only to the human operator for continuously soliciting his attention. Anyone can experience this mission by visiting the website robot-isae.isae.fr set up to collect a large amount of behavioral data by crowdsourcing. The mission is accomplished through a remote human-machine interface made of controllers and a screen displaying different areas corresponding to each task. Figure 1 shows the graphical user interface, with the 5 areas of interest (AOIs). The control station is equipped with sensing devices for human data collecting. The sensing devices which allows for human data collection are an eye-tracker (SMI), located on the bottom bar of the display and a portable bluetooth electrocardiograph (eMotion Faros 360). A specific procedure for the experiment has been defined to guarantee a statistical significance of the recorded dataset. In detail, each different human operator has to complete at least three times a ten minutes mission aiming to the best score in terms of fires extinguished. An absolute resting is imposed between missions to get a baseline for the cardiac activity. The data are collected on 17 participants of mixed sex (9 females) with average age 28.5 (S.D. = 4.52). The number and the duration of fixations per area of interest are extracted from the eye-tracker. The length of inter-beat intervals, the Heart Rate Variability (HRV) and instant Heart Rate Variability (IHRV) are computed from the ECG. Preliminary results show a lower HRV and IHRV during the mission than during the rest session: Student and Wilcoxon statistical tests ensure a difference at least equal to 6 (p

The chapter aims at analyzing the impact of the levels of talkers and their spatial separation on the cognitive resources of the listeners. For this purpose, 14 young listeners were chosen to report the speech emitted by a target talker in the presence of a concurrent masker talker and results were recorded.

In a multi-talker situation, spatial separation between talkers reduces cognitive processing load: this is the “spatial release of cognitive load”. The present study investigated the role played by the relative levels of the talkers on this spatial release of cognitive load. During the experiment, participants had to report the speech emitted by a target talker in the presence of a concurrent masker talker. The spatial separation (0° and 120° angular distance in azimuth) and the relative levels of the talkers (adverse, intermediate, and favorable target-to-masker ratio) were manipulated. The cognitive load was assessed with a prefrontal functional near-infrared spectroscopy. Data from 14 young normal- hearing listeners revealed that the target-to-masker ratio had a direct impact on the spatial release of cognitive load. Spatial separation significantly reduced the prefrontal activity only for the intermediate target-to-masker ratio and had no effect on prefrontal activity for the favorable and the adverse target-to-masker ratios. Therefore, the relative levels of the talkers might be a key point to determine the spatial release of cognitive load and more specifically the prefrontal activity induced by spatial cues in multi- talker environments.

Auditory alarm misperception is one of the critical events that lead aircraft pilots to an erroneous flying decision. The rarity of these alarms associated with their possible unreliability may play a role in this misperception. In order to investigate this hypothesis, we manipulated both audiovisual conflict and sound rarity in a simplified landing task. Behavioral data and event related potentials (ERPs) of thirteen healthy participants were analyzed. We found that the presentation of a rare auditory signal (i.e. an alarm), incongruent with visual information, led to a smaller amplitude of the auditory N100 (i.e. less negative) compared to the condition in which both signals were congruent. Moreover, the incongruity between the visual information and the rare sound did not significantly affect reaction times, suggesting that the rare sound was neglected. We propose that the lower N100 amplitude reflects an early visual-to-auditory gating that depends on the rarity of the sound. In complex aircraft environments, this early effect might be partly responsible for auditory alarm insensitivity. Our results provide a new basis for future aeronautic studies and the development of countermeasures.

International audience; In the aeronautics field, some authors have suggested that an aircraft's attitude sonification could be used by pilots to cope with spatial disorientation situations. Such a system is currently used by blind pilots to control the attitude of their aircraft. However, given the suspected higher auditory attentional capacities of blind people, the possibility for sighted individuals to use this system remains an open question. For example, its introduction may overload the auditory channel, which may in turn alter the responsiveness of pilots to infrequent but critical auditory warnings. In this study, two groups of pilots (blind versus sighted) performed a simulated flight experiment consisting of successive aircraft maneuvers, on the sole basis of an aircraft sonification. Maneuver difficulty was varied while we assessed flight performance along with subjective and electroencephalographic (EEG) measures of workload. The results showed that both groups of participants reached target-attitudes with a good accuracy. However, more complex maneuvers increased subjective workload and impaired brain responsiveness toward unexpected auditory stimuli as demonstrated by lower N1 and P3 amplitudes. Despite that the EEG signal showed a clear reorganization of the brain in the blind participants (higher alpha power), the brain responsiveness to unexpected auditory stimuli was not significantly different between the two groups. The results suggest that an auditory display might provide useful additional information to spatially disoriented pilots with normal vision. However, its use should be restricted to critical situations and simple recovery or guidance maneuvers.

The event-related potential N270 component is known to be an electrophysiological marker of the supramodal conflict processing. However little is know about the factors that may modulate its amplitude. In particular, among all studies that have investigated the N270, little or no control of the conflict strength and of the load in working memory have been done leaving a lack in the understanding of this component. We designed a spatial audiovisual conflict task with simultaneous target and cross-modal distractor to evaluate the N270 sensitivity to the conflict strength (i.e., visual target with auditory distractor or auditory target with visual distractor) and the load in working memory (goal task maintenance with frequent change in the target modality). In a first session, participants had to focus on one modality for the target position to be considered (left-hand or right-hand) while the distractor could be at the same side (compatible) or at opposite side (incompatible). In a second session, we used the same set of stimuli as in the first session with an additional distinct auditory signal that clued the participants to frequently switch between the auditory and the visual targets. We found that (1) reaction times and N270 amplitudes for conflicting situations were larger within the auditory target condition compared to the visual one, (2) the increase in target maintenance effort led to equivalent increase of both reaction times and N270 amplitudes within all conditions and (3) the right dorsolateral prefrontal cortex current density was higher for both conflicting and active maintenance of the target situations. These results provide new evidence that the N270 component is an electrophysiological marker of the supramodal conflict processing that is sensitive to the conflict strength and that conflict processing and active maintenance of the task goal are two functions of a common executive attention system.

Working memory is a key executive function for operating aircraft, especially when pilots have to recall series of air traffic control instructions. There is a need to implement tools to monitor working memory as its limitation may jeopardize flight safety. An innovative way to address this issue is to adopt a Neuroergonomics approach that merges knowledge and methods from Human Factors, System Engineering and Neuroscience. A challenge of great importance for Neuroergonomics is to implement efficient brain imaging techniques to measure the brain at work and to design Brain Computer Interfaces. We used functional near infrared spectroscopy as it has been already successfully tested to measure working memory capacity in complex environment with air traffic controllers, pilots or unmanned vehicle operators. However, the extraction of relevant features from the raw signal in ecological environment is still a critical issue due to the complexity of implementing real-time signal processing techniques without a priori knowledge. We proposed to implement the Kalman filtering approach, a signal processing technique that is efficient when the dynamics of the signal can be modeled. We based our approach on the Boynton model of hemodynamic response. We conducted a first experiment with 9 participants involving a basic working memory task to estimate the noise covariances of the Kalman filter. We then conducted a more ecological experiment in our flight simulator with 18 pilots who interacted with air traffic controller instructions (two levels of difficulty). The data was processed with the same Kalman filter settings implemented in the first experiment. This filter was benchmarked with a classical pass-band IIR filter and a Moving Average Convergence Divergence filter. Statistical analysis revealed that the Kalman filter was the most efficient to separate the two levels of load, by increasing the observed effect size in prefrontal areas involved in working memory. In addition, the use of a Kalman filter increased the performance of the classification of working memory levels based on brain signal. The results suggest that Kalman filter is a suitable approach for real-time improvement of near infrared spectroscopy signal in ecological situations and the development of Brain Computer Interfaces.

Conflicts between the pilot and the automation, when pilots detect but do not understand them, cause "automation surprise" situations and jeopardize flight safety. We conducted an experiment in a 3-axis motion flight simulator with 16 pilots equipped with an eye-tracker to analyze their behavior and eye movements during the occurrence of such a situation. The results revealed that this conflict engages participant's attentional abilities resulting in excessive and inefficient visual search patterns. This experiment confirmed the crucial need to design solutions for detecting the occurrence of conflictual situations and to assist the pilots. We therefore proposed an approach to formally identify the occurrence of "automation surprise" conflicts based on the analysis of "silent mode changes" of the autopilot. A demonstrator was implemented and allowed for the automatic trigger of messages in the cockpit that explains the autopilot behavior. We implemented a real-time demonstrator that was tested as a proof-of-concept with 7 subjects facing 3 different conflicts with automation. The results shown the efficacy of this approach which could be implemented in existing cockpits.

Summary Context Using natural connected speech, the aim of the present study was to examine the semantic congruity effect (i.e. the difference between semantically incongruous and congruous words) in sentence contexts that generate high or moderate final word expectancies. Methods We used sentences with two levels of word expectancy in the auditory modality: familiar proverbs (that generate high final word expectancy), and unfamiliar sentences (that generate only moderate final word expectancy). Results Results revealed an early congruity effect (0–200 ms) that developed across all scalp sites for familiar proverbs but not for unfamiliar sentences. By contrast, typical centro-parietal N400 and Late Positivity Component congruity effects developed later (200–500 ms and 600–900 ms ranges) for both familiar proverbs and unfamiliar sentences. Discussion We argue that the early congruity effect for proverbs comprises both a Phonological Mismatch Negativity, reflecting the processing of the acoustic/phonological mismatch between the expected (congruous) and unexpected (incongruous) sentence completions and a typical N400 semantic congruity effect with an unusual short latency because final words can be predicted from the unusually high contextual constraints of familiar proverbs. These results are considered in the light of current views of anticipation and prediction processes in sentence contexts.

Real-time solutions for noise reduction and signal processing represent a central challenge for the development of Brain Computer Interfaces (BCI). In this paper, we introduce the Moving Average Convergence Divergence (MACD) filter, a tunable digital passband filter for online noise reduction and onset detection without preliminary learning phase, used in economic markets analysis. MACD performance was tested and benchmarked with other filters using data collected with functional Near Infrared Spectoscopy (fNIRS) during a digit sequence memorization task. This filter has a good performance on filtering and real-time peak activity onset detection, compared to other techniques. Therefore, MACD could be implemented for efficient BCI design using fNIRS.

Design of future systems for flight-deck automation will reflect a trend of changing the paradigm of human-computer interaction from the master (human)- slave (machine) mode to more equilibrated cooperation. In many cases such cooperation considers several humans and computer systems, for which multi-agent dynamic cooperative systems are appropriate models. Development of such systems requires very profound analysis of mutual interactions and conflicts that may arise in such systems. Additional testing is exhaustive and expensive for such systems. In the scope of the D3CoS project these problems are addressed from the modelling point of view with ambition to create tools that will simplify the development phase and replace parts of the testing phase. In this paper we investigate common flight procedures, for which computer assistance could be developed. We show how formal modelling of procedures allows us to inspect procedural inconsistencies and workload peaks before the development starts. We show how a computer cognitive architecture (a virtual pilot) can simulate human pilot behaviour in the cockpit to address questions typical for the early phase of the development. Analysis of these questions allows us to reduce the number of candidates for the final implementation without the need of expensive experiments with human pilots. This modelling approach is demonstrated on experiments undertaken both with human pilots and a virtual pilot. The quality of the outcome from both experimental settings remains conserved as shown by physiological assessment of pilot workload, which in turn justifies the use of the modelling approach for this type of problems.

Working memory is a key executive function for flying an aircraft. This function is particularly critical when pilots have to recall series of air traffic control instructions. However, working memory limitations may jeopardize flight safety. Since the functional near-infrared spectroscopy (fNIRS) method seems promising for assessing working memory load, our objective is to implement an on-line fNIRS-based inference system that integrates two complementary estimators. The first estimator is a real-time state estimation MACD-based algorithm dedicated to identifying the pilot's instantaneous mental state (not-on-task vs. on-task). It does not require a calibration process to perform its estimation. The second estimator is an on-line SVM-based classifier that is able to discriminate task difficulty (low working memory load vs. high working memory load). These two estimators were tested with 19 pilots who were placed in a realistic flight simulator and were asked to recall air traffic control instructions. We found that the estimated pilot's mental state matched significantly better than chance with the pilot's real state (62% global accuracy, 58% specificity, and 72% sensitivity). The second estimator, dedicated to assessing single trial working memory loads, led to 80% classification accuracy, 72% specificity, and 89% sensitivity. These two estimators establish reusable blocks for further fNIRS-based passive brain computer interface development.

Introduction While conflict between instructions and distractors makes normally reactions slower, right inferior parietal lobule (IPL) damage associated with left spatial neglect leads, in a visuomotor task, to the paradoxical facilitation of rightwards movements in the presence of conflicting leftward response plans (Coulthard et al., 2008). We investigated the effects of the right IPL lesion on the interference between vision and audition in a spatial context. Methods Behavioral data and Event-Related Potentials (ERPs) of 12 healthy adults and 12 patients with a focal right parietal post-stroke lesion, matched in age and educational level, were recorded during an audio-visual task. A pure tone through a stereo system and a white circle on a screen were presented simultaneously on the same side (left or right) in congruent trials and on opposite sides in incongruent trials. Participants had to respond to the auditory or visual target's side according to the instruction, by pressing a response-pad button with their right hand. Whereas all the patients had a left spatial neglect at the acute phase of stroke, they had recovered a normal performance on “paper-and-pencil” tests when included in our study. Results Paradoxically, for right visual target with left auditory distractor, patients had a smaller reaction time “cost” than the control group. For healthy controls, two frontocentral ERPs negative components, N2 and N270, were elicited in all conflict conditions compared to congruent conditions. Whereas the amplitude of N2 and N270 was not different between the two groups for left targets, these components were not generated by the presentation of a right visual or auditory target with a left distractor in patients. Discussion N2 and N270 might be elicited by the detection of two incongruent response plans by the prefrontal cortex. Their suppression for left targets in patients supports the hypothesis that the right IPL might be a crucial site for automatic activation of competing motor plans generated by cross-modal conflicting stimuli. In patients, the behavioral facilitation for right visual targets only may be due to additional effect of visual dominance over audition in spatial tasks.