Search

Decoding natural language from noninvasive brain signals has been an exciting topic with the potential to expand the applications of brain-computer interface (BCI) systems. However, current methods face limitations in decoding sentences from electroencephalography (EEG) signals. Improving decoding performance requires the development of a more effective encoder for the EEG modality. Nonetheless, learning generalizable EEG representations remains a challenge due to the relatively small scale of existing EEG datasets. In this paper, we propose enhancing the EEG encoder to improve subsequent decoding performance. Specifically, we introduce the discrete Conformer encoder (D-Conformer) to transform EEG signals into discrete representations and bootstrap the learning process by imposing EEG-language alignment from the early training stage. The D-Conformer captures both local and global patterns from EEG signals and discretizes the EEG representation, making the representation more resilient to variations, while early-stage EEG-language alignment mitigates the limitations of small EEG datasets and facilitates the learning of the semantic representations from EEG signals. These enhancements result in improved EEG representations and decoding performance. We conducted extensive experiments and ablation studies to thoroughly evaluate the proposed method. Utilizing the D-Conformer encoder and bootstrapping training strategy, our approach demonstrates superior decoding performance across various tasks, including word-level, sentence-level, and sentiment-level decoding from EEG signals. Specifically, in word-level classification, we show that our encoding method produces more distinctive representations and higher classification performance compared to the EEG encoders from existing methods. At the sentence level, our model outperformed the baseline by 5.45%, achieving a BLEU-1 score of 42.31%. Furthermore, in sentiment classification, our model exceeded the baseline by 14%, achieving a sentiment classification accuracy of 69.3%.

A significant issue for traffic safety has been drowsy driving for decades. A number of studies have investigated the effects of acute fatigue on spectral power; and recent research has revealed that drowsy driving is associated with a variety of brain connections in a specific cortico-cortical pathway. In spite of this, it is still unclear how different brain regions are connected in drowsy driving at different levels of daily fatigue. This study identified the brain connectivity-behavior relationship among three different daily fatigue levels (low-, median- and high-fatigue) with the EEG data transfer entropy. According to the results, only low- and medium-fatigue groups demonstrated an inverted U-shaped change in connectivity from high performance to poor behavioral performance. In addition, from low- to high-fatigue groups, connectivity magnitude decreased in the frontal region and increased in the occipital region. These study results suggest that brain connectivity and driving behavior would be affected by different levels of daily fatigue.

Drowsy driving is one of the primary causes of driving fatalities. Electroencephalography (EEG), a method for detecting drowsiness directly from brain activity, has been widely used for detecting driver drowsiness in real-time. Recent studies have revealed the great potential of using brain connectivity graphs constructed based on EEG data for drowsy state predictions. However, traditional brain connectivity networks are irrelevant to the downstream prediction tasks. This article proposes a connectivity-aware graph neural network (CAGNN) using a self-attention mechanism that can generate task-relevant connectivity networks via end-to-end training. Our method achieved an accuracy of 72.6% and outperformed other convolutional neural networks (CNNs) and graph generation methods based on a drowsy driving dataset. In addition, we introduced a squeeze-and-excitation (SE) block to capture important features and demonstrated that the SE attention score can reveal the most important feature band. We compared our generated connectivity graphs in the drowsy and alert states and found drowsiness connectivity patterns, including significantly reduced occipital connectivity and interregional connectivity. Additionally, we performed a post hoc interpretability analysis and found that our method could identify drowsiness features such as alpha spindles. Our code is available online at https://github.com/ALEX95GOGO/CAGNN.

Object recognition and object identification are multifaceted cognitive operations that require various brain regions to synthesize and process information. Prior research has evidenced the activity of both visual and temporal cortices during these tasks. Notwithstanding their similarities, object recognition and identification are recognized as separate brain functions. Drawing from the two-stream hypothesis, our investigation aims to understand whether the channels within the ventral and dorsal streams contain pertinent information for effective model learning regarding object recognition and identification tasks. By utilizing the data we collected during the object recognition and identification experiment, we scrutinized EEGNet models, trained using channels that replicate the two-stream hypothesis pathways, against a model trained using all available channels. The outcomes reveal that the model trained solely using the temporal region delivered a high accuracy level in classifying four distinct object categories. Specifically, the object recognition and object identification models achieved an accuracy of 89% and 85%, respectively. By incorporating the channels that mimic the ventral stream, the model’s accuracy was further improved, with the object recognition model and object identification model achieving an accuracy of 95% and 94%, respectively. Furthermore, the Grad-CAM result of the trained models revealed a significant contribution from the ventral and dorsal stream channels toward the training of the EEGNet model. The aim of our study is to pinpoint the optimal channel configuration that provides a swift and accurate brain-computer interface system for object recognition and identification.

Identifying meaningful brain activities is critical in brain-computer interface (BCI) applications. Recently, an increasing number of neural network approaches have been proposed to recognize EEG signals. However, these approaches depend heavily on using complex network structures to improve the performance of EEG recognition and suffer from the deficit of training data. Inspired by the waveform characteristics and processing methods shared between EEG and speech signals, we propose Speech2EEG, a novel EEG recognition method that leverages pretrained speech features to improve the accuracy of EEG recognition. Specifically, a pretrained speech processing model is adapted to the EEG domain to extract multichannel temporal embeddings. Then, several aggregation methods, including the weighted average, channelwise aggregation, and channel-and-depthwise aggregation, are implemented to exploit and integrate the multichannel temporal embeddings. Finally, a classification network is used to predict EEG categories based on the integrated features. Our work is the first to explore the use of pretrained speech models for EEG signal analysis as well as the effective ways to integrate the multichannel temporal embeddings from the EEG signal. Extensive experimental results suggest that the proposed Speech2EEG method achieves state-of-the-art performance on two challenging motor imagery (MI) datasets, the BCI IV-2a and BCI IV-2b datasets, with accuracies of ${89}.{5}\%$ and ${84}.{07}\%$ , respectively. Visualization analysis of the multichannel temporal embeddings show that the Speech2EEG architecture can capture useful patterns related to MI categories, which can provide a novel solution for subsequent research under the constraints of a limited dataset scale.

Research and development of electroencephalogram (EEG) based brain-computer interfaces (BCIs) have advanced rapidly, partly due to deeper understanding of the brain and wide adoption of sophisticated machine learning approaches for decoding the EEG signals. However, recent studies have shown that machine learning algorithms are vulnerable to adversarial attacks. This paper proposes to use narrow period pulse for poisoning attack of EEG-based BCIs, which makes adversarial attacks much easier to implement. One can create dangerous backdoors in the machine learning model by injecting poisoning samples into the training set. Test samples with the backdoor key will then be classified into the target class specified by the attacker. What most distinguishes our approach from previous ones is that the backdoor key does not need to be synchronized with the EEG trials, making it very easy to implement. The effectiveness and robustness of the backdoor attack approach is demonstrated, highlighting a critical security concern for EEG-based BCIs and calling for urgent attention to address it.

Modern work environments have extensive interactions with technology and greater cognitive complexity of the tasks, which results in human operators experiencing increased mental workload. Air traffic control operators routinely work in such complex environments, and we designed tracking and collision prediction tasks to emulate their elementary tasks. The physiological response to the workload variations in these tasks was elucidated to untangle the impact of workload variations experienced by operators. Electroencephalogram (EEG), eye activity, and heart rate variability (HRV) data were recorded from 24 participants performing tracking and collision prediction tasks with three levels of difficulty. Our findings indicate that variations in task load in both these tasks are sensitively reflected in EEG, eye activity and HRV data. Multiple regression results also show that operators’ performance in both tasks can be predicted using the corresponding EEG, eye activity and HRV data. The results also demonstrate that the brain dynamics during each of these tasks can be estimated from the corresponding eye activity, HRV and performance data. Furthermore, the markedly distinct neurometrics of workload variations in the tracking and collision prediction tasks indicate that neurometrics can provide insights on the type of mental workload. These findings have applicability to the design of future mental workload adaptive systems that integrate neurometrics in deciding not just “when” but also “what” to adapt. Our study provides compelling evidence in the viability of developing intelligent closed-loop mental workload adaptive systems that ensure efficiency and safety in complex work environments.

Correct detection of peaks in electroencephalogram (EEG) signals is of essence due to the significant correlation of those potentials with cognitive performance and disorders. This paper proposes a novel and non-parametric approach to detect prediction error negativity (PEN) in cognitive conflict processing. The PEN candidates are first located from the input signal via an adaptation of a recent effective method for local maxima extraction, processed in a multi-scale manner. The found candidates are then fused and ranked based on their shape and location-based features. False positives caused by candidates’ magnitude are eliminated by rotating the sorted candidate list where the one with the second-best ranking score will be identified as PEN. The EEG data collected from a 3D object selection task have been used to verify the efficacy of the proposed approach. Compared with the state-of-the-art peak detection techniques, the proposed method shows an improvement of at least 2.67% in accuracy and 6.27% in sensitivity while requires only about 4 ms to process an epoch. The accuracy and computational efficiency of the proposed technique in the detection of PEN in cognitive conflict processing would lead to promising applications in performance improvement of brain-computer interfaces (BCIs).

Analyzing the effects landmarks have on spatial learning is an active area of research in the study of human navigation processes and one that is key to understanding the links between human brain dynamics, landmark encoding, and spatial learning outcomes. This article presents a study on whether electroencephalography (EEG) signals related to virtual global landmarks combined with deep learning can be used to predict the accuracy and efficacy of spatial learning. Virtual global landmarks are silhouettes of actual landmarks projected into the navigator’s vision via a heads-up display. They serve as a notable frame of reference in addition to the local landmarks we all typically use for route navigation. From a mobile virtual reality scenario involving 55 participants, the results of the study suggest that the EEG data associated with those who were exposed to global landmarks shows a visibly better capacity for predicting the quality of spatial learning levels than those who were not. As such, the EEG features associated with processing VGLs have a greater functional relation to the quality of spatial learning. This finding opens up a future direction of enquiry into landmark encoding and navigational ability. It may also provide a potential avenue for the early diagnosis of Alzheimer’s disease.

Motor imagery-based brain-computer interface (MI-BCI) currently represents a new trend in rehabilitation. However, individual differences in the responsive frequency bands and a poor understanding of the communication between the ipsilesional motor areas and other regions limit the use of MI-BCI therapy. Objective: Bimanual training has recently attracted attention as it achieves better outcomes as compared to repetitive one-handed training. This study compared the effects of three MI tasks with different visual feedback. Methods: Fourteen healthy subjects performed single hand motor imagery tasks while watching single static hand (traditional MI), single hand with rotation movement (rmMI), and bimanual coordination with a hand pedal exerciser (bcMI). Functional connectivity is estimated by Transfer Entropy (TE) analysis for brain information flow. Results: Brain connectivity of conducting three MI tasks showed that the bcMI demonstrated increased communications from the parietal to the bilateral prefrontal areas and increased contralateral connections between motor-related zones and spatial processing regions. Discussion/Conclusion: The results revealed bimanual coordination operation events increased spatial information and motor planning under the motor imagery task. And the proposed bimanual coordination MI-BCI (bcMI-BCI) can also achieve the effect of traditional motor imagery tasks and promotes more effective connections with different brain regions to better integrate motor-cortex functions for aiding the development of more effective MI-BCI therapy. Clinical and Translational Impact Statement The proposed bcMI-BCI provides more effective connections with different brain areas and integrates motor-cortex functions to promote motor imagery rehabilitation for patients’ impairment.

Bio-signals pattern recognition systems can be impacted by several factors with a potential to limit their associated performance and clinical translation. Among these factors, selecting the optimum feature extraction method, that can effectively exploit the interaction between the temporal and spatial information, is the most prominent. Despite the potential of deep learning (DL) models for extracting temporal, spatial, or temporal-spatial information, they are typically restricted by their need for a large amount of training data. The deep wavelet scattering transform (WST) is a relatively recent advancement within the DL literature to replace expensive convolution neural networks models with computationally less demanding methods. However, while some studies have used WST to extract features from biological signals, it has not been investigated before for electromyogram (EMG) and electroencephalogram (EEG) signals feature extraction. To investigate the hypothesis of the usefulness of WST for processing EMG and EEG signals, this study used a tremor dataset collected by the authors from people with tremor disorders. Specifically, the proposed work achieved three goals: (a) study the performance of extracting features from low-density EMG signals (8 channels), using the WST approach, (b) study the effect of extracting the features from high-density EEG signals (33 channels), using WST and study its robustness against changing the spatial and temporal aspects of classification accuracy, and (c) classify tremor severity using the WST method and compare the results with other well-known feature extraction approaches. The classification error rates were significantly reduced (maximum of nearly 12%) compared with other feature sets.

Objective: Mental stress is a major problem in our society and has become an area of interest for many psychiatric researchers. One primary research focus area is the identification of bio-markers that not only identify stress but also predict the conditions (or tasks) that cause stress. Electroencephalograms (EEGs) have been used for a long time to study and identify bio-markers. While these bio-markers have successfully predicted stress in EEG studies for binary conditions, their performance is suboptimal for multiple conditions of stress. Methods: To overcome this challenge, we propose using latent based representations of the bio-markers, which have been shown to significantly improve EEG performance compared to traditional bio-markers alone. We evaluated three commonly used EEG based bio-markers for stress, the brain load index (BLI), the spectral power values of EEG frequency bands (alpha, beta and theta), and the relative gamma (RG), with their respective latent representations using four commonly used classifiers. Results: The results show that spectral power value based bio-markers had a high performance with an accuracy of 83%, while the respective latent representations had an accuracy of 91%.

Electrooculography (EOG) is considered as the most stable physiological signal in the development of human-computer interface (HCI) for detecting eye-movement variations. EOG signal classification has gained more traction in recent years to overcome physical inconvenience in paralyzed patients. In this paper, a robust classification technique, such as eight directional movements is investigated by introducing a concept of buffer along with a variation of the slope to avoid misclassification effects in EOG signals. Blinking detection becomes complicated when the magnitude of the signals are considered. Hence, a correction technique is introduced to avoid misclassification for oblique eye movements. Meanwhile, a case study has been considered to apply these correction techniques to HCI baseball game to learn eye-movements.

In the above article [1], we detected an error in reporting the 10-fold cross-validation result. The correct 10-fold cross-validation result in Table IV is uploaded in this letter.

Polysomnography (PSG) is considered the gold standard in the diagnosis of obstructive sleep apnea (OSA). The diagnosis of OSA requires an overnight sleep experiment in a laboratory. However, due to limitations in relation to the number of labs and beds available, patients often need to wait a long time before being diagnosed and eventually treated. In addition, the unfamiliar environment and restricted mobility when a patient is being tested with a polysomnogram may disturb their sleep, resulting in an incomplete or corrupted test. Therefore, it is posed that a PSG conducted in the patient's home would be more reliable and convenient. The Internet of Things (IoT) plays a vital role in the e-Health system. In this paper, we implement an IoT-based wireless polysomnography system for sleep monitoring, which utilizes a batterypowered, miniature, wireless, portable, and multipurpose recorder. A Java-based PSG recording program in the personal computer is designed to save several bio-signals and transfer them into the European data format. These PSG records can be used to determine a patient's sleep stages and diagnose OSA. This system is portable, lightweight, and has low power-consumption. To demonstrate the feasibility of the proposed PSG system, a comparison was made between the standard PSG-Alice 5 Diagnostic Sleep System and the proposed system. Several healthy volunteer patients participated in the PSG experiment and were monitored by both the standard PSG-Alice 5 Diagnostic Sleep System and the proposed system simultaneously, under the supervision of specialists at the Sleep Laboratory in Taipei Veteran General Hospital. A comparison of the results of the time-domain waveform and sleep stage of the two systems shows that the proposed system is reliable and can be applied in practice. The proposed system can facilitate the long-term tracing and research of personal sleep monitoring at home.

Task switching is a common method to investigate executive functions such as working memory and attention. This paper investigates the effect of acute stress on brain activity using task switching. Surprisingly few studies have been conducted in this area. There is behavioral and physiological evidence to indicate that acute stress makes the participants more tense which results in a better performance. In this current study, under stressful conditions, the participants gave quick responses with high accuracy. However, unexpected results were found in relation to salivary cortisol. Furthermore, the electroencephalogram results showed that acute stress was pronounced at the frontal and parietal midline cortex, especially on the theta, alpha, and gamma bands. One possible explanation for these results may be that the participants changed their strategy in relation to executive functions during stressful conditions by paying more attention which resulted in a higher working memory capacity which enhanced performance during the task switching.

A centralized infrastructure system carries out existing data analytics and decision-making processes from our current highly virtualized platform of wireless networks and the Internet of Things (IoT) applications. There is a high possibility that these existing methods will encounter more challenges and issues in relation to network dynamics, resulting in a high overhead in the network response time, leading to latency and traffic. In order to avoid these problems in the network and achieve an optimum level of resource utilization, a new paradigm called edge computing (EC) is proposed to pave the way for the evolution of new age applications and services. With the integration of EC, the processing capabilities are pushed to the edge of network devices such as smart phones, sensor nodes, wearables, and on-board units, where data analytics and knowledge generation are performed which removes the necessity for a centralized system. Many IoT applications, such as smart cities, the smart grid, smart traffic lights, and smart vehicles, are rapidly upgrading their applications with EC, significantly improving response time as well as conserving network resources. Irrespective of the fact that EC shifts the workload from a centralized cloud to the edge, the analogy between EC and the cloud pertaining to factors such as resource management and computation optimization are still open to research studies. Hence, this paper aims to validate the efficiency and resourcefulness of EC. We extensively survey the edge systems and present a comparative study of cloud computing systems. After analyzing the different network properties in the system, the results show that EC systems perform better than cloud computing systems. Finally, the research challenges in implementing an EC system and future research directions are discussed.

Automated human action recognition has the potential to play an important role in public security, for example, in relation to the multiview surveillance videos taken in public places, such as train stations or airports. This paper compares three practical, reliable, and generic systems for multiview video-based human action recognition, namely, the nearest neighbor classifier, Gaussian mixture model classifier, and the nearest mean classifier. To describe the different actions performed in different views, view-invariant features are proposed to address multiview action recognition. These features are obtained by extracting the holistic features from different temporal scales which are modeled as points of interest which represent the global spatial-temporal distribution. Experiments and cross-data testing are conducted on the KTH, WEIZMANN, and MuHAVi datasets. The system does not need to be retrained when scenarios are changed which means the trained database can be applied in a wide variety of environments, such as view angle or background changes. The experiment results show that the proposed approach outperforms the existing methods on the KTH and WEIZMANN datasets.

Different rendering styles induce different levels of agency and user behaviors in virtual reality environments. We applied an electroencephalogram-based approach to investigate how the rendering style of the users' hands affects behavioral and cognitive responses. To this end, we introduced prediction errors due to cognitive conflicts during a 3-D object selection task by manipulating the selection distance of the target object. The results showed that, for participants with high behavioral inhibition scores, the amplitude of the negative event-related potential at approximately 50-250 ms correlated with the realism of the virtual hands. Concurring with the uncanny valley theory, these findings suggest that the more realistic the representation of the user's hand is, the more sensitive the user becomes toward subtle errors, such as tracking inaccuracies.

Improving the degree of assistance given by in-car navigation systems is an important issue for the safety of both drivers and passengers. There is a vast body of research that assesses the usability and interfaces of the existing navigation systems but very few investigations study the impact on the brain activity based on navigation-based driving. In this paper, a real-world experiment is designed to acquire the electroencephalography (EEG) and in-car information to analyze the dynamic brain activity while the driver is performing the lane-changing task based on the auditory instructions from an in-car navigation system. The results show that auditory cues can influence the speed and increase the frontal EEG delta and beta power, which is related to motor preparation and decision making during a lane change. However, there were no significant results on the alpha power. A better lane-change assessment can be obtained using specific vehicle information (lateral acceleration and heading angle) with EEG features for future naturalized driving study.

There are current limitations in the recording technologies for measuring EEG activity in clinical and experimental applications. Acquisition systems involving wet electrodes are time-consuming and uncomfortable for the user. Furthermore, dehydration of the gel affects the quality of the acquired data and reliability of long-term monitoring. As a result, dry electrodes may be used to facilitate the transition from neuroscience research or clinical practice to real-life applications. EEG signals can be easily obtained using dry electrodes on the forehead, which provides extensive information concerning various cognitive dysfunctions and disorders. This paper presents the usefulness of the forehead EEG with advanced sensing technology and signal processing algorithms to support people with healthcare needs, such as monitoring sleep, predicting headaches, and treating depression. The proposed system for evaluating sleep quality is capable of identifying five sleep stages to track nightly sleep patterns. Additionally, people with episodic migraines can be notified of an imminent migraine headache hours in advance through monitoring forehead EEG dynamics. The depression treatment screening system can predict the efficacy of rapid antidepressant agents. It is evident that frontal EEG activity is critically involved in sleep management, headache prevention, and depression treatment. The use of dry electrodes on the forehead allows for easy and rapid monitoring on an everyday basis. The advances in EEG recording and analysis ensure a promising future in support of personal healthcare solutions.

Internet of Things is smartly changing various existing research areas into new themes, including smart health, smart home, smart industry, and smart transport. Relying on the basis of “smart transport,” Internet of Vehicles (IoV) is evolving as a new theme of research and development from vehicular ad hoc networks (VANETs). This paper presents a comprehensive framework of IoV with emphasis on layered architecture, protocol stack, network model, challenges, and future aspects. Specifically, following the background on the evolution of VANETs and motivation on IoV an overview of IoV is presented as the heterogeneous vehicular networks. The IoV includes five types of vehicular communications, namely, vehicle-to-vehicle, vehicle-to-roadside, vehicle-to-infrastructure of cellular networks, vehicle-to-personal devices, and vehicle-to-sensors. A five layered architecture of IoV is proposed considering functionalities and representations of each layer. A protocol stack for the layered architecture is structured considering management, operational, and security planes. A network model of IoV is proposed based on the three network elements, including cloud, connection, and client. The benefits of the design and development of IoV are highlighted by performing a qualitative comparison between IoV and VANETs. Finally, the challenges ahead for realizing IoV are discussed and future aspects of IoV are envisioned.

Many commercially available electroencephalography (EEG) sensors, including conventional wet and dry sensors, can cause skin irritation and user discomfort owing to the foreign material. The EEG products, especially sensors, highly prioritize the comfort level during devices wear. To overcome these drawbacks for EEG sensors, this paper designs Societe Generale de Surveillance S · A · (SGS)-certified, silicon-based dry-contact EEG sensors (SBDSs) for EEG signal measurements. According to the SGS testing report, SBDSs extract does not irritate skin or induce noncytotoxic effects on L929 cells according to ISO10993-5. The SBDS is also lightweight, flexible, and nonirritating to the skin, as well as capable of easily fitting to scalps without any skin preparation or use of a conductive gel. For forehead and hairy sites, EEG signals can be measured reliably with the designed SBDSs. In particular, for EEG signal measurements at hairy sites, the acicular and flexible design of SBDS can push the hair aside to achieve satisfactory scalp contact, as well as maintain low skin-electrode interface impedance. Results of this paper demonstrate that the proposed sensors perform well in the EEG measurements and are feasible for practical applications.

Decades of heavy investment in laboratory-based brain imaging and neuroscience have led to foundational insights into how humans sense, perceive, and interact with the external world. However, it is argued that fundamental differences between laboratory-based and naturalistic human behavior may exist. Thus, it remains unclear how well the current knowledge of human brain function translates into the highly dynamic real world. While some demonstrated successes in real-world neurotechnologies are observed, particularly in the area of brain-computer interaction technologies, innovations and developments to date are limited to a small science and technology community. We posit that advancements in realworld neuroimaging tools for use by a broad-based workforce will dramatically enhance neurotechnology applications that have the potential to radically alter human-system interactions across all aspects of everyday life. We discuss the efforts of a joint government-academic-industry team to take an integrative, interdisciplinary, and multi-aspect approach to translate current technologies into devices that are truly fieldable across a range of environments. Results from initial work, described here, show promise for dramatic advances in the field that will rapidly enhance our ability to assess brain activity in real-world scenarios.