Search

Brain-computer interfaces (BCIs) enable direct communication between the brain and external devices. This review highlights the core decoding algorithms that enable multimodal BCIs, including a dissection of the elements, a unified view of diversified approaches, and a comprehensive analysis of the present state of the field. We emphasize algorithmic advancements in cross-modality mapping, sequential modeling, besides classic multi-modality fusion, illustrating how these novel AI approaches enhance decoding of brain data. The current literature of BCI applications on visual, speech, and affective decoding are comprehensively explored. Looking forward, we draw attention on the impact of emerging architectures like multimodal Transformers, and discuss challenges such as brain data heterogeneity and common errors. This review also serves as a bridge in this interdisciplinary field for experts with neuroscience background and experts that study AI, aiming to provide a comprehensive understanding for AI-powered multimodal BCIs.

An electroencephalogram (EEG) based brain-computer interface (BCI) enables direct communication between the brain and external devices. However, EEG-based BCIs face at least three major challenges in real-world applications: data scarcity and individual differences, adversarial vulnerability, and data privacy. While previous studies have addressed one or two of these issues, simultaneous accommodation of all three challenges remains challenging and unexplored. This paper fills this gap, by proposing an Augmented Robustness Ensemble (ARE) algorithm and integrating it into three privacy protection scenarios (centralized source-free transfer, federated source-free transfer, and source data perturbation), achieving simultaneously accurate decoding, adversarial robustness, and privacy protection of EEG-based BCIs. Experiments on three public EEG datasets demonstrated that our proposed approach outperformed over 10 classic and state-of-the-art approaches in both accuracy and robustness in all three privacy-preserving scenarios, even outperforming state-of-the-art transfer learning approaches that do not consider privacy protection at all. This is the first time that three major challenges in EEG-based BCIs can be addressed simultaneously, significantly improving the practicalness of EEG decoding in real-world BCIs.

A brain-computer interface (BCI) enables direct communication between the brain and an external device. Electroencephalogram (EEG) is a common input signal for BCIs, due to its convenience and low cost. Most research on EEG-based BCIs focuses on the accurate decoding of EEG signals, while ignoring their security. Recent studies have shown that machine learning models in BCIs are vulnerable to adversarial attacks. This paper proposes adversarial filtering based evasion and backdoor attacks to EEG-based BCIs, which are very easy to implement. Experiments on three datasets from different BCI paradigms demonstrated the effectiveness of our proposed attack approaches. To our knowledge, this is the first study on adversarial filtering for EEG-based BCIs, raising a new security concern and calling for more attention on the security of BCIs.

Objective: An electroencephalogram (EEG)-based brain-computer interface (BCI) is a direct communication pathway between the human brain and a computer. Most research so far studied more accurate BCIs, but much less attention has been paid to the ethics of BCIs. Aside from task-specific information, EEG signals also contain rich private information, e.g., user identity, emotion, disorders, etc., which should be protected. Approach: We show for the first time that adding user-wise perturbations can make identity information in EEG unlearnable. We propose four types of user-wise privacy-preserving perturbations, i.e., random noise, synthetic noise, error minimization noise, and error maximization noise. After adding the proposed perturbations to EEG training data, the user identity information in the data becomes unlearnable, while the BCI task information remains unaffected. Main results: Experiments on six EEG datasets using three neural network classifiers and various traditional machine learning models demonstrated the robustness and practicability of the proposed perturbations. Significance: Our research shows the feasibility of hiding user identity information in EEG data without impacting the primary BCI task information.

Machine learning has achieved great success in electroencephalogram (EEG) based brain-computer interfaces (BCIs). Most existing BCI studies focused on improving the decoding accuracy, with only a few considering the adversarial security. Although many adversarial defense approaches have been proposed in other application domains such as computer vision, previous research showed that their direct extensions to BCIs degrade the classification accuracy on benign samples. This phenomenon greatly affects the applicability of adversarial defense approaches to EEG-based BCIs. To mitigate this problem, we propose alignment-based adversarial training (ABAT), which performs EEG data alignment before adversarial training. Data alignment aligns EEG trials from different domains to reduce their distribution discrepancies, and adversarial training further robustifies the classification boundary. The integration of data alignment and adversarial training can make the trained EEG classifiers simultaneously more accurate and more robust. Experiments on five EEG datasets from two different BCI paradigms (motor imagery classification, and event related potential recognition), three convolutional neural network classifiers (EEGNet, ShallowCNN and DeepCNN) and three different experimental settings (offline within-subject cross-block/-session classification, online cross-session classification, and pre-trained classifiers) demonstrated its effectiveness. It is very intriguing that adversarial attacks, which are usually used to damage BCI systems, can be used in ABAT to simultaneously improve the model accuracy and robustness.

We propose a hybrid mixed spatial graphical model framework and novel concepts, e.g., cross-Markov Random Field (cross-MRF), to comprehensively address all feature aspects of highly multivariate high-dimensionality (HMHD) spatial data class when constructing the desired joint variance and precision matrix (where both p and n are large). Specifically, the framework accommodates any customized conditional independence (CI) among any number of p variate fields at the first stage, alleviating dynamic memory burden. Meanwhile, it facilitates parallel generation of covariance and precision matrix, with the latter's generation order scaling only linearly in p. In the second stage, we demonstrate the multivariate Hammersley-Clifford theorem from a column-wise conditional perspective and unearth the existence of cross-MRF. The link of the mixed spatial graphical framework and the cross-MRF allows for a mixed conditional approach, resulting in the sparsest possible representation of the precision matrix via accommodating the doubly CI among both p and n, with the highest possible exact-zero-value percentage. We also explore the possibility of the co-existence of geostatistical and MRF modelling approaches in one unified framework, imparting a potential solution to an open problem. The derived theories are illustrated with 1D simulation and 2D real-world spatial data., Comment: 46 pages; 11 figures

Matern correlation is of pivotal importance in spatial statistics and machine learning. This paper serves as a panoramic primer for this correlation with an emphasis on the exposition of its changing behavior and smoothness properties in response to the change of its two parameters. Such exposition is achieved through a series of simulation studies, the use of an interactive 3D visualization applet, and a practical modeling example, all tailored for a wide-ranging statistical audience. Meanwhile, the thorough understanding of these parameter-smoothness relationships, in turn, serves as a pragmatic guide for researchers in their real-world modeling endeavors, such as setting appropriate initial values for these parameters and parameter-fine-tuning in their Bayesian modeling practice or simulation studies involving the Matern correlation. Derived problems surrounding Matern, such as inconsistent parameter inference, extended forms of Matern and limitations of Matern, are also explored and surveyed to impart a panoramic view of this correlation.

Data augmentation is a crucial regularization technique for deep neural networks, particularly in medical image classification. Mainstream data augmentation (DA) methods are usually applied at the image level. Due to the specificity and diversity of medical imaging, expertise is often required to design effective DA strategies, and improper augmentation operations can degrade model performance. Although automatic augmentation methods exist, they are computationally intensive. Semantic data augmentation can implemented by translating features in feature space. However, over-translation may violate the image label. To address these issues, we propose \emph{Bayesian Random Semantic Data Augmentation} (BSDA), a computationally efficient and handcraft-free feature-level DA method. BSDA uses variational Bayesian to estimate the distribution of the augmentable magnitudes, and then a sample from this distribution is added to the original features to perform semantic data augmentation. We performed experiments on nine 2D and five 3D medical image datasets. Experimental results show that BSDA outperforms current DA methods. Additionally, BSDA can be easily assembled into CNNs or Transformers as a plug-and-play module, improving the network's performance. The code is available online at \url{https://github.com/YaoyaoZhu19/BSDA}.

We demonstrate the post-induction of high-quality microcavity on silicon photonic crystal (PC) waveguide by integrating few-layer GaSe crystal, which promises highly efficient on-chip optical frequency conversions. The integration of GaSe shifts the dispersion bands of the PC waveguide mode into the bandgap, resulting in localized modes confined by the bare PC waveguides. Thanks to the small contrast of refractive index at the boundaries of microcavity, it is reliably to obtain quality (Q) factors exceeding 10^4. With the enhanced light-GaSe interaction by the microcavity modes and high second-order nonlinearity of GaSe, remarkable second-harmonic generation (SHG) and sum-frequency generation (SFG) are achieved. A record-high on-chip SHG conversion efficiency of 131100% W^-1 is obtained, enabling the clear SHG imaging of the resonant modes with the pump of sub-milliwatts continuous-wave (CW) laser. Driven by a pump of on-resonance CW laser, strong SFGs are successfully carried out with the other pump of a CW laser spanning over the broad telecom-band. Broadband frequency conversion of an incoherent superluminescent light-emitting diode with low spectral power density is also realized in the integrated GaSe-PC waveguide. Our results are expected to provide new strategies for high-efficiency light-matter interactions, nonlinear photonics and light source generation in silicon photonic integrated circuits., Comment: 5 figures

We demonstrate an efficient nonlinear photodetector (NLPD) with quadratic response based on a few-layer InSe p-n homojunction, which is beneficial from the strong second harmonic generation (SHG) process in InSe and effective harvest of photocarriers actuated by the high-quality homojunction. The NLPD can sense light with photon energy smaller than InSe electronic bandgap because the SHG process in InSe doubles the frequency of incident light, extending InSe photodetection wavelength range to 1750 nm. The InSe p-n homojunction, which is electrostatically doped by two split back gates, presents a rectification ratio exceeding 106 with a dark current down to 2 pA and a high normalized responsivity of 0.534 A/W2 for the telecom-band pulsed light at 1550 nm. The photocurrents of the SHG-assisted photodetection have a quadratic dependence on the optical powers, making the NLPD highly sensitive to light intensity variation with improved spatial resolution. As examples, the NLPD is employed to precisely determine the localization point of a focused laser beam waist and implement spatial imaging with an improved resolution compared with the linear photodetector. These features highlight the potential of the proposed NLPD in developing advanced optical sensing and imaging systems.

Recommender systems is set up to address the issue of information overload in traditional information retrieval systems, which is focused on recommending information that is of most interest to users from massive information. Generally, there is a sequential nature and heterogeneity to the behavior of a person interacting with a system, leading to the proposal of multi-behavior sequential recommendation (MBSR). MBSR is a relatively new and worthy direction for in-depth research, which can achieve state-of-the-art recommendation through suitable modeling, and some related works have been proposed. This survey aims to shed light on the MBSR problem. Firstly, we introduce MBSR in detail, including its problem definition, application scenarios and challenges faced. Secondly, we detail the classification of MBSR, including neighborhood-based methods, matrix factorization-based methods and deep learning-based methods, where we further classify the deep learning-based methods into different learning architectures based on RNN, GNN, Transformer, and generic architectures as well as architectures that integrate hybrid techniques. In each method, we present related works based on the data perspective and the modeling perspective, as well as analyze the strengths, weaknesses and features of these works. Finally, we discuss some promising future research directions to address the challenges and improve the current status of MBSR.

This work is motivated by the ECMWF CAMS reanalysis data, a valuable resource for researchers in environmental-related areas, as they contain the most updated atmospheric composition information on a global scale. Unlike observational data obtained from monitoring equipment, such reanalysis data are produced by computers via a 4D-Var data assimilation mechanism, thus their stochastic property remains largely unclear. Such lack of knowledge in turn limits their utility scope and hinders them from wider and more flexible statistical usages, especially spatio-temporal modelling except for uncertainty quantification and data fusion. Therefore, this paper studies the stochastic property of these reanalysis outputs data. We used measure theory and proved the tangible existence of spatial and temporal stochasticity associated with these reanalysis data and revealed that they are essentially realisations from digitised versions of real-world hidden spatial and/or temporal stochastic processes. This means we can treat the reanalysis outputs data the same as observational data in practice and thus ensures more flexible spatio-temporal stochastic methodologies apply to them. We also objectively analysed different types of errors in the reanalysis data and deciphered their mutual dependence/independence, which together give clear and definite guidance on the modelling of error terms. The results of this study also serve as a solid stepping stone for spatio-temporal modellers and environmental AI researchers to embark on their research directly with these reanalysis outputs data using stochastic models.