Search

© 2021 The Authors. In this paper we propose a framework using multi-channel convolutional neural network (MC-CNN) for recognizing the grammatical class (verb or noun) of covertly-spoken words from electroencephalogram (EEG) signals. Our proposed network extracts features by taking into account spatial, temporal, and spectral properties of the EEG signal. Further, sets of signals acquired from different regions of the brain are processed separately within the proposed framework and are subsequently combined at the classification stage. This approach enables the network to effectively learn discriminative features from the locations of the brain where imagined speech is processed. Our network was tested using challenging experiments, including cases where the test subject did not take part in system training. In our main application scenario, where no instance of a specific noun or verb was used during training, our method achieved 85.7% recognition. Further, our proposed method was evaluated on a publicly available EEG dataset and achieved recognition rate of 93.8% in binary classification. These results demonstrate the potential of our method.

With the advent of new algorithms, brain-computer interfacing has been extensively used in medical and non-medical fields. In this regard, an experiment was conducted by the authors to recognize the imagined speech, the results of which are reported in this paper. This work can act as a speech prosthesis for completely paralyzed patients who cannot communicate normally. Thirteen subjects imagined five English words (sos, stop, medicine, comehere, washroom) while their electroencephalogram (EEG) signals were recorded simultaneously. The word pairs were analyzed in six natural frequencies of the brain. The envelopes of analytical signals acquired from Hilbert transform were calculated for all the frequency bands and the resulting features were classified using seven classifiers. The maximum accuracy reached up to 88.36%. The experimental study showed that alpha and theta frequency bands were able to classify the highest amount of imagined speech with a maximum average accuracy of 72.73% and 69.41% respectively. The results were comparable to state-of-the-art methods. The findings reported in this work will encourage the research community to use non-invasive modalities like EEG for exploring more in this area.

The recent investigations and advances in imagined speech decoding and recognition has tremendously improved the decoding of speech directly from brain activity with the help of several neuroimaging techniques that assist us in exploring the neurological processes of imagined speech. This development leads to assist people with disabilities to benefit from neuroprosthetic devices that improve the life of those suffering from neurological disorders. This paper presents the summary of recent progress in decoding imagined speech using Electroenceplography (EEG) signal, as this neuroimaging method enable us to monitor brain activity with high temporal resolution, it is very portable, low cost, and safer as compared to other methods. Therefore, it is a good candidate in investigating an imagined speech decoding from the human cortex which remains a challenging task. The paper also reviews some recent techniques, challenges, future recommendations and possible solutions to improve prosthetic devices and the development of brain computer interface system (BCI).

We present a transfer learning-based approach for decoding imagined speech from electroencephalogram (EEG). Features are extracted simultaneously from multiple EEG channels, rather than separately from individual channels. This helps in capturing the interrelationships between the cortical regions. To alleviate the problem of lack of enough data for training deep networks, sliding window-based data augmentation is performed. Mean phase coherence and magnitude-squared coherence, two popular measures used in EEG connectivity analysis, are used as features. These features are compactly arranged, exploiting their symmetry, to obtain a three dimensional “image-like” representation. The three dimensions of this matrix correspond to the alpha, beta and gamma EEG frequency bands. A deep network with ResNet50 as the base model is used for classifying the imagined prompts. The proposed method is tested on the publicly available ASU dataset of imagined speech EEG, comprising four different types of prompts. The accuracy of decoding the imagined prompt varies from a minimum of 79.7% for vowels to a maximum of 95.5% for short-long words across the various subjects. The accuracies obtained are better than the state-of-the-art methods, and the technique is good in decoding prompts of different complexities.

Neural oscillations synchronize with the periodicity of external stimuli such as the rhythm of the speech amplitude envelope. This synchronization induces a speech-specific, replicable neural phase pattern across trials and enables perceived speech to be classified. In this study, we hypothesized that neural oscillations during articulatory imagination of speech could also synchronize with the rhythm of speech imagery. To validate the hypothesis, after replacing the imagined speech with overt speech due to the physically unobservable nature of imagined speech, we investigated (1) whether the EEG-based regressed speech envelopes correlate with the overt speech envelope and (2) whether EEG during the imagined speech can classify speech stimuli with different envelopes. The variability of the duration of the imagined speech across trials was corrected using dynamic time warping. The classification was based on the distance between a test data and a template waveform of each class. Results showed a significant correlation between the EEG-based regressed envelope and the overt speech envelope. The average classification accuracy was 38.5%, which is significantly above the rate of chance (33.3%). These results demonstrate the synchronization between EEG during the imagined speech and the envelope of the overt counterpart.

Interpretation of neural signals to a form that is as intelligible as speech facilitates the development of communication mediums for the otherwise speech/motor-impaired individuals. Speech perception, production, and imagination often constitute phases of human communication. The primary goal of this article is to analyze the similarity between these three phases by studying electroencephalogram(EEG) patterns across these modalities, in order to establish their usefulness for brain computer interfaces. Neural decoding of speech using such non-invasive techniques necessitates the optimal choice of signal analysis and translation protocols. By employing selection-by-exclusion based temporal modeling algorithms, we discover fundamental syllable-like units that reveal similar set of signal signatures across all the three phases. Significantly higher than chance accuracies are recorded for single trial multi-unit EEG classification using machine learning approaches over three datasets across 30 subjects. Repeatability and subject independence tests performed at every step of the analysis further strengthens the findings and holds promise for translating brain signals to speech non-invasively.

The use of imagined speech with electroencephalographic (EEG) signals is a promising field of brain-computer interfaces (BCI) that seeks communication between areas of the cerebral cortex related to language and devices or machines. However, the complexity of this brain process makes the analysis and classification of this type of signals a relevant topic of research. The goals of this study were: to develop a new algorithm based on Deep Learning (DL), referred to as CNNeeg1-1, to recognize EEG signals in imagined vowel tasks, to create an imagined speech database with 50 subjects specialized in imagined vowels from the Spanish language (/a/,/e/,/i/,/o/,/u/), and to contrast the performance of the CNNeeg1-1 algorithm with the DL Shallow CNN and EEGNet benchmark algorithms using an open access database (BD1) and the newly developed database (BD2). In this study, a mixed variance analysis of variance was conducted to assess the intra-subject and inter-subject training of the proposed algorithms. The results show that for intra-subject training analysis, the best performance among the Shallow CNN, EEGNet, and CNNeeg1-1 methods in classifying imagined vowels (/a/,/e/,/i/,/o/,/u/) was exhibited by CNNeeg1-1, with an accuracy of 65.62% for BD1 database and 85.66% for BD2 database.

The recent advances in brain–computer interface (BCI) systems have resulted in successful implementations of multiple application domains using different paradigms. The imaginary speech paradigm has received less focus when compared to other paradigms. This study proposes deep learning models to classify and detect the imaginary pronunciations of vowels ‘a’ and ‘u’ using EEG data. The data was captured from three subjects with no psychological or neurological disorders. Channels Fz, Cz, C3, and C4 were used for analyzing the data. A bandpass filter with a frequency range of 1–45 Hz was used to filter the data. Three deep learning models are proposed in this study. These models are the convolution neural network (CNN) model, the long short-term memory (LSTM) model, and a model based on the combination of CNN and LSTM. The model containing combinations of CNN and LSTM performed best with the average classification accuracy of 84%. The CNN model and LSTM models yielded average classification accuracies of 51% and 63%, respectively.

There is increasing interest in developing intuitive brain-computer interfaces (BCIs) to differentiate intuitive mental tasks such as imagined speech. Both electroencephalography (EEG) and function...

Due to the problems presented in current traditional/biometric security systems, the interest to use new security systems, have been increasing. This paper explores the use of brain signals EEG-based during imagined speech in order to use it as a new biometric measure for Subjects identification and thus create a new biometric security system. The main contribution of this paper are two methods for feature extraction, first to improve the signal-to-noise ratio the Common Average Reference was applied. The first method was based on Discrete Wavelet Transform, and the second method was based on statistical features directly from the raw signal. The proposed methods were tested in a dataset of 27 Subjects who performed 33 repetitions of 5 imagined words in Spanish. The results show the feasibility of the task with accurate identification of the Subject, regardless of the imagined word used and using a commercial EEG system (EMOTIV EPOC). In addition, the scope of the method is displayed by decreasing the training data, as well as the number of active sensors for the identification task. Using the proposed method with future improvements and implementing it in a low-cost device can be a new and valuable biometric security system.

Automatic speech recognition interfaces are becoming increasingly pervasive in daily life as a means of interacting with and controlling electronic devices. Current speech interfaces, however, are infeasible for a variety of users and use cases, such as patients who suffer from locked-in syndrome or those who need privacy. In these cases, an interface that works based on envisioned speech, the idea of imagining what one wants to say, could be of benefit. Consequently, in this work, we propose an imagined speech Brain-Computer-Interface (BCI) using Electroencephalogram (EEG) signals. EEG signals are processed using a deep spatio-temporal learning architecture with 1D Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM), respectively. LSTM units are implemented in a many-to-many fashion to produce a time series of imagined speech outputs. Using this series, the performance of the system is boosted using majority vote (MV) post-processing to further improve results. The performance is evaluated on two publicly available datasets; one to test the performance of the tuned model, and another to test its generalization to a new dataset. The proposed architecture outperforms previous results with improvements of up to 23.7%.

Objective: Decoding imagined speech from brain signals could provide a more natural, user-friendly way for developing the next generation of the brain-computer interface (BCI). With the advantages of non-invasive, portable, relatively high spatial resolution and insensitivity to motion artifacts, the functional near-infrared spectroscopy (fNIRS) shows great potential for developing the non-invasive speech BCI. However, there is a lack of fNIRS evidence in uncovering the neural mechanism of imagined speech. Our goal is to investigate the specific brain regions and the corresponding cortico-cortical functional connectivity features during imagined speech with fNIRS. Approach: fNIRS signals were recorded from thirteen subjects' bilateral motor and prefrontal cortex during overtly and covertly repeating words. Cortical activation was determined through the mean oxygen-hemoglobin concentration changes, and functional connectivity was calculated by Pearson's correlation coefficient. Main results: (1) The bilateral dorsal motor cortex was significantly activated during the covert speech, whereas the bilateral ventral motor cortex was significantly activated during the overt speech. (2) As a subregion of the motor cortex, sensorimotor cortex (SMC) showed a dominant dorsal response to covert speech condition, whereas a dominant ventral response to overt speech condition. (3) Broca's area was deactivated during the covert speech but activated during the overt speech. (4) Compared to overt speech, dorsal SMC-related functional connections were enhanced during the covert speech. Significance: We provide fNIRS evidence for the involvement of dSMC in speech imagery. dSMC is the speech imagery network's key hub and is probably involved in the sensorimotor information processing during the covert speech. This study could inspire the BCI community to focus on the potential contribution of dSMC during speech imagery.

The task of classifying vowels via brain activities has been studied in many ideal direct-speech brain-computer interfaces (DS-BCIs). The vowels in those studies usually had clear acoustic differences, mainly on the first and second formants (i.e., F1 and F2). Whereas recent studies found that those speech features were difficult to be presented in DS-BCIs based on imagined speech, the spoken speech with audible output has the potential to provide insight regarding the relationship between spoken vowels' classification accuracies and their acoustic differences. This work aimed to classify four spoken Mandarin vowels (i.e., /a/, /u/, /i/ and /u/, and pronounced with different consonants and tones to form monosyllabic stimuli in Mandarin Chinese) by using electroencephalogram (EEG) signals. The F1 and F2 of each spoken vowel were extracted; the corresponding spoken EEG signals were analyzed with the Riemannian manifold method and further used to classify spoken vowels with a linear discriminant classifier. The acoustic analysis showed that in the F1-F2 plane, the ll ellipse was closed to the /u/ and /i/ ellipses. The classification results showed that vowels /a/., /u/ and /i/ were well classified (82.0%, 69.5% and 68.2%, respectively), but vowel /u/ was more easily classified into /u/, /i/ and /u/. Results in this work suggested that the spoken vowels with similar formant structures were difficult to be classified by using their spoken EEG signals.

The present work aimed to classify three speech modalities (i.e., spoken speech, intended speech and imagined speech) by using their corresponding cortical EEG signals. Eleven participants were recruited to take part in the experiments to produce 70 Mandarin-Chinese monosyllables in different speech modalities. The EEG signals were recorded during the experiments and processed by discrete wavelet transform (DWT), and the extracted features were selected by a minimum redundancy maximum relevance (MRMR) algorithm. Finally, 50 selected features were trained and tested by a multi-class Support Vector Machine for the task of speech modality classification. The average classification accuracy across participants for distinguishing three speech modalities was 66.0%, with pairwise classification accuracies ranging from 71.5% to 85.7%. The results of this work demonstrated the feasibility of identifying different speech modalities in speech-based BCIs by employing EEG signals.

Objective. Directly decoding imagined speech from electroencephalogram (EEG) signals has attracted much interest in brain-computer interface applications, because it provides a natural and intuitive communication method for locked-in patients. Several methods have been applied to imagined speech decoding, but how to construct spatial-temporal dependencies and capture long-range contextual cues in EEG signals to better decode imagined speech should be considered. Approach. In this study, we propose a novel model called hybrid-scale spatial-temporal dilated convolution network (HS-STDCN) for EEG-based imagined speech recognition. HS-STDCN integrates feature learning from temporal and spatial information into a unified end-to-end model. To characterize the temporal dependencies of the EEG sequences, we adopted a hybrid-scale temporal convolution layer to capture temporal information at multiple levels. A depthwise spatial convolution layer was then designed to construct intrinsic spatial relationships of EEG electrodes, which can produce a spatial-temporal representation of the input EEG data. Based on the spatial-temporal representation, dilated convolution layers were further employed to learn long-range discriminative features for the final classification. Main results. To evaluate the proposed method, we compared the HS-STDCN with other existing methods on our collected dataset. The HS-STDCN achieved an averaged classification accuracy of 54.31\% for decoding eight imagined words, which is significantly better than other methods at a significance level of 0.05. Significance. The proposed HS-STDCN model provided an effective approach to make use of both the temporal and spatial dependencies of the input EEG signals for imagined speech recognition. We also visualized the word semantic differences to analyze the impact of word semantics on imagined speech recognition, investigated the important regions in the decoding process, and explored the use of fewer electrodes to achieve comparable performance.

Recent advances in brain-computer interface technology have shown the potential of imagined speech and visual imagery as a robust paradigm for intuitive brain-computer interface communication. However, the internal dynamics of the two paradigms along with their intrinsic features haven't been revealed. In this paper, we investigated the functional connectivity of the two paradigms, considering various frequency ranges. The dataset of sixteen subjects performing thirteen-class imagined speech and visual imagery were used for the analysis. The phase-locking value of imagined speech and visual imagery was analyzed in seven cortical regions with four frequency ranges. We compared the functional connectivity of imagined speech and visual imagery with the resting state to investigate the brain alterations during the imagery. The phase-locking value in the whole brain region exhibited a significant decrease during both imagined speech and visual imagery. Broca and Wernicke's area along with the auditory cortex mainly exhibited a significant decrease in the imagined speech, and the prefrontal cortex and the auditory cortex have shown a significant decrease in the visual imagery paradigm. Further investigation on the brain connectivity along with the decoding performance of the two paradigms may play a crucial role as a performance predictor., Submitted to IEEE The 9th International Winter Conference on Brain-Computer Interface

Speech imagery is a mental strategy that paralyzed patients can use to control a brain-computer interface (BCI) at their own pace. Most studies that have attempted to decode speech have used scalp electroencephalography or electrocorticography. Only few studies have used stereotactic electroencephalography (SEEG), which enables the exploration of deeply located structures in the brain, in this context. In this paper, we aim to identify discriminative features for decoding speech perception and overt and imagined speech production from SEEG recordings in three patients with epilepsy. We report results for the detection of speech events and for the classification of the corresponding utterances. We propose that SEEG-based BCI systems with multiple degrees of freedom may be reliably controlled by selected phonetic features decoded from the superior temporal gyrus.

Brain-Computer Interface (BCI) is a promising technology for individuals who suffer from motor or speech disabilities due to the process of decoding brain signals. This paper uses a dataset for imagined speech to classify vowels based on the neurological areas of the brain. The normalized cross-correlation matrices between two electrodes are used as features. We demonstrate that by using the EEG from the frontal region of the brain, we obtain higher than 85 percent accuracy for correct vowel decoding by using two types of neural networks: convolutional neural network (CNN) and long short-term memory (LSTM). This accuracy is higher than previous studies that have classified the dataset using the entire brain region. This work shows great promise for task decoding where the physiological regions of the brain associated with specific tasks are exploited. The proposed approach has the potential to be deployed in future BCI applications.

An asynchronous Brain--Computer Interface (BCI) based on imagined speech is a tool that allows to control an external device or to emit a message at the moment the user desires to by decoding EEG signals of imagined speech. In order to correctly implement these types of BCI, we must be able to detect from a continuous signal, when the subject starts to imagine words. In this work, five methods of feature extraction based on wavelet decomposition, empirical mode decomposition, frequency energies, fractal dimension and chaos theory features are presented to solve the task of detecting imagined words segments from continuous EEG signals as a preliminary study for a latter implementation of an asynchronous BCI based on imagined speech. These methods are tested in three datasets using four different classifiers and the higher F1 scores obtained are 0.73, 0.79, and 0.68 for each dataset, respectively. This results are promising to build a system that automatizes the segmentation of imagined words segments for latter classification., Comment: 10 pages, 14 figures

The objective of this work is to explore the potential use of electroencephalography (EEG) as a means for silent communication by way of decoding imagined speech from measured electrical brain waves. Communication using BCI can be used for medical and non-medical application. Worldwide a large number of people suffer from disabilities which impair normal communication. Communication BCI’s are excellent tool which helps the affected patients communicate with others. BCI technology is used can be used in many different areas of life, beginning on most important field which is help people, through improving or making life easier and ending with entertainment. Also BCI technology can be used in military in different aspects, it may be used to monitor a soldier's cognitive workload, control a drone swarm, or link with a prosthetic, among other examples.

Brain-computer interface (BCI) is oriented toward intuitive systems that users can easily operate. Imagined speech and visual imagery are emerging paradigms that can directly convey a user’s intention. We investigated the underlying characteristics that affect the decoding performance of these two paradigms. Twenty-two subjects performed imagined speech and visual imagery of twelve words/phrases frequently used for patients’ communication. Spectral features were analyzed with thirteen-class classification (including rest class) using EEG filtered in six frequency ranges. In addition, cortical regions relevant to the two paradigms were analyzed by classification using single-channel and pre-defined cortical groups. Furthermore, we analyzed the word properties that affect the decoding performance based on the number of syllables, concrete and abstract concepts, and the correlation between the two paradigms. Finally, we investigated multiclass scalability in both paradigms. The high-frequency band displayed a significantly superior performance to that in the case of any other spectral features in the thirteen-class classification (imagined speech: 39.73 ± 5.64%; visual imagery: 40.14 ± 4.17%). Furthermore, the performance of Broca’s and Wernicke’s areas and auditory cortex was found to have improved among the cortical regions in both paradigms. As the number of classes increased, the decoding performance decreased moderately. Moreover, every subject exceeded the confidence level performance, implying the strength of the two paradigms in BCI inefficiency. These two intuitive paradigms were found to be highly effective for multiclass communication systems, having considerable similarities between each other. The results could provide crucial information for improving the decoding performance for practical BCI applications.

Speech cognition bears potential application as a brain computer interface that can improve the quality of life for the otherwise communication impaired people. While speech and resting state EEG are popularly studied, here we attempt to explore a "non-speech"(NS) state of brain activity corresponding to the silence regions of speech audio. Firstly, speech perception is studied to inspect the existence of such a state, followed by its identification in speech imagination. Analogous to how voice activity detection is employed to enhance the performance of speech recognition, the EEG state activity detection protocol implemented here is applied to boost the confidence of imagined speech EEG decoding. Classification of speech and NS state is done using two datasets collected from laboratory-based and commercial-based devices. The state sequential information thus obtained is further utilized to reduce the search space of imagined EEG unit recognition. Temporal signal structures and topographic maps of NS states are visualized across subjects and sessions. The recognition performance and the visual distinction observed demonstrates the existence of silence signatures in EEG.

A Brain-Computer Interface (BCI) System captures the neural activity of the Central Nervous System (CNS) and delivers an output which replaces the natural output of the CNS [1]. That helps who have lost their ability to speak, spelling words in a monitor or helps to recover the movements for people who have suffered some amputation of their limbs or a motor paralysis of their body. The main objective of the project is to control an upper limb using a Myohand twin Ottobock prosthesis ref 8E38 = 7 [2] using EEG silent speech signals. On this research, we will focus on a novel methodology that attempts to classify imagined speech based on vowels, which uses as the primary technique for artifact removal the Adaptive-Projection Intrinsically Transformed Multivariate Empirical Mode Decomposition (APIT-MEMD) [3], and for the feature generation stage the Singular Value Decomposition (SVD) [4]. For the classification stage, two classifiers where tested, the Extremely Randomized Trees classifier (ET) [5], and the Adaboost (ADB) [6]. The overall accuracy achieved per subject and per vowels’ pairwise classification, was 91.54% using ET. For a multiclass classifier, the overall accuracy over the eighteen subjects of the database was 79.06%.

Purpose: The present study which addressed adults who stutter (AWS), has been an attempt to investigate power spectral dynamics in stuttering state through answering the written questions using quantitative electroencephalography (qEEG).Materials and Methods: A 64-channel EEG setup was used for data acquisition in 9 AWS. Since speech, and especially stuttering, causes significant noise in the EEG, the three conditions of speech preparation (SP), imagined speech (IS), and simulated speech (SS) in a 7-band format were chosen to source localize the signals using the standard low-resolution electromagnetic tomography (sLORETA) tool in fluent and disfluent states. Results: Having extracted enough fluent and disfluent utterances, significant differences were noted. Consistent with previous studies, the lack of beta suppression in SP, especially in beta2 and beta3 and somewhat in gamma band, was localized in supplementary motor area (SMA) and pre motor area in disfluent state. Delta band frequency was the best marker of stuttering shared in all 3 experimental conditions. Decreased delta power in SMA of both hemispheres and right premotor area through SP, in fronto-central and right angular gyrus through IS, and in SMA of both hemispheres through SS were a notable qEEG features of disfluent speech. Conclusion: The dynamics of beta and delta frequency bands may potentially explain the neural networks involved in stuttering. Based on the above, neurorehabilitation may better be formulated in the treatment of speech disfluency, namely stuttering.

Translation of imagined speech electroencephalogram(EEG) into human understandable commands greatly facilitates the design of naturalistic brain computer interfaces. To achieve improved imagined speech unit classification, this work aims to profit from the parallel information contained in multi-phasal EEG data recorded while speaking, imagining and performing articulatory movements corresponding to specific speech units. A bi-phase common representation learning module using neural networks is designed to model the correlation and reproducibility between an analysis phase and a support phase. The trained Correlation Network is then employed to extract discriminative features of the analysis phase. These features are further classified into five binary phonological categories using machine learning models such as Gaussian mixture based hidden Markov model and deep neural networks. The proposed approach further handles the non-availability of multi-phasal data during decoding. Topographic visualizations along with result-based inferences suggest that the multi-phasal correlation modelling approach proposed in the paper enhances imagined-speech EEG recognition performance.

Imagined speech is spotlighted as a new trend in the brain-machine interface due to its application as an intuitive communication tool. However, previous studies have shown low classification performance, therefore its use in real-life is not feasible. In addition, no suitable method to analyze it has been found. Recently, deep learning algorithms have been applied to this paradigm. However, due to the small amount of data, the increase in classification performance is limited. To tackle these issues, in this study, we proposed an end-to-end framework using Siamese neural network encoder, which learns the discriminant features by considering the distance between classes. The imagined words (e.g., arriba (up), abajo (down), derecha (right), izquierda (left), adelante (forward), and atras (backward)) were classified using the raw electroencephalography (EEG) signals. We obtained a 6-class classification accuracy of 31.40 ± 2.73% for imagined speech, which significantly outperformed other methods. This was possible because the Siamese neural network, which increases the distance between dissimilar samples while decreasing the distance between similar samples, was used. In this regard, our method can learn discriminant features from a small dataset. The proposed framework would help to increase the classification performance of imagined speech for a small amount of data and implement an intuitive communication system.

Artificial speech synthesis can be done using electroencephalography (EEG) and electrocorticography (ECoG) for the brain–computer interface (BCI). This paper focuses on using EEG to classify phonological categories. Although literature is available on the identification and classification of phoneme information in the electroencephalography signals, the classification accuracy of some phonological categories is high, while that of others is too low. Thus, this chapter focuses on identifying the correlation between imagined EEG and audio signals to select the appropriate EEG features. It also identifies the EEG channels that are best suited for imagined speech. Once features are selected, phonemes are classified as vowels and consonants using a support vector machine. Experimental results suggest good accuracy when using 49 features that correlated with audio signals.

Imagined speech is an emerging paradigm for intuitive control of the brain-computer interface based communication system. Although the decoding performance of the imagined speech is improving with actively proposed architectures, the fundamental question about ‘what component are they decoding?’ is still remaining as a question mark. Considering that the imagined speech refers to an internal mechanism of producing speech, it may naturally resemble the distinct features of the overt speech. In this paper, we investigate the close relation of the spatial and temporal features between imagined speech and overt speech using electroencephalography signals. Based on the common spatial pattern feature, we acquired 16.2% and 59.9% of averaged thirteen-class classification accuracy (chance rate = 7.7%) for imagined speech and overt speech, respectively. Although the overt speech showed significantly higher classification performance compared to the imagined speech, we found potentially similar common spatial pattern of the identical classes of imagined speech and overt speech. Furthermore, in the temporal feature, we examined the analogous grand averaged potentials of the highly distinguished classes in the two speech paradigms. Specifically, the correlation of the amplitude between the imagined speech and the overt speech was 0.71 in the class with the highest true positive rate. The similar spatial and temporal features of the two paradigms may provide a key to the bottom-up decoding of imagined speech, implying the possibility of robust classification of multiclass imagined speech. It could be a milestone to comprehensive decoding of the speech-related paradigms, considering their underlying patterns.

Although a significant number of studies have been devoted to the investigation of the electrographic correlates and neurophysiological mechanisms of spoken and inner (imagined) speech, there is a question on which EEG characteristics reflect its content. Considering that speech is a complex cognitive process which requires coordinated activity of a number of cortical structures of the large hemispheres, the EEG coherence values were studied. The values were recorded from 14 channels of 10 young men in the process of real verbalization (spoken speech) and during pronunciation of imagined words designating directions in space (up, down, right, left, forward, backward). It was shown that the level of EEG coherence is generally higher for real verbalization, most significantly at gamma-2-rhythm frequencies (55–70 Hz). Spatial coherence patterns specific to a number of words are formed in the left cerebral hemisphere during imagined utterance of words at gamma-2 frequencies. The application of machine learning and neural network classification has demonstrated a significant similarity of the generated spatial coherent patterns of spoken and inner (imagined) speech. The Multi-layer Perceptron (MLP) neural network classification method has shown the accuracy of word detection in the imagined speech according to brain activity patterns up to 49–61% for 3 classes and 33–40% for 7 classes, with a random recognition rate of 33,3 and 14,2% respectively. The latter indicates a promising application of coherence values and imagined speech denoting directions in space for the development of Brain-computer interfaces (BCIs).

The way top-down and bottom-up processes interact to shape our perception and behaviour is a fundamental question and remains highly controversial. How early in a processing stream do such interactions occur, and what factors govern such interactions? The degree of abstractness of a perceptual attribute (for example, orientation versus shape in vision, or loudness versus sound identity in hearing) may determine the locus of neural processing and interaction between bottom-up and internal information. Using an imagery-perception repetition paradigm, we find that imagined speech affects subsequent auditory perception, even for a low-level attribute such as loudness. This effect is observed in early auditory responses in magnetoencephalography and electroencephalography that correlate with behavioural loudness ratings. The results suggest that the internal reconstruction of neural representations without external stimulation is flexibly regulated by task demands, and that such top-down processes can interact with bottom-up information at an early perceptual stage to modulate perception.

Communication using brain–computer interfaces (BCIs) can be non-intuitive, often requiring the performance of a conversation-irrelevant task such as hand motor imagery. In this paper, the reliability of electroencephalography (EEG) signals in discriminating between different covert speech tasks is investigated. Twelve participants, across two sessions each, were asked to perform multiple iterations of three differing mental tasks for 10 s each: unconstrained rest or the mental repetition of the words “yes” or “no.” A multilayer perceptron (MLP) artificial neural network (ANN) was used to classify all three pairwise combinations of “yes,” “no,” and rest trials and also for ternary classification. An average accuracy of 75.7% ± 9.6 was reached in the classification of covert speech trials versus rest, with all participants exceeding chance level (57.8%). The classification of “yes” versus “no” yielded an average accuracy of 63.2 ± 6.4 with ten participants surpassing chance level (57.8%). Finally, the ternary classification yielded an average accuracy of 54.1% ± 9.7 with all participants exceeding chance level (39.1%). The proposed MLP network provided significantly higher accuracies compared to some of the most common classification techniques in BCI. To our knowledge, this is the first report of using ANN for the classification of EEG covert speech across multiple sessions. Our findings support further study of covert speech as a BCI activation task, potentially leading to the development of more intuitive BCIs for communication.

Recent advances in EEG technology makes brain-computer-interface (BCI) an exciting field of research. BCI is primarily used to adopt with the paralyzed human body parts. However, BCI in envisioned speech recognition using electroencephalogram (EEG) signals has not been studied in details. Therefore, developing robust speech recognition system using EEG signals was proposed. In this paper, we propose a coarse-to-fine-level envisioned speech recognition framework with the help of EEG signals that can be thought of as a serious contribution in this field of research. Coarse-level classification is used to differentiate/categorize text and non-text classes using random forest (RF) classifier. Next, a finer-level imagined speech recognition of each class has been carried out. EEG data of 30 text and not-text classes including characters, digits, and object images have been imagined by 23 participants in this study. A recognition accuracy of 85.20 and 67.03% has been recorded at coarse- and fine-level classifications, respectively. The proposed framework outperforms the existing research work in terms of accuracy. We also show the robustness in envisioned speech recognition.

EEG-based Brain–Computer Interfaces (BCI) are an alternative technique that aims to integrate people with severe motor disabilities to their environment. However, they still are not used in everyday life because BCIs are controlled by means of few intuitive electrophysiological sources. To address this problem, work has been carried out with the objective of classifying EEG signals recorded while imagining speech. In this paper, the emerging techniques of sonification and textification are applied to EEG signals, which allows us to characterize EEG signals as either an audio signal or a text document. The aim is to analyze whether these techniques can help to do a better discrimination or to highlight patterns, and assess which of the two methods performs better in order to improve classification results of unspoken words. For proving this we processed signals, from 27 subjects, with and without the application of the proposed methods of sonification and textification. The average accuracy rate using the original EEG signals, the EEG sonified signals and the EEG textified signals are 58.41 ± 12.41%, 63.82 ± 13.44% and 83.34 ± 6.12%, respectively. It can be clearly noticed that the methods of sonification and textification of EEG improve the classification rates obtained using the EEG signals in their original form.

Recent models of speech production suggest that motor commands generate forward predictions of the auditory consequences of those commands, that these forward predications can be used to monitor and correct speech output, and that this system is hierarchically organized (Hickok, Houde, & Rong, Neuron, 69(3), 407-–422, 2011; Pickering & Garrod, Behavior and Brain Sciences, 36(4), 329-–347, 2013). Recent psycholinguistic research has shown that internally generated speech (i.e., imagined speech) produces different types of errors than does overt speech (Oppenheim & Dell, Cognition, 106(1), 528-–537, 2008; Oppenheim & Dell, Memory & Cognition, 38(8), 1147–1160, 2010). These studies suggest that articulated speech might involve predictive coding at additional levels than imagined speech. The current fMRI experiment investigates neural evidence of predictive coding in speech production. Twenty-four participants from UC Irvine were recruited for the study. Participants were scanned while they were visually presented with a sequence of words that they reproduced in sync with a visual metronome. On each trial, they were cued to either silently articulate the sequence or to imagine the sequence without overt articulation. As expected, silent articulation and imagined speech both engaged a left hemisphere network previously implicated in speech production. A contrast of silent articulation with imagined speech revealed greater activation for articulated speech in inferior frontal cortex, premotor cortex and the insula in the left hemisphere, consistent with greater articulatory load. Although both conditions were silent, this contrast also produced significantly greater activation in auditory cortex in dorsal superior temporal gyrus in both hemispheres. We suggest that these activations reflect forward predictions arising from additional levels of the perceptual/motor hierarchy that are involved in monitoring the intended speech output.

Speech production is a hierarchical mechanism involving the synchronization of the brain and the oral articulators, where the intention of linguistic concepts is transformed into meaningful sounds. Individuals with locked-in syndrome (fully paralyzed but aware) lose their motor ability completely including articulation and even eyeball movement. The neural pathway may be the only option to resume a certain level of communication for these patients. Current brain-computer interfaces (BCIs) use patients' visual and attentional correlates to build communication, resulting in a slow communication rate (a few words per minute). Direct decoding of imagined speech from the neural signals (and then driving a speech synthesizer) has the potential for a higher communication rate. In this study, we investigated the decoding of five imagined and spoken phrases from single-trial, non-invasive magnetoencephalography (MEG) signals collected from eight adult subjects. Two machine learning algorithms were used. One was an artificial neural network (ANN) with statistical features as the baseline approach. The other was convolutional neural networks (CNNs) applied on the spatial, spectral and temporal features extracted from the MEG signals. Experimental results indicated the possibility to decode imagined and spoken phrases directly from neuromagnetic signals. CNNs were found to be highly effective with an average decoding accuracy of up to 93% for the imagined and 96% for the spoken phrases.

A brain–computer interface (BCI) is a technology that uses neural features to restore or augment the capabilities of its user. A BCI for speech would enable communication in real time via neural correlates of attempted or imagined speech. Such a technology would potentially restore communication and improve quality of life for locked-in patients and other patients with severe communication disorders. There have been many recent developments in neural decoders, neural feature extraction, and brain recording modalities facilitating BCI for the control of prosthetics and in automatic speech recognition (ASR). Indeed, ASR and related fields have developed significantly over the past years, and many lend many insights into the requirements, goals, and strategies for speech BCI. Neural speech decoding is a comparatively new field but has shown much promise with recent studies demonstrating semantic, auditory, and articulatory decoding using electrocorticography (ECoG) and other neural recording modalities. Because the neural representations for speech and language are widely distributed over cortical regions spanning the frontal, parietal, and temporal lobes, the mesoscopic scale of population activity captured by ECoG surface electrode arrays may have distinct advantages for speech BCI, in contrast to the advantages of microelectrode arrays for upper-limb BCI. Nevertheless, there remain many challenges for the translation of speech BCIs to clinical populations. This review discusses and outlines the current state-of-the-art for speech BCI and explores what a speech BCI using chronic ECoG might entail. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s13311-018-00692-2) contains supplementary material, which is available to authorized users.

Speech mental imagery is a quasi-perceptual experience that occurs in the absence of real speech stimulation. How imagined speech with higher-order structures such as words, phrases and sentences is rapidly organized and internally constructed remains elusive. To address this issue, subjects were tasked with imagining and perceiving poems along with a sequence of reference sounds with a presentation rate of 4 Hz while magnetoencephalography (MEG) recording was conducted. Giving that a sentence in a traditional Chinese poem is five syllables, a sentential rhythm was generated at a distinctive frequency of 0.8 Hz. Using the frequency tagging we concurrently tracked the neural processing timescale to the top-down generation of rhythmic constructs embedded in speech mental imagery and the bottom-up sensory-driven activity that were precisely tagged at the sentence-level rate of 0.8 Hz and a stimulus-level rate of 4 Hz, respectively. We found similar neural responses induced by the internal construction of sentences from syllables with both imagined and perceived poems and further revealed shared and distinct cohorts of cortical areas corresponding to the sentence-level rhythm in imagery and perception. This study supports the view of a common mechanism between imagery and perception by illustrating the neural representations of higher-order rhythmic structures embedded in imagined and perceived speech.

Scalp electroencephalogram (EEG) is one of the most commonly used methods to acquire EEG data for brain-computer interfaces (BCIs). Worldwide a large number of people suffer from disabilities which impair normal communication. Communication BCIs are an excellent tool which helps the affected patients communicate with others. In this paper scalp EEG data is analysed to discriminate between the imagined vowel sounds /a/, /u/ and no action or rest as control state. Mean absolute deviation (MAD) and Arithmetic mean are used as features to classify data into one of the classes /a/, /u/ or rest. With high classification accuracies of 87.5-100% for two class problem and 78.33-96.67% for three class problem that have been obtained in this work, this algorithm can be used in communication BCIs, to develop speech prosthesis and in synthetic telepathy systems.

Studies on recognising unspoken speech with the use of electroencephalographic (EEG) signals vary in their designs. The participants are either asked to imagine unspoken speech within a specific time frame, or alternatively indicate the start and end of the imagined speech. Optimizing the length and training size of imagined speech is important to improve the rate and speed of recognizing unspoken speech in on-line applications. In this study, we recorded EEG data when the participants performed unspoken speech of five words using two technologies: (1) marking the start and end of the trial by using mouse clicks and (2) performing the imagination in a four-second fixed time window. Four classifiers were trained in all experiment parts: support vector machine, naive bayes, random forest, and linear discriminate analysis. The results show that the best time frame is 3.5-4 seconds length. Moreover, the increase in training size improve the average classification accuracy. However, this improvement becomes slight between 125-175 total training trials. The training data can be recorded in parts, however, the required training size should be increased to have better classification accuracy. In all analysis parts, random forest classifier shows better results among the other classifiers.

Objective: In this study, electroencephalography data of imagined words were classified using four different feature extraction approaches. Eight subjects were recruited for the recording of imagination with five different words, namely; “go,” “back,” “left,” “right,” and “stop.” Methods: One hundred trials for each word were recorded for both imagination and perception, although this study utilized only imagination data. Two different connectivity methods were applied, namely; a covariance-based and a maximum linear cross-correlation-based connectivity measure. These connectivity measures were further computed to extract the phase-only data as an additional method of feature extraction. In addition, four different channel selections were used. The final connectivity matrix from each of the four methods was vectorized and used as the feature vector for the classifier. To classify EEG data, a sigmoid activation function-based linear extreme learning machine was used. Result and Significance: We achieved a maximum classification rate of 40.30% ( p $ 0.007) and 87.90% ( p $ 0.003) in multiclass (five classes) and binary settings, respectively. Thus, our results suggested that EEG responses to imagined speech could be successfully classified using an extreme learning machine. Conclusion: This study involving the classification of imagined words can be a milestone contribution toward the development of practical brain–computer interface systems using silent speech.

Speech plays an important role in day to day life in communication. The patients suffering from severe motor neuron diseases are unable to speak due to restricted motor movement and find it difficult to communicate with others. brain computer interface (BCI) is the technology that can be used for the rehabilitation of such patients which can decode their thoughts into words/commands thus improving their quality of life. In present paper BCI has been implemented for the imagined speech, where three Hindi vowels were used that was pronounced by eight subjects covertly. Three statistical time domain features were used to classify Electroencephalographic (EEG) signals by applying linear support vector machine (SVM) as classifier. The results obtained were represented in terms of training accuracy (TA) and prediction accuracy (PA).

Imagined speech has recently become an important neuro-paradigm in the field of brain-computer interface (BCI) research. Electroencephalogram (EEG) recordings during imagined speech production are difficult to decode accurately, due to factors such as weak neural correlates and spatial specificity, and signal noise during the recording process. In this study, a dataset of imagined speech recordings obtained during production of eleven different units of imagined speech is used to investigate the relative effects of different features on classification accuracy. Three distinct feature-sets are computed from the data: a linear feature-set, a non-linear feature-set, and a feature-set comprised only of mel frequency cepstral coefficients (MFCC). Each feature-set is used to train a decision tree classifier and a Support Vector Machine classifier. The results indicate that the use of MFCC features provides greater discrimination of imagined speech EEG recordings in comparison with the other features evaluated, and that phonological differences between imagined words can serve as an aid to classification.

Most of the researches in Electroencephalogram(EEG)-based Brain-Computer Interfaces (BCI) are focused on the use of motor imagery. As an attempt to improve the control of these interfaces, the use of language instead of movement has been recently explored, in the form of imagined speech. This work aims for the discrimination of imagined words in electroencephalogram signals. For this purpose, the analysis of multiple variables of the signal and their relation is considered by means of a multivariate data analysis, i.e., Parallel Factor Analysis (PARAFAC). In previous works, this method has demonstrated to be useful for EEG analysis. Nevertheless, to the best of our knowledge, this is the first attempt to analyze imagined speech signals using this approach. In addition, a novel use of the extracted PARAFAC components is proposed in order to improve the discrimination of the imagined words. The obtained results, besides of higher accuracy rates in comparison with related works, showed lower standard deviation among subjects suggesting the effectiveness and robustness of the proposed method. These results encourage the use of multivariate analysis for BCI applications in combination with imagined speech signals.

Imagined speech is a process where a person imagines the sound of words without moving any of his or her muscles to actually say the word. If the brain signals of a person imagining the speech can be used to recognize the actual words intended to be spoken, this could be a huge step towards helping people with physical disabilities such as locked-in syndrome to have effective communication with others. This can also prove to be useful in situation where visual or audible communication is undesirable, for instance in military situation. Recent advancement in technologies and devices for capturing brain signals, particularly electroencephalogram (EEG), has made the research in recognizing imagined speech possible. While these are still in early years, published studies have shown promising results in this particular area of research. Current approaches in recognizing imagined speech can generally be divided into two, syllable-based and word-based. In this paper, we proposed a simple word-based approach using Mel Frequency Cepstral Coefficients (MFCC) and k-Nearest Neighbor (k-NN) towards recognizing two simple words using EEG signals. Despite its simplicity, the results obtained show some improvements to other studies based on dry EEG electrode device.

When brain activity ions, the potential for human capacities augmentation is promising. In this paper, EMD is used to decompose EEG signals during Imagined Speech in order to use it as a biometric marker for creating a Biometric Recognition System. For each EEG channel, the most relevant Intrinsic Mode Functions (IMFs) are decided based on the Minkowski distance, and for each IMF 4 features are computed: Instantaneous and Teager energy distribution and Higuchi and Petrosian Fractal Dimension. To test the proposed method, a dataset with 20 Subjects who imagined 30 repetitions of 5 words in Spanish, is used. Four classifiers are used for this task - random forest, SVM, naive Bayes, and k-NN - and their performances are compared. The accuracy obtained (up to 0.92 using Linear SVM) after 10-folds cross-validation suggest that the proposed method based on EMD can be valuable for creating EEG-based biometrics of imagined speech for Subject identification.

We propose a mixed deep neural network strategy, incorporating parallel combination of Convolutional (CNN) and Recurrent Neural Networks (RNN), cascaded with deep autoencoders and fully connected layers towards automatic identification of imagined speech from EEG. Instead of utilizing raw EEG channel data, we compute the joint variability of the channels in the form of a covariance matrix that provide spatio-temporal representations of EEG. The networks are trained hierarchically and the extracted features are passed onto the next network hierarchy until the final classification. Using a publicly available EEG based speech imagery database we demonstrate around 23.45% improvement of accuracy over the baseline method. Our approach demonstrates the promise of a mixed DNN approach for complex spatial-temporal classification problems., Accepted in AAAI 2019 under Student Abstract and Poster Program

Brain computer interface (BCI) technologies, which enable direct communication between the brain and external devices, have been developed. BCI technology can be utilized in neural prosthetics to restore impaired movement, including speech production. However, most of the BCI systems that have been developed are the “P300-speller” type, which can only detect objects that users direct his/her attention at. To develop more versatile BCI systems that can detect a user's intention or thoughts, the brain responses associated with verbal imagery need to be clarified. In this study, the brain magnetic fields associated with auditory verbal imagery and speech hearing were recorded using magnetoencephalography (MEG) carried out on 8 healthy adults. Although the magnetic fields lagged slightly and were long-lasting, significant deflections were observed even for verbal imagery, in the temporal regions, as well as for actual speech hearing. Also, sources for the deflections were localized in the association auditory cortices. Cross-correlations were calculated between envelopes of the imagined/presented speech sound and the evoked brain responses in the temporal areas. Measurable correlations were obtained for the presented speech sound; however, no significant correlations were observed for the imagined speech sound. These results indicate that auditory verbal imagery undoubtedly activates the auditory cortex, at least, and generates some observable neural responses.

Verbal communication makes human unique from other species. People use different modalities of speech while communicating with others. Widely practised modalities are speak loudly (utter), whispering and mumbling with closed lips. Apart from speaking, people also speak in their mind. Due to different ailments or injury, some people have lost their ability to speak and are forced to take other means to communicate. Speech restoration through Brain Computer Interfacing (BCI) is still at nascent stage. Through this study, we have explored the contrast between these modalities and it will lead to identification of imagined speech through electroencephalography (EEG). As different speech modalities are similar in nature in spatiotemporal domain, here we have proposed utilisation of nonlinearilty, more specifically multifractal nature, of the modalities present in EEGs. On the basis of the multifractal parameters we have achieved 99.7% accuracy in classification.

This work has focused on the possibilities of classifying vowels ‘a’, ‘e’, ‘i’, ‘o’, ‘u’ from EEG signals, that has been derived while imagining the vowels, with minimum input features. The EEG signals have been acquired from 5 subjects while imagining and uttering the vowels during a well defined experimental protocol, have been processed and segmented using established signal processing routines. The signals have been segmented under various sub-band frequencies and subjected to Db4 Discrete Wavelet Transform. The various conventional and derived energy based features have been acquired from the sub-band frequency signals, trained and tested using Deep Belief Networks for classifying the imagined vowels. The experiments have been repeated on various electrode combinations. Results obtained from all sub-band frequency based features have shown a good classification accuracy. Further, classification protocol employing features that have been derived from each sub-band frequency has shown that the theta and gamma band frequency features have been more effective with a vowel classification accuracy ranging between 75–100%.