Search

Direct cortical stimulation (DCS) has been used extensively as a tool in human neuroscience research and clinically in neurosurgery. Implanted electrocorticography electrodes can deliver electrical stimulation to the cortical surface, eliciting or inhibiting neural activity as an experimental manipulation, or to probe neural tissue function during epilepsy and tumor resection surgeries. Current applications of DCS pave the way for new clinical translation to restore sensation to those who have lost it through spinal-cord injury or stroke. DCS of specific sensory cortical regions in the human brain elicits sensory percepts in specific regions of the body and recent research has shown that DCS can elicit meaningful somatosensory percepts, which may be used for neuroprosthetic feedback and integrated into tasks. In the future, further research using more finely spaced electrodes, tailored patterns of stimulation, and longer term studies will shed light on the intricacies of sensory processing in humans as well as bring these neuroprosthetic applications closer to widespread clinical reality.

ObjectiveAdvances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can robustly train on pooled, multi-participant data and generalize to new participants.ApproachWe introduce a new decoder, HTNet, which uses a convolutional neural network with two innovations: (1) a Hilbert transform that computes spectral power at data-driven frequencies and (2) a layer that projects electrode-level data onto predefined brain regions. The projection layer critically enables applications with intracranial electrocorticography (ECoG), where electrode locations are not standardized and vary widely across participants. We trained HTNet to decode arm movements using pooled ECoG data from 11 of 12 participants and tested performance on unseen ECoG or electroencephalography (EEG) participants; these pretrained models were also subsequently fine-tuned to each test participant.Main resultsHTNet outperformed state-of-the-art decoders when tested on unseen participants, even when a different recording modality was used. By fine-tuning these generalized HTNet decoders, we achieved performance approaching the best tailored decoders with as few as 50 ECoG or 20 EEG events. We were also able to interpret HTNet’s trained weights and demonstrate its ability to extract physiologically-relevant features.SignificanceBy generalizing to new participants and recording modalities, robustly handling variations in electrode placement, and allowing participant-specific fine-tuning with minimal data, HTNet is applicable across a broader range of neural decoding applications compared to current state-of-the-art decoders.

Studying the neural correlates of sleep can lead to revelations in our understanding of sleep and its interplay with different neurological disorders. Sleep research relies on manual annotation of sleep stages based on rules developed for healthy adults. Automating sleep stage annotation can expedite sleep research and enable us to better understand atypical sleep patterns. Our goal was to create a fully unsupervised approach to label sleep and wake states in human electro-corticography (ECoG) data from epilepsy patients. Here, we demonstrate that with continuous data from a single ECoG electrode, hidden semi-Markov models (HSMM) perform best in classifying sleep/wake states without excessive transitions, with a mean accuracy (n=4) of 85.2% compared to using K-means clustering (72.2%) and hidden Markov models (81.5%). Our results confirm that HSMMs produce meaningful labels for ECoG data and establish the groundwork to apply this model to cluster sleep stages and potentially other behavioral states.

Electrocorticographic brain computer interfaces (ECoG-BCIs) offer tremendous opportunities for restoring function in individuals suffering from neurological damage and for advancing basic neuroscience knowledge. ECoG electrodes are already commonly used clinically for monitoring epilepsy and have greater spatial specificity in recording neuronal activity than techniques such as electroencephalography (EEG). Much work to date in the field has focused on using ECoG signals recorded from cortex as control outputs for driving end effectors. An equally important but less explored application of an ECoG-BCI is directing input into cortex using ECoG electrodes for direct electrical stimulation (DES). Combining DES with ECoG recording enables a truly bidirectional BCI, where information is both read from and written to the brain. We discuss the advantages and opportunities, as well as the barriers and challenges presented by using DES in an ECoG-BCI. In this article, we review ECoG electrodes, the physics and physiology of DES, and the use of electrical stimulation of the brain for the clinical treatment of disorders such as epilepsy and Parkinson’s disease. We briefly discuss some of the translational, regulatory, financial, and ethical concerns regarding ECoG-BCIs. Next, we describe the use of ECoG-based DES for providing sensory feedback and for probing and modifying cortical connectivity. We explore future directions, which may draw on invasive animal studies with penetrating and surface electrodes as well as non-invasive stimulation methods such as transcranial magnetic stimulation (TMS). We conclude by describing enabling technologies, such as smaller ECoG electrodes for more precise targeting of cortical areas, signal processing strategies for simultaneous stimulation and recording, and computational modeling and algorithms for tailoring stimulation to each individual brain.

Brain-Computer Interfaces (BCIs) benefit greatly from performance feedback, but current systems lack automatic, task-independent feedback. Cortical responses elicited from user error have the potential to serve as state-based feedback to BCI decoders. To gain a better understanding of local error potentials, we investigate responsive cortical power underlying error-related potentials (ErrPs) from the human cortex during a one-dimensional center-out BCI task, tracking the topography of high-gamma (70-100 Hz) band power (HBP) specific to BCI error. We measured electrocorticography (ECoG) in three human subjects during dynamic, continuous control over BCI cursor velocity. Subjects used motor imagery and rest to move the cursor towards and subsequently dwell within a target region. We then identified and labeled epochs where the BCI decoder incorrectly moved the cursor in the direction opposite of the subject’s expectations (i.e., BCI error). We found increased HBP in various cortical areas 100-500 ms following BCI error with respect to epochs of correct, intended control. Significant responses were noted in primary somatosensory, motor, premotor, and parietal areas and generally regardless of whether the subject was using motor imagery or rest to move the cursor towards the target. Parts of somatosensory, temporal, and parietal areas exclusively had increased HBP when subjects were using motor imagery. In contrast, only part of the parietal cortex near the angular gyrus exclusively had an increase in HBP during rest. This investigation is, to our knowledge, the first to explore cortical fields changes in the context of continuous control in ECoG BCI. We present topographical changes in HBP characteristic specific to the generation of error. By focusing on continuous control, instead of on discrete control for simple selection, we investigate a more naturalistic setting and provide high ecological validity for characterizing error potentials. Such potentials could be considered as design elements for co-adaptive BCIs in the future as task-independent feedback to the decoder, allowing for more robust and individualized BCIs.

We present BrainNet which, to our knowledge, is the first multi-person non-invasive direct brain-to-brain interface for collaborative problem solving. The interface combines electroencephalography (EEG) to record brain signals and transcranial magnetic stimulation (TMS) to deliver information noninvasively to the brain. The interface allows three human subjects to collaborate and solve a task using direct brain-to-brain communication. Two of the three subjects are designated as “Senders” whose brain signals are decoded using real-time EEG data analysis. The decoding process extracts each Sender’s decision about whether to rotate a block in a Tetris-like game before it is dropped to fill a line. The Senders’ decisions are transmitted via the Internet to the brain of a third subject, the “Receiver,” who cannot see the game screen. The Senders’ decisions are delivered to the Receiver’s brain via magnetic stimulation of the occipital cortex. The Receiver integrates the information received from the two Senders and uses an EEG interface to make a decision about either turning the block or keeping it in the same orientation. A second round of the game provides an additional chance for the Senders to evaluate the Receiver’s decision and send feedback to the Receiver’s brain, and for the Receiver to rectify a possible incorrect decision made in the first round. We evaluated the performance of BrainNet in terms of (1) Group-level performance during the game, (2) True/False positive rates of subjects’ decisions, and (3) Mutual information between subjects. Five groups, each with three human subjects, successfully used BrainNet to perform the collaborative task, with an average accuracy of 81.25%. Furthermore, by varying the information reliability of the Senders by artificially injecting noise into one Sender’s signal, we investigated how the Receiver learns to integrate noisy signals in order to make a correct decision. We found that like conventional social networks, BrainNet allows Receivers to learn to trust the Sender who is more reliable, in this case, based solely on the information transmitted directly to their brains. Our results point the way to future brain-to-brain interfaces that enable cooperative problem solving by humans using a “social network” of connected brains.

Cortical stimulation through electrocorticographic (ECoG) electrodes is a potential method for providing sensory feedback in future prosthetic and rehabilitative applications. Here, we evaluate human subjects’ ability to continuously modulate their motor behavior based on feedback from direct surface stimulation of the somatosensory cortex. Subjects wore a dataglove that measured their hand aperture position and received one of three stimuli over the hand sensory cortex based on their current hand position as compared to a target aperture position. Using cortical stimulation feedback, subjects adjusted their hand aperture to move towards the target aperture region. One subject was able to achieve accuracies and R2 values well above chance (best performance: R2 = 0.93; accuracy = 0.76/1). Performance dropped during the catch trial (same stimulus independent of the position) to below chance levels, suggesting that the subject had been using the varied sensory feedback to modulate their motor behavior. To our knowledge, this study represents one of the first demonstrations of using direct cortical surface stimulation of the human sensory cortex to perform a motor task, and is a first step towards developing closed-loop human sensorimotor brain-computer interfaces.

Goal: To identify and overcome barriers to creating new neurotechnologies capable of restoring both motor and sensory function in individuals with neurological conditions. Methods: This report builds upon the outcomes of a joint workshop between the US National Science Foundation and the German Research Foundation on New Perspectives in Neuroengineering and Neurotechnology convened in Arlington, VA, USA, November 13–14, 2014. Results: The participants identified key technological challenges for recording and manipulating neural activity, decoding, and interpreting brain data in the presence of plasticity, and early considerations of ethical and social issues pertinent to the adoption of neurotechnologies. Conclusions: The envisaged progress in neuroengineering requires tightly integrated hardware and signal processing efforts, advances in understanding of physiological adaptations to closed-loop interactions with neural devices, and an open dialog with stakeholders and potential end-users of neurotechnology. Significance: The development of new neurotechnologies (e.g., bidirectional brain–computer interfaces) could significantly improve the quality of life of people living with the effects of brain or spinal cord injury, or other neurodegenerative diseases. Focused efforts aimed at overcoming the remaining barriers at the electrode tissue interface, developing implantable hardware with on-board computation, and refining stimulation methods to precisely activate neural tissue will advance both our understanding of brain function and our ability to treat currently intractable disorders of the nervous system.

The human ventral temporal cortex has regions that are known to selectively process certain categories of visual inputs; they are specialized for the content (“faces,” “places,” “tools”) and not the form (“line,” “patch”) of the image being seen. In our study, human patients with implanted electrocorticography (ECoG) electrode arrays were shown sequences of simple face and house pictures. We quantified neuronal population activity, finding robust face-selective sites on the fusiform gyrus and house-selective sites on the lingual/parahippocampal gyri. The magnitude and timing of single trials were compared between novel (“house-face”) and repeated (“face-face”) stimulus-type responses. More than half of the category-selective sites showed significantly greater total activity for novel stimulus class. Approximately half of the face-selective sites (and none of the house-selective sites) showed significantly faster latency to peak (∼50 ms) for novel stimulus class. This establishes subregions within category-selective areas that are differentially tuned to novelty in sequential context, where novel stimuli are processed faster in some regions, and with increased activity in others.

Electrocorticography (ECoG) signals can provide high spatio-temporal resolution and high signal to noise ratio recordings of local neural activity from the surface of the brain. Previous studies have shown that broad-band, spatially focal, high-frequency increases in ECoG signals are highly correlated with movement and other cognitive tasks and can be volitionally modulated. However, significant additional information may be present in inter-electrode interactions, but adding additional higher order inter-electrode interactions can be impractical from a computational aspect, if not impossible.In this paper we present a new method of calculating high frequency interactions between electrodes called Short-Time Windowed Covariance (STWC) that builds on mathematical techniques currently used in neural signal analysis, along with an implementation that accelerates the algorithm by orders of magnitude by leveraging commodity, off-the-shelf graphics processing unit (GPU) hardware.Using the hardware-accelerated implementation of STWC, we identify many types of event-related inter-electrode interactions from human ECoG recordings on global and local scales that have not been identified by previous methods. Unique temporal patterns are observed for digit flexion in both low- (10mm spacing) and high-resolution (3mm spacing) electrode arrays.Covariance is a commonly used metric for identifying correlated signals, but the standard covariance calculations do not allow for temporally varying covariance. In contrast STWC allows and identifies event-driven changes in covariance without identifying spurious noise correlations.STWC can be used to identify event-related neural interactions whose high computational load is well suited to GPU capabilities.

Neural correlates of movement planning onset and direction may be present in human electrocorticography in the signal dynamics of both motor and non-motor cortical regions. We use a three-stage model of jPCA reduced-rank hidden Markov model (jPCA-RR-HMM), regularized shrunken-centroid discriminant analysis (RDA), and LASSO regression to extract direction-sensitive planning information and movement onset in an upper-limb 3D isometric force task in a human subject. This mode achieves a relatively high true positive force-onset prediction rate of 60% within 250ms, and an above-chance 36% accuracy (17% chance) in predicting one of six planned 3D directions of isometric force using pre-movement signals. We also find direction-distinguishing information up to 400ms before force onset in the pre-movement signals, captured by electrodes placed over the limb-ipsilateral dorsal premotor regions. This approach can contribute to more accurate decoding of higher-level movement goals, at earlier timescales, and inform sensor placement. Our results also contribute to further understanding of the spatiotemporal features of human motor planning., 4 pages, 3 figures, accepted to EMBC 2016 (38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society)

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Most ongoing efforts have focused on training decoders on specific, stereotyped tasks in laboratory settings. Implementing brain-computer interfaces (BCIs) in natural settings requires adaptive strategies and scalable algorithms that require minimal supervision. Here we propose an unsupervised approach to decoding neural states from human brain recordings acquired in a naturalistic context. We demonstrate our approach on continuous long-term electrocorticographic (ECoG) data recorded over many days from the brain surface of subjects in a hospital room, with simultaneous audio and video recordings. We first discovered clusters in high-dimensional ECoG recordings and then annotated coherent clusters using speech and movement labels extracted automatically from audio and video recordings. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Our results show that our unsupervised approach can discover distinct behaviors from ECoG data, including moving, speaking and resting. We verify the accuracy of our approach by comparing to manual annotations. Projecting the discovered cluster centers back onto the brain, this technique opens the door to automated functional brain mapping in natural settings.

Objective A primary control signal in brain-computer interfaces (BCIs) have been cortical signals related to movement. However, in cases where natural motor function remains, BCI control signals may interfere with other possibly simultaneous activity for useful ongoing movement. We sought to determine if the brain could learn to control both a BCI and concurrent overt movement execution in such cases. Approach We designed experiments where BCI and overt movements must be used concurrently and in coordination to achieve a 2D centre out control. Power in the 70-90 Hz band of human electrocorticography (ECoG) signals, was used to generate BCI control commands for vertical movement of the cursor. These signals were deliberately recorded from the same human cortical site that produced the strongest movement related activity associated with the concurrent overt finger movements required for the horizontal movement of the cursor. Main results We demonstrate that three subjects were able to perform the concurrent BCI task, controlling BCI and natural movements simultaneously and to a large extent independently. We conclude that the brain is capable of dissociating the original control signal dependency on movement, producing specific BCI control signals in the presence of motor related responses from the ongoing overt behaviour with which the BCI signal was initially correlated. Significance We demonstrate a novel human brain-computer interface (BCI) which can be used to control movement concurrently and in coordination with movements of the natural limbs. This demonstrates the dissociation of cortical activity from the behaviour with which it was originally associated despite the ongoing behaviour and shows the feasibility of achieving simultaneous BCI control of devices with natural movements.

High gamma (HG) power changes during motor activity, especially at frequencies above 70 Hz, play an important role in functional cortical mapping and as control signals for BCI (brain computer interface) applications. Most studies of HG activity have used ECoG (electrocorticography) which provides high-quality spatially localized signals, but is an invasive method. Recent studies have shown that non-invasive modalities such as EEG and MEG can also detect task related HG power changes. We show here that a 27 channel EEG (electroencephalography) montage provides high-quality spatially localized signals non-invasively for HG frequencies ranging from 83 to 101 Hz. We used a generic head model, a weighted minimum norm least squares (MNLS) inverse method, and a self-paced finger movement paradigm. The use of an inverse method enables us to map the EEG onto a generic cortex model. We find the HG activity during the task to be well localized in the contralateral motor area. We find HG power increases prior to finger movement, with average latencies of 462 ms and 82 ms before EMG (electromyogram) onset. We also find significant phase-locking between contra- and ipsilateral motor areas over a similar HG frequency range; here the synchronization onset precedes the EMG by 400 ms. We also compare our results to ECoG data from a similar paradigm and find EEG mapping and ECoG in good agreement. Our findings demonstrate that mapped EEG provides information on two important parameters for functional mapping and BCI which are usually only found in HG of ECoG signals: spatially localized power increases and bihemispheric phase-locking.

Human cognition is thought to be mediated by large-scale interactions between distant sites in the neocortex. Synchronization between different cortical areas has been suggested as one possible mechanism for corticocortical interaction. Here, we report robust, directional cross-frequency synchronization between distant sensorimotor sites in human neocortex during a movement task. In four subjects, electrocorticographic recordings from the cortical surface revealed a low-frequency rhythm (10–13 Hz) that combined with a higher frequency (77–82 Hz) in a ventral region of the premotor cortex to produce a third rhythm at the sum of these two frequencies in a distant motor site. Such cross-frequency coupling implies a nonlinear interaction between these cortical sites. These findings demonstrate that task-specific, phase–phase coupling can support communication between distant areas of the human neocortex.

We describe a brain-computer interface for controlling a humanoid robot directly using brain signals obtained non-invasively from the scalp through electroencephalography (EEG). EEG has previously been used for tasks such as controlling a cursor and spelling a word, but it has been regarded as an unlikely candidate for more complex forms of control owing to its low signal-to-noise ratio. Here we show that by leveraging advances in robotics, an interface based on EEG can be used to command a partially autonomous humanoid robot to perform complex tasks such as walking to specific locations and picking up desired objects. Visual feedback from the robot's cameras allows the user to select arbitrary objects in the environment for pick-up and transport to chosen locations. Results from a study involving nine users indicate that a command for the robot can be selected from four possible choices in 5 s with 95% accuracy. Our results demonstrate that an EEG-based brain-computer interface can be used for sophisticated robotic interaction with the environment, involving not only navigation as in previous applications but also manipulation and transport of objects.

Electrocorticographic spectral changes during movement show a behavioral inflection in the classic gamma band (30-70 Hz). We quantify this inflection and demonstrate that it limits classification accuracy. We call for the designation of a functionally defined band above it, which we denote the chi-band.

We demonstrate the feasibility of real-time cortical mapping from arrays of subdural electrodes using the electrocorticographic signal power in the higher spectral frequencies (76-200 Hz, or "chi-index"). Hand area was mapped offline in eight individuals using brief baseline and hand-movement measurements. In one patient, hand sensorimotor cortex was identified online during a handshake. We propose that this high-frequency component of the electrocorticogram provides a generic, reliable, clinically useful correlate of local cortical function.

Medically refractory epilepsy accounts for more than 30% of the epilepsy population. Scalp EEG electrodes have limited ability to localize seizure onset from deep structures and implantation of subdural electrodes with long term monitoring provides additional information. Apart from clinical application, this patient population provides a unique opportunity for acquiring electrocorticography data in research paradigms. We present a method for rapid localization of electrodes using lateral and anterior-posterior X-rays. Skull landmarks and proportions are used for co-registration with the standardized Talairach coordinate system. This MATLAB-based "Location on Cortex" (LOC) package facilitates rapid visualization of clinical and experimental data in a user-friendly manner.

We present, to our knowledge, the first demonstration that a non-invasive brain-to-brain interface (BBI) can be used to allow one human to guess what is on the mind of another human through an interactive question-and-answering paradigm similar to the “20 Questions” game. As in previous non-invasive BBI studies in humans, our interface uses electroencephalography (EEG) to detect specific patterns of brain activity from one participant (the “respondent”), and transcranial magnetic stimulation (TMS) to deliver functionally-relevant information to the brain of a second participant (the “inquirer”). Our results extend previous BBI research by (1) using stimulation of the visual cortex to convey visual stimuli that are privately experienced and consciously perceived by the inquirer; (2) exploiting real-time rather than off-line communication of information from one brain to another; and (3) employing an interactive task, in which the inquirer and respondent must exchange information bi-directionally to collaboratively solve the task. The results demonstrate that using the BBI, ten participants (five inquirer-respondent pairs) can successfully identify a “mystery item” using a true/false question-answering protocol similar to the “20 Questions” game, with high levels of accuracy that are significantly greater than a control condition in which participants were connected through a sham BBI.

Electrocorticography (ECoG) has been demonstrated to be an effective modality as a platform for brain-computer interfaces (BCIs). Through our experience with ten subjects, we further demonstrate evidence to support the power and flexibility of this signal for BCI usage. In a subset of four patients, closed-loop BCI experiments were attempted with the patient receiving online feedback that consisted of one-dimensional cursor movement controlled by ECoG features that had shown correlation with various real and imagined motor and speech tasks. All four achieved control, with final target accuracies between 73%-100%. We assess the methods for achieving control and the manner in which enhancing online control can be accomplished by rescreening during online tasks. Additionally, we assess the relevant issues of the current experimental paradigm in light of their clinical constraints.

Human visual processing is of such complexity that, despite decades of focused research, many basic questions remain unanswered. Although we know that the inferotemporal cortex is a key region in object recognition, we don’t fully understand its physiologic role in brain function, nor do we have the full set of tools to explore this question. Here we show that electrical potentials from the surface of the human brain contain enough information to decode a subject’s perceptual state accurately, and with fine temporal precision. Electrocorticographic (ECoG) arrays were placed over the inferotemporal cortical areas of seven subjects. Pictures of faces and houses were quickly presented while each subject performed a simple visual task. Results showed that two well-known types of brain signals—event-averaged broadband power and event-averaged raw potential—can independently or together be used to classify the presented image. When applied to continuously recorded brain activity, our decoding technique could accurately predict whether each stimulus was a face, house, or neither, with ~20 ms timing error. These results provide a roadmap for improved brain-computer interfacing tools to help neurosurgeons, research scientists, engineers, and, ultimately, patients.

High gamma oscillations (70–150 Hz; HG) are rapidly evolving, spatially localized neurophysiological signals that are believed to be the best representative signature of engaged neural populations. The HG band has been best characterized from invasive electrophysiological approaches such as electrocorticography because of the increased signal-to-noise ratio that results when by-passing the scalp and skull. Despite the recent observation that HG activity can be detected non-invasively by electroencephalography (EEG), it is unclear to what extent EEG can accurately resolve the spatial distribution of HG signals during active task engagement. We have overcome some of the limitations inherent to acquiring HG signals across the scalp by utilizing individual head anatomy in combination with an inverse modeling method. We applied a linearly constrained minimum variance (LCMV) beamformer method on EEG data during a motor imagery paradigm to extract a time-frequency spectrogram at every voxel location on the cortex. To confirm spatially distributed patterns of HG responses, we contrasted overlapping maps of the EEG HG signal with blood oxygen level dependence (BOLD) functional magnetic resonance imaging (fMRI) data acquired from the same set of neurologically normal subjects during a separate session. We show that scalp-based HG band activity detected by EEG during motor imagery spatially co-localizes with BOLD fMRI data. Taken together, these results suggest that EEG can accurately resolve spatially specific estimates of local cortical high frequency signals, potentially opening an avenue for non-invasive measurement of HG potentials from diverse sets of neurologically impaired populations for diagnostic and therapeutic purposes.

We illustrate a general principal of electrical potential measurements from the surface of the cerebral cortex, by revisiting and reanalyzing experimental work from the visual, language and motor systems. A naive decomposition technique of electrocorticographic power spectral measurements reveals that broadband spectral changes reliably track task engagement. These broadband changes are shown to be a generic correlate of local cortical function across a variety of brain areas and behavioral tasks. Furthermore, they fit a power-law form that is consistent with simple models of the dendritic integration of asynchronous local population firing. Because broadband spectral changes covary with diverse perceptual and behavioral states on the timescale of 20-50 ms, they provide a powerful and widely applicable experimental tool.

Electrocorticographic recording is now being used in a wide variety of experimental settings. We present a simple method which can be used to estimate electrode position with respect to brain gyral anatomy using a pre-implantation MRI and post-implantation coronal and sagittal x-rays. It is semi-automated, with the user manually rotating and scaling an x-ray to fit brain outline, identifying threshold values for brain surface rendering, and clicking on electrodes on an x-ray. Electrode positions can be rapidly identified and rendered in about 20 minutes from start to finish. This approach is useful when the MRI quality is poor, there is no quality CT, but one would like to understand the relationship between experimental result and brain anatomy.

Imagery of motor movement plays an important role in learning of complex motor skills, from learning to serve in tennis to perfecting a pirouette in ballet. What and where are the neural substrates that underlie motor imagery-based learning? We measured electrocorticographic cortical surface potentials in eight human subjects during overt action and kinesthetic imagery of the same movement, focusing on power in “high frequency” (76–100 Hz) and “low frequency” (8–32 Hz) ranges. We quantitatively establish that the spatial distribution of local neuronal population activity during motor imagery mimics the spatial distribution of activity during actual motor movement. By comparing responses to electrocortical stimulation with imagery-induced cortical surface activity, we demonstrate the role of primary motor areas in movement imagery. The magnitude of imagery-induced cortical activity change was ∼25% of that associated with actual movement. However, when subjects learned to use this imagery to control a computer cursor in a simple feedback task, the imagery-induced activity change was significantly augmented, even exceeding that of overt movement.

Brain rhythms are more than just passive phenomena in visual cortex. For the first time, we show that the physiology underlying brain rhythms actively suppresses and releases cortical areas on a second-to-second basis during visual processing. Furthermore, their influence is specific at the scale of individual gyri. We quantified the interaction between broadband spectral change and brain rhythms on a second-to-second basis in electrocorticographic (ECoG) measurement of brain surface potentials in five human subjects during a visual search task. Comparison of visual search epochs with a blank screen baseline revealed changes in the raw potential, the amplitude of rhythmic activity, and in the decoupled broadband spectral amplitude. We present new methods to characterize the intensity and preferred phase of coupling between broadband power and band-limited rhythms, and to estimate the magnitude of rhythm-to-broadband modulation on a trial-by-trial basis. These tools revealed numerous coupling motifs between the phase of low frequency (δ, θ, α, β, and γ band) rhythms and the amplitude of broadband spectral change. In the θ and β ranges, the coupling of phase to broadband change is dynamic during visual processing, decreasing in some occipital areas and increasing in others, in a gyrally specific pattern. Finally, we demonstrate that the rhythms interact with one another across frequency ranges, and across cortical sites.

Publisher Summary The chapter provides an over view of some of the major feature extraction, pattern recognition, and machine learning techniques that have been successfully applied to electroencephalography (EEG)- and ECoG-based Brain–computer interfaces (BCIs). BCIs augment the human ability to communicate and interact with the external world by directly linking the brain to computers and robotic devices. BCIs bypass the normal neuromuscular output pathways for translating brain signals into action. Instead, physiological brain signals are processed in real time by digital signal processing methods to allow a novel form of communication and interaction with the environment. A major goal of BCIs has been to improve the quality of life of physically impaired individuals, including those paralyzed because of degenerative neurological diseases such as amyotrophic lateral sclerosis, spinal cord injury, or stroke. BCIs allow a subject to directly control objects such as a cursor or a robot using brain signals. BCIs depend crucially on the ability to reliably identify behaviorally induced changes (or “cognitive states”) in the brain signals being recorded. Statistical pattern recognition and machine learning algorithms play an important role in identifying these changes in brain signals and mapping them to appropriate control signals.

Most brain-computer interface classification experiments from electrical potential recordings have been focused on the identification of classes of stimuli or behavior where the timing of experimental parameters is known or pre-designated. Real world experience, however, is spontaneous, and to this end we describe an experiment predicting the occurrence, timing, and types of visual stimuli perceived by a human subject from electrocorticographic recordings. All 300 of 300 presented stimuli were correctly detected, with a temporal precision of order 20 ms. The type of stimulus (face/house) was correctly identified in 95% of these cases. There were approximately 20 false alarm events, corresponding to a late 2nd neuronal response to a previously identified event.

Electrocorticography (ECoG) offers a powerful and versatile platform for developing brain-computer interfaces; it avoids the risks of brain-invasive methods such as intracortical implants while providing significantly higher signal-to-noise ratio than noninvasive techniques such as electroencephalography. The authors demonstrate that both contra- and ipsilateral finger movements can be discriminated from ECoG signals recorded from a single brain hemisphere. The ECoG activation patterns over sensorimotor areas for contra- and ipsilateral movements were found to overlap to a large degree in the recorded hemisphere. Ipsilateral movements, however, produced less pronounced activity compared with contralateral movements. The authors also found that single-trial classification of movements could be improved by selecting patient-specific frequency components in high-frequency bands (> 50 Hz). Their discovery that ipsilateral hand movements can be discriminated from ECoG signals from a single hemisphere has important implications for neurorehabilitation, suggesting in particular the possibility of regaining ipsilateral movement control using signals from an intact hemisphere after damage to the other hemisphere.

We present results of cortical activity during phoneme pronunciation, recorded using miniaturized electrocorticography grids with high spatial resolution. A patient implanted with the miniature grid was instructed to audibly pronounce one of four phonemes. For each phoneme, we observed distinct spatial correlation patterns at the 3mm electrode spacing. We applied a support vector machine classification scheme and, for the first time, were able to distinguish discrete phonemes with high accuracy. In addition, we found that sub-regions of our miniature array were specific for distinct pairs of phonemes, showing that cortical phoneme processing occurs at a higher resolution than previously though.

This paper presents a two-part study investigating the use of forearm surface electromyographic (EMG) signals for real-time control of a robotic arm. In the first part of the study, we explore and extend current classification-based paradigms for myoelectric control to obtain high accuracy (92-98%) on an eight-class offline classification problem, with up to 16 classifications/s. This offline study suggested that a high degree of control could be achieved with very little training time (under 10 min). The second part of this paper describes the design of an online control system for a robotic arm with 4 degrees of freedom. We evaluated the performance of the EMG-based real-time control system by comparing it with a keyboard-control baseline in a three-subject study for a variety of complex tasks.

This paper studies classifiability of electrocorticographic signals (ECoG) for use in a human brain-computer interface (BCI). The results show that certain spectral features can be reliably used across several subjects to accurately classify different types of movements. Sparse and nonsparse versions of the support vector machine and regularized linear discriminant analysis linear classifiers are assessed and contrasted for the classification problem. In conjunction with a careful choice of features, the classification process automatically and consistently identifies neurophysiological areas known to be involved in the movements. An average two-class classification accuracy of 95% for real movement and around 80% for imagined movement is shown. The high accuracy and generalizability of these results, obtained with as few as 30 data samples per class, support the use of classification methods for ECoG-based BCIs.

We study the problem of distinguishing between individual finger movements of one hand using electrocorticographic (ECOG) signals. In previous work, we have shown that ECOG signals have high predictive accuracy and spatial resolution for classifying hand versus tongue movements. In this paper, we significantly extend this paradigm by studying the first 5-class classification problem for ECOG, and show that an average 5-class accuracy of 23% across 6 subjects is possible using as little as 10min of training data. In addition to opening up possibilities for higher-bandwidth brain-computer interfaces, the use of finger movements for control may yield a more intuitive mapping from ECOG signals to control of a prosthetic. Although this study uses real movements, our results provide the foundation for understanding ECOG signal changes during finger movement.

Non-stationarities are ubiquitous in EEG signals. They are especially apparent in the use of EEG-based brain-computer interfaces (BCIs): (a) in the differences between the initial calibration measurement and the online operation of a BCI, or (b) caused by changes in the subject's brain processes during an experiment (e.g. due to fatigue, change of task involvement, etc). In this paper, we quantify for the first time such systematic evidence of statistical differences in data recorded during offline and online sessions. Furthermore, we propose novel techniques of investigating and visualizing data distributions, which are particularly useful for the analysis of (non-) stationarities. Our study shows that the brain signals used for control can change substantially from the offline calibration sessions to online control, and also within a single session. In addition to this general characterization of the signals, we propose several adaptive classification schemes and study their performance on data recorded during online experiments. An encouraging result of our study is that surprisingly simple adaptive methods in combination with an offline feature selection scheme can significantly increase BCI performance.

We describe the first direct brain-to-brain interface in humans and present results from experiments involving six different subjects. Our non-invasive interface, demonstrated originally in August 2013, combines electroencephalography (EEG) for recording brain signals with transcranial magnetic stimulation (TMS) for delivering information to the brain. We illustrate our method using a visuomotor task in which two humans must cooperate through direct brain-to-brain communication to achieve a desired goal in a computer game. The brain-to-brain interface detects motor imagery in EEG signals recorded from one subject (the "sender") and transmits this information over the internet to the motor cortex region of a second subject (the "receiver"). This allows the sender to cause a desired motor response in the receiver (a press on a touchpad) via TMS. We quantify the performance of the brain-to-brain interface in terms of the amount of information transmitted as well as the accuracies attained in (1) decoding the sender's signals, (2) generating a motor response from the receiver upon stimulation, and (3) achieving the overall goal in the cooperative visuomotor task. Our results provide evidence for a rudimentary form of direct information transmission from one human brain to another using non-invasive means.

Recently, CNN reported on the future of brain-computer interfaces (BCIs). BCIs are devices that process a user's brain signals to allow direct communication and interaction with the environment. BCIs bypass the normal neuromuscular output pathways and rely on digital signal processing and machine learning to translate brain signals to action (Figure 1). Historically, BCIs were developed with biomedical applications in mind, such as restoring communication in completely paralyzed individuals and replacing lost motor function. More recent applications have targeted nondisabled individuals by exploring the use of BCIs as a novel input device for entertainment and gaming. The task of the BCI is to identify and predict behaviorally induced changes or "cognitive states" in a user's brain signals. Brain signals are recorded either noninvasively from electrodes placed on the scalp [electroencephalogram (EEG)] or invasively from electrodes placed on the surface of or inside the brain. BCIs based on these recording techniques have allowed healthy and disabled individuals to control a variety of devices. In this article, we will describe different challenges and proposed solutions for noninvasive brain-computer interfacing.

Brain-computer interfaces (BCIs) have traditionally been developed for paralyzed and locked-in individuals with no motor control. However, there is a much larger population of patients with some residual motor function as well as the general population of able-bodied individuals, both of whom could benefit significantly from BCIs. An important question that has yet to be systematically studied is: can subjects use BCIs simultaneously with overt motor activity? We present results from a preliminary study aimed at exploring this question. Three subjects used hand motor imagery in an electroencephalographic (EEG) BCI while simultaneously using a joystick to control a cursor. Particular attention was paid to preventing potential muscle artifacts from influencing imagery-based control. All three subjects were able to use the hybrid "imagery+joystick" mode of control over two days, demonstrating the ability to learn and significantly improve performance. These results suggest that subjects can potentially augment their normal human sensorimotor capability by exercising direct brain control over devices concurrently with overt motor control.