Search

Background Recent technological advances in brain recording and machine learning algorithms are enabling the study of neural activity underlying spontaneous human behaviors, beyond the confines of cued, repeated trials. However, analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term. New method Here we describe an automated, behavior-first approach for analyzing simultaneously recorded long-term, naturalistic electrocorticography (ECoG) and behavior video data. We identify and characterize spontaneous human upper-limb movements by combining computer vision, discrete latent-variable modeling, and string pattern-matching on the video. Results Our pipeline discovers and annotates over 40,000 instances of naturalistic arm movements in long term (7–9 day) behavioral videos, across 12 subjects. Analysis of the simultaneously recorded brain data reveals neural signatures of movement that corroborate previous findings. Our pipeline produces large training datasets for brain–computer interfacing applications, and we show decoding results from a movement initiation detection task. Comparison with existing methods Spontaneous movements capture real-world neural and behavior variability that is missing from traditional cued tasks. Building beyond window-based movement detection metrics, our unsupervised discretization scheme produces a queryable pose representation, allowing localization of movements with finer temporal resolution. Conclusions Our work addresses the unique analytic challenges of studying naturalistic human behaviors and contributes methods that may generalize to other neural recording modalities beyond ECoG. We publish our curated dataset and believe that it will be a valuable resource for future studies of naturalistic movements.

ObjectiveAdvances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can robustly train on pooled, multi-participant data and generalize to new participants.ApproachWe introduce a new decoder, HTNet, which uses a convolutional neural network with two innovations: (1) a Hilbert transform that computes spectral power at data-driven frequencies and (2) a layer that projects electrode-level data onto predefined brain regions. The projection layer critically enables applications with intracranial electrocorticography (ECoG), where electrode locations are not standardized and vary widely across participants. We trained HTNet to decode arm movements using pooled ECoG data from 11 of 12 participants and tested performance on unseen ECoG or electroencephalography (EEG) participants; these pretrained models were also subsequently fine-tuned to each test participant.Main resultsHTNet outperformed state-of-the-art decoders when tested on unseen participants, even when a different recording modality was used. By fine-tuning these generalized HTNet decoders, we achieved performance approaching the best tailored decoders with as few as 50 ECoG or 20 EEG events. We were also able to interpret HTNet’s trained weights and demonstrate its ability to extract physiologically-relevant features.SignificanceBy generalizing to new participants and recording modalities, robustly handling variations in electrode placement, and allowing participant-specific fine-tuning with minimal data, HTNet is applicable across a broader range of neural decoding applications compared to current state-of-the-art decoders.

Studying the neural correlates of sleep can lead to revelations in our understanding of sleep and its interplay with different neurological disorders. Sleep research relies on manual annotation of sleep stages based on rules developed for healthy adults. Automating sleep stage annotation can expedite sleep research and enable us to better understand atypical sleep patterns. Our goal was to create a fully unsupervised approach to label sleep and wake states in human electro-corticography (ECoG) data from epilepsy patients. Here, we demonstrate that with continuous data from a single ECoG electrode, hidden semi-Markov models (HSMM) perform best in classifying sleep/wake states without excessive transitions, with a mean accuracy (n=4) of 85.2% compared to using K-means clustering (72.2%) and hidden Markov models (81.5%). Our results confirm that HSMMs produce meaningful labels for ECoG data and establish the groundwork to apply this model to cluster sleep stages and potentially other behavioral states.

Increasingly, neuroimaging researchers are exploring the use of real-time functional magnetic resonance imaging (rt-fMRI) as a way to access a participant’s ongoing brain function throughout a scan. This approach presents novel and exciting experimental applications ranging from monitoring data quality in real time, to delivering neurofeedback from a region of interest, to dynamically controlling experimental flow, or interfacing with remote devices. Yet, for those interested in adopting this method, the existing software options are few and limited in application. This presents a barrier for new users, as well as hinders existing users from refining techniques and methods. Here we introduce a free, open-source rt-fMRI package, the Pyneal toolkit, designed to address this limitation. The Pyneal toolkit is python-based software that offers a flexible and user friendly framework for rt-fMRI, is compatible with all three major scanner manufacturers (GE, Siemens, Phillips), and, critically, allows fully customized analysis pipelines. In this article, we provide a detailed overview of the architecture, describe how to set up and run the Pyneal toolkit during an experimental session, offer tutorials with scan data that demonstrate how data flows through the Pyneal toolkit with example analyses, and highlight the advantages that the Pyneal toolkit offers to the neuroimaging community.

A Bayesian model suggests that when interacting with a group, humans simulate the “mind of the group” to choose an action., To make decisions in a social context, humans have to predict the behavior of others, an ability that is thought to rely on having a model of other minds known as “theory of mind.” Such a model becomes especially complex when the number of people one simultaneously interacts with is large and actions are anonymous. Here, we present results from a group decision-making task known as the volunteer’s dilemma and demonstrate that a Bayesian model based on partially observable Markov decision processes outperforms existing models in quantitatively predicting human behavior and outcomes of group interactions. Our results suggest that in decision-making tasks involving large groups with anonymous members, humans use Bayesian inference to model the “mind of the group,” making predictions of others’ decisions while also simulating the effects of their own actions on the group’s dynamics in the future.

Electrocorticographic brain computer interfaces (ECoG-BCIs) offer tremendous opportunities for restoring function in individuals suffering from neurological damage and for advancing basic neuroscience knowledge. ECoG electrodes are already commonly used clinically for monitoring epilepsy and have greater spatial specificity in recording neuronal activity than techniques such as electroencephalography (EEG). Much work to date in the field has focused on using ECoG signals recorded from cortex as control outputs for driving end effectors. An equally important but less explored application of an ECoG-BCI is directing input into cortex using ECoG electrodes for direct electrical stimulation (DES). Combining DES with ECoG recording enables a truly bidirectional BCI, where information is both read from and written to the brain. We discuss the advantages and opportunities, as well as the barriers and challenges presented by using DES in an ECoG-BCI. In this article, we review ECoG electrodes, the physics and physiology of DES, and the use of electrical stimulation of the brain for the clinical treatment of disorders such as epilepsy and Parkinson’s disease. We briefly discuss some of the translational, regulatory, financial, and ethical concerns regarding ECoG-BCIs. Next, we describe the use of ECoG-based DES for providing sensory feedback and for probing and modifying cortical connectivity. We explore future directions, which may draw on invasive animal studies with penetrating and surface electrodes as well as non-invasive stimulation methods such as transcranial magnetic stimulation (TMS). We conclude by describing enabling technologies, such as smaller ECoG electrodes for more precise targeting of cortical areas, signal processing strategies for simultaneous stimulation and recording, and computational modeling and algorithms for tailoring stimulation to each individual brain.

We present BrainNet which, to our knowledge, is the first multi-person non-invasive direct brain-to-brain interface for collaborative problem solving. The interface combines electroencephalography (EEG) to record brain signals and transcranial magnetic stimulation (TMS) to deliver information noninvasively to the brain. The interface allows three human subjects to collaborate and solve a task using direct brain-to-brain communication. Two of the three subjects are designated as “Senders” whose brain signals are decoded using real-time EEG data analysis. The decoding process extracts each Sender’s decision about whether to rotate a block in a Tetris-like game before it is dropped to fill a line. The Senders’ decisions are transmitted via the Internet to the brain of a third subject, the “Receiver,” who cannot see the game screen. The Senders’ decisions are delivered to the Receiver’s brain via magnetic stimulation of the occipital cortex. The Receiver integrates the information received from the two Senders and uses an EEG interface to make a decision about either turning the block or keeping it in the same orientation. A second round of the game provides an additional chance for the Senders to evaluate the Receiver’s decision and send feedback to the Receiver’s brain, and for the Receiver to rectify a possible incorrect decision made in the first round. We evaluated the performance of BrainNet in terms of (1) Group-level performance during the game, (2) True/False positive rates of subjects’ decisions, and (3) Mutual information between subjects. Five groups, each with three human subjects, successfully used BrainNet to perform the collaborative task, with an average accuracy of 81.25%. Furthermore, by varying the information reliability of the Senders by artificially injecting noise into one Sender’s signal, we investigated how the Receiver learns to integrate noisy signals in order to make a correct decision. We found that like conventional social networks, BrainNet allows Receivers to learn to trust the Sender who is more reliable, in this case, based solely on the information transmitted directly to their brains. Our results point the way to future brain-to-brain interfaces that enable cooperative problem solving by humans using a “social network” of connected brains.

The field of brain-computer interfaces is poised to advance from the traditional goal of controlling prosthetic devices using brain signals to combining neural decoding and encoding within a single neuroprosthetic device. Such a device acts as a "co-processor" for the brain, with applications ranging from inducing Hebbian plasticity for rehabilitation after brain injury to reanimating paralyzed limbs and enhancing memory. We review recent progress in simultaneous decoding and encoding for closed-loop control and plasticity induction. To address the challenge of multi-channel decoding and encoding, we introduce a unifying framework for developing brain co-processors based on artificial neural networks and deep learning. These "neural co-processors" can be used to jointly optimize cost functions with the nervous system to achieve desired behaviors ranging from targeted neuro-rehabilitation to augmentation of brain function., Comment: Invited submission to the journal Current Opinion in Neurobiology

Cortical stimulation through electrocorticographic (ECoG) electrodes is a potential method for providing sensory feedback in future prosthetic and rehabilitative applications. Here, we evaluate human subjects’ ability to continuously modulate their motor behavior based on feedback from direct surface stimulation of the somatosensory cortex. Subjects wore a dataglove that measured their hand aperture position and received one of three stimuli over the hand sensory cortex based on their current hand position as compared to a target aperture position. Using cortical stimulation feedback, subjects adjusted their hand aperture to move towards the target aperture region. One subject was able to achieve accuracies and R2 values well above chance (best performance: R2 = 0.93; accuracy = 0.76/1). Performance dropped during the catch trial (same stimulus independent of the position) to below chance levels, suggesting that the subject had been using the varied sensory feedback to modulate their motor behavior. To our knowledge, this study represents one of the first demonstrations of using direct cortical surface stimulation of the human sensory cortex to perform a motor task, and is a first step towards developing closed-loop human sensorimotor brain-computer interfaces.

International audience; In collective decision making, members of a group need to coordinate their actions in order to achieve a desirable outcome. When there is no direct communication between group members, one should decide based on inferring others' intentions from their actions. The inference of others' intentions is called "theory of mind" and can involve different levels of reasoning, from a single inference on a hidden variable to considering others partially or fully optimal and reasoning about their actions conditioned on one's own actions (levels of "theory of mind"). In this paper, we present a new Bayesian theory of collective decision making based on a simple yet most commonly observed behavior: conformity. We show that such a Bayesian framework allows one to achieve any level of theory of mind in collective decision making. The viability of our framework is demonstrated on two different experiments, a consensus task with 120 subjects and a volunteer's dilemma task with 29 subjects, each with multiple conditions.

Objective. Electrical stimulation of the human brain is commonly used for eliciting and inhibiting neural activity for clinical diagnostics, modifying abnormal neural circuit function for therapeutics, and interrogating cortical connectivity. However, recording electrical signals with concurrent stimulation results in dominant electrical artifacts that mask the neural signals of interest. Here we develop a method to reproducibly and robustly recover neural activity during concurrent stimulation. We concentrate on signal recovery across an array of electrodes without channel-wise fine-tuning of the algorithm. Our goal includes signal recovery with trains of stimulation pulses, since repeated, high-frequency pulses are often required to induce desired effects in both therapeutic and research domains. We have made all of our code and data publicly available. Approach. We developed an algorithm that automatically detects templates of artifacts across many channels of recording, creating a dictionary of learned templates using unsupervised clustering. The artifact template that best matches each individual artifact pulse is subtracted to recover the underlying activity. To assess the success of our method, we focus on whether it extracts physiologically interpretable signals from real recordings. Main results. We demonstrate our signal recovery approach on invasive electrophysiologic recordings from human subjects during stimulation. We show the recovery of meaningful neural signatures in both electrocorticographic (ECoG) arrays and deep brain stimulation (DBS) recordings. In addition, we compared cortical responses induced by the stimulation of primary somatosensory (S1) by natural peripheral touch, as well as motor cortex activity with and without concurrent S1 stimulation. Significance. Our work will enable future advances in neural engineering with simultaneous stimulation and recording.

Objective: The objective of this paper is to estimate the parameters of a multivariate autoregressive process from noisy multichannel data. Methods: Using a multivariate generalization of the Cadzow method, we propose a new method for estimating autoregressive parameters from noisy data: the nonlinear Cadzow method. Results: We show that our method outperforms existing multivariate methods such as higher order Yule–Walker method and Kalman EM method on simulated data. We apply our method to estimation of Granger causality from noisy data and again obtain superior results compared to previous methods. Finally, when applied to experimental local field potential data from monkey somatosensory and motor cortical areas, our method produces results consistent with cortical physiology. Conclusion: The proposed nonlinear Cadzow method outperforms existing methods in obtaining denoised estimates of multivariate autoregressive parameters. Significance: Since multichannel recordings have become commonplace in biomedical applications ranging from discovering functional connectivity in the brain to speech data analysis and these recordings are inevitably contaminated by measurement noise, we believe our method has the potential for significant impact.

Determining hand grasping forces and kinetic direction and magnitude are important for the development of dexterous neural prosthetics. While many earlier decoding methods have successfully predicted upper-limb kinematic output from cortical signals in the sensorimotor parietal and premotor regions, the full extent of the regions that characterize kinetic behavior is unknown. In this study, we found that neural dynamics based on electrocorticography (ECoG) recorded from the human brain surface can successfully encode structured and unstructured grasping and arm kinetic output. We found a time-averaged linear relationship between gamma band spectral ECoG power and sustained grasping force output with visual feedback. In the kinetic grasping task, we obtained classification accuracy of 47% (25% = chance) using quadratic discriminant analysis. Additionally, we also found a similar linear relationship between spectral power and cued isometric force generation, without concurrent visual feedback, in arm force application; this feature could also classify arm force output categories with an accuracy of 41% (33% = chance). In addition, we applied quadratic discriminant analysis with top 12 principal components to attain approximately 26% accuracy (chance is 16%) in determining arm kinetic force direction. We found that the gamma band spectral power from both experiments in posterior parietal cortex, as well as projections of high gamma variability along the top principal components, can be successfully used to predict sustained kinetic outputs and to explain both structured and unstructured force output variability. Our findings contribute to a deeper understanding of neural dynamics for fine kinetic behavior.

Direct cortical stimulation (DCS) of primary somatosensory cortex (S1) could help restore sensation and provide task-relevant feedback in a neuroprosthesis. However, the psychophysics of S1 DCS is poorly studied, including any comparison to cutaneous haptic stimulation. We compare the response times to DCS of human hand somatosensory cortex through electrocorticographic grids with response times to haptic stimuli delivered to the hand in four subjects. We found that subjects respond significantly slower to S1 DCS than to natural, haptic stimuli for a range of DCS train durations. Median response times for haptic stimulation varied from 198 ms to 313 ms, while median responses to reliably perceived DCS ranged from 254 ms for one subject, all the way to 528 ms for another. We discern no significant impact of learning or habituation through the analysis of blocked trials, and find no significant impact of cortical stimulation train duration on response times. Our results provide a realistic set of expectations for latencies with somatosensory DCS feedback for future neuroprosthetic applications and motivate the study of neural mechanisms underlying human perception of somatosensation via DCS.

The human ventral temporal cortex has regions that are known to selectively process certain categories of visual inputs; they are specialized for the content (“faces,” “places,” “tools”) and not the form (“line,” “patch”) of the image being seen. In our study, human patients with implanted electrocorticography (ECoG) electrode arrays were shown sequences of simple face and house pictures. We quantified neuronal population activity, finding robust face-selective sites on the fusiform gyrus and house-selective sites on the lingual/parahippocampal gyri. The magnitude and timing of single trials were compared between novel (“house-face”) and repeated (“face-face”) stimulus-type responses. More than half of the category-selective sites showed significantly greater total activity for novel stimulus class. Approximately half of the face-selective sites (and none of the house-selective sites) showed significantly faster latency to peak (∼50 ms) for novel stimulus class. This establishes subregions within category-selective areas that are differentially tuned to novelty in sequential context, where novel stimuli are processed faster in some regions, and with increased activity in others.

An article published in Language (Sproat 2014a) questions our findings on the Indus script and Pictish symbols published in the journals Science (Rao et al. 2009a), PNAS (Rao et al. 2009b), IEEE Computer (Rao 2010), and the Proceedings of the Royal Society (Lee et al. 2010a,b). Sproat’s article does not accurately present our methods and findings, and its conclusions are based on what appears to be a misunderstanding of our proposed approach. For example, the article’s results on entropic measures seem to favor, rather than contradict, the inductive hypothesis that the Indus script may represent writing. The article selects results to draw a particular set of conclusions and convey a specific viewpoint. In light of these issues, we stand by our original findings.

Electrocorticography (ECoG) signals can provide high spatio-temporal resolution and high signal to noise ratio recordings of local neural activity from the surface of the brain. Previous studies have shown that broad-band, spatially focal, high-frequency increases in ECoG signals are highly correlated with movement and other cognitive tasks and can be volitionally modulated. However, significant additional information may be present in inter-electrode interactions, but adding additional higher order inter-electrode interactions can be impractical from a computational aspect, if not impossible.In this paper we present a new method of calculating high frequency interactions between electrodes called Short-Time Windowed Covariance (STWC) that builds on mathematical techniques currently used in neural signal analysis, along with an implementation that accelerates the algorithm by orders of magnitude by leveraging commodity, off-the-shelf graphics processing unit (GPU) hardware.Using the hardware-accelerated implementation of STWC, we identify many types of event-related inter-electrode interactions from human ECoG recordings on global and local scales that have not been identified by previous methods. Unique temporal patterns are observed for digit flexion in both low- (10mm spacing) and high-resolution (3mm spacing) electrode arrays.Covariance is a commonly used metric for identifying correlated signals, but the standard covariance calculations do not allow for temporally varying covariance. In contrast STWC allows and identifies event-driven changes in covariance without identifying spurious noise correlations.STWC can be used to identify event-related neural interactions whose high computational load is well suited to GPU capabilities.

The majority of subjects who attempt to learn control of a brain–computer interface (BCI) can do so with adequate training. Much like when one learns to type or ride a bicycle, BCI users report transitioning from a deliberate, cognitively focused mindset to near automatic control as training progresses. What are the neural correlates of this process of BCI skill acquisition? Seven subjects were implanted with electrocorticography (ECoG) electrodes and had multiple opportunities to practice a 1D BCI task. As subjects became proficient, strong initial task-related activation was followed by lessening of activation in prefrontal cortex, premotor cortex, and posterior parietal cortex, areas that have previously been implicated in the cognitive phase of motor sequence learning and abstract task learning. These results demonstrate that, although the use of a BCI only requires modulation of a local population of neurons, a distributed network of cortical areas is involved in the acquisition of BCI proficiency.

Identification of intended movement type and movement phase of hand grasp shaping are critical features for the control of volitional neuroprosthetics. We demonstrate that neural dynamics during visually-guided imagined grasp shaping can encode intended movement. We apply Procrustes analysis and LASSO regression to achieve 72% accuracy (chance = 25%) in distinguishing between visually-guided imagined grasp trajectories. Further, we can predict the stage of grasp shaping in the form of elapsed time from start of trial (R2=0.4). Our approach contributes to more accurate single-trial decoding of higher-level movement goals and the phase of grasping movements in individuals not trained with brain-computer interfaces. We also find that the overall time-varying trajectory structure of imagined movements tend to be consistent within individuals, and that transient trajectory deviations within trials return to the task-dependent trajectory mean. These overall findings may contribute to the further understanding of the cortical dynamics of human motor imagery., Comment: 4 pages, 6 figures, accepted to IEEE NER 2017 (8th International IEEE EMBS Conference on Neural Engineering)

We introduce Graph-Structured Sum-Product Networks (GraphSPNs), a probabilistic approach to structured prediction for problems where dependencies between latent variables are expressed in terms of arbitrary, dynamic graphs. While many approaches to structured prediction place strict constraints on the interactions between inferred variables, many real-world problems can be only characterized using complex graph structures of varying size, often contaminated with noise when obtained from real data. Here, we focus on one such problem in the domain of robotics. We demonstrate how GraphSPNs can be used to bolster inference about semantic, conceptual place descriptions using noisy topological relations discovered by a robot exploring large-scale office spaces. Through experiments, we show that GraphSPNs consistently outperform the traditional approach based on undirected graphical models, successfully disambiguating information in global semantic maps built from uncertain, noisy local evidence. We further exploit the probabilistic nature of the model to infer marginal distributions over semantic descriptions of as yet unexplored places and detect spatial environment configurations that are novel and incongruent with the known evidence., Comment: 9 pages, 8 figures. AAAI Conference on Artificial Intelligence (AAAI 2018)

We propose a new probabilistic framework that allows mobile robots to autonomously learn deep, generative models of their environments that span multiple levels of abstraction. Unlike traditional approaches that combine engineered models for low-level features, geometry, and semantics, our approach leverages recent advances in Sum-Product Networks (SPNs) and deep learning to learn a single, universal model of the robot's spatial environment. Our model is fully probabilistic and generative, and represents a joint distribution over spatial information ranging from low-level geometry to semantic interpretations. Once learned, it is capable of solving a wide range of tasks: from semantic classification of places, uncertainty estimation, and novelty detection, to generation of place appearances based on semantic information and prediction of missing data in partial observations. Experiments on laser-range data from a mobile robot show that the proposed universal model obtains performance superior to state-of-the-art models fine-tuned to one specific task, such as Generative Adversarial Networks (GANs) or SVMs.

Autonomous mobile robots will soon become ubiquitous in human-populated environments. Besides their typical applications in fetching, delivery, or escorting, such robots present the opportunity to assist human users in their daily tasks by gathering and reporting up-to-date knowledge about the environment. In this paper, we explore this use case and present an end-to-end framework that enables a mobile robot to answer natural language questions about the state of a large-scale, dynamic environment asked by the inhabitants of that environment. The system parses the question and estimates an initial viewpoint that is likely to contain information for answering the question based on prior environment knowledge. Then, it autonomously navigates towards the viewpoint while dynamically adapting to changes and new information. The output of the system is an image of the most relevant part of the environment that allows the user to obtain an answer to their question. We additionally demonstrate the benefits of a continuously operating information gathering robot by showing how the system can answer retrospective questions about the past state of the world using incidentally recorded sensory data. We evaluate our approach with a custom mobile robot deployed in a university building, with questions collected from occupants of the building. We demonstrate our system's ability to respond to these questions in different environmental conditions.

Motivated by the growing evidence for Bayesian computation in the brain, we show how a two-layer recurrent network of Poisson neurons can perform both approximate Bayesian inference and learning for any hidden Markov model. The lower-layer sensory neurons receive noisy measurements of hidden world states. The higher-layer neurons infer a posterior distribution over world states via Bayesian inference from inputs generated by sensory neurons. We demonstrate how such a neuronal network with synaptic plasticity can implement a form of Bayesian inference similar to Monte Carlo methods such as particle filtering. Each spike in a higher-layer neuron represents a sample of a particular hidden world state. The spiking activity across the neural population approximates the posterior distribution over hidden states. In this model, variability in spiking is regarded not as a nuisance but as an integral feature that provides the variability necessary for sampling during inference. We demonstrate how the network can learn the likelihood model, as well as the transition probabilities underlying the dynamics, using a Hebbian learning rule. We present results illustrating the ability of the network to perform inference and learning for arbitrary hidden Markov models.

Neural correlates of movement planning onset and direction may be present in human electrocorticography in the signal dynamics of both motor and non-motor cortical regions. We use a three-stage model of jPCA reduced-rank hidden Markov model (jPCA-RR-HMM), regularized shrunken-centroid discriminant analysis (RDA), and LASSO regression to extract direction-sensitive planning information and movement onset in an upper-limb 3D isometric force task in a human subject. This mode achieves a relatively high true positive force-onset prediction rate of 60% within 250ms, and an above-chance 36% accuracy (17% chance) in predicting one of six planned 3D directions of isometric force using pre-movement signals. We also find direction-distinguishing information up to 400ms before force onset in the pre-movement signals, captured by electrodes placed over the limb-ipsilateral dorsal premotor regions. This approach can contribute to more accurate decoding of higher-level movement goals, at earlier timescales, and inform sensor placement. Our results also contribute to further understanding of the spatiotemporal features of human motor planning., 4 pages, 3 figures, accepted to EMBC 2016 (38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society)

Objective A primary control signal in brain-computer interfaces (BCIs) have been cortical signals related to movement. However, in cases where natural motor function remains, BCI control signals may interfere with other possibly simultaneous activity for useful ongoing movement. We sought to determine if the brain could learn to control both a BCI and concurrent overt movement execution in such cases. Approach We designed experiments where BCI and overt movements must be used concurrently and in coordination to achieve a 2D centre out control. Power in the 70-90 Hz band of human electrocorticography (ECoG) signals, was used to generate BCI control commands for vertical movement of the cursor. These signals were deliberately recorded from the same human cortical site that produced the strongest movement related activity associated with the concurrent overt finger movements required for the horizontal movement of the cursor. Main results We demonstrate that three subjects were able to perform the concurrent BCI task, controlling BCI and natural movements simultaneously and to a large extent independently. We conclude that the brain is capable of dissociating the original control signal dependency on movement, producing specific BCI control signals in the presence of motor related responses from the ongoing overt behaviour with which the BCI signal was initially correlated. Significance We demonstrate a novel human brain-computer interface (BCI) which can be used to control movement concurrently and in coordination with movements of the natural limbs. This demonstrates the dissociation of cortical activity from the behaviour with which it was originally associated despite the ongoing behaviour and shows the feasibility of achieving simultaneous BCI control of devices with natural movements.

High gamma (HG) power changes during motor activity, especially at frequencies above 70 Hz, play an important role in functional cortical mapping and as control signals for BCI (brain computer interface) applications. Most studies of HG activity have used ECoG (electrocorticography) which provides high-quality spatially localized signals, but is an invasive method. Recent studies have shown that non-invasive modalities such as EEG and MEG can also detect task related HG power changes. We show here that a 27 channel EEG (electroencephalography) montage provides high-quality spatially localized signals non-invasively for HG frequencies ranging from 83 to 101 Hz. We used a generic head model, a weighted minimum norm least squares (MNLS) inverse method, and a self-paced finger movement paradigm. The use of an inverse method enables us to map the EEG onto a generic cortex model. We find the HG activity during the task to be well localized in the contralateral motor area. We find HG power increases prior to finger movement, with average latencies of 462 ms and 82 ms before EMG (electromyogram) onset. We also find significant phase-locking between contra- and ipsilateral motor areas over a similar HG frequency range; here the synchronization onset precedes the EMG by 400 ms. We also compare our results to ECoG data from a similar paradigm and find EEG mapping and ECoG in good agreement. Our findings demonstrate that mapped EEG provides information on two important parameters for functional mapping and BCI which are usually only found in HG of ECoG signals: spatially localized power increases and bihemispheric phase-locking.

In this paper, we provide the first demonstration that a humanoid robot can learn to walk directly by imitating a human gait obtained from motion capture (mocap) data without any prior information of its dynamics model. Programming a humanoid robot to perform an action (such as walking) that takes into account the robot's complex dynamics is a challenging problem. Traditional approaches typically require highly accurate prior knowledge of the robot's dynamics and environment in order to devise complex (and often brittle) control algorithms for generating a stable dynamic motion. Training using human mocap is an intuitive and flexible approach to programming a robot, but direct usage of mocap data usually results in dynamically unstable motion. Furthermore, optimization using high-dimensional mocap data in the humanoid full-body joint space is typically intractable. We propose a new approach to tractable imitation-based learning in humanoids without a robot's dynamic model. We represent kinematic informati...

Human cognition is thought to be mediated by large-scale interactions between distant sites in the neocortex. Synchronization between different cortical areas has been suggested as one possible mechanism for corticocortical interaction. Here, we report robust, directional cross-frequency synchronization between distant sensorimotor sites in human neocortex during a movement task. In four subjects, electrocorticographic recordings from the cortical surface revealed a low-frequency rhythm (10–13 Hz) that combined with a higher frequency (77–82 Hz) in a ventral region of the premotor cortex to produce a third rhythm at the sum of these two frequencies in a distant motor site. Such cross-frequency coupling implies a nonlinear interaction between these cortical sites. These findings demonstrate that task-specific, phase–phase coupling can support communication between distant areas of the human neocortex.

We describe a brain-computer interface for controlling a humanoid robot directly using brain signals obtained non-invasively from the scalp through electroencephalography (EEG). EEG has previously been used for tasks such as controlling a cursor and spelling a word, but it has been regarded as an unlikely candidate for more complex forms of control owing to its low signal-to-noise ratio. Here we show that by leveraging advances in robotics, an interface based on EEG can be used to command a partially autonomous humanoid robot to perform complex tasks such as walking to specific locations and picking up desired objects. Visual feedback from the robot's cameras allows the user to select arbitrary objects in the environment for pick-up and transport to chosen locations. Results from a study involving nine users indicate that a command for the robot can be selected from four possible choices in 5 s with 95% accuracy. Our results demonstrate that an EEG-based brain-computer interface can be used for sophisticated robotic interaction with the environment, involving not only navigation as in previous applications but also manipulation and transport of objects.

Several algorithms and models have recently been proposed for imitation learning in humans and robots. However, few proposals offer a framework for imitation learning in noisy stochastic environments where the imitator must learn and act under real-time performance constraints. We present a novel probabilistic framework for imitation learning in stochastic environments with unreliable sensors. Bayesian algorithms, based on Meltzoff and Moore's AIM hypothesis for action imitation, implement the core of an imitation learning framework. Our algorithms are computationally efficient, allowing real-time learning and imitation in an active stereo vision robotic head and on a humanoid robot. We present simulated and real-world robotics results demonstrating the viability of our approach. We conclude by advocating a research agenda that promotes interaction between cognitive and robotic studies of imitation.

Medically refractory epilepsy accounts for more than 30% of the epilepsy population. Scalp EEG electrodes have limited ability to localize seizure onset from deep structures and implantation of subdural electrodes with long term monitoring provides additional information. Apart from clinical application, this patient population provides a unique opportunity for acquiring electrocorticography data in research paradigms. We present a method for rapid localization of electrodes using lateral and anterior-posterior X-rays. Skull landmarks and proportions are used for co-registration with the standardized Talairach coordinate system. This MATLAB-based "Location on Cortex" (LOC) package facilitates rapid visualization of clinical and experimental data in a user-friendly manner.

In the first large study of its kind, we quantified changes in electrocorticographic signals associated with motor movement across 22 subjects with subdural electrode arrays placed for identification of seizure foci. Patients underwent a 5–7 d monitoring period with array placement, before seizure focus resection, and during this time they participated in the study. An interval-based motor-repetition task produced consistent and quantifiable spectral shifts that were mapped on a Talairach-standardized template cortex. Maps were created independently for a high-frequency band (HFB) (76–100 Hz) and a low-frequency band (LFB) (8–32 Hz) for several different movement modalities in each subject. The power in relevant electrodes consistently decreased in the LFB with movement, whereas the power in the HFB consistently increased. In addition, the HFB changes were more focal than the LFB changes. Sites of power changes corresponded to stereotactic locations in sensorimotor cortex and to the results of individual clinical electrical cortical mapping. Sensorimotor representation was found to be somatotopic, localized in stereotactic space to rolandic cortex, and typically followed the classic homunculus with limited extrarolandic representation.

Advances in mobile robotics have enabled robots that can autonomously operate in human-populated environments. Although primary tasks for such robots might be fetching, delivery, or escorting, they present an untapped potential as information gathering agents that can answer questions for the community of co-inhabitants. In this paper, we seek to better understand requirements for such information gathering robots (InfoBots) from the perspective of the user requesting the information. We present findings from two studies: (i) a user survey conducted in two office buildings and (ii) a 4-day long deployment in one of the buildings, during which inhabitants of the building could ask questions to an InfoBot through a web-based interface. These studies allow us to characterize the types of information that InfoBots can provide for their users.

Predictive coding is a unifying framework for understanding redundancy reduction and efficient coding in the nervous system. By transmitting only the unpredicted portions of an incoming sensory signal, predictive coding allows the nervous system to reduce redundancy and make full use of the limited dynamic range of neurons. Starting with the hypothesis of efficient coding as a design principle in the sensory system, predictive coding provides a functional explanation for a range of neural responses and many aspects of brain organization. The lateral and temporal antagonism in receptive fields in the retina and lateral geniculate nucleus occur naturally as a consequence of predictive coding of natural images. In the higher visual system, predictive coding provides an explanation for oriented receptive fields and contextual effects as well as the hierarchical reciprocally connected organization of the cortex. Predictive coding has also been found to be consistent with a variety of neurophysiological and psychophysical data obtained from different areas of the brain. WIREs Cogni Sci 2011 2 580-593 DOI: 10.1002/wcs.142 For further resources related to this article, please visit the WIREs website.

Robot Programming by Demonstration (PbD) allows users to program a robot by demonstrating the desired behavior. Providing these demonstrations typically involves moving the robot through a sequence of states, often by physically manipulating it. This requires users to be co-located with the robot and have the physical ability to manipulate it. In this paper, we present a natural language based interface for PbD that removes these requirements and enables hands-free programming. We focus on programming object manipulation actions—our key insight is that such actions can be decomposed into known types of manipulator movements that are naturally described using spatial language; e.g., object reference expressions and prepositions. Our method takes a natural language command and the current world state to infer the intended movement command and its parametrization. We implement this method on a two-armed mobile manipulator and demonstrate the different types of manipulation actions that can be programmed with it. We compare it to a kinesthetic PbD interface and we demonstrate our method's ability to deal with incomplete language.

We present, to our knowledge, the first demonstration that a non-invasive brain-to-brain interface (BBI) can be used to allow one human to guess what is on the mind of another human through an interactive question-and-answering paradigm similar to the “20 Questions” game. As in previous non-invasive BBI studies in humans, our interface uses electroencephalography (EEG) to detect specific patterns of brain activity from one participant (the “respondent”), and transcranial magnetic stimulation (TMS) to deliver functionally-relevant information to the brain of a second participant (the “inquirer”). Our results extend previous BBI research by (1) using stimulation of the visual cortex to convey visual stimuli that are privately experienced and consciously perceived by the inquirer; (2) exploiting real-time rather than off-line communication of information from one brain to another; and (3) employing an interactive task, in which the inquirer and respondent must exchange information bi-directionally to collaboratively solve the task. The results demonstrate that using the BBI, ten participants (five inquirer-respondent pairs) can successfully identify a “mystery item” using a true/false question-answering protocol similar to the “20 Questions” game, with high levels of accuracy that are significantly greater than a control condition in which participants were connected through a sham BBI.

Electrocorticography (ECoG) has been demonstrated to be an effective modality as a platform for brain-computer interfaces (BCIs). Through our experience with ten subjects, we further demonstrate evidence to support the power and flexibility of this signal for BCI usage. In a subset of four patients, closed-loop BCI experiments were attempted with the patient receiving online feedback that consisted of one-dimensional cursor movement controlled by ECoG features that had shown correlation with various real and imagined motor and speech tasks. All four achieved control, with final target accuracies between 73%-100%. We assess the methods for achieving control and the manner in which enhancing online control can be accomplished by rescreening during online tasks. Additionally, we assess the relevant issues of the current experimental paradigm in light of their clinical constraints.

The responses of neurons in cortical areas V2 and V4 can be significantly modulated by attention to particular locations within an input image. We show that such effects emerge naturally when perception is viewed as a probabilistic inference process governed by Bayesian principles and implemented in hierarchical cortical networks. The proposed model can explain a rich variety of attention-related responses in cortical area V4 including multiplicative modulation of tuning curves, restoration of neural responses in the presence of distracting stimuli, and influence of attention on neighboring unattended locations. Our results suggest a new interpretation of attention as a cortical mechanism for reducing perceptual uncertainty by combining top-down task-relevant information with bottom-up sensory inputs in a probabilistic manner.