Search

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Implementing Brain Computer Interfaces (BCIs) outside carefully controlled experiments in laboratory settings requires adaptive and scalable strategies with minimal supervision. Here we describe an unsupervised approach to decoding neural states from naturalistic human brain recordings. We analyzed continuous, long-term electrocorticography (ECoG) data recorded over many days from the brain of subjects in a hospital room, with simultaneous audio and video recordings. We discovered coherent clusters in high-dimensional ECoG recordings using hierarchical clustering and automatically annotated them using speech and movement labels extracted from audio and video. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Interpretable behaviors were decoded from ECoG data, including moving, speaking and resting; the results were assessed by comparison with manual annotation. Discovered clusters were projected back onto the brain revealing features consistent with known functional areas, opening the door to automated functional brain mapping in natural settings.

High gamma oscillations (70-150 Hz; HG) are rapidly evolving, spatially localized neurophysiological signals that are believed to be the best representative signature of engaged neural populations. The HG band has been best characterized from invasive electrophysiological approaches such as electrocorticography (ECoG) because of the increased signal-to-noise ratio that results when by-passing the scalp and skull. Despite the recent observation that HG activity can be detected non-invasively by electroencephalography (EEG), it is unclear to what extent EEG can accurately resolve the spatial distribution of HG signals during active task engagement. We have overcome some of the limitations inherent to acquiring HG signals across the scalp by utilizing individual head anatomy in combination with an inverse modeling method. We applied a linearly constrained minimum variance beamformer (LCMV) method on EEG data during a motor imagery paradigm to extract a time-frequency spectrogram at every voxel location on the cortex. To confirm spatially distributed patterns of HG responses, we contrasted overlapping maps of the EEG HG signal with BOLD fMRI data acquired from the same set of neurologically normal subjects during a separate session. We show that scalp-based HG band activity detected by EEG during motor imagery spatially co-localizes with BOLD fMRI data. Taken together, these results suggest that EEG can accurately resolve spatially specific estimates of local cortical high frequency signals, potentially opening an avenue for non-invasive measurement of HG potentials from diverse sets of neurologically impaired populations for diagnostic and therapeutic purposes

Abstract Humans frequently interact with agents whose intentions can fluctuate between competition and cooperation over time. It is unclear how the brain adapts to fluctuating intentions of others when the nature of the interactions (to cooperate or compete) is not explicitly and truthfully signaled. Here, we use model-based fMRI and a task in which participants thought they were playing with another player. In fact, they played with an algorithm that alternated without signaling between cooperative and competitive strategies. We show that a neurocomputational mechanism with arbitration between competitive and cooperative experts outperforms other learning models in predicting choice behavior. At the brain level, the fMRI results show that the ventral striatum and ventromedial prefrontal cortex track the difference of reliability between these experts. When attributing competitive intentions, we find increased coupling between these regions and a network that distinguishes prediction errors related to competition and cooperation. These findings provide a neurocomputational account of how the brain arbitrates dynamically between cooperative and competitive intentions when making adaptive social decisions.

We introduce dynamic predictive coding, a hierarchical model of spatiotemporal prediction and sequence learning in the neocortex. The model assumes that higher cortical levels modulate the temporal dynamics of lower levels, correcting their predictions of dynamics using prediction errors. As a result, lower levels form representations that encode sequences at shorter timescales (e.g., a single step) while higher levels form representations that encode sequences at longer timescales (e.g., an entire sequence). We tested this model using a two-level neural network, where the top-down modulation creates low-dimensional combinations of a set of learned temporal dynamics to explain input sequences. When trained on natural videos, the lower-level model neurons developed space-time receptive fields similar to those of simple cells in the primary visual cortex while the higher-level responses spanned longer timescales, mimicking temporal response hierarchies in the cortex. Additionally, the network's hierarchical sequence representation exhibited both predictive and postdictive effects resembling those observed in visual motion processing in humans (e.g., in the flash-lag illusion). When coupled with an associative memory emulating the role of the hippocampus, the model allowed episodic memories to be stored and retrieved, supporting cue-triggered recall of an input sequence similar to activity recall in the visual cortex. When extended to three hierarchical levels, the model learned progressively more abstract temporal representations along the hierarchy. Taken together, our results suggest that cortical processing and learning of sequences can be interpreted as dynamic predictive coding based on a hierarchical spatiotemporal generative model of the visual world.

Different sleep stages have been shown to be vital for a variety of brain functions, including learning, memory, and skill consolidation. However, our understanding of neural dynamics during sleep and the role of prominent LFP frequency bands remain incomplete. To elucidate such dynamics and differences between behavioral states we collected multichannel LFP and spike data in primary motor cortex of unconstrained macaques for up to 24 h using a head-fixed brain-computer interface (Neurochip3). Each 8-s bin of time was classified into awake-moving (Move), awake-resting (Rest), REM sleep (REM), or non-REM sleep (NREM) by using dimensionality reduction and clustering on the average spectral density and the acceleration of the head. LFP power showed high delta during NREM, high theta during REM, and high beta when the animal was awake. Cross-frequency phase-amplitude coupling typically showed higher coupling during NREM between all pairs of frequency bands. Two notable exceptions were high delta-high gamma and theta-high gamma coupling during Move, and high theta-beta coupling during REM. Single units showed decreased firing rate during NREM, though with increased short ISIs compared to other states. Spike-LFP synchrony showed high delta synchrony during Move, and higher coupling with all other frequency bands during NREM. These results altogether reveal potential roles and functions of different LFP bands that have previously been unexplored.

Measurement(s) Brain activity measurement • Body Position • Behavior labels • Brain electrode locations Technology Type(s) Electrocorticography • Video Recording Sample Characteristic - Organism Homo sapiens Sample Characteristic - Environment hospital Sample Characteristic - Location Harborview Medical Center

Brain–computer interfaces (BCIs) benefit greatly from performance feedback, but current systems lack automatic, task-independent feedback. Cortical responses elicited from user error have the potential to serve as state-based feedback to BCI decoders. To gain a better understanding of local error potentials, we investigate responsive cortical power underlying error-related potentials (ErrPs) from the human cortex during a one-dimensional center-out BCI task, tracking the topography of high-gamma (70–100 Hz) band power (HBP) specific to BCI error. We measured electrocorticography (ECoG) in three human subjects during dynamic, continuous control over BCI cursor velocity. Subjects used motor imagery and rest to move the cursor toward and subsequently dwell within a target region. We then identified and labeled epochs where the BCI decoder incorrectly moved the cursor in the direction opposite of the subject’s expectations (i.e., BCI error). We found increased HBP in various cortical areas 100–500 ms following BCI error with respect to epochs of correct, intended control. Significant responses were noted in primary somatosensory, motor, premotor, and parietal areas and generally regardless of whether the subject was using motor imagery or rest to move the cursor toward the target. Parts of somatosensory, temporal, and parietal areas exclusively had increased HBP when subjects were using motor imagery. In contrast, only part of the parietal cortex near the angular gyrus exclusively had an increase in HBP during rest. This investigation is, to our knowledge, the first to explore cortical fields changes in the context of continuous control in ECoG BCI. We present topographical changes in HBP characteristic specific to the generation of error. By focusing on continuous control, instead of on discrete control for simple selection, we investigate a more naturalistic setting and provide high ecological validity for characterizing error potentials. Such potentials could be considered as design elements for co-adaptive BCIs in the future as task-independent feedback to the decoder, allowing for more robust and individualized BCIs.

A motor cortex-based brain-computer interface (BCI) creates a novel real world output directly from cortical activity. Use of a BCI has been demonstrated to be a learned skill that involves recruitment of neural populations that are directly linked to BCI control as well as those that are not. The nature of interactions between these populations, however, remains largely unknown. Here, we employed a data-driven approach to assess the interaction between both local and remote cortical areas during the use of an electrocorticographic BCI, a method which allows direct sampling of cortical surface potentials. Comparing the area controlling the BCI with remote areas, we evaluated relationships between the amplitude envelopes of band limited powers as well as non-linear phase-phase interactions. We found amplitude-amplitude interactions in the high gamma (HG, 70-150 Hz) range that were primarily located in the posterior portion of the frontal lobe, near the controlling site, and non-linear phase-phase interactions involving multiple frequencies (cross-frequency coupling between 8-11 Hz and 70-90 Hz) taking place over larger cortical distances. Further, strength of the amplitude-amplitude interactions decreased with time, whereas the phase-phase interactions did not. These findings suggest multiple modes of cortical communication taking place during BCI use that are specialized for function and depend on interaction distance.

The link between object perception and neural activity in visual cortical areas is a problem of fundamental importance in neuroscience. Here we show that electrical potentials from the ventral temporal cortical surface in humans contain sufficient information for spontaneous and near-instantaneous identification of a subject's perceptual state. Electrocorticographic (ECoG) arrays were placed on the subtemporal cortical surface of seven epilepsy patients. Grayscale images of faces and houses were displayed rapidly in random sequence. We developed a template projection approach to decode the continuous ECoG data stream spontaneously, predicting the occurrence, timing and type of visual stimulus. In this setting, we evaluated the independent and joint use of two well-studied features of brain signals, broadband changes in the frequency power spectrum of the potential and deflections in the raw potential trace (event-related potential; ERP). Our ability to predict both the timing of stimulus onset and the type of image was best when we used a combination of both the broadband response and ERP, suggesting that they capture different and complementary aspects of the subject's perceptual state. Specifically, we were able to predict the timing and type of 96% of all stimuli, with less than 5% false positive rate and a ~20ms error in timing.

We present, to our knowledge, the first demonstration that a non-invasive brain-to-brain interface (BBI) can be used to allow one human to guess what is on the mind of another human through an interactive question-and-answering paradigm similar to the "20 Questions" game. As in previous non-invasive BBI studies in humans, our interface uses electroencephalography (EEG) to detect specific patterns of brain activity from one participant (the "respondent"), and transcranial magnetic stimulation (TMS) to deliver functionally-relevant information to the brain of a second participant (the "inquirer"). Our results extend previous BBI research by (1) using stimulation of the visual cortex to convey visual stimuli that are privately experienced and consciously perceived by the inquirer; (2) exploiting real-time rather than off-line communication of information from one brain to another; and (3) employing an interactive task, in which the inquirer and respondent must exchange information bi-directionally to collaboratively solve the task. The results demonstrate that using the BBI, ten participants (five inquirer-respondent pairs) can successfully identify a "mystery item" using a true/false question-answering protocol similar to the "20 Questions" game, with high levels of accuracy that are significantly greater than a control condition in which participants were connected through a sham BBI.

A fundamental challenge in robotics today is building robots that can learn new skills by observing humans and imitating human actions. We propose a new Bayesian approach to robotic learning by imitation inspired by the developmental hypothesis that children use self-experience to bootstrap the process of intention recognition and goal-based imitation. Our approach allows an autonomous agent to: (i) learn probabilistic models of actions through self-discovery and experience, (ii) utilize these learned models for inferring the goals of human actions, and (iii) perform goal-based imitation for robotic learning and human-robot collaboration. Such an approach allows a robot to leverage its increasing repertoire of learned behaviors to interpret increasingly complex human actions and use the inferred goals for imitation, even when the robot has very different actuators from humans. We demonstrate our approach using two different scenarios: (i) a simulated robot that learns human-like gaze following behavior, and (ii) a robot that learns to imitate human actions in a tabletop organization task. In both cases, the agent learns a probabilistic model of its own actions, and uses this model for goal inference and goal-based imitation. We also show that the robotic agent can use its probabilistic model to seek human assistance when it recognizes that its inferred actions are too uncertain, risky, or impossible to perform, thereby opening the door to human-robot collaboration.

We describe the first direct brain-to-brain interface in humans and present results from experiments involving six different subjects. Our non-invasive interface, demonstrated originally in August 2013, combines electroencephalography (EEG) for recording brain signals with transcranial magnetic stimulation (TMS) for delivering information to the brain. We illustrate our method using a visuomotor task in which two humans must cooperate through direct brain-to-brain communication to achieve a desired goal in a computer game. The brain-to-brain interface detects motor imagery in EEG signals recorded from one subject (the "sender") and transmits this information over the internet to the motor cortex region of a second subject (the "receiver"). This allows the sender to cause a desired motor response in the receiver (a press on a touchpad) via TMS. We quantify the performance of the brain-to-brain interface in terms of the amount of information transmitted as well as the accuracies attained in (1) decoding the sender's signals, (2) generating a motor response from the receiver upon stimulation, and (3) achieving the overall goal in the cooperative visuomotor task. Our results provide evidence for a rudimentary form of direct information transmission from one human brain to another using non-invasive means.

A key problem in neuroscience is understanding how the brain makes decisions under uncertainty. Important insights have been gained using tasks such as the random dots motion discrimination task in which the subject makes decisions based on noisy stimuli. A descriptive model known as the drift diffusion model has previously been used to explain psychometric and reaction time data from such tasks but to fully explain the data, one is forced to make ad-hoc assumptions such as a time-dependent collapsing decision boundary. We show that such assumptions are unnecessary when decision making is viewed within the framework of partially observable Markov decision processes (POMDPs). We propose an alternative model for decision making based on POMDPs. We show that the motion discrimination task reduces to the problems of (1) computing beliefs (posterior distributions) over the unknown direction and motion strength from noisy observations in a bayesian manner, and (2) selecting actions based on these beliefs to maximize the expected sum of future rewards. The resulting optimal policy (belief-to-action mapping) is shown to be equivalent to a collapsing decision threshold that governs the switch from evidence accumulation to a discrimination decision. We show that the model accounts for both accuracy and reaction time as a function of stimulus strength as well as different speed-accuracy conditions in the random dots task.

The Indus script is one of the major undeciphered scripts of the ancient world. The small size of the corpus, the absence of bilingual texts, and the lack of definite knowledge of the underlying language has frustrated efforts at decipherment since the discovery of the remains of the Indus civilization. Building on previous statistical approaches, we apply the tools of statistical language processing, specifically n-gram Markov chains, to analyze the syntax of the Indus script. We find that unigrams follow a Zipf-Mandelbrot distribution. Text beginner and ender distributions are unequal, providing internal evidence for syntax. We see clear evidence of strong bigram correlations and extract significant pairs and triplets using a log-likelihood measure of association. Highly frequent pairs and triplets are not always highly significant. The model performance is evaluated using information-theoretic measures and cross-validation. The model can restore doubtfully read texts with an accuracy of about 75%. We find that a quadrigram Markov chain saturates information theoretic measures against a held-out corpus. Our work forms the basis for the development of a stochastic grammar which may be used to explore the syntax of the Indus script in greater detail.

A Bayesian framework based on partially observable Markov decision processes (POMDPs) not only predicts subjects’ confidence in a perceptual decision making task but also explains well-known discrepancies between confidence and choice accuracy as arising from incomplete knowledge of the environment.

Traditionally, recording from and stimulating the brain with high spatial and temporal resolution required invasive means. However, recently, the technical capabilities of less invasive and non-invasive neuro-interfacing technology have been dramatically improving, and laboratories and funders aim to further improve these capabilities. These technologies can facilitate functions such as multi-person communication, mood regulation and memory recall. We consider a potential future where the less invasive technology is in high demand. Will this demand match that the current-day demand for a smartphone? Here, we draw upon existing research to project which particular neuroethics issues may arise in this potential future and what preparatory steps may be taken to address these issues.

In the ongoing COVID-19 pandemic, public health experts have produced guidelines to limit the spread of the coronavirus, but individuals do not always comply with experts' recommendations. Here, we tested whether a specific psychological belief-identification with all humanity-predicts cooperation with public health guidelines as well as helpful behavior during the COVID-19 pandemic. We hypothesized that peoples' endorsement of this belief-their relative perception of a connection and moral commitment to other humans-would predict their tendencies to adopt World Health Organization (WHO) guidelines and to help others. To assess this, we conducted a global online study (N = 2537 participants) of four WHO-recommended health behaviors and four pandemic-related moral dilemmas that we constructed to be relevant to helping others at a potential cost to oneself. We used generalized linear mixed models (GLMM) that included 10 predictor variables (demographic, contextual, and psychological) for each of five outcome measures (a WHO cooperative health behavior score, plus responses to each of our four moral, helping dilemmas). Identification with all humanity was the most consistent and consequential predictor of individuals' cooperative health behavior and helpful responding. Analyses showed that the identification with all humanity significantly predicted each of the five outcomes while controlling for the other variables (Prange < 10-22 to < 0.009). The mean effect size of the identification with all humanity predictor on these outcomes was more than twice as large as the effect sizes of other predictors. Identification with all humanity is a psychological construct that, through targeted interventions, may help scientists and policymakers to better understand and promote cooperative health behavior and help-oriented concern for others during the current pandemic as well as in future humanitarian crises.

Increasingly, neuroimaging researchers are exploring the use of real-time functional magnetic resonance imaging (rt-fMRI) as a way to access a participant’s ongoing brain function throughout a scan. This approach presents novel and exciting experimental applications ranging from monitoring data quality in real time, to delivering neurofeedback from a region of interest, to dynamically controlling experimental flow, or interfacing with remote devices. Yet, for those interested in adopting this method, the existing software options are few and limited in application. This presents a barrier for new users, as well as hinders existing users from refining techniques and methods. Here we introduce a free, open-source rt-fMRI package, the Pyneal toolkit, designed to address this limitation. The Pyneal toolkit is python-based software that offers a flexible and user friendly framework for rt-fMRI, is compatible with all three major scanner manufacturers (GE, Siemens, Phillips), and, critically, allows fully customized analysis pipelines. In this article, we provide a detailed overview of the architecture, describe how to set up and run the Pyneal toolkit during an experimental session, offer tutorials with scan data that demonstrate how data flows through the Pyneal toolkit with example analyses, and highlight the advantages that the Pyneal toolkit offers to the neuroimaging community.

Electrocorticographic brain computer interfaces (ECoG-BCIs) offer tremendous opportunities for restoring function in individuals suffering from neurological damage and for advancing basic neuroscience knowledge. ECoG electrodes are already commonly used clinically for monitoring epilepsy and have greater spatial specificity in recording neuronal activity than techniques such as electroencephalography (EEG). Much work to date in the field has focused on using ECoG signals recorded from cortex as control outputs for driving end effectors. An equally important but less explored application of an ECoG-BCI is directing input into cortex using ECoG electrodes for direct electrical stimulation (DES). Combining DES with ECoG recording enables a truly bidirectional BCI, where information is both read from and written to the brain. We discuss the advantages and opportunities, as well as the barriers and challenges presented by using DES in an ECoG-BCI. In this article, we review ECoG electrodes, the physics and physiology of DES, and the use of electrical stimulation of the brain for the clinical treatment of disorders such as epilepsy and Parkinson’s disease. We briefly discuss some of the translational, regulatory, financial, and ethical concerns regarding ECoG-BCIs. Next, we describe the use of ECoG-based DES for providing sensory feedback and for probing and modifying cortical connectivity. We explore future directions, which may draw on invasive animal studies with penetrating and surface electrodes as well as non-invasive stimulation methods such as transcranial magnetic stimulation (TMS). We conclude by describing enabling technologies, such as smaller ECoG electrodes for more precise targeting of cortical areas, signal processing strategies for simultaneous stimulation and recording, and computational modeling and algorithms for tailoring stimulation to each individual brain.

Brain rhythms are more than just passive phenomena in visual cortex. For the first time, we show that the physiology underlying brain rhythms actively suppresses and releases cortical areas on a second-to-second basis during visual processing. Furthermore, their influence is specific at the scale of individual gyri. We quantified the interaction between broadband spectral change and brain rhythms on a second-to-second basis in electrocorticographic (ECoG) measurement of brain surface potentials in five human subjects during a visual search task. Comparison of visual search epochs with a blank screen baseline revealed changes in the raw potential, the amplitude of rhythmic activity, and in the decoupled broadband spectral amplitude. We present new methods to characterize the intensity and preferred phase of coupling between broadband power and band-limited rhythms, and to estimate the magnitude of rhythm-to-broadband modulation on a trial-by-trial basis. These tools revealed numerous coupling motifs between the phase of low frequency (δ, θ, α, β, and γ band) rhythms and the amplitude of broadband spectral change. In the θ and β ranges, the coupling of phase to broadband change is dynamic during visual processing, decreasing in some occipital areas and increasing in others, in a gyrally specific pattern. Finally, we demonstrate that the rhythms interact with one another across frequency ranges, and across cortical sites.