Search

Tracking an odour plume to locate its source under variable wind and plume statistics is a complex task. Flying insects routinely accomplish such tracking, often over long distances, in pursuit of food or mates. Several aspects of this remarkable behaviour and its underlying neural circuitry have been studied experimentally. Here we take a complementary in silico approach to develop an integrated understanding of their behaviour and neural computations. Specifically, we train artificial recurrent neural network agents using deep reinforcement learning to locate the source of simulated odour plumes that mimic features of plumes in a turbulent flow. Interestingly, the agents’ emergent behaviours resemble those of flying insects, and the recurrent neural networks learn to compute task-relevant variables with distinct dynamic structures in population activity. Our analyses put forward a testable behavioural hypothesis for tracking plumes in changing wind direction, and we provide key intuitions for memory requirements and neural dynamics in odour plume tracking.

In perceptual decisions, subjects infer hidden states of the environment based on noisy sensory information. Here we show that both choice and its associated confidence are explained by a Bayesian framework based on partially observable Markov decision processes (POMDPs). We test our model on monkeys performing a direction-discrimination task with post-decision wagering, demonstrating that the model explains objective accuracy and predicts subjective confidence. Further, we show that the model replicates well-known discrepancies of confidence and accuracy, including the hard-easy effect, opposing effects of stimulus variability on confidence and accuracy, dependence of confidence ratings on simultaneous or sequential reports of choice and confidence, apparent difference between choice and confidence sensitivity, and seemingly disproportionate influence of choice-congruent evidence on confidence. These effects may not be signatures of sub-optimal inference or discrepant computational processes for choice and confidence. Rather, they arise in Bayesian inference with incomplete knowledge of the environment., A Bayesian framework based on partially observable Markov decision processes (POMDPs) not only predicts subjects’ confidence in a perceptual decision making task but also explains well-known discrepancies between confidence and choice accuracy as arising from incomplete knowledge of the environment.

Objective: A major challenge in designing closed-loop brain-computer interfaces is finding optimal stimulation patterns as a function of ongoing neural activity for different subjects and objectives. Approach: To achieve goal-directed closed-loop neurostimulation, we propose "neural co-processors" which use artificial neural networks and deep learning to learn optimal closed-loop stimulation policies, shaping neural activity and bridging injured neural circuits for targeted repair and rehabilitation. The co-processor adapts the stimulation policy as the biological circuit itself adapts to the stimulation, achieving a form of brain-device co-adaptation. Here we use simulations to lay the groundwork for future in vivo tests of neural co-processors. We leverage a cortical model of grasping, to which we applied various forms of simulated lesions, allowing us to develop the critical learning algorithms and study adaptations to non-stationarity. Main results: Our simulations show the ability of a neural co-processor to learn a stimulation policy using a supervised learning approach, and to adapt that policy as the underlying brain and sensors change. Our co-processor successfully co-adapted with the simulated brain to accomplish the reach-and-grasp task after a variety of lesions were applied, achieving recovery towards healthy function. Significance: Our results provide the first proof-of-concept demonstration of a co-processor for adaptive activity-dependent closed-loop neurostimulation, optimizing for a rehabilitation goal. While a gap remains between simulations and applications, our results provide insights on how co-processors may be developed for learning complex adaptive stimulation policies for a variety of neural rehabilitation and neuroprosthetic applications., 45 pages, 19 figures. Submitted the IOP Journal of Neural Engineering

We present a windowed technique to learn parsimonious time-varying autoregressive models from multivariate timeseries. This unsupervised method uncovers interpretable spatiotemporal structure in data via non-smooth and non-convex optimization. In each time window, we assume the data follow a linear model parameterized by a system matrix, and we model this stack of potentially different system matrices as a low rank tensor. Because of its structure, the model is scalable to high-dimensional data and can easily incorporate priors such as smoothness over time. We find the components of the tensor using alternating minimization and prove that any stationary point of this algorithm is a local minimum. We demonstrate on a synthetic example that our method identifies the true rank of a switching linear system in the presence of noise. We illustrate our model's utility and superior scalability over extant methods when applied to several synthetic and real-world example: two types of time-varying linear systems, worm behavior, sea surface temperature, and monkey brain datasets.

Tracking a turbulent plume to locate its source under variable wind and plume statistics is a complex task; flying insects routinely accomplish such tracking, often over long distances, in pursuit of food or mates. Several aspects of this remarkable behavior and its underlying neural circuitry have been studied experimentally. Here, we take a complementary in silico approach to develop an integrated understanding of behavior and neural computations. Specifically, we train artificial recurrent neural network (RNN) agents using deep reinforcement learning (DRL) to locate the source of simulated turbulent plumes. Interestingly, the agents' emergent behaviors resemble those of flying insects, and the RNNs learn to compute task-relevant variables with distinct dynamic structures in population activity. Our analyses put forward a testable behavioral hypothesis for tracking plumes in changing wind direction, and we provide key intuitions for memory requirements and neural dynamics in turbulent plume tracking.

Intracortical microstimulation (ICMS) is commonly used in many experimental and clinical paradigms; however, its effects on the activation of neurons are still not completely understood. To document the responses of cortical neurons in non-human primates to stimulation, we recorded single unit activity while delivering single-pulse stimulation via Utah arrays implanted in primary motor cortex of three macaque monkeys. Stimuli between 5-50 μA delivered to single channels reliably evoked spikes in neurons recorded throughout the array with delays of up to 12 milliseconds. ICMS pulses also induced a period of inhibition lasting up to 150 ms that typically followed the initial excitatory response. Higher current amplitudes led to a greater probability of evoking a spike and extended the duration of inhibition. The likelihood of evoking a spike in a neuron was dependent on the spontaneous firing rate as well as the delay between its most recent spike time and stimulus onset. Tonic repetitive stimulation between 2 and 20 Hz often modulated both the probability of evoking spikes and the duration of inhibition, although high frequency stimulation in particular was more likely to change both responses. On a trial-by-trial basis, whether a stimulus evoked a spike did not affect the subsequent inhibitory response; however, their changes over time were frequently positively or negatively correlated. Our results document the complex dynamics of cortical neural responses to electrical stimulation that need to be considered when utilizing ICMS for scientific and clinical applications.Significance statementIntracortical microstimulation (ICMS) is commonly used to probe the cortex, and previous studies have characterized the responses of single neurons to ICMS. However, these studies typically explored the averaged effects of ICMS throughout each experimental session, rather than by a trial-by-trial basis for each stimulation pulse. By shifting the approach, we explored the dependence of neural responses to ICMS on the spontaneous neural activity as well as the dynamics of responses over time due to repetitive stimulation. Our results highlight how the responses of neurons to ICMS are likely the result of interactions between local excitatory and inhibitory cortical circuits. These results will help inform the design of ICMS for both basic research and clinically relevant stimulation protocols.

We introduce Active Predictive Coding Networks (APCNs), a new class of neural networks that solve a major problem posed by Hinton and others in the fields of artificial intelligence and brain modeling: how can neural networks learn intrinsic reference frames for objects and parse visual scenes into part-whole hierarchies by dynamically allocating nodes in a parse tree? APCNs address this problem by using a novel combination of ideas: (1) hypernetworks are used for dynamically generating recurrent neural networks that predict parts and their locations within intrinsic reference frames conditioned on higher object-level embedding vectors, and (2) reinforcement learning is used in conjunction with backpropagation for end-to-end learning of model parameters. The APCN architecture lends itself naturally to multi-level hierarchical learning and is closely related to predictive coding models of cortical function. Using the MNIST, Fashion-MNIST and Omniglot datasets, we demonstrate that APCNs can (a) learn to parse images into part-whole hierarchies, (b) learn compositional representations, and (c) transfer their knowledge to unseen classes of objects. With their ability to dynamically generate parse trees with part locations for objects, APCNs offer a new framework for explainable AI that leverages advances in deep learning while retaining interpretability and compositionality.

Traditionally, recording from and stimulating the brain with high spatial and temporal resolution required invasive means. However, recently, the technical capabilities of less invasive and non-invasive neuro-interfacing technology have been dramatically improving, and laboratories and funders aim to further improve these capabilities. These technologies can facilitate functions such as multi-person communication, mood regulation and memory recall. We consider a potential future where the less invasive technology is in high demand. Will this demand match that the current-day demand for a smartphone? Here, we draw upon existing research to project which particular neuroethics issues may arise in this potential future and what preparatory steps may be taken to address these issues.

Objective. Recent advances in neural decoding have accelerated the development of brain–computer interfaces aimed at assisting users with everyday tasks such as speaking, walking, and manipulating objects. However, current approaches for training neural decoders commonly require large quantities of labeled data, which can be laborious or infeasible to obtain in real-world settings. Alternatively, self-supervised models that share self-generated pseudo-labels between two data streams have shown exceptional performance on unlabeled audio and video data, but it remains unclear how well they extend to neural decoding. Approach. We learn neural decoders without labels by leveraging multiple simultaneously recorded data streams, including neural, kinematic, and physiological signals. Specifically, we apply cross-modal, self-supervised deep clustering to train decoders that can classify movements from brain recordings. After training, we then isolate the decoders for each input data stream and compare the accuracy of decoders trained using cross-modal deep clustering against supervised and unimodal, self-supervised models. Main results. We find that sharing pseudo-labels between two data streams during training substantially increases decoding performance compared to unimodal, self-supervised models, with accuracies approaching those of supervised decoders trained on labeled data. Next, we extend cross-modal decoder training to three or more modalities, achieving state-of-the-art neural decoding accuracy that matches or slightly exceeds the performance of supervised models. Significance. We demonstrate that cross-modal, self-supervised decoding can be applied to train neural decoders when few or no labels are available and extend the cross-modal framework to share information among three or more data streams, further improving self-supervised training.

Motor behaviors are central to many functions and dysfunctions of the brain, and understanding their neural basis has consequently been a major focus in neuroscience. However, most studies of motor behaviors have been restricted to artificial, repetitive paradigms, far removed from natural movements performed “in the wild.” Here, we leveraged recent advances in machine learning and computer vision to analyze intracranial recordings from 12 human subjects during thousands of spontaneous, unstructured arm reach movements, observed over several days for each subject. These naturalistic movements elicited cortical spectral power patterns consistent with findings from controlled paradigms, but with considerable neural variability across subjects and events. We modeled interevent variability using 10 behavioral and environmental features; the most important features explaining this variability were reach angle and day of recording. Our work is among the first studies connecting behavioral and neural variability across cortex in humans during unstructured movements and contributes to our understanding of long-term naturalistic behavior. Significance Statement Understanding the neural basis of human movement has long been a key focus in neuroscience. However, researchers often study constrained, monotonous tasks that differ greatly from the rich and diverse natural movements we actually make. Here, we use data-intensive computational approaches to reveal patterns in neural activity underlying naturalistic human arm movements. While such movements match previous experimental findings on average, there is substantial neural variability from one movement to the next. We partially explain this variability by aspects of the movement observed, though much variability remains unaccounted for. Our study sheds light on how the brain generates natural arm movements and emphasizes the critical need to study brain activity in more unstructured settings that mimic daily life.

The original predictive coding model of Rao & Ballard [1] focused on spatial prediction to explain spatial receptive fields and contextual effects in the visual cortex. Here, we introduce a new dynamic predictive coding model that achieves spatiotemporal prediction of complex natural image sequences using time-varying transition matrices. We overcome the limitations of static linear transition models (as in, e.g., Kalman filters) using a hypernetwork to adjust the transition matrix dynamically for every time step, allowing the model to predict using a time-varying mixture of possible transition dynamics. We developed a single level model with recurrent modulation of transition weights by a hypernetwork and a two-level hierarchical model with top-down modulation based on a hypernetwork. At each time step, the model predicts the next input and estimates a sparse neural code by minimizing prediction error. When exposed to natural movies, the model learned localized, oriented spatial filters as well as both separable and inseparable (direction-selective) space-time receptive fields at the first level, similar to those found in the primary visual cortex (V1). Longer timescale responses and stability at the second level also emerged naturally from minimizing prediction errors for the first level dynamics. Our results suggest that the multiscale temporal response properties of cortical neurons could be the result of the cortex learning a hierarchical generative model of the visual world with higher order areas predicting the transition dynamics of lower order areas.

ObjectiveAdvances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can robustly train on pooled, multi-participant data and generalize to new participants.ApproachWe introduce a new decoder, HTNet, which uses a convolutional neural network with two innovations: (1) a Hilbert transform that computes spectral power at data-driven frequencies and (2) a layer that projects electrode-level data onto predefined brain regions. The projection layer critically enables applications with intracranial electrocorticography (ECoG), where electrode locations are not standardized and vary widely across participants. We trained HTNet to decode arm movements using pooled ECoG data from 11 of 12 participants and tested performance on unseen ECoG or electroencephalography (EEG) participants; these pretrained models were also subsequently fine-tuned to each test participant.Main resultsHTNet outperformed state-of-the-art decoders when tested on unseen participants, even when a different recording modality was used. By fine-tuning these generalized HTNet decoders, we achieved performance approaching the best tailored decoders with as few as 50 ECoG or 20 EEG events. We were also able to interpret HTNet’s trained weights and demonstrate its ability to extract physiologically-relevant features.SignificanceBy generalizing to new participants and recording modalities, robustly handling variations in electrode placement, and allowing participant-specific fine-tuning with minimal data, HTNet is applicable across a broader range of neural decoding applications compared to current state-of-the-art decoders.

Increasingly, neuroimaging researchers are exploring the use of real-time functional magnetic resonance imaging (rt-fMRI) as a way to access a participant’s ongoing brain function throughout a scan. This approach presents novel and exciting experimental applications ranging from monitoring data quality in real time, to delivering neurofeedback from a region of interest, to dynamically controlling experimental flow, or interfacing with remote devices. Yet, for those interested in adopting this method, the existing software options are few and limited in application. This presents a barrier for new users, as well as hinders existing users from refining techniques and methods. Here we introduce a free, open-source rt-fMRI package, the Pyneal toolkit, designed to address this limitation. The Pyneal toolkit is python-based software that offers a flexible and user friendly framework for rt-fMRI, is compatible with all three major scanner manufacturers (GE, Siemens, Phillips), and, critically, allows fully customized analysis pipelines. In this article, we provide a detailed overview of the architecture, describe how to set up and run the Pyneal toolkit during an experimental session, offer tutorials with scan data that demonstrate how data flows through the Pyneal toolkit with example analyses, and highlight the advantages that the Pyneal toolkit offers to the neuroimaging community.

The promise of robots assisting humans in everyday tasks has led to a variety of research questions and challenges in human-robot collaboration. Here, we address the question of whether and when a ...

A Bayesian model suggests that when interacting with a group, humans simulate the “mind of the group” to choose an action., To make decisions in a social context, humans have to predict the behavior of others, an ability that is thought to rely on having a model of other minds known as “theory of mind.” Such a model becomes especially complex when the number of people one simultaneously interacts with is large and actions are anonymous. Here, we present results from a group decision-making task known as the volunteer’s dilemma and demonstrate that a Bayesian model based on partially observable Markov decision processes outperforms existing models in quantitatively predicting human behavior and outcomes of group interactions. Our results suggest that in decision-making tasks involving large groups with anonymous members, humans use Bayesian inference to model the “mind of the group,” making predictions of others’ decisions while also simulating the effects of their own actions on the group’s dynamics in the future.

Electrocorticographic brain computer interfaces (ECoG-BCIs) offer tremendous opportunities for restoring function in individuals suffering from neurological damage and for advancing basic neuroscience knowledge. ECoG electrodes are already commonly used clinically for monitoring epilepsy and have greater spatial specificity in recording neuronal activity than techniques such as electroencephalography (EEG). Much work to date in the field has focused on using ECoG signals recorded from cortex as control outputs for driving end effectors. An equally important but less explored application of an ECoG-BCI is directing input into cortex using ECoG electrodes for direct electrical stimulation (DES). Combining DES with ECoG recording enables a truly bidirectional BCI, where information is both read from and written to the brain. We discuss the advantages and opportunities, as well as the barriers and challenges presented by using DES in an ECoG-BCI. In this article, we review ECoG electrodes, the physics and physiology of DES, and the use of electrical stimulation of the brain for the clinical treatment of disorders such as epilepsy and Parkinson’s disease. We briefly discuss some of the translational, regulatory, financial, and ethical concerns regarding ECoG-BCIs. Next, we describe the use of ECoG-based DES for providing sensory feedback and for probing and modifying cortical connectivity. We explore future directions, which may draw on invasive animal studies with penetrating and surface electrodes as well as non-invasive stimulation methods such as transcranial magnetic stimulation (TMS). We conclude by describing enabling technologies, such as smaller ECoG electrodes for more precise targeting of cortical areas, signal processing strategies for simultaneous stimulation and recording, and computational modeling and algorithms for tailoring stimulation to each individual brain.

We present BrainNet which, to our knowledge, is the first multi-person non-invasive direct brain-to-brain interface for collaborative problem solving. The interface combines electroencephalography (EEG) to record brain signals and transcranial magnetic stimulation (TMS) to deliver information noninvasively to the brain. The interface allows three human subjects to collaborate and solve a task using direct brain-to-brain communication. Two of the three subjects are designated as “Senders” whose brain signals are decoded using real-time EEG data analysis. The decoding process extracts each Sender’s decision about whether to rotate a block in a Tetris-like game before it is dropped to fill a line. The Senders’ decisions are transmitted via the Internet to the brain of a third subject, the “Receiver,” who cannot see the game screen. The Senders’ decisions are delivered to the Receiver’s brain via magnetic stimulation of the occipital cortex. The Receiver integrates the information received from the two Senders and uses an EEG interface to make a decision about either turning the block or keeping it in the same orientation. A second round of the game provides an additional chance for the Senders to evaluate the Receiver’s decision and send feedback to the Receiver’s brain, and for the Receiver to rectify a possible incorrect decision made in the first round. We evaluated the performance of BrainNet in terms of (1) Group-level performance during the game, (2) True/False positive rates of subjects’ decisions, and (3) Mutual information between subjects. Five groups, each with three human subjects, successfully used BrainNet to perform the collaborative task, with an average accuracy of 81.25%. Furthermore, by varying the information reliability of the Senders by artificially injecting noise into one Sender’s signal, we investigated how the Receiver learns to integrate noisy signals in order to make a correct decision. We found that like conventional social networks, BrainNet allows Receivers to learn to trust the Sender who is more reliable, in this case, based solely on the information transmitted directly to their brains. Our results point the way to future brain-to-brain interfaces that enable cooperative problem solving by humans using a “social network” of connected brains.

The field of brain-computer interfaces is poised to advance from the traditional goal of controlling prosthetic devices using brain signals to combining neural decoding and encoding within a single neuroprosthetic device. Such a device acts as a "co-processor" for the brain, with applications ranging from inducing Hebbian plasticity for rehabilitation after brain injury to reanimating paralyzed limbs and enhancing memory. We review recent progress in simultaneous decoding and encoding for closed-loop control and plasticity induction. To address the challenge of multi-channel decoding and encoding, we introduce a unifying framework for developing brain co-processors based on artificial neural networks and deep learning. These "neural co-processors" can be used to jointly optimize cost functions with the nervous system to achieve desired behaviors ranging from targeted neuro-rehabilitation to augmentation of brain function., Comment: Invited submission to the journal Current Opinion in Neurobiology

Cortical stimulation through electrocorticographic (ECoG) electrodes is a potential method for providing sensory feedback in future prosthetic and rehabilitative applications. Here, we evaluate human subjects’ ability to continuously modulate their motor behavior based on feedback from direct surface stimulation of the somatosensory cortex. Subjects wore a dataglove that measured their hand aperture position and received one of three stimuli over the hand sensory cortex based on their current hand position as compared to a target aperture position. Using cortical stimulation feedback, subjects adjusted their hand aperture to move towards the target aperture region. One subject was able to achieve accuracies and R2 values well above chance (best performance: R2 = 0.93; accuracy = 0.76/1). Performance dropped during the catch trial (same stimulus independent of the position) to below chance levels, suggesting that the subject had been using the varied sensory feedback to modulate their motor behavior. To our knowledge, this study represents one of the first demonstrations of using direct cortical surface stimulation of the human sensory cortex to perform a motor task, and is a first step towards developing closed-loop human sensorimotor brain-computer interfaces.

Goal: To identify and overcome barriers to creating new neurotechnologies capable of restoring both motor and sensory function in individuals with neurological conditions. Methods: This report builds upon the outcomes of a joint workshop between the US National Science Foundation and the German Research Foundation on New Perspectives in Neuroengineering and Neurotechnology convened in Arlington, VA, USA, November 13–14, 2014. Results: The participants identified key technological challenges for recording and manipulating neural activity, decoding, and interpreting brain data in the presence of plasticity, and early considerations of ethical and social issues pertinent to the adoption of neurotechnologies. Conclusions: The envisaged progress in neuroengineering requires tightly integrated hardware and signal processing efforts, advances in understanding of physiological adaptations to closed-loop interactions with neural devices, and an open dialog with stakeholders and potential end-users of neurotechnology. Significance: The development of new neurotechnologies (e.g., bidirectional brain–computer interfaces) could significantly improve the quality of life of people living with the effects of brain or spinal cord injury, or other neurodegenerative diseases. Focused efforts aimed at overcoming the remaining barriers at the electrode tissue interface, developing implantable hardware with on-board computation, and refining stimulation methods to precisely activate neural tissue will advance both our understanding of brain function and our ability to treat currently intractable disorders of the nervous system.

International audience; In collective decision making, members of a group need to coordinate their actions in order to achieve a desirable outcome. When there is no direct communication between group members, one should decide based on inferring others' intentions from their actions. The inference of others' intentions is called "theory of mind" and can involve different levels of reasoning, from a single inference on a hidden variable to considering others partially or fully optimal and reasoning about their actions conditioned on one's own actions (levels of "theory of mind"). In this paper, we present a new Bayesian theory of collective decision making based on a simple yet most commonly observed behavior: conformity. We show that such a Bayesian framework allows one to achieve any level of theory of mind in collective decision making. The viability of our framework is demonstrated on two different experiments, a consensus task with 120 subjects and a volunteer's dilemma task with 29 subjects, each with multiple conditions.

Objective. Electrical stimulation of the human brain is commonly used for eliciting and inhibiting neural activity for clinical diagnostics, modifying abnormal neural circuit function for therapeutics, and interrogating cortical connectivity. However, recording electrical signals with concurrent stimulation results in dominant electrical artifacts that mask the neural signals of interest. Here we develop a method to reproducibly and robustly recover neural activity during concurrent stimulation. We concentrate on signal recovery across an array of electrodes without channel-wise fine-tuning of the algorithm. Our goal includes signal recovery with trains of stimulation pulses, since repeated, high-frequency pulses are often required to induce desired effects in both therapeutic and research domains. We have made all of our code and data publicly available. Approach. We developed an algorithm that automatically detects templates of artifacts across many channels of recording, creating a dictionary of learned templates using unsupervised clustering. The artifact template that best matches each individual artifact pulse is subtracted to recover the underlying activity. To assess the success of our method, we focus on whether it extracts physiologically interpretable signals from real recordings. Main results. We demonstrate our signal recovery approach on invasive electrophysiologic recordings from human subjects during stimulation. We show the recovery of meaningful neural signatures in both electrocorticographic (ECoG) arrays and deep brain stimulation (DBS) recordings. In addition, we compared cortical responses induced by the stimulation of primary somatosensory (S1) by natural peripheral touch, as well as motor cortex activity with and without concurrent S1 stimulation. Significance. Our work will enable future advances in neural engineering with simultaneous stimulation and recording.

Objective: The objective of this paper is to estimate the parameters of a multivariate autoregressive process from noisy multichannel data. Methods: Using a multivariate generalization of the Cadzow method, we propose a new method for estimating autoregressive parameters from noisy data: the nonlinear Cadzow method. Results: We show that our method outperforms existing multivariate methods such as higher order Yule–Walker method and Kalman EM method on simulated data. We apply our method to estimation of Granger causality from noisy data and again obtain superior results compared to previous methods. Finally, when applied to experimental local field potential data from monkey somatosensory and motor cortical areas, our method produces results consistent with cortical physiology. Conclusion: The proposed nonlinear Cadzow method outperforms existing methods in obtaining denoised estimates of multivariate autoregressive parameters. Significance: Since multichannel recordings have become commonplace in biomedical applications ranging from discovering functional connectivity in the brain to speech data analysis and these recordings are inevitably contaminated by measurement noise, we believe our method has the potential for significant impact.

Direct cortical stimulation (DCS) of primary somatosensory cortex (S1) could help restore sensation and provide task-relevant feedback in a neuroprosthesis. However, the psychophysics of S1 DCS is poorly studied, including any comparison to cutaneous haptic stimulation. We compare the response times to DCS of human hand somatosensory cortex through electrocorticographic grids with response times to haptic stimuli delivered to the hand in four subjects. We found that subjects respond significantly slower to S1 DCS than to natural, haptic stimuli for a range of DCS train durations. Median response times for haptic stimulation varied from 198 ms to 313 ms, while median responses to reliably perceived DCS ranged from 254 ms for one subject, all the way to 528 ms for another. We discern no significant impact of learning or habituation through the analysis of blocked trials, and find no significant impact of cortical stimulation train duration on response times. Our results provide a realistic set of expectations for latencies with somatosensory DCS feedback for future neuroprosthetic applications and motivate the study of neural mechanisms underlying human perception of somatosensation via DCS.

The human ventral temporal cortex has regions that are known to selectively process certain categories of visual inputs; they are specialized for the content (“faces,” “places,” “tools”) and not the form (“line,” “patch”) of the image being seen. In our study, human patients with implanted electrocorticography (ECoG) electrode arrays were shown sequences of simple face and house pictures. We quantified neuronal population activity, finding robust face-selective sites on the fusiform gyrus and house-selective sites on the lingual/parahippocampal gyri. The magnitude and timing of single trials were compared between novel (“house-face”) and repeated (“face-face”) stimulus-type responses. More than half of the category-selective sites showed significantly greater total activity for novel stimulus class. Approximately half of the face-selective sites (and none of the house-selective sites) showed significantly faster latency to peak (∼50 ms) for novel stimulus class. This establishes subregions within category-selective areas that are differentially tuned to novelty in sequential context, where novel stimuli are processed faster in some regions, and with increased activity in others.

We present here a browser-based application for visualizing patterns of connectivity in 3D stacked data matrices with large numbers of pairwise relations. Visualizing a connectivity matrix, looking for trends and patterns, and dynamically manipulating these values is a challenge for scientists from diverse fields, including neuroscience and genomics. In particular, high-dimensional neural data include those acquired via electroencephalography (EEG), electrocorticography (ECoG), magnetoencephalography (MEG), and functional MRI. Neural connectivity data contains multivariate attributes for each edge between different brain regions, which motivated our lightweight, open source, easy-to-use visualization tool for the exploration of these connectivity matrices to highlight connections of interest. Here we present a client-side, mobile-compatible visualization tool written entirely in HTML5/JavaScript that allows in-browser manipulation of user-defined files for exploration of brain connectivity. Visualizations can highlight different aspects of the data simultaneously across different dimensions. Input files are in JSON format, and custom Python scripts have been written to parse MATLAB or Python data files into JSON-loadable format. We demonstrate the analysis of connectivity data acquired via human ECoG recordings as a domain-specific implementation of our application. We envision applications for this interactive tool in fields seeking to visualize pairwise connectivity., 4 pages, IEEE NER 2017

Electrocorticography (ECoG) signals can provide high spatio-temporal resolution and high signal to noise ratio recordings of local neural activity from the surface of the brain. Previous studies have shown that broad-band, spatially focal, high-frequency increases in ECoG signals are highly correlated with movement and other cognitive tasks and can be volitionally modulated. However, significant additional information may be present in inter-electrode interactions, but adding additional higher order inter-electrode interactions can be impractical from a computational aspect, if not impossible.In this paper we present a new method of calculating high frequency interactions between electrodes called Short-Time Windowed Covariance (STWC) that builds on mathematical techniques currently used in neural signal analysis, along with an implementation that accelerates the algorithm by orders of magnitude by leveraging commodity, off-the-shelf graphics processing unit (GPU) hardware.Using the hardware-accelerated implementation of STWC, we identify many types of event-related inter-electrode interactions from human ECoG recordings on global and local scales that have not been identified by previous methods. Unique temporal patterns are observed for digit flexion in both low- (10mm spacing) and high-resolution (3mm spacing) electrode arrays.Covariance is a commonly used metric for identifying correlated signals, but the standard covariance calculations do not allow for temporally varying covariance. In contrast STWC allows and identifies event-driven changes in covariance without identifying spurious noise correlations.STWC can be used to identify event-related neural interactions whose high computational load is well suited to GPU capabilities.

The majority of subjects who attempt to learn control of a brain–computer interface (BCI) can do so with adequate training. Much like when one learns to type or ride a bicycle, BCI users report transitioning from a deliberate, cognitively focused mindset to near automatic control as training progresses. What are the neural correlates of this process of BCI skill acquisition? Seven subjects were implanted with electrocorticography (ECoG) electrodes and had multiple opportunities to practice a 1D BCI task. As subjects became proficient, strong initial task-related activation was followed by lessening of activation in prefrontal cortex, premotor cortex, and posterior parietal cortex, areas that have previously been implicated in the cognitive phase of motor sequence learning and abstract task learning. These results demonstrate that, although the use of a BCI only requires modulation of a local population of neurons, a distributed network of cortical areas is involved in the acquisition of BCI proficiency.

Identification of intended movement type and movement phase of hand grasp shaping are critical features for the control of volitional neuroprosthetics. We demonstrate that neural dynamics during visually-guided imagined grasp shaping can encode intended movement. We apply Procrustes analysis and LASSO regression to achieve 72% accuracy (chance = 25%) in distinguishing between visually-guided imagined grasp trajectories. Further, we can predict the stage of grasp shaping in the form of elapsed time from start of trial (R2=0.4). Our approach contributes to more accurate single-trial decoding of higher-level movement goals and the phase of grasping movements in individuals not trained with brain-computer interfaces. We also find that the overall time-varying trajectory structure of imagined movements tend to be consistent within individuals, and that transient trajectory deviations within trials return to the task-dependent trajectory mean. These overall findings may contribute to the further understanding of the cortical dynamics of human motor imagery., Comment: 4 pages, 6 figures, accepted to IEEE NER 2017 (8th International IEEE EMBS Conference on Neural Engineering)

We introduce Graph-Structured Sum-Product Networks (GraphSPNs), a probabilistic approach to structured prediction for problems where dependencies between latent variables are expressed in terms of arbitrary, dynamic graphs. While many approaches to structured prediction place strict constraints on the interactions between inferred variables, many real-world problems can be only characterized using complex graph structures of varying size, often contaminated with noise when obtained from real data. Here, we focus on one such problem in the domain of robotics. We demonstrate how GraphSPNs can be used to bolster inference about semantic, conceptual place descriptions using noisy topological relations discovered by a robot exploring large-scale office spaces. Through experiments, we show that GraphSPNs consistently outperform the traditional approach based on undirected graphical models, successfully disambiguating information in global semantic maps built from uncertain, noisy local evidence. We further exploit the probabilistic nature of the model to infer marginal distributions over semantic descriptions of as yet unexplored places and detect spatial environment configurations that are novel and incongruent with the known evidence., Comment: 9 pages, 8 figures. AAAI Conference on Artificial Intelligence (AAAI 2018)

Can the human brain learn to interpret inputs from a virtual world delivered directly through brain stimulation? We answer this question by describing the first demonstration of humans playing a computer game utilizing only direct brain stimulation and no other sensory inputs. The demonstration also provides the first instance of artificial sensory information, in this case depth, being delivered directly to the human brain through noninvasive methods. Our approach utilizes transcranial magnetic stimulation (TMS) of the human visual cortex to convey binary information about obstacles in a virtual maze. At certain intensities, TMS elicits visual percepts known as phosphenes, which transmits information to the subject about their current location within the maze. Using this computer-brain interface (CBI), five subjects successfully navigated an average of 92% of all the steps in a variety of virtual maze worlds. They also became more accurate in solving the task over time. These results suggest that humans can learn to utilize information delivered directly and noninvasively to their brains to solve tasks that cannot be solved using their natural senses, opening the door to human sensory augmentation and novel modes of human-computer interaction.

We propose a new probabilistic framework that allows mobile robots to autonomously learn deep, generative models of their environments that span multiple levels of abstraction. Unlike traditional approaches that combine engineered models for low-level features, geometry, and semantics, our approach leverages recent advances in Sum-Product Networks (SPNs) and deep learning to learn a single, universal model of the robot's spatial environment. Our model is fully probabilistic and generative, and represents a joint distribution over spatial information ranging from low-level geometry to semantic interpretations. Once learned, it is capable of solving a wide range of tasks: from semantic classification of places, uncertainty estimation, and novelty detection, to generation of place appearances based on semantic information and prediction of missing data in partial observations. Experiments on laser-range data from a mobile robot show that the proposed universal model obtains performance superior to state-of-the-art models fine-tuned to one specific task, such as Generative Adversarial Networks (GANs) or SVMs.

Neural correlates of movement planning onset and direction may be present in human electrocorticography in the signal dynamics of both motor and non-motor cortical regions. We use a three-stage model of jPCA reduced-rank hidden Markov model (jPCA-RR-HMM), regularized shrunken-centroid discriminant analysis (RDA), and LASSO regression to extract direction-sensitive planning information and movement onset in an upper-limb 3D isometric force task in a human subject. This mode achieves a relatively high true positive force-onset prediction rate of 60% within 250ms, and an above-chance 36% accuracy (17% chance) in predicting one of six planned 3D directions of isometric force using pre-movement signals. We also find direction-distinguishing information up to 400ms before force onset in the pre-movement signals, captured by electrodes placed over the limb-ipsilateral dorsal premotor regions. This approach can contribute to more accurate decoding of higher-level movement goals, at earlier timescales, and inform sensor placement. Our results also contribute to further understanding of the spatiotemporal features of human motor planning., 4 pages, 3 figures, accepted to EMBC 2016 (38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society)

Images usually contain multiple objects that are semantically related to one another. Mapping from low-level visual features to mutually dependent high-level semantics can be formulated as a structured prediction problem. Current statistical models for structured prediction make simplifying assumptions about the underlying output graph structure, such as assuming a low-order Markov chain, because exact inference becomes intractable as the tree-width of the underlying graph increases. Approximate inference algorithms, on the other hand, force one to trade off representational power with computational efficiency. In this paper, we present large margin sigmoid belief networks (LMSBNs) for structured prediction in images. LMSBNs allow a very fast inference algorithm for arbitrary graph structures that runs in polynomial time with high probability. This probability is data-distribution dependent and is maximized in learning. The new approach overcomes the representation-efficiency trade-off in previous models and allows fast structured prediction with complicated graph structures. We present results from applying a fully connected model to semantic image annotation, image retrieval and optical character recognition (OCR) problems, and demonstrate that the proposed approach can yield significant performance gains over current state-of-the-art methods.