Search

In perceptual decisions, subjects infer hidden states of the environment based on noisy sensory information. Here we show that both choice and its associated confidence are explained by a Bayesian framework based on partially observable Markov decision processes (POMDPs). We test our model on monkeys performing a direction-discrimination task with post-decision wagering, demonstrating that the model explains objective accuracy and predicts subjective confidence. Further, we show that the model replicates well-known discrepancies of confidence and accuracy, including the hard-easy effect, opposing effects of stimulus variability on confidence and accuracy, dependence of confidence ratings on simultaneous or sequential reports of choice and confidence, apparent difference between choice and confidence sensitivity, and seemingly disproportionate influence of choice-congruent evidence on confidence. These effects may not be signatures of sub-optimal inference or discrepant computational processes for choice and confidence. Rather, they arise in Bayesian inference with incomplete knowledge of the environment., A Bayesian framework based on partially observable Markov decision processes (POMDPs) not only predicts subjects’ confidence in a perceptual decision making task but also explains well-known discrepancies between confidence and choice accuracy as arising from incomplete knowledge of the environment.

Objective. Recent advances in neural decoding have accelerated the development of brain–computer interfaces aimed at assisting users with everyday tasks such as speaking, walking, and manipulating objects. However, current approaches for training neural decoders commonly require large quantities of labeled data, which can be laborious or infeasible to obtain in real-world settings. Alternatively, self-supervised models that share self-generated pseudo-labels between two data streams have shown exceptional performance on unlabeled audio and video data, but it remains unclear how well they extend to neural decoding. Approach. We learn neural decoders without labels by leveraging multiple simultaneously recorded data streams, including neural, kinematic, and physiological signals. Specifically, we apply cross-modal, self-supervised deep clustering to train decoders that can classify movements from brain recordings. After training, we then isolate the decoders for each input data stream and compare the accuracy of decoders trained using cross-modal deep clustering against supervised and unimodal, self-supervised models. Main results. We find that sharing pseudo-labels between two data streams during training substantially increases decoding performance compared to unimodal, self-supervised models, with accuracies approaching those of supervised decoders trained on labeled data. Next, we extend cross-modal decoder training to three or more modalities, achieving state-of-the-art neural decoding accuracy that matches or slightly exceeds the performance of supervised models. Significance. We demonstrate that cross-modal, self-supervised decoding can be applied to train neural decoders when few or no labels are available and extend the cross-modal framework to share information among three or more data streams, further improving self-supervised training.

The original predictive coding model of Rao & Ballard [1] focused on spatial prediction to explain spatial receptive fields and contextual effects in the visual cortex. Here, we introduce a new dynamic predictive coding model that achieves spatiotemporal prediction of complex natural image sequences using time-varying transition matrices. We overcome the limitations of static linear transition models (as in, e.g., Kalman filters) using a hypernetwork to adjust the transition matrix dynamically for every time step, allowing the model to predict using a time-varying mixture of possible transition dynamics. We developed a single level model with recurrent modulation of transition weights by a hypernetwork and a two-level hierarchical model with top-down modulation based on a hypernetwork. At each time step, the model predicts the next input and estimates a sparse neural code by minimizing prediction error. When exposed to natural movies, the model learned localized, oriented spatial filters as well as both separable and inseparable (direction-selective) space-time receptive fields at the first level, similar to those found in the primary visual cortex (V1). Longer timescale responses and stability at the second level also emerged naturally from minimizing prediction errors for the first level dynamics. Our results suggest that the multiscale temporal response properties of cortical neurons could be the result of the cortex learning a hierarchical generative model of the visual world with higher order areas predicting the transition dynamics of lower order areas.

Direct cortical stimulation (DCS) has been used extensively as a tool in human neuroscience research and clinically in neurosurgery. Implanted electrocorticography electrodes can deliver electrical stimulation to the cortical surface, eliciting or inhibiting neural activity as an experimental manipulation, or to probe neural tissue function during epilepsy and tumor resection surgeries. Current applications of DCS pave the way for new clinical translation to restore sensation to those who have lost it through spinal-cord injury or stroke. DCS of specific sensory cortical regions in the human brain elicits sensory percepts in specific regions of the body and recent research has shown that DCS can elicit meaningful somatosensory percepts, which may be used for neuroprosthetic feedback and integrated into tasks. In the future, further research using more finely spaced electrodes, tailored patterns of stimulation, and longer term studies will shed light on the intricacies of sensory processing in humans as well as bring these neuroprosthetic applications closer to widespread clinical reality.

Background Recent technological advances in brain recording and machine learning algorithms are enabling the study of neural activity underlying spontaneous human behaviors, beyond the confines of cued, repeated trials. However, analyzing such unstructured data lacking a priori experimental design remains a significant challenge, especially when the data is multi-modal and long-term. New method Here we describe an automated, behavior-first approach for analyzing simultaneously recorded long-term, naturalistic electrocorticography (ECoG) and behavior video data. We identify and characterize spontaneous human upper-limb movements by combining computer vision, discrete latent-variable modeling, and string pattern-matching on the video. Results Our pipeline discovers and annotates over 40,000 instances of naturalistic arm movements in long term (7–9 day) behavioral videos, across 12 subjects. Analysis of the simultaneously recorded brain data reveals neural signatures of movement that corroborate previous findings. Our pipeline produces large training datasets for brain–computer interfacing applications, and we show decoding results from a movement initiation detection task. Comparison with existing methods Spontaneous movements capture real-world neural and behavior variability that is missing from traditional cued tasks. Building beyond window-based movement detection metrics, our unsupervised discretization scheme produces a queryable pose representation, allowing localization of movements with finer temporal resolution. Conclusions Our work addresses the unique analytic challenges of studying naturalistic human behaviors and contributes methods that may generalize to other neural recording modalities beyond ECoG. We publish our curated dataset and believe that it will be a valuable resource for future studies of naturalistic movements.

ObjectiveAdvances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can robustly train on pooled, multi-participant data and generalize to new participants.ApproachWe introduce a new decoder, HTNet, which uses a convolutional neural network with two innovations: (1) a Hilbert transform that computes spectral power at data-driven frequencies and (2) a layer that projects electrode-level data onto predefined brain regions. The projection layer critically enables applications with intracranial electrocorticography (ECoG), where electrode locations are not standardized and vary widely across participants. We trained HTNet to decode arm movements using pooled ECoG data from 11 of 12 participants and tested performance on unseen ECoG or electroencephalography (EEG) participants; these pretrained models were also subsequently fine-tuned to each test participant.Main resultsHTNet outperformed state-of-the-art decoders when tested on unseen participants, even when a different recording modality was used. By fine-tuning these generalized HTNet decoders, we achieved performance approaching the best tailored decoders with as few as 50 ECoG or 20 EEG events. We were also able to interpret HTNet’s trained weights and demonstrate its ability to extract physiologically-relevant features.SignificanceBy generalizing to new participants and recording modalities, robustly handling variations in electrode placement, and allowing participant-specific fine-tuning with minimal data, HTNet is applicable across a broader range of neural decoding applications compared to current state-of-the-art decoders.

Increasingly, neuroimaging researchers are exploring the use of real-time functional magnetic resonance imaging (rt-fMRI) as a way to access a participant’s ongoing brain function throughout a scan. This approach presents novel and exciting experimental applications ranging from monitoring data quality in real time, to delivering neurofeedback from a region of interest, to dynamically controlling experimental flow, or interfacing with remote devices. Yet, for those interested in adopting this method, the existing software options are few and limited in application. This presents a barrier for new users, as well as hinders existing users from refining techniques and methods. Here we introduce a free, open-source rt-fMRI package, the Pyneal toolkit, designed to address this limitation. The Pyneal toolkit is python-based software that offers a flexible and user friendly framework for rt-fMRI, is compatible with all three major scanner manufacturers (GE, Siemens, Phillips), and, critically, allows fully customized analysis pipelines. In this article, we provide a detailed overview of the architecture, describe how to set up and run the Pyneal toolkit during an experimental session, offer tutorials with scan data that demonstrate how data flows through the Pyneal toolkit with example analyses, and highlight the advantages that the Pyneal toolkit offers to the neuroimaging community.

The promise of robots assisting humans in everyday tasks has led to a variety of research questions and challenges in human-robot collaboration. Here, we address the question of whether and when a ...

The ability to track odor plumes in dynamic environments is critical for flying insects following attractive odors to localize food or mates. This remarkable tracking behavior requires multimodal i...

The field of brain-computer interfaces is poised to advance from the traditional goal of controlling prosthetic devices using brain signals to combining neural decoding and encoding within a single neuroprosthetic device. Such a device acts as a "co-processor" for the brain, with applications ranging from inducing Hebbian plasticity for rehabilitation after brain injury to reanimating paralyzed limbs and enhancing memory. We review recent progress in simultaneous decoding and encoding for closed-loop control and plasticity induction. To address the challenge of multi-channel decoding and encoding, we introduce a unifying framework for developing brain co-processors based on artificial neural networks and deep learning. These "neural co-processors" can be used to jointly optimize cost functions with the nervous system to achieve desired behaviors ranging from targeted neuro-rehabilitation to augmentation of brain function., Comment: Invited submission to the journal Current Opinion in Neurobiology

Cortical stimulation through electrocorticographic (ECoG) electrodes is a potential method for providing sensory feedback in future prosthetic and rehabilitative applications. Here, we evaluate human subjects’ ability to continuously modulate their motor behavior based on feedback from direct surface stimulation of the somatosensory cortex. Subjects wore a dataglove that measured their hand aperture position and received one of three stimuli over the hand sensory cortex based on their current hand position as compared to a target aperture position. Using cortical stimulation feedback, subjects adjusted their hand aperture to move towards the target aperture region. One subject was able to achieve accuracies and R2 values well above chance (best performance: R2 = 0.93; accuracy = 0.76/1). Performance dropped during the catch trial (same stimulus independent of the position) to below chance levels, suggesting that the subject had been using the varied sensory feedback to modulate their motor behavior. To our knowledge, this study represents one of the first demonstrations of using direct cortical surface stimulation of the human sensory cortex to perform a motor task, and is a first step towards developing closed-loop human sensorimotor brain-computer interfaces.

Objective: The objective of this paper is to estimate the parameters of a multivariate autoregressive process from noisy multichannel data. Methods: Using a multivariate generalization of the Cadzow method, we propose a new method for estimating autoregressive parameters from noisy data: the nonlinear Cadzow method. Results: We show that our method outperforms existing multivariate methods such as higher order Yule–Walker method and Kalman EM method on simulated data. We apply our method to estimation of Granger causality from noisy data and again obtain superior results compared to previous methods. Finally, when applied to experimental local field potential data from monkey somatosensory and motor cortical areas, our method produces results consistent with cortical physiology. Conclusion: The proposed nonlinear Cadzow method outperforms existing methods in obtaining denoised estimates of multivariate autoregressive parameters. Significance: Since multichannel recordings have become commonplace in biomedical applications ranging from discovering functional connectivity in the brain to speech data analysis and these recordings are inevitably contaminated by measurement noise, we believe our method has the potential for significant impact.

Determining hand grasping forces and kinetic direction and magnitude are important for the development of dexterous neural prosthetics. While many earlier decoding methods have successfully predicted upper-limb kinematic output from cortical signals in the sensorimotor parietal and premotor regions, the full extent of the regions that characterize kinetic behavior is unknown. In this study, we found that neural dynamics based on electrocorticography (ECoG) recorded from the human brain surface can successfully encode structured and unstructured grasping and arm kinetic output. We found a time-averaged linear relationship between gamma band spectral ECoG power and sustained grasping force output with visual feedback. In the kinetic grasping task, we obtained classification accuracy of 47% (25% = chance) using quadratic discriminant analysis. Additionally, we also found a similar linear relationship between spectral power and cued isometric force generation, without concurrent visual feedback, in arm force application; this feature could also classify arm force output categories with an accuracy of 41% (33% = chance). In addition, we applied quadratic discriminant analysis with top 12 principal components to attain approximately 26% accuracy (chance is 16%) in determining arm kinetic force direction. We found that the gamma band spectral power from both experiments in posterior parietal cortex, as well as projections of high gamma variability along the top principal components, can be successfully used to predict sustained kinetic outputs and to explain both structured and unstructured force output variability. Our findings contribute to a deeper understanding of neural dynamics for fine kinetic behavior.

Direct cortical stimulation (DCS) of primary somatosensory cortex (S1) could help restore sensation and provide task-relevant feedback in a neuroprosthesis. However, the psychophysics of S1 DCS is poorly studied, including any comparison to cutaneous haptic stimulation. We compare the response times to DCS of human hand somatosensory cortex through electrocorticographic grids with response times to haptic stimuli delivered to the hand in four subjects. We found that subjects respond significantly slower to S1 DCS than to natural, haptic stimuli for a range of DCS train durations. Median response times for haptic stimulation varied from 198 ms to 313 ms, while median responses to reliably perceived DCS ranged from 254 ms for one subject, all the way to 528 ms for another. We discern no significant impact of learning or habituation through the analysis of blocked trials, and find no significant impact of cortical stimulation train duration on response times. Our results provide a realistic set of expectations for latencies with somatosensory DCS feedback for future neuroprosthetic applications and motivate the study of neural mechanisms underlying human perception of somatosensation via DCS.

We present here a browser-based application for visualizing patterns of connectivity in 3D stacked data matrices with large numbers of pairwise relations. Visualizing a connectivity matrix, looking for trends and patterns, and dynamically manipulating these values is a challenge for scientists from diverse fields, including neuroscience and genomics. In particular, high-dimensional neural data include those acquired via electroencephalography (EEG), electrocorticography (ECoG), magnetoencephalography (MEG), and functional MRI. Neural connectivity data contains multivariate attributes for each edge between different brain regions, which motivated our lightweight, open source, easy-to-use visualization tool for the exploration of these connectivity matrices to highlight connections of interest. Here we present a client-side, mobile-compatible visualization tool written entirely in HTML5/JavaScript that allows in-browser manipulation of user-defined files for exploration of brain connectivity. Visualizations can highlight different aspects of the data simultaneously across different dimensions. Input files are in JSON format, and custom Python scripts have been written to parse MATLAB or Python data files into JSON-loadable format. We demonstrate the analysis of connectivity data acquired via human ECoG recordings as a domain-specific implementation of our application. We envision applications for this interactive tool in fields seeking to visualize pairwise connectivity., 4 pages, IEEE NER 2017

Electrocorticography (ECoG) signals can provide high spatio-temporal resolution and high signal to noise ratio recordings of local neural activity from the surface of the brain. Previous studies have shown that broad-band, spatially focal, high-frequency increases in ECoG signals are highly correlated with movement and other cognitive tasks and can be volitionally modulated. However, significant additional information may be present in inter-electrode interactions, but adding additional higher order inter-electrode interactions can be impractical from a computational aspect, if not impossible.In this paper we present a new method of calculating high frequency interactions between electrodes called Short-Time Windowed Covariance (STWC) that builds on mathematical techniques currently used in neural signal analysis, along with an implementation that accelerates the algorithm by orders of magnitude by leveraging commodity, off-the-shelf graphics processing unit (GPU) hardware.Using the hardware-accelerated implementation of STWC, we identify many types of event-related inter-electrode interactions from human ECoG recordings on global and local scales that have not been identified by previous methods. Unique temporal patterns are observed for digit flexion in both low- (10mm spacing) and high-resolution (3mm spacing) electrode arrays.Covariance is a commonly used metric for identifying correlated signals, but the standard covariance calculations do not allow for temporally varying covariance. In contrast STWC allows and identifies event-driven changes in covariance without identifying spurious noise correlations.STWC can be used to identify event-related neural interactions whose high computational load is well suited to GPU capabilities.

Identification of intended movement type and movement phase of hand grasp shaping are critical features for the control of volitional neuroprosthetics. We demonstrate that neural dynamics during visually-guided imagined grasp shaping can encode intended movement. We apply Procrustes analysis and LASSO regression to achieve 72% accuracy (chance = 25%) in distinguishing between visually-guided imagined grasp trajectories. Further, we can predict the stage of grasp shaping in the form of elapsed time from start of trial (R2=0.4). Our approach contributes to more accurate single-trial decoding of higher-level movement goals and the phase of grasping movements in individuals not trained with brain-computer interfaces. We also find that the overall time-varying trajectory structure of imagined movements tend to be consistent within individuals, and that transient trajectory deviations within trials return to the task-dependent trajectory mean. These overall findings may contribute to the further understanding of the cortical dynamics of human motor imagery., Comment: 4 pages, 6 figures, accepted to IEEE NER 2017 (8th International IEEE EMBS Conference on Neural Engineering)

Can the human brain learn to interpret inputs from a virtual world delivered directly through brain stimulation? We answer this question by describing the first demonstration of humans playing a computer game utilizing only direct brain stimulation and no other sensory inputs. The demonstration also provides the first instance of artificial sensory information, in this case depth, being delivered directly to the human brain through noninvasive methods. Our approach utilizes transcranial magnetic stimulation (TMS) of the human visual cortex to convey binary information about obstacles in a virtual maze. At certain intensities, TMS elicits visual percepts known as phosphenes, which transmits information to the subject about their current location within the maze. Using this computer-brain interface (CBI), five subjects successfully navigated an average of 92% of all the steps in a variety of virtual maze worlds. They also became more accurate in solving the task over time. These results suggest that humans can learn to utilize information delivered directly and noninvasively to their brains to solve tasks that cannot be solved using their natural senses, opening the door to human sensory augmentation and novel modes of human-computer interaction.

We propose a new probabilistic framework that allows mobile robots to autonomously learn deep, generative models of their environments that span multiple levels of abstraction. Unlike traditional approaches that combine engineered models for low-level features, geometry, and semantics, our approach leverages recent advances in Sum-Product Networks (SPNs) and deep learning to learn a single, universal model of the robot's spatial environment. Our model is fully probabilistic and generative, and represents a joint distribution over spatial information ranging from low-level geometry to semantic interpretations. Once learned, it is capable of solving a wide range of tasks: from semantic classification of places, uncertainty estimation, and novelty detection, to generation of place appearances based on semantic information and prediction of missing data in partial observations. Experiments on laser-range data from a mobile robot show that the proposed universal model obtains performance superior to state-of-the-art models fine-tuned to one specific task, such as Generative Adversarial Networks (GANs) or SVMs.

Autonomous mobile robots will soon become ubiquitous in human-populated environments. Besides their typical applications in fetching, delivery, or escorting, such robots present the opportunity to assist human users in their daily tasks by gathering and reporting up-to-date knowledge about the environment. In this paper, we explore this use case and present an end-to-end framework that enables a mobile robot to answer natural language questions about the state of a large-scale, dynamic environment asked by the inhabitants of that environment. The system parses the question and estimates an initial viewpoint that is likely to contain information for answering the question based on prior environment knowledge. Then, it autonomously navigates towards the viewpoint while dynamically adapting to changes and new information. The output of the system is an image of the most relevant part of the environment that allows the user to obtain an answer to their question. We additionally demonstrate the benefits of a continuously operating information gathering robot by showing how the system can answer retrospective questions about the past state of the world using incidentally recorded sensory data. We evaluate our approach with a custom mobile robot deployed in a university building, with questions collected from occupants of the building. We demonstrate our system's ability to respond to these questions in different environmental conditions.

Motivated by the growing evidence for Bayesian computation in the brain, we show how a two-layer recurrent network of Poisson neurons can perform both approximate Bayesian inference and learning for any hidden Markov model. The lower-layer sensory neurons receive noisy measurements of hidden world states. The higher-layer neurons infer a posterior distribution over world states via Bayesian inference from inputs generated by sensory neurons. We demonstrate how such a neuronal network with synaptic plasticity can implement a form of Bayesian inference similar to Monte Carlo methods such as particle filtering. Each spike in a higher-layer neuron represents a sample of a particular hidden world state. The spiking activity across the neural population approximates the posterior distribution over hidden states. In this model, variability in spiking is regarded not as a nuisance but as an integral feature that provides the variability necessary for sampling during inference. We demonstrate how the network can learn the likelihood model, as well as the transition probabilities underlying the dynamics, using a Hebbian learning rule. We present results illustrating the ability of the network to perform inference and learning for arbitrary hidden Markov models.

Neural correlates of movement planning onset and direction may be present in human electrocorticography in the signal dynamics of both motor and non-motor cortical regions. We use a three-stage model of jPCA reduced-rank hidden Markov model (jPCA-RR-HMM), regularized shrunken-centroid discriminant analysis (RDA), and LASSO regression to extract direction-sensitive planning information and movement onset in an upper-limb 3D isometric force task in a human subject. This mode achieves a relatively high true positive force-onset prediction rate of 60% within 250ms, and an above-chance 36% accuracy (17% chance) in predicting one of six planned 3D directions of isometric force using pre-movement signals. We also find direction-distinguishing information up to 400ms before force onset in the pre-movement signals, captured by electrodes placed over the limb-ipsilateral dorsal premotor regions. This approach can contribute to more accurate decoding of higher-level movement goals, at earlier timescales, and inform sensor placement. Our results also contribute to further understanding of the spatiotemporal features of human motor planning., 4 pages, 3 figures, accepted to EMBC 2016 (38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society)

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Most ongoing efforts have focused on training decoders on specific, stereotyped tasks in laboratory settings. Implementing brain-computer interfaces (BCIs) in natural settings requires adaptive strategies and scalable algorithms that require minimal supervision. Here we propose an unsupervised approach to decoding neural states from human brain recordings acquired in a naturalistic context. We demonstrate our approach on continuous long-term electrocorticographic (ECoG) data recorded over many days from the brain surface of subjects in a hospital room, with simultaneous audio and video recordings. We first discovered clusters in high-dimensional ECoG recordings and then annotated coherent clusters using speech and movement labels extracted automatically from audio and video recordings. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Our results show that our unsupervised approach can discover distinct behaviors from ECoG data, including moving, speaking and resting. We verify the accuracy of our approach by comparing to manual annotations. Projecting the discovered cluster centers back onto the brain, this technique opens the door to automated functional brain mapping in natural settings.

Generative models for vision and pattern recognition have been overshadowed in recent years by powerful non-parametric discriminative models. These discriminative models can learn arbitrary decision boundaries between classes and have proved very effective in classification and detection problems. However, unlike generative models, they do not lend themselves naturally to more general vision tasks such as rendering novel images, de-noising, and in-painting. In this paper we introduce Complementary Kernel Density Estimation (CKDE), a new generative model that adopts many of the features of non-parametric discriminative models: (1) CKDE allows complex decision surfaces and arbitrary class conditional distributions to be learned, (2) it is easy to train because the log likelihood of the model is concave, so it has no local maxima, and (3) one can train its class conditional distributions jointly to share information among the different classes. We first demonstrate that CKDE is more accurate in benchmark classification tasks than a purely discriminative method such as the SVM. We then show that the posterior probability of class labels is more accurately estimated than kernelized logistic regression. Our other results demonstrate that partial images can be accurately classified by marginalizing unobserved pixels from the class conditional distributions, and missing parts of the image can be painted in using the learned generative model.

Probabilistic methods for analyzing sequences are providing new insights into the 4,000-year-old undeciphered script of the Indus civilization.

In this paper, we provide the first demonstration that a humanoid robot can learn to walk directly by imitating a human gait obtained from motion capture (mocap) data without any prior information of its dynamics model. Programming a humanoid robot to perform an action (such as walking) that takes into account the robot's complex dynamics is a challenging problem. Traditional approaches typically require highly accurate prior knowledge of the robot's dynamics and environment in order to devise complex (and often brittle) control algorithms for generating a stable dynamic motion. Training using human mocap is an intuitive and flexible approach to programming a robot, but direct usage of mocap data usually results in dynamically unstable motion. Furthermore, optimization using high-dimensional mocap data in the humanoid full-body joint space is typically intractable. We propose a new approach to tractable imitation-based learning in humanoids without a robot's dynamic model. We represent kinematic informati...

Although no historical information exists about the Indus civilization (flourished ca . 2600–1900 B.C.), archaeologists have uncovered about 3,800 short samples of a script that was used throughout the civilization. The script remains undeciphered, despite a large number of attempts and claimed decipherments over the past 80 years. Here, we propose the use of probabilistic models to analyze the structure of the Indus script. The goal is to reveal, through probabilistic analysis, syntactic patterns that could point the way to eventual decipherment. We illustrate the approach using a simple Markov chain model to capture sequential dependencies between signs in the Indus script. The trained model allows new sample texts to be generated, revealing recurring patterns of signs that could potentially form functional subunits of a possible underlying language. The model also provides a quantitative way of testing whether a particular string belongs to the putative language as captured by the Markov model. Application of this test to Indus seals found in Mesopotamia and other sites in West Asia reveals that the script may have been used to express different content in these regions. Finally, we show how missing, ambiguous, or unreadable signs on damaged objects can be filled in with most likely predictions from the model. Taken together, our results indicate that the Indus script exhibits rich synactic structure and the ability to represent diverse content. both of which are suggestive of a linguistic writing system rather than a nonlinguistic symbol system.

We describe a brain-computer interface for controlling a humanoid robot directly using brain signals obtained non-invasively from the scalp through electroencephalography (EEG). EEG has previously been used for tasks such as controlling a cursor and spelling a word, but it has been regarded as an unlikely candidate for more complex forms of control owing to its low signal-to-noise ratio. Here we show that by leveraging advances in robotics, an interface based on EEG can be used to command a partially autonomous humanoid robot to perform complex tasks such as walking to specific locations and picking up desired objects. Visual feedback from the robot's cameras allows the user to select arbitrary objects in the environment for pick-up and transport to chosen locations. Results from a study involving nine users indicate that a command for the robot can be selected from four possible choices in 5 s with 95% accuracy. Our results demonstrate that an EEG-based brain-computer interface can be used for sophisticated robotic interaction with the environment, involving not only navigation as in previous applications but also manipulation and transport of objects.

Electrocorticographic spectral changes during movement show a behavioral inflection in the classic gamma band (30-70 Hz). We quantify this inflection and demonstrate that it limits classification accuracy. We call for the designation of a functionally defined band above it, which we denote the chi-band.

We demonstrate the feasibility of real-time cortical mapping from arrays of subdural electrodes using the electrocorticographic signal power in the higher spectral frequencies (76-200 Hz, or "chi-index"). Hand area was mapped offline in eight individuals using brief baseline and hand-movement measurements. In one patient, hand sensorimotor cortex was identified online during a handshake. We propose that this high-frequency component of the electrocorticogram provides a generic, reliable, clinically useful correlate of local cortical function.

Several algorithms and models have recently been proposed for imitation learning in humans and robots. However, few proposals offer a framework for imitation learning in noisy stochastic environments where the imitator must learn and act under real-time performance constraints. We present a novel probabilistic framework for imitation learning in stochastic environments with unreliable sensors. Bayesian algorithms, based on Meltzoff and Moore's AIM hypothesis for action imitation, implement the core of an imitation learning framework. Our algorithms are computationally efficient, allowing real-time learning and imitation in an active stereo vision robotic head and on a humanoid robot. We present simulated and real-world robotics results demonstrating the viability of our approach. We conclude by advocating a research agenda that promotes interaction between cognitive and robotic studies of imitation.

Medically refractory epilepsy accounts for more than 30% of the epilepsy population. Scalp EEG electrodes have limited ability to localize seizure onset from deep structures and implantation of subdural electrodes with long term monitoring provides additional information. Apart from clinical application, this patient population provides a unique opportunity for acquiring electrocorticography data in research paradigms. We present a method for rapid localization of electrodes using lateral and anterior-posterior X-rays. Skull landmarks and proportions are used for co-registration with the standardized Talairach coordinate system. This MATLAB-based "Location on Cortex" (LOC) package facilitates rapid visualization of clinical and experimental data in a user-friendly manner.

Advances in mobile robotics have enabled robots that can autonomously operate in human-populated environments. Although primary tasks for such robots might be fetching, delivery, or escorting, they present an untapped potential as information gathering agents that can answer questions for the community of co-inhabitants. In this paper, we seek to better understand requirements for such information gathering robots (InfoBots) from the perspective of the user requesting the information. We present findings from two studies: (i) a user survey conducted in two office buildings and (ii) a 4-day long deployment in one of the buildings, during which inhabitants of the building could ask questions to an InfoBot through a web-based interface. These studies allow us to characterize the types of information that InfoBots can provide for their users.

Robot Programming by Demonstration (PbD) allows users to program a robot by demonstrating the desired behavior. Providing these demonstrations typically involves moving the robot through a sequence of states, often by physically manipulating it. This requires users to be co-located with the robot and have the physical ability to manipulate it. In this paper, we present a natural language based interface for PbD that removes these requirements and enables hands-free programming. We focus on programming object manipulation actions—our key insight is that such actions can be decomposed into known types of manipulator movements that are naturally described using spatial language; e.g., object reference expressions and prepositions. Our method takes a natural language command and the current world state to infer the intended movement command and its parametrization. We implement this method on a two-armed mobile manipulator and demonstrate the different types of manipulation actions that can be programmed with it. We compare it to a kinesthetic PbD interface and we demonstrate our method's ability to deal with incomplete language.

Autonomous mobile robots equipped with a number of sensors will soon be ubiquitous in human populated environments. In this paper we present an initial exploration into the potential of using such robots for information gathering. We present findings from a formative user survey and a 4-day long Wizard-of-Oz deployment of a robot that answers questions such as "Is there free food on the kitchen table?" Our studies allow us to characterize the types of information that InfoBots might be most useful for.

The responses of neurons in cortical areas V2 and V4 can be significantly modulated by attention to particular locations within an input image. We show that such effects emerge naturally when perception is viewed as a probabilistic inference process governed by Bayesian principles and implemented in hierarchical cortical networks. The proposed model can explain a rich variety of attention-related responses in cortical area V4 including multiplicative modulation of tuning curves, restoration of neural responses in the presence of distracting stimuli, and influence of attention on neighboring unattended locations. Our results suggest a new interpretation of attention as a cortical mechanism for reducing perceptual uncertainty by combining top-down task-relevant information with bottom-up sensory inputs in a probabilistic manner.

Short-term synaptic plasticity has been proposed as a way for cortical neurons to process temporal information. We present a model network that uses short-term plasticity to implement a temporal clustering algorithm. The model's facilitory synapses learn temporal signals drawn from mixtures of nonlinear processes. Units in the model correspond to populations of cortical pyramidal cells arranged in columns; each column consists of neurons with similar spatiotemporal receptive fields. Clustering is based on mutual inhibition similar to Kohonen's SOMs. A generalized expectation maximization (GEM) algorithm, guaranteed to increase model likelihood with each iteration, learns the synaptic parameters.

Recent experiments have shown that neocortical synapses exhibit both short-term plasticity and spike-timing dependent long-term plasticity. It has been suggested that changes in short-term plasticity are mediated by a redistribution of synaptic efficacy. Here we propose a simple model of the interaction between spike-timing dependent plasticity and short-term plasticity. We show that the model captures the synaptic behavior seen in experiments on redistribution of synaptic efficacy. Results from our simulations suggest that spike-timing dependent redistribution of synaptic efficacy offers neocortical neurons a potentially powerful mechanism for learning spatiotemporal patterns in the input stream.

Complex cells in primary visual cortex exhibit highly nonlinear receptive field properties such as phase-invariant direction selectivity and antagonistic interactions between individually excitatory stimuli. Traditional models assume that these properties are governed by the outputs of antecedent simple cells, but these models are at odds with studies showing that complex cells may receive direct inputs from the lateral geniculate nucleus (LGN) or can be driven by stimuli that fail to activate simple cells. Using a biophysically detailed model of recurrently connected cortical neurons, we show that complex cell-like direction selectivity may emerge without antecedent simple cell inputs, as a consequence of spike-timing dependent synaptic plasticity during visual development. The directionally-selective receptive fields of model neurons, as determined by reverse correlation and 2-bar interaction maps, were similar to those obtained from complex cells in awake monkey primary visual cortex. These results suggest a new interpretation of complex cells as integral components of an adaptive cortical circuit for motion detection and prediction.

A fundamental challenge in robotics today is building robots that can learn new skills by observing humans and imitating human actions. We propose a new Bayesian approach to robotic learning by imitation inspired by the developmental hypothesis that children use self-experience to bootstrap the process of intention recognition and goal-based imitation. Our approach allows an autonomous agent to: (i) learn probabilistic models of actions through self-discovery and experience, (ii) utilize these learned models for inferring the goals of human actions, and (iii) perform goal-based imitation for robotic learning and human-robot collaboration. Such an approach allows a robot to leverage its increasing repertoire of learned behaviors to interpret increasingly complex human actions and use the inferred goals for imitation, even when the robot has very different actuators from humans. We demonstrate our approach using two different scenarios: (i) a simulated robot that learns human-like gaze following behavior, and (ii) a robot that learns to imitate human actions in a tabletop organization task. In both cases, the agent learns a probabilistic model of its own actions, and uses this model for goal inference and goal-based imitation. We also show that the robotic agent can use its probabilistic model to seek human assistance when it recognizes that its inferred actions are too uncertain, risky, or impossible to perform, thereby opening the door to human-robot collaboration.

Human visual processing is of such complexity that, despite decades of focused research, many basic questions remain unanswered. Although we know that the inferotemporal cortex is a key region in object recognition, we don’t fully understand its physiologic role in brain function, nor do we have the full set of tools to explore this question. Here we show that electrical potentials from the surface of the human brain contain enough information to decode a subject’s perceptual state accurately, and with fine temporal precision. Electrocorticographic (ECoG) arrays were placed over the inferotemporal cortical areas of seven subjects. Pictures of faces and houses were quickly presented while each subject performed a simple visual task. Results showed that two well-known types of brain signals—event-averaged broadband power and event-averaged raw potential—can independently or together be used to classify the presented image. When applied to continuously recorded brain activity, our decoding technique could accurately predict whether each stimulus was a face, house, or neither, with ~20 ms timing error. These results provide a roadmap for improved brain-computer interfacing tools to help neurosurgeons, research scientists, engineers, and, ultimately, patients.

Visual cognition depends critically on the moment-to-moment orientation of gaze. To change the gaze to a new location in space, that location must be computed and used by the oculomotor system. One of the most common sources of information for this computation is the visual appearance of an object. A crucial question is: How is the appearance information contained in the photometric array is converted into a target position? This paper proposes a such a model that accomplishes this calculation. The model uses iconic scene representations derived from oriented spatiochromatic filters at multiple scales. Visual search for a target object proceeds in a coarse-to-fine fashion with the target's largest scale filter responses being compared first. Task-relevant target locations are represented as saliency maps which are used to program eye movements. A central feature of the model is that it separates the targeting process, which changes gaze, from the decision process, which extracts information at or near the new gaze point to guide behavior. The model provides a detailed explanation for center-of-gravity saccades that have been observed in many previous experiments. In addition, the model's targeting performance has been compared with the eye movements of human subjects under identical conditions in natural visual search tasks. The results show good agreement both quantitatively (the search paths are strikingly similar) and qualitatively (the fixations of false targets are comparable).

Although imitation learning is a powerful tech- nique for robot learning and knowledge acquisition from na¨ human users, it often suffers from the need for expensive human demonstrations. In some cases the robot has an insufficient number of useful demonstrations, while in others its learning ability is limited by the number of users it directly interacts with. We propose an approach that overcomes these short- comings by using crowdsourcing to collect a wider variety of examples from a large pool of human demonstrators online. We present a new goal-based imitation learning framework which utilizes crowdsourcing as a major source of human demonstration data. We demonstrate the effectiveness of our approach experimentally on a scenario where the robot learns to build 2D object models on a table from basic building blocks using knowledge gained from locals and online crowd workers. In addition, we show how the robot can use this knowledge to support human-robot collaboration tasks such as goal inference through object-part classification and missing-part prediction. We report results from a user study involving fourteen local demonstrators and hundreds of crowd workers on 16 different model building tasks.

The standard cortical model assumes that the meaning of a neuron's signal is contained in its firing rate. While that model has been used to interpret a voluminous amount of experimental data, it does not address the question of timing, or how recipient neurons can decode this signal in time to predict behavioral results. We propose a model based on coincident firing of large groups of neurons. We show using an example of predictive coding, how the cortex can support vast amounts of non-interfering parallel computation.

We describe a model of visual processing in which feedback connections from a higher- to a lower- order visual cortical area carry predictions of lower-level neural activities, whereas the feedforward connections carry the residual errors between the predictions and the actual lower-level activities. When exposed to natural images, a hierarchical network of model neurons implementing such a model developed simple-cell-like receptive fields. A subset of neurons responsible for carrying the residual errors showed endstopping and other extra-classical receptive-field effects. These results suggest that rather than being exclusively feedforward phenomena, nonclassical surround effects in the visual cortex may also result from cortico-cortical feedback as a consequence of the visual system using an efficient hierarchical strategy for encoding natural images.

Neurons in the mammalian primary visual cortex are known to possess spatially localized, oriented receptive fields. It has previously been suggested that these distinctive properties may reflect an efficient image encoding strategy based on maximizing the sparseness of the distribution of output neuronal activities or alternately, extracting the independent components of natural image ensembles. Here, we show that a strategy for transformation-invariant coding of images based on a first-order Taylor series expansion of an image also causes localized, oriented receptive fields to be learned from natural image inputs. These receptive fields, which approximate localized first-order differential operators at various orientations, allow a pair of cooperating neural networks, one estimating object identity (‘what’) and the other estimating object transformations (‘where’), to simultaneously recognize an object and estimate its pose by jointly maximizing the a posteriori probability of generating the observed vi...

We describe a general framework for learning perception-based navigational behaviors in autonomous mobile robots. A hierarchical behavior-based decomposition of the control architecture is used to facilitate efficient modular learning. Lower level reactive behaviors such as collision detection and obstacle avoidance are learned using a stochastic hill-climbing method while higher level goal-directed navigation is achieved using a self-organizing sparse distributed memory. The memory is initially trained by teleoperating the robot on a small number of paths within a given domain of interest. During training, the vectors in the sensory space as well as the motor space are continually adapted using a form of competitive learning to yield basis vectors that efficiently span the sensorimotor space. After training, the robot navigates from arbitrary locations to a desired goal location using motor output vectors computed by a saliency-based weighted averaging scheme. The pervasive problem of perceptual aliasing in finite-order Markovian environments is handled by allowing both current as well as the set of immediately preceding perceptual inputs to predict the motor output vector for the current time instant. We describe experimental and simulation results obtained using a mobile robot equipped with bump sensors, photosensors and infrared receivers, navigating within an enclosed obstacle-ridden arena. The results indicate that the method performs successfully in a number of navigational tasks exhibiting varying degrees of perceptual aliasing.

Given that attention precedes an eye movement to a target it becomes possible to use fixation sequences to probe the spatiotemporal dynamics of search Applying this method to a realistic search task, we found eye movements directed to the geometric centers of progressively smaller groups of objects rather than accurate fixations to individual objects in a display Such a binary search strategy is consistent with zoom-lens models positing an initially broad distribution of search, followed by a narrowing of this search region until only the target is selected We also interpret this oculomotor averaging behavior as evidence for an initially parallel search analysis that becomes increasingly serial as the search process converges on the target

Objective. Brain–computer interfaces (BCIs) are confronted with two fundamental challenges: (a) the uncertainty associated with decoding noisy brain signals, and (b) the need for co-adaptation between the brain and the interface so as to cooperatively achieve a common goal in a task. We seek to mitigate these challenges. Approach. We introduce a new approach to brain–computer interfacing based on partially observable Markov decision processes (POMDPs). POMDPs provide a principled approach to handling uncertainty and achieving co-adaptation in the following manner: (1) Bayesian inference is used to compute posterior probability distributions (‘beliefs’) over brain and environment state, and (2) actions are selected based on entire belief distributions in order to maximize total expected reward; by employing methods from reinforcement learning, the POMDP’s reward function can be updated over time to allow for co-adaptive behaviour. Main results. We illustrate our approach using a simple non-invasive BCI which optimizes the speed–accuracy trade-off for individual subjects based on the signal-to-noise characteristics of their brain signals. We additionally demonstrate that the POMDP BCI can automatically detect changes in the user’s control strategy and can co-adaptively switch control strategies on-the-fly to maximize expected reward. Significance. Our results suggest that the framework of POMDPs offers a promising approach for designing BCIs that can handle uncertainty in neural signals and co-adapt with the user on an ongoing basis. The fact that the POMDP BCI maintains a probability distribution over the user’s brain state allows a much more powerful form of decision making than traditional BCI approaches, which have typically been based on the output of classifiers or regression techniques. Furthermore, the co-adaptation of the system allows the BCI to make online improvements to its behaviour, adjusting itself automatically to the user’s changing circumstances.