Search

We present two studies in which brain-computer interfaces (BCIs) used two related EEG signals, the readiness potential (RP) and the lateralized readiness potential (LRP), in order to predict and feed back motor intentions in real-time. In each of the studies, the experimental task was designed as a game that the subjects played against the BCI. In one of the experiments, subjects played a “duel” game against the BCI. They were challenged to perform spontaneous button presses but to withhold any movement when interrupted by a stop signal. This stop signal was controlled by the BCI that had been trained to predict movements by detecting the occurrence of RPs in the ongoing EEG. In the other experiment, participants played a “matching pennies” game. They won a point if they raised a different hand than the opponent at the end of a countdown and lost a point otherwise. The opponent was played by the BCI, who had been trained to predict from the LRP in the ongoing EEG which hand the subject would move at the end of the countdown. Hence, in both experiments a key feature of the BCI was its closed-loop nature, that is the ability to predict the motor intention in real-time and provide an immediate feedback of the prediction to the subject. In both experiments, prediction accuracies of the BCI were substantially higher compared to random predictions, thereby demonstrating the success of this approach. This allows researchers to use BCIs as research tools to address questions from cognitive neuroscience and provide new insights into the coupling of motor preparatory signals and the corresponding actions.

For successful communication, conversational partners need to estimate each other’s current knowledge state. Nonverbal facial and bodily cues can reveal relevant information about how confident a speaker is about what they are saying. Using functional magnetic resonance imaging, we aimed to identify brain regions that encode how confident a speaker is perceived to be. Participants viewed videos of people answering general knowledge questions and judged each respondent’s confidence in their answer. Our results suggest a distinct role of two neural networks known to support social inferences, the so-called mentalizing and the mirroring network. While activation in both networks underlies the processing of nonverbal cues, only activity in the mentalizing network, most notably the medial prefrontal cortex and the bilateral temporoparietal junction, is modulated by how confident the respondent is judged to be. Our results support an integrative account of the mirroring and mentalizing network, in which the two systems support each other in aiding pragmatic processing.

Humans recognize faces and objects with high speed and accuracy regardless of their orientation. Recent studies have proposed that orientation invariance in face recognition involves an intermediate representation where neural responses are similar for mirror-symmetric views. Here, we used fMRI, multivariate pattern analysis, and computational modeling to investigate the neural encoding of faces and vehicles at different rotational angles. Corroborating previous studies, we demonstrate a representation of face orientation in the fusiform face-selective area (FFA). We go beyond these studies by showing that this representation is category-selective and tolerant to retinal translation. Critically, by controlling for low-level confounds, we found the representation of orientation in FFA to be compatible with a linear angle code. Aspects of mirror-symmetric coding cannot be ruled out when FFA mean activity levels are considered as a dimension of coding. Finally, we used a parametric family of computational models, involving a biased sampling of view-tuned neuronal clusters, to compare different face angle encoding models. The best fitting model exhibited a predominance of neuronal clusters tuned to frontal views of faces. In sum, our findings suggest a category-selective and monotonic code of face orientation in the human FFA, in line with primate electrophysiology studies that observed mirror-symmetric tuning of neural responses at higher stages of the visual system, beyond the putative homolog of human FFA.

There has been a long history of research on visual working memory. Whereas early studies have focused on the role of lateral prefrontal cortex in the storage of sensory information, this has been challenged by research in humans that has directly assessed the encoding of perceptual contents, pointing towards a role of visual and parietal regions during storage. In a previous study we used pattern classification to investigate the storage of complex visual color patterns across delay periods. This revealed coding of such contents in early visual and parietal brain regions. Here we aim to investigate whether the involvement of visual and parietal cortex is also observable for other types of complex, visuo-spatial pattern stimuli. Specifically, we used a combination of fMRI and multivariate classification to investigate the retention of complex flow-field stimuli defined by the spatial patterning of motion trajectories of random dots. Subjects were trained to memorize the precise spatial layout of these stimuli and to retain this information during an extended delay. We used a multivariate decoding approach to identify brain regions where spatial patterns of activity encoded the memorized stimuli. Content-specific memory signals were observable in motion sensitive visual area MT+ and in posterior parietal cortex that might encode spatial information in a modality independent manner. Interestingly, we also found information about the memorized visual stimulus in somatosensory cortex, suggesting a potential crossmodal contribution to memory. Our findings thus indicate that working memory storage of visual percepts might be distributed across unimodal, multimodal and even crossmodal brain regions.

A large body of evidence has implicated the posterior parietal and orbitofrontal cortex in the processing of value. However, value correlates perfectly with salience when appetitive stimuli are investigated in isolation. Accordingly, considerable uncertainty has remained about the precise nature of the previously identified signals. In particular, recent evidence suggests that neurons in the primate parietal cortex signal salience instead of value. To investigate neural signatures of value and salience, here we apply multivariate (pattern-based) analyses to human functional MRI data acquired during a noninstrumental outcome-prediction task involving appetitive and aversive outcomes. Reaction time data indicated additive and independent effects of value and salience. Critically, we show that multivoxel ensemble activity in the posterior parietal cortex encodes predicted value and salience in superior and inferior compartments, respectively. These findings reinforce the earlier reports of parietal value signals and reconcile them with the recent salience report. Moreover, we find that multivoxel patterns in the orbitofrontal cortex correlate with value. Importantly, the patterns coding for the predicted value of appetitive and aversive outcomes are similar, indicating a common neural scale for appetite and aversive values in the orbitofrontal cortex. Thus orbitofrontal activity patterns satisfy a basic requirement for a neural value signal.

In functional magnetic resonance imaging (fMRI), model quality of general linear models (GLMs) for first-level analysis is rarely assessed. In recent work (Soch et al., 2016: “How to avoid mismodelling in GLM-based fMRI data analysis: cross-validated Bayesian model selection”, NeuroImage, vol. 141, pp. 469-489; DOI: 10.1016/j. neuroimage.2016.07.047), we have introduced cross-validated Bayesian model selection (cvBMS) to infer the best model for a group of subjects and use it to guide second-level analysis. While this is the optimal approach given that the same GLM has to be used for all subjects, there is a much more efficient procedure when model selection only addresses nuisance variables and regressors of interest are included in all candidate models. In this work, we propose cross-validated Bayesian model averaging (cvBMA) to improve parameter estimates for these regressors of interest by combining information from all models using their posterior probabilities. This is particularly useful as different models can lead to different conclusions regarding experimental effects and the most complex model is not necessarily the best choice. We find that cvBMS can prevent not detecting established effects and that cvBMA can be more sensitive to experimental effects than just using even the best model in each subject or the model which is best in a group of subjects.

Prospective clinical studies support a link between psychological stress and multiple sclerosis (MS) disease severity, and peripheral stress systems are frequently dysregulated in MS patients. However, the exact link between neurobiological stress systems and MS symptoms is unknown. To evaluate the link between neural stress responses and disease parameters, we used an arterial-spin-labeling functional MRI stress paradigm in 36 MS patients and 21 healthy controls. Specifically, we measured brain activity during a mental arithmetic paradigm with performance-adaptive task frequency and performance feedback and related this activity to disease parameters. Across all participants, stress increased heart rate, perceived stress, and neural activity in the visual, cerebellar and insular cortex areas compared with a resting condition. None of these responses was related to cognitive load (task frequency). Consistently, although performance and cognitive load were lower in patients than in controls, stress responses did not differ between groups. Insula activity elevated during stress compared with rest was negatively linked to impairment of pyramidal and cerebral functions in patients. Cerebellar activation was related negatively to gray matter (GM) atrophy (i.e., positively to GM volume) in patients. Interestingly, this link was also observed in overlapping areas in controls. Cognitive load did not contribute to these associations. The results show that our task induced psychological stress independent of cognitive load. Moreover, stress-induced brain activity reflects clinical disability in MS. Finally, the link between stress-induced activity and GM volume in patients and controls in overlapping areas suggests that this link cannot be caused by the disease alone.

Machine learning-based imaging diagnostics has recently reached or even surpassed the level of clinical experts in several clinical domains. However, classification decisions of a trained machine learning system are typically non-transparent, a major hindrance for clinical integration, error tracking or knowledge discovery. In this study, we present a transparent deep learning framework relying on 3D convolutional neural networks (CNNs) and layer-wise relevance propagation (LRP) for diagnosing multiple sclerosis (MS), the most widespread autoimmune neuroinflammatory disease. MS is commonly diagnosed utilizing a combination of clinical presentation and conventional magnetic resonance imaging (MRI), specifically the occurrence and presentation of white matter lesions in T2-weighted images. We hypothesized that using LRP in a naive predictive model would enable us to uncover relevant image features that a trained CNN uses for decision-making. Since imaging markers in MS are well-established this would enable us to validate the respective CNN model. First, we pre-trained a CNN on MRI data from the Alzheimer's Disease Neuroimaging Initiative (n = 921), afterwards specializing the CNN to discriminate between MS patients (n = 76) and healthy controls (n = 71). Using LRP, we then produced a heatmap for each subject in the holdout set depicting the voxel-wise relevance for a particular classification decision. The resulting CNN model resulted in a balanced accuracy of 87.04% and an area under the curve of 96.08% in a receiver operating characteristic curve. The subsequent LRP visualization revealed that the CNN model focuses indeed on individual lesions, but also incorporates additional information such as lesion location, non-lesional white matter or gray matter areas such as the thalamus, which are established conventional and advanced MRI markers in MS. We conclude that LRP and the proposed framework have the capability to make diagnostic decisions of CNN models transparent, which could serve to justify classification decisions for clinical review, verify diagnosis-relevant features and potentially gather new disease knowledge., Highlights • LRP helps in explaining individual CNN decisions for diagnosing multiple sclerosis (MS) based on conventional MRI data • CNNs learn to identify hyperintense lesions as an important biomarker of MS • CNNs learn to identify relevant areas beyond lesions • Transfer learning improves learning across diseases and MRI sequences • Transparent CNNs show potential in validating models, verifying diagnosis-relevant features and gathering disease knowledge

Visual stimuli can "pop out" if they are different to their background. There has been considerable debate as to the role of primary visual cortex (V1) versus higher visual areas (esp. V4) in pop-out processing. Here we parametrically modulated the relative orientation of stimuli and their backgrounds to investigate the neural correlates of pop-out in visual cortex while subjects were performing a demanding fixation task in a scanner. Whole brain and region of interest analyses confirmed a representation of orientation contrast in extrastriate visual cortex (V4), but not in striate visual cortex (V1). Thus, although previous studies have shown that human V1 can be involved in orientation pop-out, our findings demonstrate that there are cases where V1 is "blind" and pop-out detection is restricted to higher visual areas. Pop-out processing is presumably a distributed process across multiple visual regions.

Delusions are unfounded yet tenacious beliefs and a symptom of psychotic disorder. Varying degrees of delusional ideation are also found in the healthy population. Here, we empirically validated a neurocognitive model that explains both the formation and the persistence of delusional beliefs in terms of altered perceptual inference. In a combined behavioral and functional neuroimaging study in healthy participants, we used ambiguous visual stimulation to probe the relationship between delusion-proneness and the effect of learned predictions on perception. Delusional ideation was associated with less perceptual stability, but a stronger belief-induced bias on perception, paralleled by enhanced functional connectivity between frontal areas that encoded beliefs and sensory areas that encoded perception. These findings suggest that weakened lower-level predictions that result in perceptual instability are implicated in the emergence of delusional beliefs. In contrast, stronger higher-level predictions that sculpt perception into conformity with beliefs might contribute to the tenacious persistence of delusional beliefs.

Unconscious neural activity has been repeatedly shown to precede and potentially even influence subsequent free decisions. However, to date, such findings have been mostly restricted to simple motor choices, and despite considerable debate, there is no evidence that the outcome of more complex free decisions can be predicted from prior brain signals. Here, we show that the outcome of a free decision to either add or subtract numbers can already be decoded from neural activity in medial prefrontal and parietal cortex 4 s before the participant reports they are consciously making their choice. These choice-predictive signals co-occurred with the so-called default mode brain activity pattern that was still dominant at the time when the choice-predictive signals occurred. Our results suggest that unconscious preparation of free choices is not restricted to motor preparation. Instead, decisions at multiple scales of abstraction evolve from the dynamics of preceding brain activity.

Humans are able to flexibly devise and implement rules to reach their desired goals. For simple situations, we can use single rules, such as “if traffic light is green then cross the street.” In most cases, however, more complex rule sets are required, involving the integration of multiple layers of control. Although it has been shown that prefrontal cortex is important for rule representation, it has remained unclear how the brain encodes more complex rule sets. Here, we investigate how the brain represents the order in which different parts of a rule set are evaluated. Participants had to follow compound rule sets that involved the concurrent application of two single rules in a specific order, where one of the rules always had to be evaluated first. The rules and their assigned order were independently manipulated. By applying multivariate decoding to fMRI data, we found that the identity of the current rule was encoded in a frontostriatal network involving right ventrolateral prefrontal cortex, right superior frontal gyrus, and dorsal striatum. In contrast, rule order could be decoded in the dorsal striatum and in the right premotor cortex. The nonhomogeneous distribution of information across brain areas was confirmed by follow-up analyses focused on relevant regions of interest. We argue that the brain encodes complex rule sets by “decomposing” them in their constituent features, which are represented in different brain areas, according to the aspect of information to be maintained.

Perceptual decision-making entails the transformation of graded sensory signals into categorical judgments. Often, there is a direct mapping between these judgments and specific motor responses. However, when stimulus-response mappings are fixed, neural activity underlying decision-making cannot be separatedfrom neural activity reflecting motor planning. Several human neuroimaging studies have reported changes in brain activity associated with perceptual decisions. Nevertheless, to date it has remained unknown where and how specific choices are encoded in the human brain when motor planning is decoupled from the decisionprocess. We addressed this question by having subjects judge the direction of motion of dynamic random dot patterns at various levels of motion strength while measuring their brain activity with fMRI. We used multivariate decoding analyses to search the whole brain for patterns of brain activity encoding subjects' choices. To decouple the decision process from motor planning, subjects were informed about the required motor response only after stimulus presentation. Patterns of fMRI signals in early visual and inferior parietal cortex predicted subjects' perceptual choices irrespective of motor planning. This was true across several levels of motion strength and even in the absence of any coherent stimulus motion. We also found that the cortical distribution of choice-selective brain signals depended on stimulus strength: While visual cortex carried most choice-selective information for strong motion, information in parietal cortex decreased with increasing motion coherence. These results demonstrate that human visual and inferior parietal cortexcarry information about the visual decision in a more abstract format than can be explained by simple motor intentions. Both brain regions may be differentially involved in perceptual decision-making in the face of strong and weak sensory evidence.

Humans use rules to organize their actions to achieve specific goals. Although simple rules that link a sensory stimulus to one response may suffice in some situations, often, the application of multiple, hierarchically organized rules is required. Recent theories suggest that progressively higher level rules are encoded along an anterior-to-posterior gradient within PFC. Although some evidence supports the existence of such a functional gradient, other studies argue for a lesser degree of specialization within PFC. We used fMRI to investigate whether rules at different hierarchical levels are represented at distinct locations in the brain or encoded by a single system. Thirty-seven male and female participants represented and applied hierarchical rule sets containing one lower-level stimulus–response rule and one higher-level selection rule. We used multivariate pattern analysis to investigate directly the representation of rules at each hierarchical level in absence of information about rules from other levels or other task-related information, thus providing a clear identification of low- and high-level rule representations. We could decode low- and high-level rules from local patterns of brain activity within a wide frontoparietal network. However, no significant difference existed between regions encoding representations of rules from both levels except for precentral gyrus, which represented only low-level rule information. Our findings show that the brain represents conditional rules regardless of their level in the explored hierarchy, so the human control system did not organize task representation according to this dimension. Our paradigm represents a promising approach to identifying critical principles that shape this control system.SIGNIFICANCE STATEMENTSeveral recent studies investigating the organization of the human control system propose that rules at different control levels are organized along an anterior-to-posterior gradient within PFC. In this study, we used multivariate pattern analysis to explore independently the representation of formally identical conditional rules belonging to different levels of a cognitive hierarchy and provide for the first time a clear identification of low- and high-level rule representations. We found no major spatial differences between regions encoding rules from different hierarchical levels. This suggests that the human brain does not use levels in the investigated hierarchy as a topographical organization principle to represent rules controlling our behavior. Our paradigm represents a promising approach to identifying which principles are critical.

Focused attention meditations have been shown to improve psychological health and wellbeing and are nowadays an integral part of many psychotherapies. While research on the neural correlates of focused attention meditation is increasing, findings vary on whether meditations are associated with high or low activity in the default mode network (DMN). To clarify the relationship between focused attention meditation and the activity in DMN regions, it may be helpful to distinguish internal and external attention as well as different phases within one meditation: During focused attention meditation, the practitioner switches between mindful attention, mind-wandering and refocusing. Here, we employed a thought-probe paradigm to study the neural correlates of these different phases. Twenty healthy, meditation naive participants were introduced to external (mindfulness of sound) and internal (mindfulness of breathing) attention meditation and then practiced the meditation at home for four consecutive days. They then performed the same focused attention meditations during fMRI scanning, in four runs alternating between internal and external attention. At pseudorandom intervals, participants were asked whether they had just been focused on the task (mindful attention) or had been distracted (mind-wandering). During mindful attention, brain regions typically associated with the DMN, such as the medial prefrontal cortex, posterior cingulate cortex and left temporoparietal junction showed significantly less neural activation compared to mind-wandering phases. Reduced activity of the DMN was found during both external and internal attention, with stronger deactivation in the posterior cingulate cortex during internal attention compared to external attention. Moreover, refocusing after mind-wandering was associated with activity in the left inferior frontal gyrus. Our results support the theory that mindful attention is associated with reduced DMN activity compared to mind-wandering, independent of the practitioner's attention focus (i.e., internal vs. external).

We would like to thank Deecke and Soekadar (1) for their insightful comments on our paper (2). We agree that the voluntary movements in our study were not fully unconstrained. A green light was used to indicate to the participant that a trial had started. This indeed raises the question whether our results generalize to movements without external cueing. Please note that, in our study, the green light is not a response trigger. It indicates the onset of a self-timed waiting period that is followed by an open-ended period during which movements could be freely made. In contrast to many studies on the readiness potential (RP), our study did not require participants to move within a specific time. Thus, …

Being able to comprehend another person's intentions and emotions is essential for successful social interaction. However, it is currently unknown whether the human brain possesses a neural mechanism that attracts people to others whose mental states they can easily understand. Here we show that the degree to which a person feels attracted to another person can change while they observe the other's affective behavior, and that these changes depend on the observer's confidence in having correctly understood the other's affective state. At the neural level, changes in interpersonal attraction were predicted by activity in the reward system of the observer's brain. Importantly, these effects were specific to individual observer-target pairs and could not be explained by a target's general attractiveness or expressivity. Furthermore, using multivoxel pattern analysis (MVPA), we found that neural activity in the reward system of the observer's brain varied as a function of how well the target's affective behavior matched the observer's neural representation of the underlying affective state: The greater the match, the larger the brain's intrinsic reward signal. Taken together, these findings provide evidence that reward-related neural activity during social encounters signals how well an individual's "neural vocabulary" is suited to infer another person's affective state, and that this intrinsic reward might be a source of changes in interpersonal attraction.

Voxel-wise general linear models (GLMs) are a standard approach for analyzing functional magnetic resonance imaging (fMRI) data. An advantage of GLMs is that they are flexible and can be adapted to the requirements of many different data sets. However, the specification of first-level GLMs leaves the researcher with many degrees of freedom which is problematic given recent efforts to ensure robust and reproducible fMRI data analysis. Formal model comparisons that allow a systematic assessment of GLMs are only rarely performed. On the one hand, too simple models may underfit data and leave real effects undiscovered. On the other hand, too complex models might overfit data and also reduce statistical power. Here we present a systematic approach termed cross-validated Bayesian model selection (cvBMS) that allows to decide which GLM best describes a given fMRI data set. Importantly, our approach allows for non-nested model comparison, i.e. comparing more than two models that do not just differ by adding one or more regressors. It also allows for spatially heterogeneous modelling, i.e. using different models for different parts of the brain. We validate our method using simulated data and demonstrate potential applications to empirical data. The increased use of model comparison and model selection should increase the reliability of GLM results and reproducibility of fMRI studies.

Perceptual confidence refers to the degree to which we believe in the accuracy of our percepts. Signal detection theory suggests that perceptual confidence is computed from an internal "decision variable," which reflects the amount of available information in favor of one or another perceptual interpretation of the sensory input. The neural processes underlying these computations have, however, remained elusive. Here, we used fMRI and multivariate decoding techniques to identify regions of the human brain that encode this decision variable and confidence during a visual motion discrimination task. We used observers' binary perceptual choices and confidence ratings to reconstruct the internal decision variable that governed the subjects' behavior. A number of areas in prefrontal and posterior parietal association cortex encoded this decision variable, and activity in the ventral striatum reflected the degree of perceptual confidence. Using a multivariate connectivity analysis, we demonstrate that patterns of brain activity in the right ventrolateral prefrontal cortex reflecting the decision variable were linked to brain signals in the ventral striatum reflecting confidence. Our results suggest that the representation of perceptual confidence in the ventral striatum is derived from a transformation of the continuous decision variable encoded in the cerebral cortex.

It is a vital ability of humans to flexibly adapt their behavior to different environmental situations. Constantly, the rules for our sensory-to-motor mappings need to be adapted to the current task demands. For example, the same sensory input might require two different motor responses depending on the actual situation. How does the brain prepare for such different responses? It has been suggested that the functional connections within cortex are biased according to the present rule to guide the flow of information in accordance with the required sensory-to-motor mapping. Here, we investigated with fMRI whether task settings might indeed change the functional connectivity structure in a large-scale brain network. Subjects performed a visuomotor response task that required an interaction between visual and motor cortex: either within each hemisphere or across the two hemispheres of the brain depending on the task condition. A multivariate analysis on the functional connectivity graph of a cortical visuomotor network revealed that the functional integration, i.e., the connectivity structure, is altered according to the task condition already during a preparatory period before the visual cue and the actual movement. Our results show that the topology of connection weights within a single network changes according to and thus predicts the upcoming task. This suggests that the human brain prepares to respond in different conditions by altering its large scale functional connectivity structure even before an action is required.

Rules are widely used in everyday life to organize actions and thoughts in accordance with our internal goals. At the simplest level, single rules can be used to link individual sensory stimuli to their appropriate responses. However, most tasks are more complex and require the concurrent application of multiple rules. Experiments on humans and monkeys have shown the involvement of a frontoparietal network in rule representation. Yet, a fundamental issue still needs to be clarified: Is the neural representation of multiple rules compositional, that is, built on the neural representation of their simple constituent rules? Subjects were asked to remember and apply either simple or compound rules. Multivariate decoding analyses were applied to functional magnetic resonance imaging data. Both ventrolateral frontal and lateral parietal cortex were involved in compound representation. Most importantly, we were able to decode the compound rules by training classifiers only on the simple rules they were composed of. This shows that the code used to store rule information in prefrontal cortex is compositional. Compositional coding in rule representation suggests that it might be possible to decode other complex action plans by learning the neural patterns of the known composing elements.

The primate orbitofrontal cortex (OFC) is involved in reward processing, learning, and decision making. Research in monkeys has shown that this region is densely connected with higher sensory, limbic, and subcortical regions. Moreover, a parcellation of the monkey OFC into two subdivisions has been suggested based on its intrinsic anatomical connections. However, in humans, little is known about any functional subdivisions of the OFC except for a rather coarse medial/lateral distinction. Here, we used resting-state fMRI in combination with unsupervised clustering techniques to investigate whether OFC subdivisions can be revealed based on their functional connectivity profiles with other brain regions. Examination of different cluster solutions provided support for a parcellation into two parts as observed in monkeys, but it also highlighted a much finer hierarchical clustering of the orbital surface. Specifically, we identified (1) a medial, (2) a posterior-central, (3) a central, and (4–6) three lateral clusters spanning the anterior–posterior gradient. Consistent with animal tracing studies, these OFC clusters were connected to other cortical regions such as prefrontal, temporal, and parietal cortices but also subcortical areas in the striatum and the midbrain. These connectivity patterns provide important implications for identifying specific functional roles of OFC subdivisions for reward processing, learning, and decision making. Moreover, this parcellation schema can provide guidance to report results in future studies.

The present magnetoencephalography study investigated whether the brain states of early syntactic and auditory-perceptual processes can be decoded from single-trial recordings with a multivariate pattern classification approach. In particular, it was investigated whether the early neural activation patterns in response to rule violations in basic auditory perception and in high cognitive processes (syntax) reflect a functional organization that largely generalizes across individuals or is subject-specific. On this account, subjects were auditorily presented with correct sentences, syntactically incorrect sentences, correct sentences including an interaural time difference change, and sentences containing both violations. For the analysis, brain state decoding was carried out within and across subjects with three pairwise classifications. Neural patterns elicited by each of the violation sentences were separately classified with the patterns elicited by the correct sentences. The results revealed the highest decoding accuracies over temporal cortex areas for all three classification types. Importantly, both the magnitude and the spatial distribution of decoding accuracies for the early neural patterns were very similar for within-subject and across-subject decoding. At the same time, across-subject decoding suggested a hemispheric bias, with the most consistent patterns in the left hemisphere. Thus, the present data show that not only auditory-perceptual processing brain states but also cognitive brain states of syntactic rule processing can be decoded from single-trial brain activations. Moreover, the findings indicate that the neural patterns in response to syntactic cognition and auditory perception reflect a functional organization that is highly consistent across individuals.

Patients suffering from obsessive-compulsive disorder (OCD) are characterized by dysregulated neuronal processing of disorder-specific and also unspecific affective stimuli. In the present study, we investigated whether generic fear-inducing, disgust-inducing, and neutral stimuli can be decoded from brain patterns of single fMRI time samples of individual OCD patients and healthy controls. Furthermore, we tested whether differences in the underlying encoding provide information to classify subjects into groups (OCD patients or healthy controls). Two pattern classification analyses were conducted. In analysis 1, we used a classifier to decode the category of a currently viewed picture from extended fMRI patterns of single time samples (TR=3s) in individual subjects for several pairs of categories. In analysis 2, we used a searchlight approach to predict subjects' diagnostic status based on local brain patterns. In analysis 1, we obtained significant accuracies for the separation of fear-eliciting from neutral pictures in OCD patients and healthy controls. Separation of disgust-inducing from neutral pictures was significant in healthy controls. In analysis 2, we identified diagnostic information for the presence of OCD in the orbitofrontal cortex, and in the caudate nucleus. Accuracy obtained in these regions was 100% (p

Zusammenfassung Seit dem Aufkommen der funktionellen Magnetresonanztomografie (fMRT) vor 20 Jahren steht eine neue Methode zur nicht invasiven Messung von Gehirnfunktionen zur Verfügung, welche in den kognitiven Neurowissenschaften inzwischen weit verbreitet ist. Traditionell wurden fMRT-Daten vor allem verwendet, um globale Änderungen der Aktivität in bestimmten Gehirnregionen zu messen, wie sie etwa während einer kognitiven Verarbeitung auftreten. Die Entwicklung neuer Methoden ermöglicht nun einen verfeinerten, inhaltsbasierten Ansatz. Das „multivariate Decoding“ erlaubt es, die kognitive Information zu untersuchen, die in feinkörnigen fMRT-Aktivitätsmustern enthalten ist. Damit lässt sich die Kodierung spezifischer kognitiver Inhalte und Repräsentationen im Gehirn näher bestimmen. Hier wird ein Überblick über verschiedene Entwicklungen des multivariaten Decoding gegeben von der Anwendung in den kognitiven Neurowissenschaften (Wahrnehmung, Aufmerksamkeit, Belohnung, Entscheidungsfindung, emotionale Kommunikation) über neuere methodische Entwicklungen (Informationsfluss, oberflächenbasiertes Searchlight-Decoding) bis hin zur medizinischen Diagnostik gegeben.

The advent of functional magnetic resonance imaging (fMRI) of brain function 20 years ago has provided a new methodology for non-in­vasive measurement of brain function that is now widely used in cognitive neurosci­ence. Traditionally, fMRI data has been an­alyzed looking for overall activity chang­es in brain regions in response to a stimu­lus or a cognitive task. Now, recent develop­ments have introduced more elaborate, con­tent-based analysis techniques. When mul­tivariate decoding is applied to the detailed patterning of regionally-specific fMRI signals, it can be used to assess the amount of infor­mation these encode about specific task-vari­ables. Here we provide an overview of sev­eral developments, spanning from applica­tions in cognitive neuroscience (perception, attention, reward, decision making, emotion­al communication) to methodology (informa­tion flow, surface-based searchlight decod­ing) and medical diagnostics.

Human perception depends heavily on the quality of sensory information. When objects are hard to see we often believe ourselves to be purely guessing. Here we investigated whether such guesses use brain networks involved in perceptual decision making or independent networks. We used a combination of fMRI and pattern classification to test how visibility affects the signals, which determine choices. We found that decisions regarding clearly visible objects are predicted by signals in sensory brain regions, whereas different regions in parietal cortex became predictive when subjects were shown invisible objects and believed themselves to be purely guessing. This parietal network was highly overlapping with regions, which have previously been shown to encode free decisions. Thus, the brain might use a dedicated network for determining choices when insufficient sensory information is available.

Intentional action is essential to human behavior, yet its neural basis remains poorly understood. In order to identify neural networks specifically involved in intentional action, freely chosen and externally cued intentions have previously been contrasted. This has led to the identification of a fronto-parietal network, which is involved in freely choosing one's intentions. However, it remains unclear whether this network encodes specific intentions, or whether it merely reflects general preparatory or control processes correlated with intentional action. Here, we used MVPA on fMRI data to identify brain regions encoding non-motor intentions that were either freely chosen or externally cued. We found that a fronto-parietal network, including the lateral prefrontal cortex, premotor, and parietal cortex, contained information about both freely chosen and externally cued intentions. Importantly, MVPA cross-classification indicated that this network represents the content of our intentions similarly, regardless of whether these intentions are freely chosen or externally cued. This finding suggests that the intention network has a general role in processing and representing intentions independent of their origin.

In humans, spontaneous movements are often preceded by early brain signals. One such signal is the readiness potential (RP) that gradually arises within the last second preceding a movement. An important question is whether people are able to cancel movements after the elicitation of such RPs, and if so until which point in time. Here, subjects played a game where they tried to press a button to earn points in a challenge with a brain-computer interface (BCI) that had been trained to detect their RPs in real time and to emit stop signals. Our data suggest that subjects can still veto a movement even after the onset of the RP. Cancellation of movements was possible if stop signals occurred earlier than 200 ms before movement onset, thus constituting a point of no return.

In multivariate pattern analysis of neuroimaging data, 'second-level' inference is often performed by entering classification accuracies into a $t$-test vs chance level across subjects. We argue that while the random-effects analysis implemented by the $t$-test does provide population inference if applied to activation differences, it fails to do so in the case of classification accuracy or other 'information-like' measures, because the true value of such measures can never be below chance level. This constraint changes the meaning of the population-level null hypothesis being tested, which becomes equivalent to the global null hypothesis that there is no effect in any subject in the population. Consequently, rejecting it only allows to infer that there are some subjects in which there is an information effect, but not that it generalizes, rendering it effectively equivalent to fixed-effects analysis. This statement is supported by theoretical arguments as well as simulations. We review possible alternative approaches to population inference for information-based imaging, converging on the idea that it should not target the mean, but the prevalence of the effect in the population. One method to do so, 'permutation-based information prevalence inference using the minimum statistic', is described in detail and applied to empirical data., Comment: manuscript accepted by NeuroImage, plus minor fixes and a note added after publication

When choosing actions, humans have to balance carefully between different task demands. On the one hand, they should perform tasks repeatedly to avoid frequent and effortful switching between different tasks. On the other hand, subjects have to retain their flexibility to adapt to changes in external task demands such as switching away from an increasingly difficult task. Here, we developed a difficulty-based choice task to investigate how subjects voluntarily select task-sets in predictably changing environments. Subjects were free to choose 1 of the 3 task-sets on a trial-by-trial basis, while the task difficulty changed dynamically over time. Subjects self-sequenced their behavior in this environment while we measured brain responses with functional magnetic resonance imaging (fMRI). Using multivariate decoding, we found that task choices were encoded in the medial prefrontal cortex (dorso-medial prefrontal cortex, dmPFC, and dorsal anterior cingulate cortex, dACC). The same regions were found to encode task difficulty, a major factor influencing choices. Importantly, the present paradigm allowed us to disentangle the neural code for task choices and task difficulty, ensuring that activation patterns in dmPFC/dACC independently encode these 2 factors. This finding provides new evidence for the importance of the dmPFC/dACC for task-selection and motivational functions in highly dynamic environments.

Several brain regions appear to play a role in representing different body identities. The specific contribution of each of these regions is still unclear, however. Here we investigated which brain areas enable the visual distinction between self and other bodies of different familiarity, and between familiar and unfamiliar other individuals, and moreover, where identity-specific information on the three individuals was encoded. Participants were confronted with standardized headless human body stimuli either showing the participant's own, a personally familiar or an unfamiliar other body, while performing a luminance discrimination task. Employing multivariate pattern analysis, we were able to identify areas that allowed for the distinction of self from personal familiar other bodies within the medial prefrontal cortex (mPFC) and posterior cingulate cortex/precuneus. Successful distinction of self from unfamiliar others was possible in the left middle frontal gyrus, the right inferior frontal gyrus, the left pre-supplementary motor area and the right putamen. Personally familiar others could be distinguished from unfamiliar others in the right temporoparietal junction (TPJ). An analysis of identity-specific information revealed a spatial gradient ranging from inferior posterior to superior anterior portions of the mPFC that was associated with encoding identity-related information for self via familiar to unfamiliar other bodies, respectively. Furthermore, several midline and frontal regions encoded information on more than one identity. The TPJ's role in deviance detection was underlined, as only identity-specific information on unfamiliar others was encoded here. Together, our findings suggest substantial spatial overlap in neural correlates of self and other body representation and thus, support the hypothesis of a socially-related representation of the self.

The predicted reward of different behavioral options plays an important role in guiding decisions. Previous research has identified reward predictions in prefrontal and striatal brain regions. Moreover, it has been shown that the neural representation of a predicted reward is similar to the neural representation of the actual reward outcome. However, it has remained unknown how these representations emerge over the course of learning and how they relate to decision making. Here, we sought to investigate learning of predicted reward representations using functional magnetic resonance imaging and multivariate pattern classification. Using a pavlovian conditioning procedure, human subjects learned multiple novel cue–outcome associations in each scanning run. We demonstrate that across learning activity patterns in the orbitofrontal cortex, the dorsolateral prefrontal cortex (DLPFC), and the dorsal striatum, coding the value of predicted rewards become similar to the patterns coding the value of actual reward outcomes. Furthermore, we provide evidence that predicted reward representations in the striatum precede those in prefrontal regions and that representations in the DLPFC are linked to subsequent value-based choices. Our results show that different brain regions represent outcome predictions by eliciting the neural representation of the actual outcome. Furthermore, they suggest that reward predictions in the DLPFC are directly related to value-based choices.

SummaryAn important requirement for vision is to identify interesting and relevant regions of the environment for further processing. Some models assume that salient locations from a visual scene are encoded in a dedicated spatial saliency map [1, 2]. Then, a winner-take-all (WTA) mechanism [1, 2] is often believed to threshold the graded saliency representation and identify the most salient position in the visual field. Here we aimed to assess whether neural representations of graded saliency and the subsequent WTA mechanism can be dissociated. We presented images of natural scenes while subjects were in a scanner performing a demanding fixation task, and thus their attention was directed away. Signals in early visual cortex and posterior intraparietal sulcus (IPS) correlated with graded saliency as defined by a computational saliency model. Multivariate pattern classification [3, 4] revealed that the most salient position in the visual field was encoded in anterior IPS and frontal eye fields (FEF), thus reflecting a potential WTA stage. Our results thus confirm that graded saliency and WTA-thresholded saliency are encoded in distinct neural structures. This could provide the neural representation required for rapid and automatic orientation toward salient events in natural environments.

This study addresses how visual food cues are encoded in reward related brain areas and whether this encoding might provide information to differentiate between patients suffering from eating disorders [binge‐eating disorder (BED) and bulimia nervosa (BN)], overweight controls (C‐OW), and normal‐weight controls (C‐NW). Participants passively viewed pictures of food stimuli and neutral stimuli in a cue reactivity design. Two classification analyses were conducted. First, we used multivariate pattern recognition techniques to decode the category of a currently viewed picture from local brain activity patterns. In the second analysis, we applied an ensemble classifier to predict the clinical status of subjects (BED, BN, C‐OW, and C‐NW) based on food‐related brain response patterns. The left insular cortex separated between food and neutral contents in all four groups. Patterns in the right insular cortex provided a maximum diagnostic accuracy for the separation of BED patients and C‐NW (86% accuracy, P < 10(−5), 82% sensitivity, and 90% specificity) as well as BN patients and C‐NW (78% accuracy, P = 0.001, 86% sensitivity, and 70% specificity). The right ventral striatum separated maximally between BED patients and C‐OW (71% accuracy, P = 0.013, 59% sensitivity, and 82% specificity). The right lateral orbitofrontal cortex separated maximally between BN patients and C‐OW (86% accuracy, P < 10(−4), 79% sensitivity, and 94% specificity). The best differential diagnostic separation between BED and BN patients was obtained in the left ventral striatum (84% accuracy, P < 10(−3), 82% sensitivity, and 86% specificity). Our results indicate that pattern recognition techniques are able to contribute to a reliable differential diagnosis of BN and BED. Hum Brain Mapp 33:2135–2146, 2012. © 2011 Wiley Periodicals, Inc.

The ability to detect where a person is attending is fundamental for brain-computer-interfaces. We explore how the attentional focus can be decoded from brain signals noninvasively acquired with functional magnetic resonance imaging (fMRI). Several cortical regions have previously been reported to have topographic maps reflecting the focus of visual attention. Interestingly, attentional maps were observed to be gradually less topographic when moving from early visual areas toward extra-occipital areas. Recent studies suggested that this might indicate a shift from topographically represented local processing to a global processing dominated by laterality. However, it remains unclear, to which extent the topographical representation of a region characterizes its quality to encode visuospatial attention. Here we addressed this problem by applying multivoxel pattern analysis to fMRI signals. In combination with a cortical surface-based mapping of spatial preference, our analysis revealed a broad cortical network that locally contains information about the locus of visual attention. The informative regions are not restricted to topographic areas, but even in frontal areas, where topographic organization is almost indiscernible, the attentional locus can be decoded from brain activity. Specifically, we find attentional information in the right middle frontal gyrus and the right ventrolateral prefrontal cortex. Furthermore, in these two areas information is sufficient to distinguish attentional loci within the ipsi- as well as the contra-lateral visual hemifields. The laterality dominance decreases when moving from occipital to frontal areas. Our results suggest that information about visuospatial attention is encoded beyond topographically organized regions by local patterns of brain activity. VVC 2011 Wiley Periodicals, Inc. Int J Imaging Syst Technol, 21, 201–210, 2011 (This work was funded by the Max Planck Society and the Bernstein Computational Neuroscience Program of the German Federal Ministry of Education and Research (BMBF Grant 01GQ0411) and the Excellency Initiative of the German Federal Ministry of Education and Research (DFG Grant GSC86/1-2009).)

Local voxel patterns of fMRI signals contain specific information about cognitive processes ranging from basic sensory processing to high level decision making. These patterns can be detected using multivariate pattern classification, and localization of these patterns can be achieved with searchlight methods in which the information content of spherical sub-volumes of the fMRI signal is assessed. The only assumption made by this approach is that the patterns are spatially local. We present a cortical surface-based searchlight approach to pattern localization. Voxels are grouped according to distance along the cortical surface—the intrinsic metric of cortical anatomy—rather than Euclidean distance as in volumetric searchlights. Using a paradigm in which the category of visually presented objects is decoded, we compare the surface-based method to a standard volumetric searchlight technique. Group analyses of accuracy maps produced by both methods show similar distributions of informative regions. The surface-based method achieves a finer spatial specificity with comparable peak values of significance, while the volumetric method appears to be more sensitive to small informative regions and might also capture information not located directly within the gray matter. Furthermore, our findings show that a surface centered in the middle of the gray matter contains more information than to the white–gray boundary or the pial surface.

There has been a long debate on the existence of brain signals that precede the outcome of decisions, even before subjects believe they are consciously making up their mind. The framework of multivariate decoding provides a novel tool for investigating such choice-predictive information contained in neural signals leading up to a decision. New results show that the specific outcome of free choices between different plans can be interpreted from brain activity, not only after a decision has been made, but even several seconds before it is made. This suggests that a causal chain of events can occur outside subjective awareness even before a subject makes up his/her mind. An important future line of research would be to develop paradigms that allow feedback of real-time predictions of future decisions to reveal whether such decisions can still be reverted. This would shed light on how tight the causal link is between early predictive brain signals and subsequent decisions.

In language processing, the relative contribution of early sensory and higher cognitive brain areas is still an open issue. A recent controversial hypothesis proposes that sensory cortices show sensitivity to syntactic processes, whereas other studies suggest a wider neural network outside sensory regions. The goal of the current event‐related fMRI study is to clarify the contribution of sensory cortices in auditory syntactic processing in a 2 × 2 design. Two‐word utterances were presented auditorily and varied both in perceptual markedness (presence or absence of an overt word category marking “‐t”), and in grammaticality (syntactically correct or incorrect). A multivariate pattern classification approach was applied to the data, flanked by conventional cognitive subtraction analyses. The combination of methods and the 2 × 2 design revealed a clear picture: The cognitive subtraction analysis found initial syntactic processing signatures in a neural network including the left IFG, the left aSTG, the left superior temporal sulcus (STS), as well as the right STS/STG. Classification of local multivariate patterns indicated the left‐hemispheric regions in IFG, aSTG, and STS to be more syntax‐specific than the right‐hemispheric regions. Importantly, auditory sensory cortices were only sensitive to the overt perceptual marking, but not to the grammaticality, speaking against syntax‐inflicted sensory cortex modulations. Instead, our data provide clear evidence for a distinction between regions involved in pure perceptual processes and regions involved in initial syntactic processes. Hum Brain Mapp, 2012. © 2011 Wiley Periodicals, Inc.