Search

Conflict has been proposed to act as a cost in action selection, implying a general function of medio-frontal cortex in the adaptation to aversive events. Here we investigate if response conflict acts as a cost during reinforcement learning by modulating experienced reward values in cortical and striatal systems. Electroencephalography recordings show that conflict diminishes the relationship between reward-related frontal theta power and cue preference yet it enhances the relationship between punishment and cue avoidance. Individual differences in the cost of conflict on reward versus punishment sensitivity are also related to a genetic polymorphism associated with striatal D1 versus D2 pathway balance (DARPP-32). We manipulate these patterns with the D2 agent cabergoline, which induces a strong bias to amplify the aversive value of punishment outcomes following conflict. Collectively, these findings demonstrate that interactive cortico-striatal systems implicitly modulate experienced reward and punishment values as a function of conflict.

The field of cognitive science studies latent, unobservable cognitive processes that generate observable behaviors. Similarly, cognitive neuroscience attempts to link latent cognitive processes with the neural mechanisms that generate them. Although neural processes are partially observable (with imaging and electrophysiology), it would be a mistake to ‘skip’ the cognitive level and pursue a purely neuroscientific enterprise to studying behavior. In fact, virtually all of the major advances in understanding the neural basis of behavior over the last century have relied fundamentally on principles of cognition for guiding the appropriate measurements, manipulations, tasks, and interpretations. We provide several examples from the domains of episodic memory, working memory and cognitive control, and decision making in which cognitive theorizing and prior experimentation has been essential in guiding neuroscientific investigations and discoveries.

ObjectiveImpairments in learning are central to autism spectrum disorders. The authors investigated the cognitive and neural basis of these deficits in young adults with autism spectrum disorders using a well-characterized probabilistic reinforcement learning paradigm.MethodThe probabilistic selection task was implemented among matched participants with autism spectrum disorders (N=22) and with typical development (N=25), aged 18-40 years, using rapid event-related functional MRI. Participants were trained to choose the correct stimulus in high-probability (AB), medium-probability (CD), and low-probability (EF) pairs, presented with valid feedback 80%, 70%, and 60% of the time, respectively. Whole-brain voxel-wise and parametric modulator analyses examined early and late learning during the stimulus and feedback epochs of the task.ResultsThe groups exhibited comparable performance on medium- and low-probability pairs. Typically developing persons showed higher accuracy on the high-probability pair, better win-stay performance (selection of the previously rewarded stimulus on the next trial of that type), and more robust recruitment of the anterior and medial prefrontal cortex during the stimulus epoch, suggesting development of an intact reward-based working memory for recent stimulus values. Throughout the feedback epoch, individuals with autism spectrum disorders exhibited greater recruitment of the anterior cingulate and orbito-frontal cortices compared with individuals with typical development, indicating continuing trial-by-trial activity related to feedback processing.ConclusionsIndividuals with autism spectrum disorders exhibit learning deficits reflecting impaired ability to develop an effective reward-based working memory to guide stimulus selection. Instead, they continue to rely on trial-by-trial feedback processing to support learning dependent upon engagement of the anterior cingulate and orbito-frontal cortices.

The striatal dopaminergic system has been implicated in reinforcement learning (RL), motor performance, and incentive motivation. Various computational models have been proposed to account for each of these effects individually, but a formal analysis of their interactions is lacking. Here we present a novel algorithmic model expanding the classical actor-critic architecture to include fundamental interactive properties of neural circuit models, incorporating both incentive and learning effects into a single theoretical framework. The standard actor is replaced by a dual opponent actor system representing distinct striatal populations, which come to differentially specialize in discriminating positive and negative action values. Dopamine modulates the degree to which each actor component contributes to both learning and choice discriminations. In contrast to standard frameworks, this model simultaneously captures documented effects of dopamine on both learning and choice incentive-and their interactions-across a variety of studies, including probabilistic RL, effort-based choice, and motor skill learning.

pFC is proposed to implement cognitive control via directed “top–down” influence over behavior. But how is this feat achieved? The virtue of such a descriptive model is contingent on a mechanistic understanding of how motor execution is altered in specific circumstances. In this report, we provide evidence that the well-known phenomenon of slowed RTs following mistakes (post-error slowing) is directly influenced by the degree of subthalamic nucleus (STN) activity. The STN is proposed to act as a brake on motor execution following conflict or errors, buying time so a more cautious response can be made on the next trial. STN local field potentials from nine Parkinson disease patients undergoing deep brain stimulation surgery were recorded while they performed a response conflict task. In a 2.5- to 5-Hz frequency range previously associated with conflict and error processing, the degree phase consistency preceding the response was associated with increasingly slower RTs specifically following errors. These findings provide compelling evidence that post-error slowing is in part mediated by a corticosubthalamic “hyperdirect” pathway for increased response caution.

We introduce the symmetric approximation of frames by normalized tight frames extending the concept of the symmetric orthogonalization of bases by orthonormal bases in Hilbert spaces. We prove existence and uniqueness results for the symmetric approximation of frames by normalized tight frames. Even in the case of the symmetric orthogonalization of bases, our techniques and results are new. A crucial role is played by whether or not a certain operator related to the initial frame or basis is Hilbert-Schmidt.

In this report the authors present a review of complex regional pain syndrome and two case reports of complex regional pain syndrome that were recalcitrant to the usual treatments. The first case presented is a middle-aged female who developed signs and symptoms of post-traumatic complex regional pain syndrome. The second case is a woman with a pre-existing history of complex regional pain syndrome whose condition worsened after surgery despite appropriate perioperative precautions. These cases are unique because in both cases an early diagnosis of complex regional pain syndrome was established, yet they were both resistant to the usual treatment protocols.

The frontal cortex and the basal ganglia interact via a relatively well understood and elaborate system of interconnections. In the context of motor function, these interconnections can be understood as disinhibiting, or "releasing the brakes," on frontal motor action plans: The basal ganglia detect appropriate contexts for performing motor actions and enable the frontal cortex to execute such actions at the appropriate time. We build on this idea in the domain of working memory through the use of computational neural network models of this circuit. In our model, the frontal cortex exhibits robust active maintenance, whereas the basal ganglia contribute a selective, dynamic gating function that enables frontal memory representations to be rapidly updated in a task-relevant manner. We apply the model to a novel version of the continuous performance task that requires subroutine-like selective working memory updating and compare and contrast our model with other existing models and theories of frontal-cortex-basal-ganglia interactions.

The aim of the present paper is to solve some major open problems of Hilbert C*-module theory by applying various aspects of multiplier C*-theory. The key result is the equivalence established between positive invertible quasi-multipliers of the C*-algebra of “compact” operators on a Hilbert C*-module {ℳ, 〈., 〉} and A-valued inner products on ℳ, inducing an equivalent norm to the given one. The problem of unitary isomorphism of C*-valued inner products on a Hilbert C*-module is considered and new criteria are formulated. Countably generated Hilbert C*-modules turn out to be unitarily isomorphic if they are isomorphic as Banach C*-modules. The property of bounded module operators on Hilbert C*-modules of being “compact” and/or adjointable is unambiguously connected to operators with respect to any choice of the C*-valued inner product on a fixed Hilbert C*-module if every bounded module operator possesses an adjoint operator on the module. Every bounded module operator on a given full Hilbert C*-module turns out to be adjointable if the Hilbert C*-module is orthogonally complementary. Moreover, if the unit ball of the Hilbert C*-module is complete with respect to a certain locally convex topology, then these two properties are shown to be equivalent to self-duality.

Can you predict what people are going to do just by watching them? This is certainly difficult: it would require a clear mapping between observable indicators and unobservable cognitive states. In this report, we demonstrate how this is possible by monitoring eye gaze and pupil dilation, which predict dissociable biases during decision making. We quantified decision making using the drift diffusion model (DDM), which provides an algorithmic account of how evidence accumulation and response caution contribute to decisions through separate latent parameters of drift rate and decision threshold, respectively. We used a hierarchical Bayesian estimation approach to assess the single trial influence of observable physiological signals on these latent DDM parameters. Increased eye gaze dwell time specifically predicted an increased drift rate toward the fixated option, irrespective of the value of the option. In contrast, greater pupil dilation specifically predicted an increase in decision threshold during difficult decisions. These findings suggest that eye tracking and pupillometry reflect the operations of dissociated latent decision processes.

Summary Convergent evidence suggests that corticostriatal interactions act as a gate to select the input to working memory (WM). However, not all information in WM is relevant for behavior simultaneously. For this reason, a second "output gate" might advantageously govern which contents of WM influence behavior. Here, we test whether frontostriatal circuits previously implicated in input gating also support output gating during selection from WM. fMRI of a hierarchical rule task with dissociable input and output gating demands demonstrated greater lateral prefrontal cortex (PFC) recruitment and frontostriatal connectivity during output gating. Moreover, PFC and striatum correlated with distinct behavioral profiles. Whereas PFC recruitment correlated with mean efficiency of selection from WM, striatal recruitment and frontostriatal interactions correlated with its reliability, as though such dynamics stochastically gate WM's output. These results support the output gating hypothesis, suggesting that contextual representations in PFC influence striatum to select which information in WM drives responding.

The aim of the present study was to examine the contributions to decision making (DM) deficits in schizophrenia (SZ) patients, of expected value (EV) estimation and loss aversion. Patients diagnosed with SZ (n=46) and healthy controls (n=34) completed two gambling tasks. In one task, participants chose between two options with the same EV across two conditions: Loss frames and Keep frames. A second task involved accepting or rejecting gambles, in which gain and loss amounts varied, determining the EV of each trial. SZ patients showed a reduced “framing effect” relative to controls, as they did not show an increased tendency to gamble when faced with a certain loss. SZ patients also showed a reduced tendency to modify behavior as a function of EV. The degree to which choices tracked EV correlated significantly with several cognitive measures in both patients and controls. SZ patients show distinct deviations from normal behavior under risk when their decisions are based on prospective outcomes. These deviations are two-fold: cognitive deficits prevent value-based DM in more-impaired patients, and in less-impaired patients there is a lack of influence from well-established subjective biases found in healthy people. These abnormalities likely affect every-day DM strategies in schizophrenia patients.

The main goal of the present paper is to generalize the results of~\cite{TroLNM,TroBoch} in the following way: To be able to define $K_0(A)o\C$-valued Lefschetz numbers of the first type of an endomorphism $V$ on a C*-elliptic complex one usually assumes that $V=T_g$ for some representation $T_g$ of a compact group $G$ on the C*-elliptic complex. We try to refuse this restriction in the present paper. The price to pay for this is twofold: (i) $ $ We have to define Lefschetz numbers valued in some larger group as $K_0(A)o\C$. (ii) We have to deal with W*-algebras instead of general unital C*-algebras. To obtain these results we have got a number of by-product facts on the theory of Hilbert W*- and C*-modules and on bounded module operators on them which are of independent interest.

Background: There is a rapidly amassing body of evidence suggesting that reinforcement learning may be impaired in schizophrenia (SZ). Reinforcement learning not only depends on the ability to signal reward prediction errors (RPEs), but also on the ability to modulate the impact that RPEs have on learning, especially in a volatile environment. Here, we investigated the ability of individuals with SZ to use RPEs to rapidly update their decision-making strategies and the neural correlates thereof.

The aim of the present paper is to describe self-duality and C*-reflexivity of Hilbert A-modules ℳ over monotone complete C*-algebras A by the completeness of the unit ball of ℳ with respect to two types of convergence being defined, and by a structural criterion. The derived results generalize earlier results ofH. Widom [Duke Math. J. 23, 309-324, MR 17 # 1228] and W. L. Paschke [Trans. Amer. Mat. Soc. 182, 443-468, MR 50 # 8087, Canadian J. Math. 26, 1272-1280, MR 57 # 10433]. For Hilbert C*-modules over commutative AW*-algebras the equivalence of the self-duality property and of the Kaplansky-Hilbert property is reproved, (cf. M. Ozawa [J. Math. Soc. Japan 36, 589-609, MR 85 # 46068]). Especially, one derives that for a C*-algebra A the A-valued inner product of every Hilbert A-module ℳ can be continued to an A-valued inner product on it's A-dual Banach A-module ℳ' turning ℳ' to a self-dual Hilbert A-module if and only if A is monotone complete (or, equivalently, additively complete) generalizing a result of M. Hamana [Internat. J. Math. 3 (1992), 185 - 204]. A classification of countably generated self-dual Hilbert A-modules over monotone complete C*-algebras A is established. The set of all bounded module operators End′(ℳ) on self-dual Hilbert A-modules ℳ over monotone complete C*-algebras A is proved again to be a monotone complete C*-algebra. Applying these results a Weyl-Berg type theorem is proved.

SummaryHow do individuals decide to act based on a rewarding status quo versus an unexplored choice that might yield a better outcome? Recent evidence suggests that individuals may strategically explore as a function of the relative uncertainty about the expected value of options. However, the neural mechanisms supporting uncertainty-driven exploration remain underspecified. The present fMRI study scanned a reinforcement learning task in which participants stop a rotating clock hand in order to win points. Reward schedules were such that expected value could increase, decrease, or remain constant with respect to time. We fit several mathematical models to subject behavior to generate trial-by-trial estimates of exploration as a function of relative uncertainty. These estimates were used to analyze our fMRI data. Results indicate that rostrolateral prefrontal cortex tracks trial-by-trial changes in relative uncertainty, and this pattern distinguished individuals who rely on relative uncertainty for their exploratory decisions versus those who do not.Video Abstract

Context Negative symptoms are a core feature of schizophrenia, but their pathogenesis remains unclear. Negative symptoms are defined by the absence of normal function. However, there must be a productive mechanism that leads to this absence. Objective To test a reinforcement learning account suggesting that negative symptoms result from a failure in the representation of the expected value of rewards coupled with preserved loss-avoidance learning. Design Participants performed a probabilistic reinforcement learning paradigm involving stimulus pairs in which choices resulted in reward or in loss avoidance. Following training, participants indicated their valuation of the stimuli in a transfer test phase. Computational modeling was used to distinguish between alternative accounts of the data. Setting A tertiary care research outpatient clinic. Patients In total, 47 clinically stable patients with a diagnosis of schizophrenia or schizoaffective disorder and 28 healthy volunteers participated in the study. Patients were divided into a high-negative symptom group and a low-negative symptom group. Main Outcome Measures The number of choices leading to reward or loss avoidance, as well as performance in the transfer test phase. Quantitative fits from 3 different models were examined. Results Patients in the high-negative symptom group demonstrated impaired learning from rewards but intact loss-avoidance learning and failed to distinguish rewarding stimuli from loss-avoiding stimuli in the transfer test phase. Model fits revealed that patients in the high-negative symptom group were better characterized by an“actor-critic” model, learning stimulus-response associations, whereas control subjects and patients in the low-negative symptom group incorporated expected value of their actions (“Q learning”) into the selection process. Conclusions Negative symptoms in schizophrenia are associated with a specific reinforcement learning abnormality: patients with high-negative symptoms do not represent the expected value of rewards when making decisions but learn to avoid punishments through the use of prediction errors. This computational framework offers the potential to understand negative symptoms at a mechanistic level.

Item does not contain fulltext Functional imaging paradigms hold great promise as biomarkers for schizophrenia research as they can detect altered neural activity associated with the cognitive and emotional processing deficits that are so disabling to this patient population. In an attempt to identify the most promising functional imaging biomarkers for research on long-term memory (LTM), the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) initiative selected "item encoding and retrieval," "relational encoding and retrieval," and "reinforcement learning" as key LTM constructs to guide the nomination process. This manuscript reports on the outcome of the third CNTRICS biomarkers meeting in which nominated paradigms in each of these domains were discussed by a review panel to arrive at a consensus on which of the nominated paradigms could be recommended for immediate translational development. After briefly describing this decision process, information is presented from the nominating authors describing the 4 functional imaging paradigms that were selected for immediate development. In addition to describing the tasks, information is provided on cognitive and neural construct validity, sensitivity to behavioral or pharmacological manipulations, availability of animal models, psychometric characteristics, effects of schizophrenia, and avenues for future development. 01 januari 2012

In order to understand the exploitation/exploration trade-off in reinforcement learning, previous theoretical and empirical accounts have suggested that increased uncertainty may precede the decision to explore an alternative option. To date, the neural mechanisms that support the strategic application of uncertainty-driven exploration remain underspecified. In this study, electroencephalography (EEG) was used to assess trial-to-trial dynamics relevant to exploration and exploitation. Theta-band activities over middle and lateral frontal areas have previously been implicated in EEG studies of reinforcement learning and strategic control. It was hypothesized that these areas may interact during top-down strategic behavioral control involved in exploratory choices. Here, we used a dynamic reward–learning task and an associated mathematical model that predicted individual response times. This reinforcement-learning model generated value-based prediction errors and trial-by-trial estimates of exploration as a function of uncertainty. Mid-frontal theta power correlated with unsigned prediction error, although negative prediction errors had greater power overall. Trial-to-trial variations in response-locked frontal theta were linearly related to relative uncertainty and were larger in individuals who used uncertainty to guide exploration. This finding suggests that theta-band activities reflect prefrontal-directed strategic control during exploratory choices.

Individuals with autism spectrum disorders (ASDs) exhibit intact rote learning with impaired generalization. A transitive inference paradigm, involving training on four sequentially presented stimulus pairs containing overlapping items, with subsequent testing on two novel pairs, was used to investigate this pattern of learning in 27 young adults with ASDs and 31 matched neurotypical individuals (TYPs). On the basis of findings about memory and neuropathology, we hypothesized that individuals with ASDs would use a relational flexibility/conjunctive strategy reliant on an intact hippocampus, versus an associative strength/value transfer strategy requiring intact interactions between the prefrontal cortex and the striatum. Hypotheses were largely confirmed. ASDs demonstrated reduced interference from intervening pairs in early training; only TYPs formed a serial position curve by test; and ASDs exhibited impairments on the novel test pair consisting of end items with intact performance on the inner test pair. However, comparable serial position curves formed for both groups by the end of the first block.

The frontal lobes may be organized hierarchically such that more rostral frontal regions modulate cognitive control operations in caudal regions. In our companion paper (Frank MJ, Badre D. 2011. Mechanisms of hierarchical reinforcement learning in corticostriatal circuits I: computational analysis. 22:509–526), we provide novel neural circuit and algorithmic models of hierarchical cognitive control in cortico–striatal circuits. Here, we test key model predictions using functional magnetic resonance imaging (fMRI). Our neural circuit model proposes that contextual representations in rostral frontal cortex influence the striatal gating of contextual representations in caudal frontal cortex. Reinforcement learning operates at each level, such that the system adaptively learns to gate higher order contextual information into rostral regions. Our algorithmic Bayesian “mixture of experts” model captures the key computations of this neural model and provides trial-by-trial estimates of the learner’s latent hypothesis states. In the present paper, we used these quantitative estimates to reanalyze fMRI data from a hierarchical reinforcement learning task reported in Badre D, Kayser AS, D’Esposito M. 2010. Frontal cortex and the discovery of abstract action rules. Neuron. 66:315--326. Results validate key predictions of the models and provide evidence for an individual cortico–striatal circuit for reinforcement learning of hierarchical structure at a specific level of policy abstraction. These findings are initially consistent with the proposal that hierarchical control in frontal cortex may emerge from interactions among nested cortico–striatal circuits at different levels of abstraction.

The striatum is critical for the incremental learning of values associated with behavioral actions. The prefrontal cortex (PFC) represents abstract rules and explicit contingencies to support rapid behavioral adaptation in the absence of cumulative experience. Here we test two alternative models of the interaction between these systems, and individual differences thereof, when human subjects are instructed with prior information about reward contingencies that may or may not be accurate. Behaviorally, subjects are overly influenced by prior instructions, at the expense of learning true reinforcement statistics. Computational analysis found that this pattern of data is best accounted for by a confirmation bias mechanism in which prior beliefs—putatively represented in PFC—influence the learning that occurs in the striatum such that reinforcement statistics are distorted. We assessed genetic variants affecting prefrontal and striatal dopaminergic neurotransmission. A polymorphism in the COMT gene (rs4680), associated with prefrontal dopaminergic function, was predictive of the degree to which participants persisted in responding in accordance with prior instructions even as evidence against their veracity accumulated. Polymorphisms in genes associated with striatal dopamine function (DARPP-32, rs907094, and DRD2, rs6277) were predictive of learning from positive and negative outcomes. Notably, these same variants were predictive of the degree to which such learning was overly inflated or neglected when outcomes are consistent or inconsistent with prior instructions. These findings indicate dissociable neurocomputational and genetic mechanisms by which initial biases are strengthened by experience.

Computational models of the basal ganglia have matured and received increasing attention over the last decade. This article reviews some of the theoretical advances offered by these models, focusing on motor and cognitive action selection, learning, and the interaction between multiple corticostriatal circuits in selection and learning.

Due to an example indicated to us in September 2009 we have to add one more restriction to the suppositions on the imprimitivity bimodules treated in Proposition 4.1, Theorem 5.1, Theorem 6.2 and Proposition 6.3. In the situation when the Banach-Saks property holds for the imprimitivity bimodule we can describe all possible additional examples violating the newly invented supposition. So the classification of Hilbert $C^*$-modules with the Banach-Saks property is complete. Beyond that, there is still an open problem for a certain class of imprimitivity bimodules with the weak or uniform weak Banach-Saks property which might violate the additional condition.

Planning and executing volitional actions in the face of conflicting habitual responses is a critical aspect of human behavior. At the core of the interplay between these two control systems lies an override mechanism that can suppress the habitual action selection process and allow executive control to take over. Here, we construct a neural circuit model informed by behavioral and electrophysiological data collected on various response inhibition paradigms. This model extends a well established model of action selection in the basal ganglia by including a frontal executive control network which integrates information about sensory input and task rules to facilitate well-informed decision making via the oculomotor system. Our simulations of the antisaccade, Simon and saccade-override task ensue in conflict between a prepotent and controlled response which causes the network to pause action selection via projections to the subthalamic nucleus. Our model reproduces key behavioral and electrophysiological patterns and their sensitivity to lesions and pharmacological manipulations. Finally, we show how this network can be extended to include the inferior frontal cortex to simulate key qualitative patterns of global response inhibition demands as required in the stop-signal task., Comment: 3rd submission (now accepted at Psychological Review). Removed switch-DDM and some other data points, restructured some graphics. Added systematic accuracy-RT analysis of speed-accuracy trade-off

Background Negative symptoms are core features of schizophrenia (SZ); however, the cognitive and neural basis for individual negative symptom domains remains unclear. Converging evidence suggests a role for striatal and prefrontal dopamine in reward learning and the exploration of actions that might produce outcomes that are better than the status quo. The current study examines whether deficits in reinforcement learning and uncertainty-driven exploration predict specific negative symptom domains. Methods We administered a temporal decision-making task, which required trial-by-trial adjustment of reaction time to maximize reward receipt, to 51 patients with SZ and 39 age-matched healthy control subjects. Task conditions were designed such that expected value (probability × magnitude) increased, decreased, or remained constant with increasing response times. Computational analyses were applied to estimate the degree to which trial-by-trial responses are influenced by reinforcement history. Results Individuals with SZ showed impaired Go learning but intact NoGo learning relative to control subjects. These effects were most pronounced in patients with higher levels of negative symptoms. Uncertainty-based exploration was substantially reduced in individuals with SZ and selectively correlated with clinical ratings of anhedonia. Conclusions Schizophrenia patients, particularly those with high negative symptoms, failed to speed reaction times to increase positive outcomes and showed reduced tendency to explore when alternative actions could lead to better outcomes than the status quo. Results are interpreted in the context of current computational, genetic, and pharmacological data supporting the roles of striatal and prefrontal dopamine in these processes.

We review the contributions of biologically constrained computational models to our understanding of motor and cognitive deficits in Parkinson's disease (PD). The loss of dopaminergic neurons innervating the striatum in PD, and the well-established role of dopamine (DA) in reinforcement learning (RL), enable neural network models of the basal ganglia (BG) to derive concrete and testable predictions. We focus in this review on one simple underlying principle - the notion that reduced DA increases activity and causes long-term potentiation in the indirect pathway of the BG. We show how this theory can provide a unified account of diverse and seemingly unrelated phenomena in PD including progressive motor degeneration as well as cognitive deficits in RL, decision making and working memory. DA replacement therapy and deep brain stimulation can alleviate some aspects of these impairments, but can actually introduce negative effects such as motor dyskinesias and cognitive impulsivity. We discuss these treatment effects in terms of modulation of specific mechanisms within the computational framework. In addition, we review neurocomputational interpretations of increased impulsivity in the face of response conflict in patients with deep-brain-stimulation.

Previous studies have typically found that individuals with schizophrenia (SZ) report levels of emotional experience that are similar to controls (CN) when asked to view a single evocative stimulus and make an absolute judgment of stimulus “value.” However, value is rarely assigned in absolute terms in real-life situations, where one alternative or experience is often evaluated alongside others, and value judgments are made in relative terms. In the current study, we examined performance on a preference task that requires individuals to differentiate between the relative values of different stimuli. In this task, subjects were presented with many pairs of moderately positive stimuli and asked to indicate which stimulus they preferred in each pair. Resulting data indicated the rank order of preference across stimuli and the consistency of their transitive mapping (ie, if A > B and B > C, then A should be > C). Individuals with SZ (n = 38) were both less consistent in their rankings of stimuli and more likely to have larger magnitudes of discrepant responses than control subjects (n = 27). Furthermore, CN showed clear differentiation between different valence categories of stimuli (ie, highly positive > mildly positive > mildly negative > highly negative); while individuals with SZ showed the same general pattern of results but with less differentiation between the valence levels. These data suggest that individuals with SZ are impaired in developing or maintaining nuanced representations of the different attributes of a stimulus, thus making stimuli of similar general value easily confusable.

Larger error-related negativities (ERNs) have been consistently found in obsessive-compulsive disorder (OCD) patients, and are thought to reflect the activities of a hyperactive cortico-striatal circuit during action monitoring. We previously observed that obsessive-compulsive (OC) symptomatic students (non-patients) have larger ERNs during errors in a response competition task, yet smaller ERNs in a reinforcement learning task. The finding of a task-specific dissociation suggests that distinct yet partially overlapping medio-frontal systems underlie the ERN in different tasks, and that OC symptoms are associated with functional differences in these systems. Here, we used EEG source localization to identify why OC symptoms are associated with hyperactive ERNs to errors yet hypoactive ERNs when selecting maladaptive actions. At rest, OC symptomatology predicted greater activity in rostral anterior cingulate cortex (rACC) and lower activity in dorsal anterior cingulate cortex (dACC). When compared to a group with low OC symptom scores, the high OC group had greater rACC reactivity during errors in the response competition task and less deactivation of dACC activity during errors in the reinforcement learning task. The degree of activation in these areas correlated with ERN amplitudes during both tasks in the high OC group, but not in the low group. Interactive anterior cingulate cortex (ACC) systems associated avoidance of maladaptive actions were intact in the high OC group, but were related to poorer performance on a third task: probabilistic reversal learning. These novel findings link both tonic and phasic activities in the ACC to action monitoring alterations, including dissociation in performance deficits, in OC symptomatic participants.

We review the contributions of biologically constrained computational models to our understanding of motor and cognitive deficits in Parkinson’s disease (PD). The loss of dopaminergic neurons innervating the striatum in PD, and the well-established role of dopamine (DA) in reinforcement learning (RL), enable neural network models of the basal ganglia (BG) to derive concrete and testable predictions. We focus in this review on one simple underlying principle – the notion that reduced DA increases activity and causes long-term potentiation in the indirect pathway of the BG. We show how this theory can provide a unified account of diverse and seemingly unrelated phenomena in PD including progressive motor degeneration as well as cognitive deficits in RL, decision making and working memory. DA replacement therapy and deep brain stimulation can alleviate some aspects of these impairments, but can actually introduce negative effects such as motor dyskinesias and cognitive impulsivity. We discuss these treatment effects in terms of modulation of specific mechanisms within the computational framework. In addition, we review neurocomputational interpretations of increased impulsivity in the face of response conflict in patients with deep-brain-stimulation.

Item does not contain fulltext Recent efforts to understand the mechanisms underlying human cooperation have focused on the notion of trust, with research illustrating that both initial impressions and previous interactions impact the amount of trust people place in a partner. Less is known, however, about how these two types of information interact in iterated exchanges. The present study examined how implicit initial trustworthiness information interacts with experienced trustworthiness in a repeated Trust Game. Consistent with our hypotheses, these two factors reliably influence behavior both independently and synergistically, in terms of how much money players were willing to entrust to their partner and also in their post-game subjective ratings of trustworthiness. To further understand this interaction, we used Reinforcement Learning models to test several distinct processing hypotheses. These results suggest that trustworthiness is a belief about probability of reciprocation based initially on implicit judgments, and then dynamically updated based on experiences. This study provides a novel quantitative framework to conceptualize the notion of trustworthiness.

Reinforcement learning is ubiquitous. Unlike other forms of learning, it involves the processing of fast yet content-poor feedback information to correct assumptions about the nature of a task or of a set of stimuli. This feedback information is often delivered as generic rewards or punishments, and has little to do with the stimulus features to be learned. How can such low-content feedback lead to such an efficient learning paradigm? Through a review of existing neuro-computational models of reinforcement learning, we suggest that the efficiency of this type of learning resides in the dynamic and synergistic cooperation of brain systems that use different levels of computations. The implementation of reward signals at the synaptic, cellular, network and system levels give the organism the necessary robustness, adaptability and processing speed required for evolutionary and behavioral success.

The activity of dopaminergic neurons are thought to be increased by stimuli that predict reward and decreased by stimuli that predict aversive outcomes. Recent work by Matsumoto and Hikosaka challenges this model by asserting that stimuli associated with either rewarding or aversive outcomes increase the activity of dopaminergic neurons in the substantia nigra pars compacta.

BackgroundCentral to understanding of the behavioural consequences of depression has been the theory that the disorder is accompanied by an increased sensitivity to negative compared with positive reinforcement (negative bias), whereas other theorists have emphasized a global reduction in sensitivity to reinforcement in depression (blunting).MethodIn this study, we used a probabilistic selection task that was designed to examine independently rates of learning to predict both positive and negative reinforcement. Twenty-three depressed out-patients and 23 healthy controls from the local population participated in the study.ResultsNo evidence for a negative bias was observed on the task, either during acquisition of the task or during generalization of the learned information. Depressed patients responded slower on the task than controls but showed a similar modulation of reaction times (RTs) as controls following reinforcement. Evidence for blunting was observed on the training phase, as reflected in reduced trial-by-trial adjustment during this phase. However, this effect was related specifically to the severity of anhedonia, as measured by the Snaith–Hamilton Pleasure Scale (SHAPS), and was independent of overall depression severity.ConclusionsWe argue that the observation of a negative bias or blunting in a group of depressed patients may be dependent on the neuropsychological task and the symptoms of the patients tested. Our results provide insight into how these theories might be further tested.

Investigations into action monitoring have consistently detailed a fronto-central voltage deflection in the Event-Related Potential (ERP) following the presentation of negatively valenced feedback, sometimes termed the Feedback Related Negativity (FRN). The FRN has been proposed to reflect a neural response to prediction errors during reinforcement learning, yet the single trial relationship between neural activity and the quanta of expectation violation remains untested. Although ERP methods are not well suited to single trial analyses, the FRN has been associated with theta band oscillatory perturbations in the medial prefrontal cortex. Medio-frontal theta oscillations have been previously associated with expectation violation and behavioral adaptation and are well suited to single trial analysis. Here, we recorded EEG activity during a probabilistic reinforcement learning task and fit the performance data to an abstract computational model (Q-learning) for calculation of single-trial reward prediction errors. Single-trial theta oscillatory activities following feedback were investigated within the context of expectation (prediction error) and adaptation (subsequent reaction time change). Results indicate that interactive medial and lateral frontal theta activities reflect the degree of negative and positive reward prediction error in the service of behavioral adaptation. These different brain areas use prediction error calculations for different behavioral adaptations: with medial frontal theta reflecting the utilization of prediction errors for reaction time slowing (specifically following errors), but lateral frontal theta reflecting prediction errors leading to working memory-related reaction time speeding for the correct choice.

The capacity to anticipate and prepare for future events is thought to be critical for cognitive control. Dominant accounts of cognitive control treat the developing system as merely a weaker version of the adult system, progressively strengthening over time. Using the AX Continuous Performance Task (AX-CPT) in combination with high-resolution pupillometry, we find that whereas 8-year-old children resemble adults in their proactive use of cognitive control, 3.5-year-old children exhibit a qualitatively different, reactive form of cognitive control, responding to events only as they unfold and retrieving information from memory as needed in the moment. These results demonstrate the need to reconsider the origins of cognitive control and the basis for children's behaviors across domains.

Hyperactive cortico-striatal circuits including the anterior cingulate cortex (ACC) have been implicated to underlie obtrusive thoughts and repetitive behaviors in obsessive-compulsive disorder (OCD). Larger error-related negativities (ERNs) in OCD patients during simple flanker tasks have been proposed to reflect an amplified error signal in these hyperactive circuits. Such amplified error signals typically are associated with an adaptive change in response, yet in OCD these same repetitive responses persist to the point of distress and impairment. In contrast to this repetitive character of OC behavior, larger ERN amplitudes have been linked to better avoidance learning in reinforcement learning tasks. Study I thus investigated if OC symptomatology in non-patients predicted an enhanced ERN after suboptimal choices in a probabilistic learning task. Absent any behavioral differences, higher OC symptoms predicted smaller ERNs. Study II replicated this effect in an independent sample while also replicating findings of a larger ERN in a flanker task. There were no relevant behavioral differences in reinforcement learning or error monitoring as a function of symptom score. These findings implicate different, yet overlapping neural mechanisms underlying the negative deflection in the ERP following the execution of an erroneous motor response and the one following a suboptimal choice in a reinforcement learning paradigm. OC symptomatology may be dissociated in these neural systems, with hypoactivity in a system that enables learning to avoid maladaptive choices, and hyperactivity in another system that enables the same behavior to be repeated when it was assessed as not quite good enough the first time.

Individual variability in reward-based learning has been ascribed to quantitative variation in baseline levels of striatal dopamine. However, direct evidence for this pervasive hypothesis has hitherto been unavailable. We demonstrate that individual differences in reward-based reversal learning reflect variation in baseline striatal dopamine synthesis capacity, as measured with neurochemical positron emission tomography. Subjects with high baseline dopamine synthesis in the striatum showed relatively better reversal learning from unexpected rewards than from unexpected punishments, whereas subjects with low baseline dopamine synthesis in the striatum showed the reverse pattern. In addition, baseline dopamine synthesis predicted the direction of dopaminergic drug effects. The D2receptor agonist bromocriptine improved reward-based relative to punishment-based reversal learning in subjects with low baseline dopamine synthesis capacity, while impairing it in subjects with high baseline dopamine synthesis capacity in the striatum. Finally, this pattern of drug effects was outcome-specific, and driven primarily by drug effects on punishment-, but not reward-based reversal learning. These data demonstrate that the effects of D2receptor stimulation on reversal learning in humans depend on task demands and baseline striatal dopamine synthesis capacity.

Publisher Summary In recent years, computational models of learning and decision-making have become increasingly prevalent in psychology and neuroscience. These models describe brain function across a wide range of levels, from highly detailed models of ion channels and compartments of individual neurons, to abstract models that focus on the cognitive machinations the brain appears to produce. This chapter reviews a series of neurocomputational models that focus on the action selection and reinforcement learning functions of basal ganglia (BG), and their modulation by dopamine, as constrained by a broad range of data. The models have been successful in predicting behavioral outcomes resulting from manipulations of BG functionality via medications, diseases, disorders, and genetics. Furthermore, the chapter discusses how core computational principles can be extracted from complex neural models to develop simplified models in abstract mathematical form, which in turn can be quantitatively fit to behavioral data to test specific hypotheses. Such models are also useful for deriving best-fitting model parameters to correlate with biological signals, which can be used for further refinement and development of mechanistic principles.

Converging evidence implicates striatal dopamine (DA) in reinforcement learning, such that DA increases enhance “Go learning” to pursue actions with rewarding outcomes, whereas DA decreases enhance “NoGo learning” to avoid non-rewarding actions. Here we test whether these effects apply to the response time domain. We employ a novel paradigm which requires the adjustment of response times to a single response. Reward probability varies as a function of response time, whereas reward magnitude changes in the opposite direction. In the control condition, these factors exactly cancel, such that the expected value across time is constant (CEV). In two other conditions, expected value increases (IEV) or decreases (DEV), such that reward maximization requires either speeding up (Go learning) or slowing down (NoGo learning) relative to the CEV condition. We tested patients with Parkinson's disease (depleted striatal DA levels) on and off dopaminergic medication, compared with age-matched controls. While medicated, patients were better at speeding up in the DEV relative to CEV conditions. Conversely, nonmedicated patients were better at slowing down to maximize reward in the IEV condition. These effects of DA manipulation on cumulative Go/NoGo response time adaptation were captured with our a priori computational model of the basal ganglia, previously applied only to forced-choice tasks. There were also robust trial-to-trial changes in response time, but these single trial adaptations were not affected by disease or medication and are posited to rely on extrastriatal, possibly prefrontal, structures.

As one of the most complex neurocognitive disorders, schizophrenia (SZ) is a devastating condition for which the underlying sources are far from being fully understood. Indeed, it is likely that there are multiple etiologies to the disease and heterogeneity within the population. Moreover, it is impossible to understand from a purely mechanistic basis how a patient would come to believe so strongly in delusions as to, for example, gouge out his own eyes. Nevertheless, science marches forward, and the last 30 years or so have produced a wealth of knowledge regarding some of the risk factors, genetics, pharmacology, cognitive deficits, and underlying neurobiology associated with the disease. In part because of the efficacy of antipsychotic treatments via dopamine D2 receptor blockade, the majority of this research focuses on dysfunctions of the dopaminergic system, in both frontal cortex and basal ganglia, thought to be related to negative and positive symptoms, respectively. At the neurocognitive level,muchof the focushasbeenondysfunctionwithin dorsolateralprefrontal cortical circuits and their contributions to working memory, cognitive control, and attentional shifting. While dopamine plays a critical role in all these processes, it is perhaps more centrally related to aspects of motivational processing, which is surprisingly understudied in SZ. Indeed, it is possible to account for many of the frontal-dependent cognitive deficits in SZbypositingamore coredeficit in themotivational ‘‘gating’’ system for determining which information patients should ‘‘care’’ about and what they might ignore. Given the complexity of neural circuits involved in both cognitive and motivational functions, it becomes dauntingly difficult to capture the possible interactions of these circuits, and particularly how they are disrupted in SZ, with simple verbal depictions and static anatomical diagrams. Here I consider the potential application of computational neural network models as a principled and dynamic tool for exploring these interactions and psychopathology associated with dopaminergic dysfunction in SZ and which can lead to new testable predictions at both the neural and behavioral levels. These models enable one to simulate various anatomical and physiological pieces of data, using mathematical equations that capture how groups of neurons communicate activity to other neurons within and between brain areas. By incorporating aspects of neuronal physiology, connectivity, and synaptic plasticity within the basal ganglia–frontal cortical system, one can examine dynamics of this circuitry and how it may go awry. At the same time, it is not tractable to try to incorporate every known biological detail into a model, particularly when the goal is to discover how an entire system of brain regions interact to produce behavior. Thus, the models are also constrained by the need to account for existing data at these higher levels, such as effects of focal lesions or pharmacological manipulation on behavior. Critically, the models make new predictions about how the system works that would likely not have emerged otherwise and often were not conceived by the modeler prior to being built. Models can then be tested and refined and their implications explored in neurological conditions. To sum up a large body of basic research, models of frontostriatal function have generally suggested that these circuits support the following—(1) action selection: as in when making a choice among multiple competing alternatives and (2) reinforcement learning: as in modifying expectations and behavior following positive and negative outcomes. For the former process, ‘‘actions’’ to be selected include both lower level motor programs, consistent with the traditionally ascribed role of the basal ganglia in motor control, and higher level cognitive actions, such as when and when not to update/manipulate the contents of working memory. Reinforcement learning then operates on these actions such that adaptive actions are more likely to be repeated, whereas maladaptive actions are suppressed. Critically, according to both the models and available electrophysiological evidence, positive outcomes are reflected in terms of deviations from current expectations, a term referred to as a ‘‘positive prediction error,’’ and are encoded by phasic bursts of dopamine. Similarly, negative prediction errors are encoded by phasic dips or pauses in dopaminergic activity. These phasic bursts and dips modify corticostriatal synaptic plasticity, allowing the system To whom correspondence should be addressed; tel: 520-626-4787, fax: 520-621-9306, e-mail: mfrank@u.arizona.edu. Schizophrenia Bulletin doi:10.1093/schbul/sbn123

Animal findings have highlighted the modulatory role of phasic dopamine (DA) signaling in incentive learning, particularly in the acquisition of reward-related behavior. In humans, these processes remain largely unknown. In a recent study we demonstrated that a single low dose of a D2/D3 agonist (pramipexole) – assumed to activate DA autoreceptors and thus reduce phasic DA bursts – impaired reward learning in healthy subjects performing a probabilistic reward task. The purpose of the present study was to extend these behavioral findings using event-related potentials and computational modeling. Compared to the placebo group, participants receiving pramipexole showed increased feedback-related negativity to probabilistic rewards and decreased activation in dorsal anterior cingulate regions previously implicated in integrating reinforcement history over time. Additionally, findings of blunted reward learning in participants receiving pramipexole were simulated by reduced presynaptic DA signaling in response to reward in a neural network model of striatal-cortical function. These preliminary findings offer important insights on the role of phasic DA signals on reinforcement learning in humans, and provide initial evidence regarding the spatio-temporal dynamics of brain mechanisms underlying these processes.

The dopamine hypothesis of aging suggests that a monotonic dopaminergic decline accounts for many of the changes found in cognitive aging. The authors tested 44 older adults with a probabilistic selection task sensitive to dopaminergic function and designed to assess relative biases to learn more from positive or negative feedback. Previous studies demonstrated that low levels of dopamine lead to avoidance of those choices that lead to negative outcomes, whereas high levels of dopamine result in an increased sensitivity to positive outcomes. In the current study, age had a significant effect on the bias to avoid negative outcomes: Older seniors showed an enhanced tendency to learn from negative compared with positive consequences of their decisions. Younger seniors failed to show this negative learning bias. Moreover, the enhanced probabilistic integration of negative outcomes in older seniors was accompanied by a reduction in trial-to-trial learning from positive outcomes, thought to rely on working memory. These findings are consistent with models positing multiple neural mechanisms that support probabilistic integration and trial-to-trial behavior, which may be differentially impacted by older age.

The basal ganglia (BG) are critical for the coordination of several motor, cognitive, and emotional functions and become dysfunctional in several pathological states ranging from Parkinson's disease to Schizophrenia. Here we review principles developed within a neurocomputational framework of BG and related circuitry which provide insights into their functional roles in behavior. We focus on two classes of models: those that incorporate aspects of biological realism and constrained by functional principles, and more abstract mathematical models focusing on the higher level computational goals of the BG. While the former are arguably more "realistic", the latter have a complementary advantage in being able to describe functional principles of how the system works in a relatively simple set of equations, but are less suited to making specific hypotheses about the roles of specific nuclei and neurophysiological processes. We review the basic architecture and assumptions of these models, their relevance to our understanding of the neurobiological and cognitive functions of the BG, and provide an update on the potential roles of biological details not explicitly incorporated in existing models. Empirical studies ranging from those in transgenic mice to dopaminergic manipulation, deep brain stimulation, and genetics in humans largely support model predictions and provide the basis for further refinement. Finally, we discuss possible future directions and possible ways to integrate different types of models.