Search

Importance: The tools and insights of behavioral neuroscience grow apace, yet their clinical application is lagging., Observations: This article suggests that associative learning theory may be the algorithmic bridge to connect a burgeoning understanding of the brain with the challenges to the mind with which all clinicians and researchers are concerned., Conclusions and Relevance: Instead of giving up, talking past one another, or resting on the laurels of face validity, a consilient and collaborative approach is suggested: visiting laboratory meetings and clinical rounds and attempting to converse in the language of behavior and cognition to better understand and ultimately treat patients.

Reward-evoked dopamine transients are well established as prediction errors. However, the central tenet of temporal difference accounts-that similar transients evoked by reward-predictive cues also function as errors-remains untested. In the present communication we addressed this by showing that optogenetically shunting dopamine activity at the start of a reward-predicting cue prevents second-order conditioning without affecting blocking. These results indicate that cue-evoked transients function as temporal-difference prediction errors rather than reward predictions.

The phasic dopamine error signal is currently argued to be synonymous with the prediction error in Sutton and Barto (1987, 1998) model-free reinforcement learning algorithm (Schultz et al., 1997). This theory argues that phasic dopamine reflects a cached-value signal that endows reward-predictive cues with the scalar value inherent in reward. Such an interpretation does not envision a role for dopamine in more complex cognitive representations between events which underlie many forms of associative learning, restricting the role dopamine can play in learning. The cached-value hypothesis of dopamine makes three concrete predictions about when a phasic dopamine response should be seen and what types of learning this signal should be able to promote. We discuss these predictions in light of recent evidence which we believe provide particularly strong tests of their validity. In doing so, we find that while the phasic dopamine signal conforms to a cached-value account in some circumstances, other evidence demonstrate that this signal is not restricted to a model-free cached-value reinforcement learning signal. In light of this evidence, we argue that the phasic dopamine signal functions more generally to signal violations of expectancies to drive real-world associations between events., (Published by Elsevier Inc.)

Associative learning is driven by prediction errors. Dopamine transients correlate with these errors, which current interpretations limit to endowing cues with a scalar quantity reflecting the value of future rewards. We tested whether dopamine might act more broadly to support learning of an associative model of the environment. Using sensory preconditioning, we show that prediction errors underlying stimulus-stimulus learning can be blocked behaviorally and reinstated by optogenetically activating dopamine neurons. We further show that suppressing the firing of these neurons across the transition prevents normal stimulus-stimulus learning. These results establish that the acquisition of model-based information about transitions between nonrewarding events is also driven by prediction errors and that, contrary to existing canon, dopamine transients are both sufficient and necessary to support this type of learning. Our findings open new possibilities for how these biological signals might support associative learning in the mammalian brain in these and other contexts.

Unlabelled: When conditions change, organisms need to learn about the changed conditions without interfering with what they already know. To do so, they can assign the new learning to a new "state" and the old learning to a previous state. This state assignment is fundamental to behavioral flexibility. Cholinergic interneurons (CINs) in the dorsomedial striatum (DMS) are necessary for associative information to be compartmentalized in this way, but the mechanism by which they do so is unknown. Here we addressed this question by recording putative CINs from the DMS in rats performing a task consisting of a series of trial blocks, or states, that required the recall and application of contradictory associative information. We found that individual CINs in the DMS represented the current state throughout each trial. These state correlates were not observed in dorsolateral striatal CINs recorded in the same rats. Notably, DMS CIN ensembles tracked rats' beliefs about the current state such that, when states were miscoded, rats tended to make suboptimal choices reflecting the miscoding. State information held by the DMS CINs also depended completely on the orbitofrontal cortex, an area that has been proposed to signal environmental states. These results suggest that CINs set the stage for recalling associative information relevant to the current environment by maintaining a real-time representation of the current state. Such a role has novel implications for understanding the neural basis of a variety of psychiatric diseases, such as addiction or anxiety disorders, in which patients generalize inappropriately (or fail to generalize) between different environments., Significance Statement: Striatal cholinergic interneurons (CINs) are thought to be identical to tonically active neurons. These neurons have long been thought to have an important influence on striatal processing during reward-related learning. Recently, a more specific function for striatal CINs has been suggested, which is that they are necessary for striatal learning to be compartmentalized into different states as the state of the environment changes. Here we report that putative CINs appear to track rats' beliefs about which environmental state is current. We further show that this property of CINs depends on orbitofrontal cortex input and is correlated with choices made by rats. These findings could provide new insight into neuropsychiatric diseases that involve improper generalization between different contexts., (Copyright © 2016 the authors 0270-6474/16/366242-16$15.00/0.)

Underlying many complex behaviors are simple learned associations that allow humans and animals to anticipate the consequences of their actions. The orbitofrontal cortex and basolateral amygdala are two regions which are crucial to this process. In this review, we go back to basics and discuss the literature implicating both these regions in simple paradigms requiring the development of associations between stimuli and the motivationally-significant outcomes they predict. Much of the functional research surrounding this ability has suggested that the orbitofrontal cortex and basolateral amygdala play very similar roles in making these predictions. However, electrophysiological data demonstrates critical differences in the way neurons in these regions respond to predictive cues, revealing a difference in their functional role. On the basis of these data and theories that have come before, we propose that the basolateral amygdala is integral to updating information about cue-outcome contingencies whereas the orbitofrontal cortex is critical to forming a wider network of past and present associations that are called upon by the basolateral amygdala to benefit future learning episodes. The tendency for orbitofrontal neurons to encode past and present contingencies in distinct neuronal populations may facilitate its role in the formation of complex, high-dimensional state-specific associations., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Background: Neural activity in basolateral amygdala has recently been shown to reflect surprise or attention as predicted by the Pearce-Kaye-Hall model (PKH)--an influential model of associative learning. Theoretically, a PKH attentional signal originates in prediction errors of the kind associated with phasic firing of dopamine neurons. This requirement for prediction errors, coupled with projections from the midbrain dopamine system into basolateral amygdala, suggests that the PKH signal in amygdala may depend on dopaminergic input., Methods: To test this, we recorded single unit activity in basolateral amygdala in rats with 6-hydroxydopamine or sham lesions of the ipsilateral midbrain region. Neurons were recorded as the rats performed a task previously used to demonstrate both dopaminergic reward prediction errors and attentional signals in basolateral amygdala neurons., Results: We found that neurons recorded in sham lesioned rats exhibited the same attention-related PKH signal observed in previous studies. By contrast, neurons recorded in rats with ipsilateral 6-hydroxydopamine lesions failed to show attentional signaling., Conclusions: These results indicate a linkage between the neural instantiations of the basolateral complex of the amygdala attentional signal and dopaminergic prediction errors. Such a linkage would have important implications for understanding both normal and aberrant learning and behavior, particularly in diseases thought to have a primary effect on dopamine systems, such as addiction and schizophrenia., (Copyright © 2012 Society of Biological Psychiatry. Published by Elsevier Inc. All rights reserved.)

Although the human amygdala and striatum have both been implicated in associative learning, only the striatum's contribution has been consistently computationally characterized. Using a reversal learning task, we found that amygdala blood oxygen level-dependent activity tracked associability as estimated by a computational model, and dissociated it from the striatal representation of reinforcement prediction error. These results extend the computational learning approach from striatum to amygdala, demonstrating their complementary roles in aversive learning., (© 2011 Nature America, Inc. All rights reserved.)

In many cases, learning is thought to be driven by differences between the value of rewards we expect and rewards we actually receive. Yet learning can also occur when the identity of the reward we receive is not as expected, even if its value remains unchanged. Learning from changes in reward identity implies access to an internal model of the environment, from which information about the identity of the expected reward can be derived. As a result, such learning is not easily accounted for by model-free reinforcement learning theories such as temporal difference reinforcement learning (TDRL), which predicate learning on changes in reward value, but not identity. Here, we used unblocking procedures to assess learning driven by value- versus identity-based prediction errors. Rats were trained to associate distinct visual cues with different food quantities and identities. These cues were subsequently presented in compound with novel auditory cues and the reward quantity or identity was selectively changed. Unblocking was assessed by presenting the auditory cues alone in a probe test. Consistent with neural implementations of TDRL models, we found that the ventral striatum was necessary for learning in response to changes in reward value. However, this area, along with orbitofrontal cortex, was also required for learning driven by changes in reward identity. This observation requires that existing models of TDRL in the ventral striatum be modified to include information about the specific features of expected outcomes derived from model-based representations, and that the role of orbitofrontal cortex in these models be clearly delineated.

Attention or variations in event processing help drive learning. Lesion studies have implicated the central nucleus of the amygdala (CeA) in this process, particularly when expected rewards are omitted. However, lesion studies cannot specify how information processing in CeA supports such learning. To address these questions, we recorded CeA neurons in rats performing a task in which rewards were delivered or omitted unexpectedly. We found that activity in CeA neurons increased selectively at the time of omission and declined again with learning. Increased firing correlated with CeA-inactivation sensitive measures of attention. Notably CeA neurons did not fire to the cues or in response to unexpected rewards. These results indicate that CeA contributes to learning in response to reward omission due to a specific role in signaling actual omission rather than a more general involvement in signaling expectancies, errors, or reward value., (Copyright © 2010 Elsevier Inc. All rights reserved.)

The recognition that computational ideas from reinforcement learning are relevant to the study of neural circuits has taken the cognitive neuroscience community by storm. A central tenet of these models is that discrepancies between actual and expected outcomes can be used for learning. Neural correlates of such prediction-error signals have been observed now in midbrain dopaminergic neurons, striatum, amygdala and even prefrontal cortex, and models incorporating prediction errors have been invoked to explain complex phenomena such as the transition from goal-directed to habitual behavior. Yet, like any revolution, the fast-paced progress has left an uneven understanding in its wake. Here, we provide answers to ten simple questions about prediction errors, with the aim of exposing both the strengths and the limitations of this active area of neuroscience research.

Recent proposals have conceptualized piriform cortex as an association cortex, capable of integrating incoming olfactory information with descending input from higher order associative regions such as orbitofrontal cortex and basolateral amygdala (ABL). If true, encoding in piriform cortex should reflect associative features prominent in these areas during associative learning involving olfactory cues. We recently reported that neurons in anterior piriform cortex (APC) in rats exhibited significant plasticity in their responses to odor cues during associative learning. Here, we have repeated this study, recording from neurons in posterior piriform cortex (PPC), a region of piriform cortex that receives much stronger input from ABL. If associative encoding in piriform cortex is driven by inputs from ABL, then we should see more plasticity in PPC neurons than we observed in APC. Consistent with this hypothesis, we found that PPC neurons were highly associative and appeared to be somewhat more likely than neurons recorded in APC to alter their responses to the odor cues after reversal of the odor-outcome associations in the task. Further, odor-selective PPC populations exhibited markedly different firing patterns based on the valence of the odor cue. These results suggest associative encoding in piriform cortex is represented in a topographical fashion, reflecting the stronger and more specific input from olfactory bulb concerning the sensory features of odors in anterior regions and stronger input from ABL concerning the meaning of odors in posterior regions.

Recent proposals have conceptualized piriform cortex as an association cortex, capable of integrating incoming olfactory information with descending input from higher order associative regions such as orbitofrontal cortex (OFC). If true, encoding in piriform cortex should reflect associative features prominent in these areas during associative learning involving olfactory cues. To test this hypothesis, we recorded from neurons in OFC and anatomically related parts of the anterior piriform cortex (APC) in rats, learning and reversing novel odor discriminations. Findings in OFC were similar to what we have reported previously, with nearly all the cue-selective neurons exhibiting substantial plasticity during learning and reversal. Also, many of the cue-selective neurons were originally responsive in anticipation of the outcomes early in learning, thereby providing a single-unit representation of the cue-outcome associations. Some of these features were also evident in firing activity in APC, including some plasticity across learning and reversal. However, APC neurons failed to reverse cue selectivity when the associated outcome was changed, and the cue-selective population did not include neurons that were active prior to outcome delivery. Thus, although representations in APC are substantially more associative than expected in a purely sensory region, they do appear to be somewhat more constrained by the sensory features of the odor cues than representations in downstream areas of OFC.

Recent evidence has linked exposure to addictive drugs to an inability to employ information about adverse consequences, or outcomes, to control behavior. For instance, addicts and drug-experienced animals fail to adapt their behavior to avoid adverse outcomes in gambling and reversal tasks or after changes in the value of expected rewards. These deficits are similar to those caused by damage to the orbitofrontal cortex, suggesting that addictive drugs may cause long-lasting changes in the representation of outcome associations in a circuit that includes the orbitofrontal cortex. Here we test this hypothesis by recording from orbitofrontal neurons in a discrimination task in rats previously exposed to cocaine (30 mg/kg i.p. for 14 days). We found that orbitofrontal neurons recorded in cocaine-experienced rats failed to signal the adverse outcome at the time a decision was made in the task. The loss of this signal was associated with abnormal changes in response latencies on aversive trials. Furthermore, upon reversal of the cue-outcome associations, orbitofrontal neurons in cocaine-treated rats with enduring reversal impairments failed to reverse their cue-selectivity, while orbitofrontal neurons in cocaine-treated rats with normal performance showed an increase in the plasticity of cue-selective firing after reversal. These results provide direct neurophysiological evidence that exposure to cocaine can cause behaviorally relevant changes in the processing of associative information in a circuit that includes the orbitofrontal cortex.

Orbitofrontal cortex is characterized by its unique pattern of connections with subcortical areas, such as basolateral amygdala. Here we distinguish between the critical role of these areas in associative learning and the pivotal contribution of OFC to the manipulation of this information to control behavior. This contribution reflects the ability of OFC to signal the desirability of expected outcomes, which requires the integration of associative information with information concerning internal states and goals in representational memory.

Clinical evidence indicates that damage to ventromedial prefrontal cortex disrupts goal-directed actions that are guided by motivational and emotional factors. As a consequence, patients with such damage characteristically engage in maladaptive behaviors. Other research has shown that neurons in the corresponding orbital region of prefrontal cortex in laboratory animals encode information regarding the incentive properties of goals or expected events. The present study investigates the effect of neurotoxic orbitofrontal cortex (OFC) lesions in the rat on responses that are normally influenced by associations between a conditioned stimulus (CS) and the incentive value of reinforcement. Rats were first trained to associate a visual CS with delivery of food pellets to a food cup. As a consequence of learning, rats approached the food cup during the CS in anticipation of reinforcement. In a second training phase, injection of LiCl followed consumption of the food unconditioned stimulus (US) in the home cage, a procedure used to alter the incentive value of the US. Subsequently, rats were returned to the conditioning chamber, and their responding to the CS in the absence of the food US was tested. Lesions of OFC did not affect either the initial acquisition of a conditioned response to the light CS in the first training phase or taste aversion learning in the second training phase. In the test for devaluation, however, OFC rats exhibited no change in conditioned responding to the visual CS. This outcome contrasts with the behavior of control rats; after devaluation of the US a significant decrease occurred in approach to the food cup during presentation of the CS. The results reveal an inability of a cue to access representational information about the incentive value of associated reinforcement after OFC damage.