Search

Chronic stress evokes wide-ranging behavioral alterations, including risk avoidance, increased motoric output, and reduced consummatory behaviors. These are often interpreted as dysfunctions, but they may subserve adaptations for coping with existential threats. We tested this in a cohort of rats previously exposed to mild unpredictable stress for 5 weeks. Previously stressed rats exhibited the typically increased avoidance of open field and altered responses to predator odor, suggesting enhanced sensitivity to threatening contexts and cues. Interestingly, these animals collected rewards at a higher rate than controls, because they locomoted faster, spent less time in off-task (exploratory) behavior, and committed fewer licks at feeders. Further, they were not impaired in flexibly shifting choice as reward probabilities changed among feeders, suggesting that behavioral adaptations are not simply of transference to behavioral control to neural systems insensitive to reward (e.g. habits). These data add to a small but growing body of evidence indicating that stress shifts responses away from exploration and toward exploitation of resources, possibly to reduce threat exposure.HighlightsRats with a history of stress collected reward at a higher rate than controls on an operant task, owing to increase locomotion speed, reduced off-task behavior, and reduced time licking at feeders.Previously stressed rats exhibited increased win-stay responses than controls, suggesting the involvement of neural circuits related to goal-directed responding.Previously stressed rats performed equally to controls on a task requiring a shift of preferences based on reward probability, suggesting that they are not simply relying more on habit-based neural systems.

Multiple neural systems contribute to choice adaptation following reinforcement. Recent evidence suggests that the lateral habenula (LHb) plays a key role in such adaptations, particularly when reinforcements are worse than expected. Here, we investigated the effects of bilateral LHb lesions on responding in a binary choice task with no discriminatory cues. LHb lesions in rats decreased win-stay responses but surprisingly left lose-shift responses intact. This same dissociated effect was also observed after systemic administration of d-amphetamine in a separate cohort of animals. These results suggest that at least some behavioural responses triggered by reward omission do not depend on an intact LHb.

Animals tend to repeat actions that are associated with reward delivery, whereas they tend to shift responses to alternate choices following reward omission. These so-called win-stay and lose-shift responses are employed by a wide range of animals in a variety of decision-making scenarios, and depend on dissociated regions of the striatum. Specifically, lose-shift responding is impaired by extensive excitotoxic lesions of the lateral striatum. Here we used focal lesions to assess whether dorsal and ventral regions of the lateral striatum contribute differently to this effect. We found that damage to ventrolateral striatum reduced lose-shift responding without impairing win-stay, motoric, or motivational aspects of behaviour in the task, whereas lesions confined to the dorsolateral striatum significantly impaired the ability of rats to complete trials of the task. Moreover, lesions to the dorsomedial striatum had no effect on either lose-shift or win-stay responding. Together, these data suggest a novel role of the ventral portion of the lateral striatum in driving lose-shift decisions.

The brain reciprocally communicates with the rest of the body via neural, endocrine, immune, and other systems. This is crucial for coordinating the complex behavioral and physiological responses needed to cope with the many challenges of life. The Anterior Cingulate Cortex (ACC) is a key brain structure involved in assessing rewards and threats, as well as activating appropriate responses. This is a dynamic process that depends on evolving needs and challenges. Important challenges include illness or injury. These typically involve inflammation and pain, which evoke neuroinflammatory processes in the brain to drive sickness behaviours. In the short term, sickness behaviours are considered adaptive, as they promote convalescence (e.g. low mood; lethargy, fatigue, social withdrawal), and enhanced threat appraisal (e.g. anxiety) to combat increased risk/vulnerability associated with sickness. Chronic inflammation, however, appears to remodel the system to inappropriately activate threat-coping responses, resulting in depressive and/or anxious phenotypes. These mood disorders are particularly pronounced in diseases and disorders associated with gut dysfunction, which feature chronic inflammation and altered ACC function. We propose that chronic inflammation remodels ACC physiology such that it errantly predicts heightened danger based on a mental model (a.k.a ‘schema’) of the world. This evokes chronic activation of threat-coping systems, including endocrine signaling (e.g. adrenaline), and anxiety. Inflammation can be driven by brain systems involving ACC, leading to a feedback-cycle that self-reinforces pathological states. This theory accounts for a wealth of clinical and preclinical data that implicate the ACC in disorders of mood and gastrointestinal function, and reveals a key player in the gut-brain axis that may represent a novel therapeutic target.

The anterior cingulate cortex (ACC) appears to support decisions by encoding the effort-reward utility of choice options. We show here that d-amphetamine (AMPH) has dose-dependent effects on this encoding and on neural dynamics in rat ACC that are concordant with its behavioral effects. Low-dose AMPH increased task engagement and had mild effects on neural encoding, whereas high doses disrupted utility signaling and decreased task engagement. The disruption involved reduced reward signaling and compressed effort-reward encoding of utility cells, which corresponded with reduced reward consumption behaviors. Furthermore, low-dose AMPH stabilized and accelerated trajectories of neural activity in state-space, whereas high-dose AMPH destabilized trajectories. We propose that low-dose AMPH increases both excitability and stability, which preserves information and accelerates evolution of a neural ‘script’ for task execution. Excessive excitability at high doses overcomes stability enhancement to suppress weakly encoded features (e.g. reward) and cause deviation from the script, which interrupts task performance.Significance StatementAmphetamine reduced reward signaling by individual neurons in rat prefrontal cortex, but increased the stability of ensemble dynamics. These effects account for animals’ increased task engagement, despite reduced reward intake.

Psychostimulants such as d-amphetamine (AMPH) often have behavioral effects that appear paradoxical within the framework of optimal choice theory. AMPH typically increases task engagement and the effort animals exert for reward, despite decreasing reward valuation. We investigated neural correlates of this phenomenon in the anterior cingulate cortex (ACC), a brain structure implicated in signaling cost-benefit utility. AMPH decreased signaling of reward, but not effort, in the ACC of freely-moving rats. Ensembles of simultaneously recorded neurons generated task-specific trajectories of neural activity encoding past, present, and future events. Low-dose AMPH contracted these trajectories and reduced their variance, whereas high-dose AMPH expanded both. We propose that under low-dose AMPH, increased network stability balances moderately increased excitability, which promotes accelerated unfolding of a neural ‘script’ for task execution, despite reduced reward valuation. Noise from excessive excitability at high doses overcomes stability enhancement to drive frequent deviation from the script, impairing task execution.

Behavior provides important insights into neuronal processes. For example, analysis of reaching movements can give a reliable indication of the degree of impairment in neurological disorders such as stroke, Parkinson disease, or Huntington disease. The analysis of such movement abnormalities is notoriously difficult and requires a trained evaluator. Here, we show that a deep neural network is able to score behavioral impairments with expert accuracy in rodent models of stroke. The same network was also trained to successfully score movements in a variety of other behavioral tasks. The neural network also uncovered novel movement alterations related to stroke, which had higher predictive power of stroke volume than the movement components defined by human experts. Moreover, when the regression network was trained only on categorical information (control = 0; stroke = 1), it generated predictions with intermediate values between 0 and 1 that matched the human expert scores of stroke severity. The network thus offers a new data-driven approach to automatically derive ratings of motor impairments. Altogether, this network can provide a reliable neurological assessment and can assist the design of behavioral indices to diagnose and monitor neurological disorders., This study demonstrates that a state-of-the-art neural network can provide automated scoring of motor deficits with an accuracy equivalent to that of human experts and has the potential teach us to develop more-sensitive behavioral tests.

The reinforcement learning (RL) paradigm allows agents to solve tasks through trial-and-error learning. To be capable of efficient, long-term learning, RL agents should be able to apply knowledge gained in the past to new tasks they may encounter in the future. The ability to predict actions' consequences may facilitate such knowledge transfer. We consider here domains where an RL agent has access to two kinds of information: agent-centric information with constant semantics across tasks, and environment-centric information, which is necessary to solve the task, but with semantics that differ between tasks. For example, in robot navigation, environment-centric information may include the robot's geographic location, while agent-centric information may include sensor readings of various nearby obstacles. We propose that these situations provide an opportunity for a very natural style of knowledge transfer, in which the agent learns to predict actions' environmental consequences using agent-centric information. These predictions contain important information about the affordances and dangers present in a novel environment, and can effectively transfer knowledge from agent-centric to environment-centric learning systems. Using several example problems including spatial navigation and network routing, we show that our knowledge transfer approach can allow faster and lower cost learning than existing alternatives.

Although male and female rats appear to perform differently in some tasks, a clear picture of sex differences in decision-making has yet to develop. This is in part due to significant variability arising from differences in strains and tasks. The aim of this study was to characterize the effects of sex on specific response elements in a reinforcement learning task so as to help identify potential explanations for this variability. We found that the primary difference between sexes was the propensity to approach feeders out of the task context. This extraneous feeder sampling affects choice on subsequent trials in both sexes by promoting a lose-shift response away from the last feeder sampled. Female rats, however, were more likely to engage in this extraneous feeder sampling, and therefore exhibited a greater rate of this effect. Once trials following extraneous sampling were removed, there were no significant sex differences in any of the tested measures. These data suggest that feeder approach outside of the task context, which is often not recorded, could produce a confound in sex-based differences of reinforcement sensitivity in some tasks.

Observations of neurons in a resting brain and neurons in cultures often display spontaneous scale-free (SF) collective dynamics in the form of information cascades, also called ‘neuronal avalanches’. This has motivated the so called critical brain hypothesis which posits that the brain is self-tuned to a critical point or regime, separating exponentially-growing dynamics from quiescent states, to achieve optimality. Yet, how such optimality of information transmission is related to behavior and whether it persists under behavioral transitions has remained a fundamental knowledge gap. Here, we aim to tackle this challenge by studying behavioral transitions in mice using two-photon calcium imaging of the retrosplenial cortex (RSC)—an area of the brain well positioned to integrate sensory, mnemonic, and cognitive information by virtue of its strong connectivity with the hippocampus, medial prefrontal cortex, and primary sensory cortices. Our work shows that the response of the underlying neural population to behavioral transitions can vary significantly between different sub-populations such that one needs to take the structural and functional network properties of these sub-populations into account to understand the properties at the total population level. Specifically, we show that the RSC contains at least one sub-population capable of switching between two different SF regimes, indicating an intricate relationship between behavior and the optimality of neuronal response at the subgroup level. This asks for a potential reinterpretation of the emergence of self-organized criticality in neuronal systems.

Amphetamine and other drugs of abuse have both short-term and long-lasting effects on brain function, and drug sensitization paradigms often result in chronic impairments in behavioral flexibility. Here we show that acute amphetamine administration temporarily renders rats less sensitive to reward omission, as revealed by a decrease in lose-shift responding during a binary choice task. Intracerebral infusions of amphetamine into the ventral striatum did not affect lose-shift responding but did increase impulsive behavior in which rats chose to check both reward feeders before beginning the next trial. In contrast to acute systemic and intracerebral infusions, sensitization through repeated exposure induced long-lasting increased sensitivity to reward omission. These treatments did not affect choices on trials following reward delivery (i.e. win-stay responding), and sensitization increased spine density in the sensorimotor striatum. The dichotomous effects of amphetamine on short-term and long-term loss sensitivity, and the null effect on win-stay responding, are consistent with a shift of behavioral control to the sensorimotor striatum after drug sensitization. These data provide a new demonstration of such a shift in a novel task unrelated to drug administration, and suggests that the dominance of sensorimotor control persists over many hundreds of trials after sensitization.

Inflammatory bowel disease (IBD) is characterized by relapsing periods of gut inflammation, and is comorbid with depression, anxiety, and cognitive deficits. Animal models of IBD that explore the behavioral consequences almost exclusively use acute models of gut inflammation, which fails to recapitulate the cyclic, chronic nature of IBD. This study sought to identify behavioral differences in digging, memory, and stress-coping strategies in mice exposed to one (acute) or three (chronic) cycles of gut inflammation, using the dextran sodium sulfate (DSS) model of colitis. Similar levels of gut pathology were observed between acute and chronically exposed mice, although mice in the chronic treatment had significantly shorter colons, suggesting more severe disease. Behavioral measures revealed an unexpected pattern in which chronic treatment evoked fewer deficits than acute treatment. Specifically, acutely-treated mice showed alterations in measures of object burying, novel object recognition, object location memory, and stress-coping (forced swim task). Chronically-treated animals, however, showed similar alterations in object burying, but not the other measures. These data suggest an adaptive or tolerizing effect of repeated cycles of peripheral gut inflammation on mnemonic function and stress-coping, whereas some other behaviors continue to be affected by gut inflammation. We speculate that the normalization of some functions may involve the reversion to the baseline state of the hypothalamic-pituitary-adrenal axis and/or levels of neuroinflammation, which are both activated by the first exposure to the colitic agent.

Decision-making in the mammalian brain typically involves multiple brain structures within the midbrain, thalamus, striatum, limbic system, and cortex. Although task specific contributions of each brain region have been identified, neurons responding to reinforcement have been found throughout these structures. We sought to determine if any brain area, or cluster of areas, are the source of information, and if the fidelity of information varies among the areas. We recorded simultaneous field potentials (FPs) in rats from seven brain regions as they completed a binary choice task. The FPs of a 0.5 s window following reinforcement were given as input to a classifier that attempted to predict whether or not the rat received reward on each trial. The classifier correctly categorized reward on 77% of trials. Any region-specific signal could be omitted without lowering accuracy. Frequencies above 40 Hz and signals recorded later than 0.25 s following reinforcement were necessary to achieve this accuracy. Further, the classifier was able to predict reinforcement outcome above chance levels when using FPs from any single recorded brain region. Some combinations of structures, however, were more predictive than others. Analysis of FPs prior to reward revealed most regions reflected the prior probability of reward. Lastly, analyses of information flow suggested reinforcement information does not originate within a single structure of the network, within the resolution afforded by FP recordings. These data suggest reward delivery information is rapidly distributed non-uniformly across the network, and there is no canonical flow of information about reward events in the recorded structures.

Oscillatory activity is a ubiquitous property of brain signals, and yet relatively few studies have investigated how the phase of such ongoing oscillations affects our cognition. One of the main findings in this field is that the phase of electroencephalography (EEG) in the alpha band can affect perception of milliseconds-long stimuli. However, the importance of the phase of EEG for processing more naturalistic stimuli, which have a much longer duration, is still not clear. To address this question here, we presented word-nonword pairs, each of which was visible for 5 s and measured the effect of EEG phase during stimulus onset on later memory recall. The task consisted of an encoding (learning) phase in which 20 novel word-nonword pairs were presented, followed by a test phase in which participants were shown one of the seen words with four target nonwords to choose from. We found that memory recall performance was higher when the words during encoding were presented at a descending phase of the theta oscillation. This effect was the strongest in the frontal cortex. These results suggest that the phase of ongoing cortical activity can affect memorization of seconds-long stimuli that are an integral part of many daily tasks.

The anterior cingulate cortex (ACC) encodes information supporting mnemonic and cognitive processes. We show here that a rat’s position can be decoded with high spatiotemporal resolution from ACC activity. ACC neurons encoded the current state of the animal and task, except for brief excursions that sometimes occurred at target feeders. During excursions, the decoded position became more similar to a remote target feeder than the rat’s physical position. Excursions recruited activation of neurons encoding choice and reward, and the likelihood of excursions at a feeder was inversely correlated with feeder preference. These data suggest that the excursion phenomenon was related to evaluating real or fictive choice outcomes, particularly after disfavoured reinforcements. We propose that the multiplexing of position with choice-related information forms a mental model isomorphic with the task space, which can be mentally navigated via excursions to recall multimodal information about the utility of remote locations.

The propensity of animals to shift choices immediately after unexpectedly poor reinforcement outcomes is a pervasive strategy across species and tasks. We report here on the memory supporting such lose-shift responding in humans, assessed using a binary choice task in which random responding is the optimal strategy. Participants exhibited little lose-shift responding when fully attending to the task, but this increased by 30%-40% in participants that performed with additional cognitive load that is known to tax executive systems. Lose-shift responding in the cognitively loaded adults persisted throughout the testing session, despite being a sub-optimal strategy, but was less likely as the time increased between reinforcement and the subsequent choice. Furthermore, children (5-9 years old) without load performed similarly to the cognitively loaded adults. This effect disappeared in older children aged 11-13 years old. These data provide evidence supporting our hypothesis that lose-shift responding is a default and reflexive strategy in the mammalian brain, likely mediated by a decaying memory trace, and is normally suppressed by executive systems. Reducing the efficacy of executive control by cognitive load (adults) or underdevelopment (children) increases its prevalence. It may therefore be an important component to consider when interpreting choice data, and may serve as an objective behavioral assay of executive function in humans that is easy to measure.

Background The physical distance between predator and prey is a primary determinant of behavior, yet few paradigms exist to study this reliably in rodents. New method The utility of a robotically controlled laser for use in a predator–prey-like (PPL) paradigm was explored for use in rats. This involved the construction of a robotic two-dimensional gimbal to dynamically position a laser beam in a behavioral test chamber. Custom software was used to control the trajectory and final laser position in response to user input on a console. The software also detected the location of the laser beam and the rodent continuously so that the dynamics of the distance between them could be analyzed. When the animal or laser beam came within a fixed distance the animal would either be rewarded with electrical brain stimulation or shocked subcutaneously. Results Animals that received rewarding electrical brain stimulation could learn to chase the laser beam, while animals that received aversive subcutaneous shock learned to actively avoid the laser beam in the PPL paradigm. Mathematical computations are presented which describe the dynamic interaction of the laser and rodent. Comparison with existing methods The robotic laser offers a neutral stimulus to train rodents in an open field and is the first device to be versatile enough to assess distance between predator and prey in real time. Conclusions With ongoing behavioral testing this tool will permit the neurobiological investigation of predator/prey-like relationships in rodents, and may have future implications for prosthetic limb development through brain–machine interfaces.

Problem gambling is thought to be highly comorbid with attention-deficit hyperactivity disorder (ADHD). We propose that the neurobiological pathologies underlying problem gambling overlap with those in ADHD. In this study, we used a simplified computerized version of the Iowa Gambling Task (IGT) to assess differences in reinforcement-driven choice adaptation among participants with pathological gambling and/or ADHD. The task contained two choice options with different net payouts over the session; a good bet that resulted in a win of +50 points on 60% of trials (and -50 points on 40%), and a bad bet that resulted in +100 points on 40% of the trials (and -100 points on 60%). We quantified participants' preference for the good bet over the session and their sensitivity to reinforcement. Both the control subjects and medicated ADHD nongamblers significantly increased the proportion of good bets over the 400-trial session. Subjects with problem gambling performed worse than controls and ADHD nongamblers, but better than our limited sample of unmedicated ADHD gamblers. Control subjects, medicated ADHD nongamblers, and unmedicated ADHD nongamblers tended to tolerate losses following good bets, whereas unmedicated ADHD gamblers tended to tolerate losses following bad bets. These data reveal that ADHD, particularly when treated with medication, is not associated with poor choices on the IGT, but may exacerbate pathological choices in problem gamblers. It seems that stabilization of dopamine signaling that occurs when ADHD is treated is itself also a treatment for certain forms of problem gambling.

Gambling studies have described a "near-miss effect" wherein the experience of almost winning increases gambling persistence. The near-miss has been proposed to inflate the value of preceding actions through its perceptual similarity to wins. We demonstrate here, however, that it acts as a conditioned stimulus to positively or negatively influence valuation, dependent on reward expectation and cognitive engagement. When subjects are asked to choose between two simulated slot machines, near-misses increase valuation of machines with a low payout rate, whereas they decrease valuation of high payout machines. This contextual effect impairs decisions and persists regardless of manipulations to outcome feedback or financial incentive provided for good performance. It is consistent with proposals that near-misses cause frustration when wins are expected, and we propose that it increases choice stochasticity and overrides avoidance of low-valued options. Intriguingly, the near-miss effect disappears when subjects are required to explicitly value machines by placing bets, rather than choosing between them. We propose that this task increases cognitive engagement and recruits participation of brain regions involved in cognitive processing, causing inhibition of otherwise dominant systems of decision-making. Our results reveal that only implicit, rather than explicit strategies of decision-making are affected by near-misses, and that the brain can fluidly shift between these strategies according to task demands.

Δ-9-Tetrahydrocannabinol (THC) is the main psychoactive component of marijuana and has potent effects on decision-making, including a proposed reduction in cognitive flexibility. We demonstrate here that acute THC administration differentially affects some of the processes that contribute to cognitive flexibility. Specifically, THC reduces lose-shift responding in which female rats tend to immediately shift choice responses away from options that result in reward omission on the previous trial. THC, however, did not impair the ability of rats to flexibly bias responses toward feeders with higher probability of reward in a reversal task. This response adaptation developed over several trials, suggesting that THC did not impair slower forms of reinforcement learning needed to choose among options with unequal utility. This dissociation of THC's effects on innate/rapid and learned/gradual decision-making processes was unexpected, but is supported by emerging evidence that lose-shift responding is mediated by neural mechanisms distinct from those involved in other forms of reinforcement learning. The present data suggest that, at least in some tasks, the apparent reductions in cognitive flexibility by THC may be explained by the immediate effects on loss sensitivity, rather than impairments of all processes used for choice adaptation.

We used focal brain lesions in rats to examine how dorsomedial (DMS) and dorsolateral (DLS) regions of the striatum differently contribute to response adaptation driven by the delivery or omission of rewards. Rats performed a binary choice task under two modes: one in which responses were rewarded on half of the trials regardless of choice; and another 'competitive' one in which only unpredictable choices were rewarded. In both modes, control animals were more likely to use a predictable lose-switch strategy than animals with lesions of either DMS or DLS. Animals with lesions of DMS presumably relied more on DLS for behavioural control, and generated repetitive responses in the first mode. These animals then shifted to a random response strategy in the competitive mode, thereby performing better than controls or animals with DLS lesions. Analysis using computational models of reinforcement learning indicated that animals with striatal lesions, particularly of the DLS, had blunted reward sensitivity and less stochasticity in the choice mechanism. These results provide further evidence that the rodent DLS is involved in rapid response adaptation that is more sophisticated than that embodied by the classic notion of habit formation driven by gradual stimulus-response learning.

The mammalian brain is thought to use a version of Model-based Reinforcement Learning (MBRL) to guide “goal-directed” behavior, wherein animals consider goals and make plans to acquire desired outcomes. However, conventional MBRL algorithms do not fully explain animals' ability to rapidly adapt to environmental changes, or learn multiple complex tasks. They also require extensive computation, suggesting that goal-directed behavior is cognitively expensive. We propose here that key features of processing in the hippocampus support a flexible MBRL mechanism for spatial navigation that is computationally efficient and can adapt quickly to change. We investigate this idea by implementing a computational MBRL framework that incorporates features inspired by computational properties of the hippocampus: a hierarchical representation of space, “forward sweeps” through future spatial trajectories, and context-driven remapping of place cells. We find that a hierarchical abstraction of space greatly reduces the computational load (mental effort) required for adaptation to changing environmental conditions, and allows efficient scaling to large problems. It also allows abstract knowledge gained at high levels to guide adaptation to new obstacles. Moreover, a context-driven remapping mechanism allows learning and memory of multiple tasks. Simulating dorsal or ventral hippocampal lesions in our computational framework qualitatively reproduces behavioral deficits observed in rodents with analogous lesions. The framework may thus embody key features of how the brain organizes model-based RL to efficiently solve navigation and other difficult tasks.

Some have claimed that the medial prefrontal cortex (mPFC) mediates decision making. Others suggest mPFC is selectively involved in the retrieval of remote long-term memory. Yet others suggests mPFC supports memory and consolidation on time-scales ranging from seconds to days. How can all these roles be reconciled? We propose that the function of the mPFC is to learn associations between context, locations, events, and corresponding adaptive responses, particularly emotional responses. Thus, the ubiquitous involvement of mPFC in both memory and decision making may be due to the fact that almost all such tasks entail the ability to recall the best action or emotional response to specific events in a particular place and time. An interaction between multiple memory systems may explain the changing importance of mPFC to different types of memories over time. In particular, mPFC likely relies on the hippocampus to support rapid learning and memory consolidation.

Experience Versus Models There is an ongoing debate over what the orbitofrontal cortex contributes to behavior, learning, and decision-making. Jones et al. (p. 953 ) found that the orbitofrontal cortex was important for value-based computations when value must be inferred from an associative model of the task but not when value estimates based on previous experience are sufficient. This result calls into question the assumption that this region simply signals economic value. However, it would be consistent with a concept of the orbitofrontal cortex as being important for constructing model-based representations of the world that are orthogonal to value.

Reinforcement learning (RL) allows an intelligent agent to learn optimal behavior as it interacts with its environment. Conventional model-based RL algorithms learn rapidly, but can be slow to adapt to sudden changes in the environment. Animals' brains, however, are thought to employ model-based RL mechanisms for learning, but are able to adapt to changes with relative ease. By employing “transfer learning”, they can recycle previously learned information to solve new problems with minimal new learning. We developed two brain-inspired methods that can allow model-based RL to cope with changes to the underlying process being learned: hierarchical state abstraction, and context-switching. Hierarchical state abstraction allows a previously-learned model to be efficiently adapted for use in a new task, while context switching allows learned models to be saved and recalled at the appropriate times. We test these mechanisms using grid-world simulations in which the goal remains constant, but contingencies for reaching it frequently change. These mechanisms allow an agent to significantly outperform a conventional model-based RL algorithm in the task.

SummaryAdult brain function and behavior are influenced by neuronal network formation during development. Genetic susceptibility factors for adult psychiatric illnesses, such as Neuregulin-1 and Disrupted-in-Schizophrenia-1 (DISC1), influence adult high brain functions, including cognition and information processing. These factors have roles during neurodevelopment and are likely to cooperate, forming pathways or “signalosomes.” Here we report the potential to generate an animal model via in utero gene transfer in order to address an important question of how nonlethal deficits in early development may affect postnatal brain maturation and high brain functions in adulthood, which are impaired in various psychiatric illnesses such as schizophrenia. We show that transient knockdown of DISC1 in the pre- and perinatal stages, specifically in a lineage of pyramidal neurons mainly in the prefrontal cortex, leads to selective abnormalities in postnatal mesocortical dopaminergic maturation and behavioral abnormalities associated with disturbed cortical neurocircuitry after puberty.PaperFlick

Animals quickly learn to approach sources of food. Here, we report on a form of approach in which rats made volitional orofacial contact with inactive feeders between trials of a self-paced operant task. This extraneous feeder sampling (EFS) was never reinforced and therefore imposed an opportunity and effort cost. EFS decreased during initial training but persisted thereafter. The relative rate of EFS to operant responding increased with novel changes to the operant chamber, reward devaluation by prefeeding, or lesions to the dorsolateral striatum. We speculate that this may function to increase exploration when the task is uncertain (early in learning or introduction of novel apparatus components), when the opportunity cost is low, or when the learned sensorimotor solution is compromised. Moreover, EFS strongly affected subsequent choices by triggering a lose-shift response away from the sampled feeder, even though it occurred outside of the trial context. This indicates that at least some behaviors occurring between trials impact future behaviors and should be considered in decision-making studies.

Basal ganglia circuits are organized as parallel loops that have been proposed to compete in a winner-take-all fashion to determine the appropriate behavioral outcome. However, limited experimental support for strong lateral inhibition mechanisms within striatal regions questions this model. Here, stimulation of the prefrontal cortex (PFC) using naturally occurring bursty patterns inhibited firing in most nucleus accumbens (NA) projection neurons. When an excitatory response was observed for one stimulation site, neighboring PFC sites evoked inhibition in the same neuron. Furthermore, PFC stimulation activated interneurons, and PFC-evoked inhibition was blocked by GABAA antagonists in corticoaccumbens slice preparations. Thus bursting PFC activity recruits local inhibition in the NA, shaping responses of projection neurons with a topographical arrangement that allows inhibition among parallel corticoaccumbens channels. The data indicate a high order of information processing within striatal circuits that should be considered in models of basal ganglia function and disease.

Hippocampal inputs to the nucleus accumbens (NA) have been proposed to implement a gating mechanism by driving NA medium spiny neurons (MSNs) to depolarized up states that facilitate action potential firing in response to brief activation of the prefrontal cortex (PFC). Brief PFC stimulation alone, on the other hand, could not drive NA up states. As these studies were conducted using single-pulse PFC stimulation, it remains possible that PFC activation with naturalistic, bursty patterns can also drive up states in NA MSNs. Here, we assessed NA responses to PFC stimulation with a pattern similar to what is typically observed in awake animals during PFC-relevant behaviors. In vivo intracellular recordings from NA MSNs revealed that brief 20-50 Hz PFC stimulus trains evoked depolarizations that were similar to spontaneous up states in NA MSNs and were sustained beyond stimulus offset. Similar train stimulation of corticoaccumbens afferents in a parasagittal slice preparation evoked large amplitude depolarizations in NA MSNs that were sustained during stimulation but decayed rapidly following stimulation offset, suggesting that activation of cortical afferents can drive MSN depolarizations but other mechanisms may contribute to sustaining up states. These data suggest that NA MSNs integrate temporal features of PFC activation and that the NA gating model can be reformulated to include a PFC-driven gating mechanism during periods of high PFC firing, such as during cognitively demanding tasks. Synapse 63:173-180, 2009. (c) 2008 Wiley-Liss, Inc.

Prenatal stress (PNS) is a significant risk factor for the development of psychopathology in adulthood such as anxiety, depression, schizophrenia and addiction. Animal models of PNS resemble many of the effects of PNS on humans and provide a means to study the accumulated effects of PNS over several generations on brain function. Here, we examined how mild PNS delivered during the third week in utero over four consecutive generations affects behavioral flexibility and functional signaling among cortical and limbic structures. These multi-generational prenatally stressed (MGPNS) rats were not impaired on an odor-cued reversal learning task as compared to control animals. Unilateral field potential (FP) recordings from the medial prefrontal cortex, basolateral amygdala, ventral hippocampus, and striatal territories revealed widespread differences in brain signaling between these groups during the odor sampling phase of the task. The FP power was significantly lower in most structures across most frequency bands in MGPNS animals, and the relative increase in power from baseline during the task was lower for the beta band (12–30 Hz) in MGPNS animals as compared to controls. The coherence of FPs between brain regions, however, was much higher in MGPNS animals among all structures and for most frequency bands. We propose that this pattern of changes in brain signaling reflects a simplification of network processing, which is consistent with reports of reduced spine density and dendritic complexity in the brains of animals receiving PNS. Our data support the proposal that recurrent ancestral stress leads to adaptations in the brain, and that these may confer adaptive behavior in some circumstances as compared to single-generation PNS.

The propensity of animals to shift choices immediately after unexpectedly poor reinforcement outcomes is a pervasive strategy across species and tasks. We report here that the memory supporting such lose-shift responding in rats rapidly decays during the intertrial interval and persists throughout training and testing on a binary choice task, despite being a suboptimal strategy. Lose-shift responding is not positively correlated with the prevalence and temporal dependence of win-stay responding, and it is inconsistent with predictions of reinforcement learning on the task. These data provide further evidence that win-stay and lose-shift are mediated by dissociated neural mechanisms and indicate that lose-shift responding presents a potential confound for the study of choice in the many operant choice tasks with short intertrial intervals. We propose that this immediate lose-shift responding is an intrinsic feature of the brain’s choice mechanisms that is engaged as a choice reflex and works in parallel with reinforcement learning and other control mechanisms to guide action selection.

SummaryMemory formation is hypothesized to involve the generation of event-specific neural activity patterns during learning and the subsequent spontaneous reactivation of these patterns. Here, we present evidence that these processes can also be observed in urethane-anesthetized rats and are enhanced by desynchronized brain state evoked by tail pinch, subcortical carbachol infusion, or systemic amphetamine administration. During desynchronization, we found that repeated tactile or auditory stimulation evoked unique sequential patterns of neural firing in somatosensory and auditory cortex and that these patterns then reoccurred during subsequent spontaneous activity, similar to what we have observed in awake animals. Furthermore, the formation of these patterns was blocked by an NMDA receptor antagonist, suggesting that the phenomenon depends on synaptic plasticity. These results suggest that anesthetized animals with a desynchronized brain state could serve as a convenient model for studying stimulus-induced plasticity to improve our understanding of memory formation and replay in the brain.

Motivated behavior exhibits properties that change with experience and partially dissociate among a number of brain structures. Here, we review evidence from rodent experiments demonstrating that multiple brain systems acquire information in parallel and either cooperate or compete for behavioral control. We propose a conceptual model of systems interaction wherein a ventral emotional memory network involving ventral striatum (VS), amygdala, ventral hippocampus, and ventromedial prefrontal cortex triages behavioral responding to stimuli according to their associated affective outcomes. This system engages autonomic and postural responding (avoiding, ignoring, approaching) in accordance with associated stimulus valence (negative, neutral, positive), but does not engage particular operant responses. Rather, this emotional system suppresses or invigorates actions that are selected through competition between goal-directed control involving dorsomedial striatum (DMS) and habitual control involving dorsolateral striatum (DLS). The hippocampus provides contextual specificity to the emotional system, and provides an information rich input to the goal-directed system for navigation and discriminations involving ambiguous contexts, complex sensory configurations, or temporal ordering. The rapid acquisition and high capacity for episodic associations in the emotional system may unburden the more complex goal-directed system and reduce interference in the habit system from processing contingencies of neutral stimuli. Interactions among these systems likely involve inhibitory mechanisms and neuromodulation in the striatum to form a dominant response strategy. Innate traits, training methods, and task demands contribute to the nature of these interactions, which can include incidental learning in non-dominant systems. Addition of these features to reinforcement learning models of decision-making may better align theoretical predictions with behavioral and neural correlates in animals.

The prefrontal cortex (PFC) is critical for decision making, and it becomes dysfunctional in many neuropsychiatric disorders. Studies in schizophrenia patients and relevant animal models suggest loss of PFC inhibitory interneuron function. For instance, rats with a neonatal ventral hippocampal lesion (NVHL) show a deficient modulation of PFC interneurons by dopamine (DA). Whether the PFC becomes disinhibited in this model and alters decision making remains to be determined. Here, we recorded neural activity in the medial PFC of NVHL rats during a reward-discounting choice task that activated DA systems. Rats were trained to sample odors that instructed them to select one of two feeders that delivered unequal amounts of liquid. Putative pyramidal neurons in the PFC were hyperactive whereas task-related field potential oscillations were significantly reduced in NVHL rats, consistent with impaired interneuron activation by DA during odor sampling leading to disorganized processing. Cognitive flexibility was tested by examining response bias and errors after reversing reward outcomes. NVHL rats demonstrated impaired flexibility as they were less able to track changes in reward outcome and made more response errors than controls did. Reducing cortical excitability with the metabotropic glutamate receptor 2/3 agonist LY379268 (1 mg/kg, i.p.) improved behavioral flexibility in NVHL rats but not controls. Furthermore, D2 dopamine receptors were involved, as the antagonist eticlopride (0.02 mg/kg, i.p.) reduced the ability to switch only in control animals. We conclude that NVHL rats present PFC disinhibition, which affects neural information processing and the selection of appropriate behavioral responses.