Search

Obsessive-compulsive disorder (OCD) and pathological gambling (PG) are accompanied by deficits in behavioural flexibility. In reinforcement learning, this inflexibility can reflect asymmetric learning from outcomes above and below expectations. In alternative frameworks, it reflects perseveration independent of learning. Here, we examine evidence for asymmetric reward-learning in OCD and PG by leveraging model-based functional magnetic resonance imaging (fMRI). Compared with healthy controls (HC), OCD patients exhibited a lower learning rate for worse-than-expected outcomes, which was associated with the attenuated encoding of negative reward prediction errors in the dorsomedial prefrontal cortex and the dorsal striatum. PG patients showed higher and lower learning rates for better- and worse-than-expected outcomes, respectively, accompanied by higher encoding of positive reward prediction errors in the anterior insula than HC. Perseveration did not differ considerably between the patient groups and HC. These findings elucidate the neural computations of reward-learning that are altered in OCD and PG, providing a potential account of behavioural inflexibility in those mental disorders. A brain imaging study reveals that obsessive-compulsive disorder and problem gambling are associated with distinct patterns of learning from positive and negative reward prediction errors, providing a neurocomputational account of abnormal inflexible behaviors in those disorders. [ABSTRACT FROM AUTHOR]

From an associative perspective the acquisition of new goal-directed actions requires the encoding of specific action-outcome (AO) associations and, therefore, sensitivity to the validity of an action as a predictor of a specific outcome relative to other events. Although competitive architectures have been proposed within associative learning theory to achieve this kind of identity-based selection, whether and how these architectures are implemented by the brain is still a matter of conjecture. To investigate this issue, we trained human participants to encode various AO associations while undergoing functional neuroimaging (fMRI). We then degraded one AO contingency by increasing the probability of the outcome in the absence of its associated action while keeping other AO contingencies intact. We found that this treatment selectively reduced performance of the degraded action. Furthermore, when a signal predicted the unpaired outcome, performance of the action was restored, suggesting that the degradation effect reflects competition between the action and the context for prediction of the specific outcome. We used a Kalman filter to model the contribution of different causal variables to AO learning and found that activity in the medial prefrontal cortex (mPFC) and the dorsal anterior cingulate cortex (dACC) tracked changes in the association of the action and context, respectively, with regard to the specific outcome. Furthermore, we found the mPFC participated in a network with the striatum and posterior parietal cortex to segregate the influence of the various competing predictors to establish specific AO associations. [ABSTRACT FROM AUTHOR]

It is now commonly accepted that instrumental actions can reflect goal-directed control; i.e., they can show sensitivity to changes in the relationship to and the value of their consequences. With overtraining, stress, neurodegeneration, psychiatric conditions, or after exposure to various drugs of abuse, goal-directed control declines and instrumental actions are performed independently of their consequences. Although this latter insensitivity has been argued to reflect the development of habitual control, the lack of a positive definition of habits has rendered this conclusion controversial. Here we consider various alternative definitions of habit, including recent suggestions they reflect chunked action sequences, to derive criteria with which to categorize responses as habitual. We consider various theories regarding the interaction between goal-directed and habitual controllers and propose a collaborative model based on their hierarchical integration. We argue that this model is consistent with the available data, can be instantiated both at an associative level and computationally and generates interesting predictions regarding the influence of this collaborative integration on behavior. [ABSTRACT FROM AUTHOR]

State-space and action representations form the building blocks of decision-making processes in the brain; states map external cues to the current situation of the agent whereas actions provide the set of motor commands from which the agent can choose to achieve specific goals. Although these factors differ across environments, it is currently unknown whether or how accurately state and action representations are acquired by the agent because previous experiments have typically provided this information a priori through instruction or pre-training. Here we studied how state and action representations adapt to reflect the structure of the world when such a priori knowledge is not available. We used a sequential decision-making task in rats in which they were required to pass through multiple states before reaching the goal, and for which the number of states and how they map onto external cues were unknown a priori. We found that, early in training, animals selected actions as if the task was not sequential and outcomes were the immediate consequence of the most proximal action. During the course of training, however, rats recovered the true structure of the environment and made decisions based on the expanded state-space, reflecting the multiple stages of the task. Similarly, we found that the set of actions expanded with training, although the emergence of new action sequences was sensitive to the experimental parameters and specifics of the training procedure. We conclude that the profile of choices shows a gradual shift from simple representations to more complex structures compatible with the structure of the world. [ABSTRACT FROM AUTHOR]

Computational modeling plays an important role in modern neuroscience research. Much previous research has relied on statistical methods, separately, to address two problems that are actually interdependent. First, given a particular computational model, Bayesian hierarchical techniques have been used to estimate individual variation in parameters over a population of subjects, leveraging their population-level distributions. Second, candidate models are themselves compared, and individual variation in the expressed model estimated, according to the fits of the models to each subject. The interdependence between these two problems arises because the relevant population for estimating parameters of a model depends on which other subjects express the model. Here, we propose a hierarchical Bayesian inference (HBI) framework for concurrent model comparison, parameter estimation and inference at the population level, combining previous approaches. We show that this framework has important advantages for both parameter estimation and model comparison theoretically and experimentally. The parameters estimated by the HBI show smaller errors compared to other methods. Model comparison by HBI is robust against outliers and is not biased towards overly simplistic models. Furthermore, the fully Bayesian approach of our theory enables researchers to make inference on group-level parameters by performing HBI t-test. [ABSTRACT FROM AUTHOR]

Popular computational models of decision-making make specific assumptions about learning processes that may cause them to underfit observed behaviours. Here we suggest an alternative method using recurrent neural networks (RNNs) to generate a flexible family of models that have sufficient capacity to represent the complex learning and decision- making strategies used by humans. In this approach, an RNN is trained to predict the next action that a subject will take in a decision-making task and, in this way, learns to imitate the processes underlying subjects’ choices and their learning abilities. We demonstrate the benefits of this approach using a new dataset drawn from patients with either unipolar (n = 34) or bipolar (n = 33) depression and matched healthy controls (n = 34) making decisions on a two-armed bandit task. The results indicate that this new approach is better than baseline reinforcement-learning methods in terms of overall performance and its capacity to predict subjects’ choices. We show that the model can be interpreted using off-policy simulations and thereby provides a novel clustering of subjects’ learning processes—something that often eludes traditional approaches to modelling and behavioural analysis. [ABSTRACT FROM AUTHOR]

Summary Choice between actions often requires the ability to retrieve action consequences in circumstances where they are only partially observable. This capacity has recently been argued to depend on orbitofrontal cortex; however, no direct evidence for this hypothesis has been reported. Here, we examined whether activity in the medial orbitofrontal cortex (mOFC) underlies this critical determinant of decision-making in rats. First, we simulated predictions from this hypothesis for various tests of goal-directed action by removing the assumption that rats could retrieve partially observable outcomes and then tested those predictions experimentally using manipulations of the mOFC. The results closely followed predictions; consistent deficits only emerged when action consequences had to be retrieved. Finally, we put action selection based on observable and unobservable outcomes into conflict and found that whereas intact rats selected actions based on the value of retrieved outcomes, mOFC rats relied solely on the value of observable outcomes. [ABSTRACT FROM AUTHOR]

Behavioral evidence suggests that instrumental conditioning is governed by two forms of action control: a goal-directed and a habit learning process. Model-based reinforcement learning (RL) has been argued to underlie the goal-directed process; however, the way in which it interacts with habits and the structure of the habitual process has remained unclear. According to a flat architecture, the habitual process corresponds to model-free RL, and its interaction with the goal-directed process is coordinated by an external arbitration mechanism. Alternatively, the interaction between these systems has recently been argued to be hierarchical, such that the formation of action sequences underlies habit learning and a goal-directed process selects between goal-directed actions and habitual sequences of actions to reach the goal. Here we used a two-stage decision-making task to test predictions from these accounts. The hierarchical account predicts that, because they are tied to each other as an action sequence, selecting a habitual action in the first stage will be followed by a habitual action in the second stage, whereas the flat account predicts that the statuses of the first and second stage actions are independent of each other. We found, based on subjects' choices and reaction times, that human subjects combined single actions to build action sequences and that the formation of such action sequences was sufficient to explain habitual actions. Furthermore, based on Bayesian model comparison, a family of hierarchical RL models, assuming a hierarchical interaction between habit and goal-directed processes, provided a better fit of the subjects' behavior than a family of flat models. Although these findings do not rule out all possible model-free accounts of instrumental conditioning, they do show such accounts are not necessary to explain habitual actions and provide a new basis for understanding how goal-directed and habitual action control interact. [ABSTRACT FROM AUTHOR]

Mammalian embryogenesis is a dynamic process involving gene expression and mechanical forces between proliferating cells. The exact nature of these interactions, which determine the lineage patterning of the trophectoderm and endoderm tissues occurring in a highly regulated manner at precise periods during the embryonic development, is an area of debate. We have developed a computational modeling framework for studying this process, by which the combined effects of mechanical and genetic interactions are analyzed within the context of proliferating cells. At a purely mechanical level, we demonstrate that the perpendicular alignment of the animal-vegetal (a-v) and embryonic-abembryonic (eb-ab) axes is a result of minimizing the total elastic conformational energy of the entire collection of cells, which are constrained by the zona pellucida. The coupling of gene expression with the mechanics of cell movement is important for formation of both the trophectoderm and the endoderm. In studying the formation of the trophectoderm, we contrast and compare quantitatively two hypotheses: (1) The position determines gene expression, and (2) the gene expression determines the position. Our model, which couples gene expression with mechanics, suggests that differential adhesion between different cell types is a critical determinant in the robust endoderm formation. In addition to differential adhesion, two different testable hypotheses emerge when considering endoderm formation: (1) A directional force acts on certain cells and moves them into forming the endoderm layer, which separates the blastocoel and the cells of the inner cell mass (ICM). In this case the blastocoel simply acts as a static boundary. (2) The blastocoel dynamically applies pressure upon the cells in contact with it, such that cell segregation in the presence of differential adhesion leads to the endoderm formation. To our knowledge, this is the first attempt to combine cell-based spatial mechanical simulations with genetic networks to explain mammalian embryogenesis. Such a framework provides the means to test hypotheses in a controlled in silico environment. [ABSTRACT FROM AUTHOR]

Clinical and experimental observations show individual differences in the development of addiction. Increasing evidence supports the hypothesis that dopamine receptor availability in the nucleus accumbens (NAc) predisposes drug reinforcement. Here,modeling striatal-midbrain dopaminergic circuit, we propose a reinforcement learning model for addiction based on the actor-critic model of striatum. Modeling dopamine receptors in the NAc as modulators of learning rate for appetitive-but not aversive-stimuli in the critic-but not the actor-we define vulnerability to addiction as a relatively lower learning rate for the appetitive stimuli, compared to aversive stimuli, in the critic. We hypothesize that an imbalance in this learning parameter used by appetitive and aversive learning systems can result in addiction. We elucidate that the interaction between the degree of individual vulnerability and the duration of exposure to drug has two progressive consequences: deterioration of the imbalance and establishment of an abnormal habitual response in the actor. Using computational language, the proposed model describes how development of compulsive behavior can be a function of both degree of drug exposure and individual vulnerability. Moreover, the model describes how involvement of the dorsal striatum in addiction can be augmented progressively. The model also interprets other forms of addiction, such as obesity and pathological gambling, in a common mechanism with drug addiction. Finally, the model provides an answer for the question of why behavioral addictions are triggered in Parkinson's disease patients byD2 dopamine agonist treatments. [ABSTRACT FROM AUTHOR]

Based on the dopamine hypotheses of cocaine addiction and the assumption of decrement of brain reward system sensitivity after long-term drug exposure, we propose a computational model for cocaine addiction. Utilizing average reward temporal difference reinforcement learning, we incorporate the elevation of basal reward threshold after long-term drug exposure into the model of drug addiction proposed by Redish. Our model is consistent with the animal models of drug seeking under punishment. In the case of nondrug reward, the model explains increased impulsivity after long-term drug exposure. Furthermore, the existence of a blocking effect for cocaine is predicted by our model. [ABSTRACT FROM AUTHOR]

Methadone detoxification is among the widely used treatment programs for opioid dependence. The aims of this study were to identify which patient baseline factors and treatment regimen features are predictors of the treatment outcome in an outpatient flexible dose-duration methadone detoxification program. We studied 126 opioid dependents in a naturalistic nonexperimental clinical setting. The patients were assessed for baseline demographic characteristics, and drug abuse characteristics. Treatment regimen features were recorded during the program. Successful treatment completion was defined as the last daily dose of methadone being less than 15 mg, negative urine analysis in the last two weeks of treatment, and based on the final clinician-client's decision. Out of 126 patients, 60 patients completed detoxification successfully. Younger age, longer duration of the opioid abuse, and higher subjective opiate intoxication severity before treatment entry were all significantly associated with negative treatment outcome. Among treatment regimen features, higher maximum methadone dose had a marginally significant independent effect on treatment failure. Patients with maximum methadone dose of more than 75 mg per day had around ten times worse success rate when compared to those who received lesser doses. The study findings could be used to predict treatment outcome and prognosis in a more individualized and patient-tailored approach in the real clinical setting. Guideline development for treatment selection and outcome monitoring in addiction medicine based on similar studies could enhance treatment outcome in clinical services. [ABSTRACT FROM AUTHOR]

We develop an automated variational method for inference in models with Gaussian process (gp) priors and general likelihoods. The method supports multiple outputs and multiple latent functions and does not require detailed knowledge of the conditional likelihood, only needing its evaluation as a black-box function. Using a mixture of Gaussians as the variational distribution, we show that the evidence lower bound and its gradients can be estimated efficiently using samples from univariate Gaussian distributions. Furthermore, the method is scalable to large datasets which is achieved by using an augmented prior via the inducing-variable approach underpinning most sparse gp approximations, along with parallel computation and stochastic optimization. We evaluate our approach quantitatively and qualitatively with experiments on small datasets, medium-scale datasets and large datasets, showing its competitiveness under different likelihood models and sparsity levels. On the large-scale experiments involving prediction of airline delays and classification of handwritten digits, we show that our method is on par with the state-of-the-art hard-coded approaches for scalable gp regression and classification. [ABSTRACT FROM AUTHOR]

Evaluating the future consequences of actions is achievable by simulating a mental search tree into the future. Expanding deep trees, however, is computationally taxing. Therefore, machines and humans use a plan-until-habit scheme that simulates the environment up to a limited depth and then exploits habitual values as proxies for consequences that may arise in the future. Two outstanding questions in this scheme are "in which directions the search tree should be expanded?", and "when should the expansion stop?". Here we propose a principled solution to these questions based on a speed/accuracy tradeoff: deeper expansion in the appropriate directions leads to more accurate planning, but at the cost of slower decision-making. Our simulation results show how this algorithm expands the search tree effectively and efficiently in a grid-world environment. We further show that our algorithm can explain several behavioral patterns in animals and humans, namely the effect of time-pressure on the depth of planning, the effect of reward magnitudes on the direction of planning, and the gradual shift from goal-directed to habitual behavior over the course of training. The algorithm also provides several predictions testable in animal/human experiments., Competing Interests: The authors have declared that no competing interests exist.

Goal-directed action involves making high-level choices that are implemented using previously acquired action sequences to attain desired goals. Such a hierarchical schema is necessary for goal-directed actions to be scalable to real-life situations, but results in decision-making that is less flexible than when action sequences are unfolded and the decision-maker deliberates step-by-step over the outcome of each individual action. In particular, from this perspective, the offline revaluation of any outcomes that fall within action sequence boundaries will be invisible to the high-level planner resulting in decisions that are insensitive to such changes. Here, within the context of a two-stage decision-making task, we demonstrate that this property can explain the emergence of habits. Next, we show how this hierarchical account explains the insensitivity of over-trained actions to changes in outcome value. Finally, we provide new data that show that, under extended extinction conditions, habitual behaviour can revert to goal-directed control, presumably as a consequence of decomposing action sequences into single actions. This hierarchical view suggests that the development of action sequences and the insensitivity of actions to changes in outcome value are essentially two sides of the same coin, explaining why these two aspects of automatic behaviour involve a shared neural structure., (© 2014 The Author(s) Published by the Royal Society. All rights reserved.)