Search

Contemporary behavioral learning theory provides a comprehensive description of how we and other animals learn, and places behavioral flexibility and automaticity at heart of adaptive behaviors. However, to our knowledge, the computations supporting the interactions between deliberative and habitual decision-making systems are still poorly understood. Previous functional magnetic resonance imaging (fMRI) results suggest that the dorsal striatum host complementary computations that may differentially support deliberative and habitual processes [1] in the form of a dynamical interplay rather than a serial recruitment of strategies. From the same instrumental task, we develop a dual-system computational model of the two systems that can predict both performance (i.e., participant choices) and modulations in reaction times during learning. The instrumental task is a trial-and-error learning task requiring participants to find the correct associations between color stimuli and finger responses. To model the habitual system, we use a simple Q-learning algorithm (QL) [2] whose properties are fast responses, but slow convergence. For the deliberative (i.e goal-directed) system, we propose a new Bayesian Working Memory (BWM) which searches for information in the history of previous trials and stops as soon as the uncertainty on the action to perform decreases below a certain threshold. Last, we also propose a model for QL and BWM coordination. Currently, most models of system selection tend to control action selection concurrently, using either the deliberative or habitual model according to uncertainty criteria [3,4]. Only one model has investigated the relation between working memory and reinforcement learning [5] without, however explicitly modeling the temporal aspect of memory manipulation. In our approach, we propose a model for QL and BWM coordination. QL and BWM are merged such that the expensive memory manipulation is under control of, among others, the level of convergence of the habitual learning. Consequently, we also predict specific reaction times for each model that can be compared with the evolution of reaction times in instrumental learning tasks. Models are optimized for each subject with the NSGA-2 multi-objective evolutionary algorithm. The first fitness function is the Bayesian Information Criterion for individual choices. The second fitness function is also a likelihood that maximizes the probability of performing reaction times similar to humans. We compare the ability of the new model to explain human behavior with the QL or BWM only, as well as with a combination of these models based on [4], which reveals that the proposed model is in general more accurate. To conclude, we suggest that a close combination of BWM and QL better explains both choices and reaction times for most participants.

Perceptual decision-making has been widely studied using tasks in which subjects are presented with a visual stimulus and are instructed to select between two alternative options, based on a visual feature of the stimulus. Several studies have reported that the subjects' performance depends on the difficulty to indentify that specific feature. However, if we view decision-making as a continuous process that includes not only perceptual discrimination but also motor action one would expect that the cost of this action will have an impact on decision-making during the action preparation phase. We investigated this hypothesis with a combined theoretical and experimental approach. We setup a psychophysical task with human subjects, consisting of two blocks. In the first block, subjects either had to perform one of four right- or left-ward reaching movements, each with different biomechanical costs, or to make decisions between these right or left reaching movements. The movements were organised in two arrangements: one in which the movement towards the right implied a larger biomechanical cost than the movement towards the left, and one in which the costs were reversed. In the second block, subjects performed a moving dots perceptual discrimination task with different levels of coherence (right or left motion), and reported their decision with a reaching movement to the right or to the left using one of the aforementioned manipulations. As previously reported [1], we observed that when subjects freely chose between reaching movements of varying biomechanical cost, they exhibited a strong bias towards those movements with lower cost. Here, we report that this bias also influenced the motion-discrimination task, as subjects exhibited a tendency to perform the less costly movement when the ambiguity of the stimulus was high. The influence of the biomechanical bias declined as the ambiguity decreased. These results suggest that biomechanical costs are assessed and influence decisions even when the correct response does not depend on the properties of the motor apparatus. To study how visual and biomechanical information might be integrated to make decisions we use a mean-field approach of a binary decision-making model [2].The model consists of two populations of excitatory neurons that are responsive to left or right motion and that compete through mutual inhibition. We propose that even if the decision itself should exclusively depend on a single factor (motion direction), additional factors, such as biomechanics, may also influence the decision-making process. Simulation results show that a constant signal related to the biomechanical cost is necessary and sufficient to explain the probability of choice and reaction time in our motion discrimination task. Interestingly, the model shows that for same motion coherence the activation of the network is higher when the dots move toward the direction reported with the less costly movement than when they move towards the more costly one. An open question is whether the neural activity in the supplementary motor area, dorsal premotor cortex and primary motor cortex exhibit correlates of this biomechanical factor.

Gated accumulators, a.k.a race models, have been successfully used to describe neural selection processes, especially in visual selection [1,2]. The elements competing in the visual field feed evidence to counters, up to some selection threshold: the first counter reaching it is selected. Yet, these models do not explain which neurophysiological mechanism could account for the selection threshold itself, as well as how the activity in the accumulators, after threshold crossing, is transformed into a non-ambiguous motor command. Thus, we propose to create a model of saccade target selection including the Superior Colliculus (SC) and the Basal Ganglia (BG): their reciprocal connectivity has recently been emphasized [5], as well as their implication in visual selection processes [6]. By creating a model linking together recent SC [7] and BG [8] models, we investigate the possibility that selection in the SC results from SC-BG interaction rather than purely lateral inhibitions within the SC and we propose a new role for the Visuo-Motor prelude neurons of the intermediate layers of the SC [4], as noisy evidence integrators for each of the targets presented in the visual input layers. We also propose that the this integrator layer feed the BG, and that the BG disinhibitory feedback modulates the rate of integration in order to stochastically bias the selection process when targets of similar saliences compete. A second feedback loop from the BG to the SC applies direct inhibition to the ouptut of the integration layer to the deepers layers of the SC, and will selectively disinhibit only the winning target and suppress the activity of the distractors or unselected targets, so that the deep motor layer of the SC can produce an unbiaised motor command to the eyeplant. Our model's architecture offers a mechanism of gluing of the two hemifields for near-vertical saccades, and is able to reliably reproduce selection data and neurons activity profiles gathered in-vivo [3,4], by selecting one target among several distractors of inferior salience as well as one target among several other targets of similar salience. Furthermore, the model provides specific temporal and spatial discrimination predictions that can be tested in-vivo.

The activity of dopaminergic (DA) neurons has been hypothesized to encode a reward prediction error (RPE) which corresponds to the error signal in Temporal Difference (TD) learning algorithms. This hypothesis has been reinforced by numerous studies showing the relevance of TD learning algorithms to describe the role of basal ganglia in classical conditioning. However, recent recordings of DA neurons during multi-choice tasks raised contradictory interpretations on whether DA’s RPE signal is action dependent or not. Thus the precise TD algorithm (i.e. Actor-Critic, Q-learning or SARSA) that best describes DA signals remains unknown. Here we simulate and precisely analyze these TD algorithms on a multi-choice task performed by rats. We find that DA activity previously reported in this task is best fitted by a TD error which has not fully converged, and which converged faster than observed behavioral adaptation.

We elaborated computational models of the whole primate basal ganglia (BG) with the aim of answering to the long standing questions [1,2] of the relationships between the external part of the globus pallidus (GPe) and its internal part (GPi). In particular, we aimed to solve the apparent contradiction between a) anatomical studies showing that the GPi is an important target of a gabaergic inhibition from GPe [3] that is further thought to be powerful, as the synaptic targets in GPi are close to the soma [4], and b) electrophysiological studies that record very similar activities in both nuclei, at rest [5], in directional arm reaching tasks [6] and during eye closures [7]. Furthermore, the BG are supposedly the substrates of a generic selection process among cortical inputs, but the classical explanation of this selection stemming from the segregation between a direct pathway (with the "direct" striatal neurons targeting straight the GPi) and an indirect pathway (with the "indirect" striatal neurons targeting the GPe) is not compatible with primate anatomical data showing a nearly total overlap in the efferences of the striatal neurons. To address these problems, we parameterized mean-field models with the aim of fitting to an unprecedented large body of quantitative data obtained in primates. These data fall into two categories: a) anatomical data that directly constrain the parameter numerical value (they are the mean number of varicosities along the axons or the localization of the synapses along the dendrites) and b) electrophysiological data that can be used to constrain the model in simulation (they are the firing rate recorded either at rest, or when a given neurotransmitter is experimentally blocked by an antagonist). We built two matching fitness functions assessing how models were respecting these data, and we optimized them with the NSGA-2 multi-objective evolutionary algorithm [8]. We obtained more than one thousand different sets of parameters maximizing both fitnesses, and each of these models constitute a different answer to the initial paradox. By specifically examining the GPe -> GPi parameters, we found that a moderate strength of this projection is not antagonist with similar activities in both nuclei. Broader analysis led to the prediction that the GPe and GPi inputs are unbalanced, the former receiving stronger afferences from the striatum and the subthalamic nucleus than the latter. We also tested the selection capability of our plausible models by mimicking a classical arm-reaching task performed in monkeys [6], and we show that (1) the plausible basal ganglia models that we obtained are performing selections, indicating that the mainstream theory of selection is still valid without the postulate of a segregation into direct and indirect pathways, and that (2) this selection is performed more efficiently with a diffused projection pattern from the GPe into the GPi, as compared to a focused projection.