Search

International audience; In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine contributes to the control of this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple non-stationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in random exploration rate. this study shows that dopamine could adapt the exploration-exploitation trade-off in decision-making when facing changing environmental contingencies. All organisms need to make choices for their survival while being confronted to uncertainty in their environment. Animals and humans tend to exploit actions likely to provide desirable outcomes, but they must also take into account the possibility that environmental contingencies and the outcome of their actions may vary with time. Behavioral flexibility is thus needed in volatile environments in order to detect and learn new contingencies 1. This requires a delicate balance between exploitation of known resources and exploration of alternative options that may have become advantageous. How this exploration/exploitation dilemma may be resolved and regulated is still a subject of active research in the fields of Neuroscience and Machine Learning 2-5. Dopamine holds a fundamental place in contemporary theories of learning and decision-making. The temporal evolution of phasic dopamine signals across learning has been extensively replicated, and is most of the time considered as evidence of a role in learning 6-8 , but see alternative views in Coddington et al. 9. Dopamine reward prediction error (RPE) signals have been identified in a variety of instrumental and Pavlovian conditioning tasks 10-13. They affect plasticity and action value learning in cortico-basal networks 14-16 and have been directly related to behavioral adaptation in a number of decision-making tasks in humans, non-human primates 17 and rodents 18-21. Accordingly, it is commonly assumed that manipulations of dopamine activity affect the rate of learning, but this could represent a misconception. Besides learning, the role of dopamine in the control of behavioral performance is still unclear. Dopamine is known to modulate incentive choice (the tendency to differentially weigh costs and benefits) 22,23 , and risk-taking behavior 24 , as well as other motivational aspects such as effort and response vigour 25. Because dopamine is one of the key factors that may encode success or uncertainty, it might modulate decisions by biasing them toward options that present the largest uncertainty 26,27. This would correspond to a "directed" exploration strategy 5,28,29. Alternatively, success and failure could affect tonic dopamine levels and control random exploration of all options, as recently proposed by Humphries et al. 30. This form of undirected exploration, which is often difficult

Virtual reality (VR) technologies allow for the creation of 3D environments that can be exploited at the human level, maximizing humans’ use of perceptual skills through their sensory channels, and enabling them to actively influence the course of events that take place in the virtual environment (VE). As such, they constitute a significant asset in the treatment of post-traumatic stress disorder (PTSD) via exposure therapy. In this article, we review the VR tools that have been developed to date for the treatment of PTSD. The article aims to analyze how VR technologies can be exploited from a sensorimotor and interactive perspective. The findings from this analysis suggest a significant emphasis on sensory stimulation to the detriment of interaction. Finally, we propose new ideas regarding the more successful integration of sensorimotor activities and interaction into VR exposure therapy for PTSD.

An important current challenge in Human-Robot Interaction (HRI) is to enable robots to learn on-the-fly from human feedback. However, humans show a great variability in the way they reward robots. We propose to address this issue by enabling the robot to combine different learning strategies, namely model-based (MB) and model-free (MF) reinforcement learning. We simulate two HRI scenarios: a simple task where the human congratulates the robot for putting the right cubes in the right boxes, and a more complicated version of this task where cubes have to be placed in a specific order. We show that our existing MB-MF coordination algorithm previously tested in robot navigation works well here without retuning parameters. It leads to the maximal performance while producing the same minimal computational cost as MF alone. Moreover, the algorithm gives a robust performance no matter the variability of the simulated human feedback, while each strategy alone is impacted by this variability. Overall, the results suggest a promising way to promote robot learning flexibility when facing variable human feedback., Comment: 6 pages, 5 figures, written for the RO-MAN 2020 conference. arXiv admin note: text overlap with arXiv:2004.14698

Action selection has been hypothesized to be a key function of the basal ganglia, yet the nuclei involved, their interactions and the importance of the direct/indirect pathway segregation in such process remain debated. Here, we design a spiking computational model of the monkey basal ganglia derived from a previously published population model, initially parameterized to reproduce electrophysiological activity at rest and to embody as much quantitative anatomical data as possible. As a particular feature, both models exhibit the strong overlap between the direct and indirect pathways that has been documented in non‐human primates. Here, we first show how the translation from a population to an individual neuron model was achieved, with the addition of a minimal number of parameters. We then show that our model performs action selection, even though it was built without any assumption on the activity carried out during behaviour. We investigate the mechanisms of this selection through circuit disruptions and found an instrumental role of the off‐centre/on‐surround structure of the MSN‐STN‐GPi circuit, as well as of the MSN‐MSN and FSI‐MSN projections. This validates their potency in enabling selection. We finally study the pervasive centromedian and parafascicular thalamic inputs that reach all basal ganglia nuclei and whose influence is therefore difficult to anticipate. Our model predicts that these inputs modulate the responsiveness of action selection, making them a candidate for the regulation of the speed–accuracy trade‐off during decision‐making., We designed a spiking model of the basal ganglia of the monkey, derived from a mean‐field function‐agnostic model. Following monkey neuroanatomic data, it is characterized by strongly overlapping direct and indirect pathways. We show action selection is an emerging property of the resulting system and that it does not depend on direct/indirect pathway segregation.

Computational Intelligence and Artificial Intelligence are both aiming at building machines and softwares capable of intelligent behavior. They are consequently prone to interactions, even if the latter is not necessarily interested in understanding how cognition emerges from the brain substrate. In this chapter, we enumerate, describe and discuss the most important fields of interactions. Some are methodological and are concerned with information representation, processing and learning. At the functional level, the focus is set on major cognitive functions like perception, navigation, decision making and language. Among the salient characteristics of the critical contributions of Computational Neuroscience to the development of intelligent systems, its systemic view of the cerebral functioning is particularly precious to model highly multimodal cognitive functions like decision making and language and to design cognitive architectures for the autonomous behavior of robots.

International audience; Hippocampal offline reactivations during reward-based learning, usually categorized as replay events, have been found to be important for performance improvement over time and for memory consolidation. Recent computational work has linked these phenomena to the need to transform reward information into state-action values for decision-making and to propagate it to all relevant states of the environment. Nevertheless, it is still unclear whether an integrated reinforcement learning mechanism could account for the variety of awake hippocampal reacti-vations, including variety in order (forward and reverse reactivated trajectories) and variety in the location where they occur (reward site or decision-point). Here we present a model-based bidirectional search model which accounts for a variety of hippocampal reactivations. The model combines forward trajectory sampling from current position and backward sampling through prioritized sweeping from states associated with large reward prediction errors until the two trajectories connect. This is repeated until stabilization of state-action values (convergence), which could explain why hippocampal reactivations drastically diminish when the animal's performance stabilizes. Simulations in a multiple T-maze task show that forward reactivations are prominently found at decision-points while backward reactivations are exclusively generated at reward sites. Finally, the model can generate imaginary trajectories that are not allowed to the agent during task performance. We raise some experimental predictions and implications for future studies of the role of the hippocampo-prefronto-striatal network in learning.

International audience; Multiple in vivo studies have shown that place cells from the hippocampus replay previously experienced trajectories. These replays are commonly considered to mainly reflect memory consolidation processes. Some data, however , have highlighted a functional link between replays and reinforcement learning (RL). This theory, extensively used in machine learning, has introduced efficient algorithms and can explain various behavioral and physiological measures from different brain regions. RL algorithms could constitute a mechanistic description of replays, and explain how replays can reduce the number of iterations required to explore the environment during learning. We review here the main findings concerning the different hippocampal replay types and the possible associated RL models (either model-based, model-free or hybrid model types). We conclude by tying these frameworks together. We illustrate the link between data and RL through a series of model simulations. This review, at the frontier between informatics and biology, paves the way for future work on replays.

This work describes a truck-driving simulator designed for the care of truckers suffering from post-traumatic stress disorder (PTSD). This simulator exploits a new approach using virtual reality (VR) technologies in the treatment of PTSD. Indeed, we exploit full capabilities of VR technologies to not only expose the user to the frightening stimuli but also to allow him to be active in this environment that may contain anxiety-provoking stimuli. These two aspects are supported throughout a gamification process where the game mechanics are mapped to the expected learning mechanics. Moreover, the developed simulator is fully customizable. Two truck drivers suffering from PTSD have experimented it. The intervention allowed a desensitization and a verbalization of fears associated to the past trauma.

Taking inspiration from how the brain coordinates multiple learning systems is an appealing strategy to endow robots with more flexibility. One of the expected advantages would be for robots to autonomously switch to the least costly system when its performance is satisfying. However, to our knowledge no study on a real robot has yet shown that the measured computational cost is reduced while performance is maintained with such brain-inspired algorithms. We present navigation experiments involving paths of different lengths to the goal, dead-end, and non-stationarity (i.e., change in goal location and apparition of obstacles). We present a novel arbitration mechanism between learning systems that explicitly measures performance and cost. We find that the robot can adapt to environment changes by switching between learning systems so as to maintain a high performance. Moreover, when the task is stable, the robot also autonomously shifts to the least costly system, which leads to a drastic reduction in computation cost while keeping a high performance. Overall, these results illustrates the interest of using multiple learning systems.

How do we translate self-motion into goal-directed actions? Here we investigate the cognitive architecture underlying self-motion processing during exploration and goal-directed behaviour. The task, performed in an environment with limited and ambiguous external landmarks, constrained mice to use self-motion based information for sequence-based navigation. The post-behavioural analysis combined brain network characterization based on c-Fos imaging and graph theory analysis as well as computational modelling of the learning process. The study revealed a widespread network centred around the cerebral cortex and basal ganglia during the exploration phase, while a network dominated by hippocampal and cerebellar activity appeared to sustain sequence-based navigation. The learning process could be modelled by an algorithm combining memory of past actions and model-free reinforcement learning, which parameters pointed toward a central role of hippocampal and cerebellar structures for learning to translate self-motion into a sequence of goal-directed actions.

In the context of Pavlovian conditioning, two types of behaviour may emerge within the population (Flagel et al. Nature, 469(7328): 53–57, 2011). Animals may choose to engage either with the conditioned stimulus (CS), a behaviour known as sign-tracking (ST) which is sensitive to dopamine inhibition for its acquisition, or with the food cup in which the reward or unconditioned stimulus (US) will eventually be delivered, a behaviour known as goal-tracking (GT) which is dependent on dopamine for its expression only. Previous work by Lesaint et al. (PLoS Comput Biol, 10(2), 2014) offered a computational explanation for these phenomena and led to the prediction that varying the duration of the inter-trial interval (ITI) would change the relative ST-GT proportion in the population as well as phasic dopamine responses. A recent study verified this prediction, but also found a rich variance of ST and GT behaviours within the trial which goes beyond the original computational model. In this paper, we provide a computational perspective on these novel results. Electronic supplementary material The online version of this article (10.1007/s00213-019-05323-y) contains supplementary material, which is available to authorized users.

Accumulating evidence suggest that human behavior in trial-and-error learning tasks based on decisions between discrete actions may involve a combination of reinforcement learning (RL) and working-memory (WM). While the understanding of brain activity at stake in this type of tasks often involve the comparison with non-human primate neurophysiological results, it is not clear whether monkeys use similar combined RL and WM processes to solve these tasks. Here we analyzed the behavior of five monkeys with computational models combining RL and WM. Our model-based analysis approach enables to not only fit trial-by-trial choices but also transient slowdowns in reaction times, indicative of WM use. We found that the behavior of the five monkeys was better explained in terms of a combination of RL and WM despite inter-individual differences. The same coordination dynamics we used in a previous study in humans best explained the behavior of some monkeys while the behavior of others showed the opposite pattern, revealing a possible different dynamics of WM process. We further analyzed different variants of the tested models to open a discussion on how the long pretraining in these tasks may have favored particular coordination dynamics between RL and WM. This points towards either inter-species differences or protocol differences which could be further tested in humans., Comment: Behavioural Brain Research, Elsevier, 2017

In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine controls this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple non-stationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in exploration rate. This study shows that dopamine could adapt the exploration-exploitation trade-off in decision making when facing changing environmental contingencies.

During sleep and awake rest, the hippocampus replays sequences of place cells that have been activated during prior experiences. These have been interpreted as a memory consolidation process, but recent results suggest a possible interpretation in terms of reinforcement learning. The Dyna reinforcement learning algorithms use off-line replays to improve learning. Under limited replay budget, a prioritized sweeping approach, which requires a model of the transitions to the predecessors, can be used to improve performance. We investigate whether such algorithms can explain the experimentally observed replays. We propose a neural network version of prioritized sweeping Q-learning, for which we developed a growing multiple expert algorithm, able to cope with multiple predecessors. The resulting architecture is able to improve the learning of simulated agents confronted to a navigation task. We predict that, in animals, learning the world model should occur during rest periods, and that the corresponding replays should be shuffled., Living Machines 2018 (Paris, France)

Parkinson's disease is characterized by the death of dopaminergic neurons and the emergence of strong Beta-band oscillations throughout the basal ganglia nuclei. According to the mainstream theory, this synchrony is mediated by a dopamine deficit within the striatum creating a functional imbalance between the D1-expressing medium spiny neurons, which project to the internal segment of the globus pallidus, and D2-expressing one, which target its external segment, and ultimately leads to oscillatory activity. However, anatomical evidence gathered in rodents and primates has shown that striatal neurons are for the most part not organized into independent populations differentially targeting the two segments of the globus pallidus, nor alternatively expressing D1 or D2 receptors, thus calling for an alternative mechanism through which the lack of dopamine may cause oscillations. Here we adopt a computational approach in which we investigate a model whose parameters are fit to an extensive set of anatomical and physiological constraints from non-human primates, including axonal transmission delays gathered from eight experimental studies. Investigating the lack of dopamine in this model revealed that in the absence of segregated pathways, Beta-band oscillations emerge as a consequence of the extra-striate dopaminergic receptors reduced activity. These oscillations are caused by synchronous activity within the external globus pallidus-subthalamic nucleus loop, and their frequency are modulated by the transmission delays between these nuclei. Our model delivers a parsimonious explanation of oscillations that does not require any external driving influence from cortex, nor specific medium spiny neuron properties.

Victims of fatal injured accidents involving large trucks are mainly non-occupants of the trucks more than 80i¾?%. Therefore, it appears that truck drivers who had accidents are likely to suffer from Post Traumatic Stress Disorder while no research does address this need. This paper tackles this point by the mean of an exposition therapy through the use of Virtual Reality technologies. Our main contribution concerns the fact that we do not only exploit Virtual Reality to expose the user. We do create an environment that offers to the patient the possibility to act in order to recover his ability to face the traumatic situation. For this, we rely on the game mechanics and the personalization of the virtual environment to bring the trucker to be active while being exposed to the cues that are related to its fear. Doing so, he should reactivate his capacities and thus be able to act differently in such a traumatic situation.

International audience; Numerous brain regions encode variables using spatial distribution of activity in neuronal maps. Their specific geometry is usually explained by sensory considerations only. We provide here, for the first time, a theory involving the motor function of the superior colliculus to explain the geometry of its maps. We use six hypotheses in accordance with neurobiology to show that linear and logarithmic mappings are the only ones compatible with the generation of saccadic motor command. This mathematical proof gives a global coherence to the neurobiological studies on which it is based. Moreover, a new solution to the problem of saccades involving both colliculi is proposed. Comparative simulations show that it is more precise than the classical one.

International audience; Sensorimotor synchronization is a fundamental skill involved in the performance of many individual and ensemble artistic tasks (e.g., music, dance). Separate theories may explain how time intervals are estimated, and how bodily displacements toward spatial goals are controlled. However, the manner in which the nervous system produces movements toward concomitant temporal and spatial goals remains poorly understood. Previous studies have shown that typical rhythmic movements involve a motion and motionless phase (dwell). The dwell phase represents a sizeable fraction of the rhythm period, and scales with this period. The rationale for this specific organization remains unexplained, and is the object of this study. Two groups of participants (drummers, non-drummers) performed tapping movements paced at 0.5-2.5 Hz. The participants consistently organized their behavior between acoustic cues into dwell and movement phases, and movement kinematics varied with the period of the rhythm, yielding velocity profiles becoming increasingly asymmetric as the period expanded. The main new results were that the temporal variability of both the dwell and movement phases were consistent with Weber's law, and the longest phase exhibited always the smallest variability. We developed an optimal statistical model that formalized the distribution of time into dwell and movement intervals as a function of their temporal variability. The model accurately predicted the participants' dwell and movement durations irrespective of musical skill, strongly suggesting that the distribution of dwell and movement intervals results from an optimization process, dependent on each participant's skill to predict time during rest and movement.

International audience; Since 1995, numerous Actor–Critic architectures for reinforcement learning have been proposed as models of dopamine-like reinforcement learning mechanisms in the rat's basal ganglia. However, these models were usually tested in different tasks, and it is then difficult to compare their efficiency for an autonomous animat. We present here the comparison of four architectures in an animat as it per forms the same reward-seeking task. This will illustrate the consequences of different hypotheses about the management of different Actor sub-modules and Critic units, and their more or less autono mously determined coordination. We show that the classical method of coordination of modules by mixture of experts, depending on each module's performance, did not allow solving our task. Then we address the question of which principle should be applied efficiently to combine these units. Improve ments for Critic modeling and accuracy of Actor–Critic models for a natural task are finally discussed in the perspective of our Psikharpax project—an artificial rat having to survive autonomously in unpredictable environments.

The superior colliculus, a laminar structure involved in the retinotopic mapping of the visual field, plays a cardinal role in several cortical and subcortical pathways of the saccadic system. Although the selection of saccade targets has long been thought to be mainly the product of cortical processes, a growing body of evidence hints at the implication of the superior colliculus in selection processes independent from cortical inputs, capable of producing saccades at latencies incompatible with the cortical pathways. This selection ability could be produced firstly by the lateral connections between the neurons of its maps, and secondly by its interactions with the midbrain basal ganglia, already renowned for their role in decision making. We propose a biomimetic population-coded race model of selection based on a dynamic tecto-basal loop that reproduces the observed ability of the superior colliculus to stochastically select between similar stimuli. Our model's selection accuracy depends on the discriminability of the target and the distractors. Our model also offers an explanation for the phenomenon of Remote Distractor Effect based on the lateral connectivity within the basal ganglia circuitry rather than on lateral inhibitions within the collicular maps. Finally, we propose a role for the intermediate layers of the superior colliculus, as stochastic integrators dynamically gated by the selective disinhibition of the basal ganglia channels that is consistent with the recorded activity profiles of these neurons.

International audience

International audience; Researches in psychology and neuroscience have identified multiple decision systems in mammals, enabling control of behavior to shift with training and familiarity of the environment from a goal-directed system to a habitual system. The former relies on the explicit estimation of future consequences of actions through planning towards a particular goal, which makes decision time longer but produces rapid adaptation to changes in the environment. The latter learns to associate values to particular stimulus-response associations, leading to quick reactive decision- making but slow relearning in response to environmental changes. Computational neuroscience models have formalized this as a coordination of model-based and model-free reinforcement learning. From this inspiration we hypothesize that it could enable robots to learn habits, detect when these habits are appropriate and thus avoid long and costly computations of the planning system. We illustrate this in a simple repetitive cube-pushing task on a conveyor belt, where a speed-accuracy trade-off is required. We show that the two systems have complementary advantages in these tasks, which can be combined for performance improvement.

International audience; Recent experiments showed that the bio-mechanical ease and end-point stability associated to reaching movements are predicted prior to movement onset, and that these factors exert a significant influence on the choice of movement. As an extension of these results, here we investigate whether the knowledge about biomechanical costs and their influence on decision-making are the result of an adaptation process taking place during each experimental session or whether this knowledge was learned at an earlier stage of development. Specifically, we analysed both the pattern of decision-making and its fluctuations during each session, of several human subjects making free choices between two reaching movements that varied in path distance (target relative distance), biomechanical cost, aiming accuracy and stopping requirement. Our main result shows that the effect of biomechanics is well established at the start of the session, and that, consequently, the learning of biomechanical costs in decision-making occurred at an earlier stage of development. As a means to characterise the dynamics of this learning process, we also developed a model-based reinforcement learning model, which generates a possible account of how biomechanics may be incorporated into the motor plan to select between reaching movements. Results obtained in simulation showed that, after some pre-training corresponding to a motor babbling phase, the model can reproduce the subjects' overall movement preferences. Although preliminary, this supports that the knowledge about biomechanical costs may have been learned in this manner, and supports the hypothesis that the fluctuations observed in the subjects' behaviour may adapt in a similar fashion.

International audience; Otolith end organs of vertebrates sense linear accelerations of the head and gravitation. The hair cells on their epithelia are responsible for transduction. In mammals, the striola, parallel to the line where hair cells reverse their polarization, is a narrow region centered on a curve with curvature and torsion. It has been shown that the striolar region is functionally different from the rest, being involved in a phasic vestibular pathway. We propose a mathematical and computational model that explains the necessity of this amazing geometry for the striola to be able to carry out its function. Our hypothesis, related to the biophysics of the hair cells and to the physiology of their afferent neurons, is that striolar afferents collect information from several type I hair cells to detect the jerk in a large domain of acceleration directions. This predicts a mean number of two calyces for afferent neurons, as measured in rodents. The domain of acceleration directions sensed by our striolar model is compatible with the experimental results obtained on monkeys considering all afferents. Therefore, the main result of our study is that phasic and tonic vestibular afferents cover the same geometrical fields, but at different dynamical and frequency domains.

The study of decision-making between goal directed actions with rodents has been often based on experimental tasks in which animals were trained to perform specific sequences of actions, such as lever presses or nose pokes [4], to attain reward. This supported the hypothesis of reinforcement learning as the underlying mechanism to acquire those behavioural sequences, putatively implemented by the basal-ganglia circuitry [1,3]. However, experimental evidence suggests that whenever we extend the complexity of the motor responses towards timely constrained behaviour, it starts reflecting an influence of costs related not only to reward, but rather a compromise between the motor factors relevant to the task, and the timely requirements to attain the goal [6]. To investigate this further, we took advantage of new behavioral protocol in which rats running on a treadmill need to estimate a fixed-temporal interval to obtain a reward [5]. Interestingly rats became proficients in this task by developping very stereotyped running trajectories. The establishment of these precise running kinematics occured progressively in a trial-and-error process that lasted between 2 to 3 months. At this point if we shortened the treadmill length, animals persisted in reproducing the previously learned kinematics even if doing so they stopped receiving reward. This is consistent with that these stereotyped running kinematics are motor habit [8]. To provide a theoretical backend for these results, we developed a model-free reinforcement learning model [7]. We excluded model-based algorithms because of the inability of the rats to exploit the previously learned behavior to accelerate their learning rate when the task changes. The specificity of this model is to count reward delivery as positive reward, but also efforts generated at each time step as negative rewards. The problem is thus a speed-accuracy trade-off process: the goal of the model is to generate the motor sequence that optimizes the ratio discounted reward/effort. The main result shows that, as long as the local time and speed are included into the characterization of the kinematic state, the model can replicate the same motor sequences. This suggests that these two pieces information are required to learn time-constrained motor sequences, and predicts that if a brain structure indeed learns these habitual sequences as the model does (our suggestion would be the sensorimotor circuits of the basal ganglia [2]), it should exhibit correlates with the same variables during the entire sequence.

International audience; A biologically inspired navigation system for the mobile rat-like robot named Psikharpax is presented, allowing for self-localization and autonomous navigation in an initially unknown environment. The ability of parts of the model (e.g. the strategy selection mechanism) to reproduce rat behavioral data in various maze tasks has been validated before in simulations. But the capacity of the model to work on a real robot platform had not been tested. This paper presents our work on the implementation on the Psikharpax robot of two independent navigation strategies (a place-based planning strategy and a cue-guided taxon strategy) and a strategy selection meta-controller. We show how our robot can memorize which was the optimal strategy in each situation, by means of a reinforcement learning algorithm. Moreover, a context detector enables the controller to quickly adapt to changes in the environment--recognized as new contexts--and to restore previously acquired strategy preferences when a previously experienced context is recognized. This produces adaptivity closer to rat behavioral performance and constitutes a computational proposition of the role of the rat prefrontal cortex in strategy shifting. Moreover, such a brain-inspired meta-controller may provide an advancement for learning architectures in robotics.

International audience; Rodents have been widely studied for their adaptive navigation capabilities. They are able to exhibit multiple navigation strategies; some based on simple sensory-motor associations, while others rely on the construction of cognitive maps. We previously proposed a computational model of parallel learning processes during navigation which could reproduce in simulation a wide set of rat behavioral data and which could adaptively control a robot in a changing environment. In this previous robotic implementation the visual approach (or taxon) strategy was however paying attention to the intra-maze landmark only and learned to approach it. Here we replaced this mechanism by a more realistic one where the robot autonomously learns to select relevant landmarks. We show experimentally that the new taxon strategy is efficient, and that it combines robustly with the planning strategy, so as to choose the most efficient strategy given the available sensory information.

International audience; The hypothesis of multiple memory systems involved in different learning of navigation strategies has gained strong arguments through biological experiments. However, it remains difficult for experimentalists to understand how these systems interact. We propose a new computational model of selection between parallel systems involving cue-guided and place-based navigation strategies that allows analyses of selection switches between both control systems, while providing information that is not directly accessible in experiments with animals. Contrary to existing models of navigation, its module of selection is adaptive and uses a criterion which allows the comparison of strategies having different learning processes. Moreover, the spatial representation used by the place-based strategy is based on a recent hippocampus model. We illustrate the ability of this navigation model to analyze animal behavior in experiments in which the availability of sensory cues, together with the amount of training, influence the competitive or cooperative nature of their interactions.

International audience; The basal ganglia (BG) are a set of subcortical nuclei involved in action selection processes. We explore here the automatic parameterization of two models of the basal ganglia (the GPR and the CBG) using multi-objective evolutionary algorithms. We define two objective functions characterizing the supposed winner-takes-all functionality of the BG and obtain a set of solutions lying on the Pareto front for each model. We show that the CBG architecture leads to solutions dominating the GPR ones, this highlights the usefulness of the CBG additional connections with regards to the GPR. We then identify the most satisfying solutions on the fronts in terms of both functionality and plausibility. We finally define critical and indifferent parameters by analyzing their variations and values on the fronts, helping us to understand the dynamics governing the selection process in the BG models.

International audience; We propose an integrated model of the saccadic circuitry involved in target selection and motor command. It includes the Superior Colliculus and the Basal Ganglia in both cortical and subcortical loops. This model has spatial and feature-based learning capabilities which are demonstrated on various saccade tasks on a robotic platform. Results show that it is possible to learn to select saccades based on spatial information, feature-based information and combinations of both, without the necessity to explicitly pre-define eye-movement strategies.

International audience; Neuro-evolution and computational neuroscience are two scientific domains that produce surprisingly different artificial neural networks. Inspired by the "toolbox" used by neuroscientists to create their models, this paper argues two main points: (1) neural maps (spatially-organized identical neurons) should be the building blocks to evolve neural networks able to perform cognitive functions and (2) well-identified modules of the brain for which there exists computational neuroscience models provide well-defined benchmarks for neuro-evolution. To support these claims, a method to evolve networks of neural maps is introduced then applied to evolve neural networks with a similar functionality to basal ganglia in animals (i.e. action selection). Results show that: (1) the map-based encoding easily achieves this task while a direct encoding never solves it; (2) this encoding is independent of the size of maps and can therefore be used to evolve large and brain-like neural networks; (3) the failure of direct encoding to solve the task validates the relevance of action selection as a benchmark for neuro-evolution.

International audience; In this article, we describe a new computational model of switching between path-planning and cue-guided navigation strategies. It is based on three main assumptions: (i) the strategies are mediated by separate memory systems that learn independently and in parallel; (ii) the learning algorithms are different in the two memory systems—the cue-guided strategy uses a temporal-difference (TD) learning rule to approach a visible goal, whereas the path-planning strategy relies on a place-cell-based graph-search algorithm to learn the location of a hidden goal; (iii) a strategy selection mechanism uses TD-learning rule to choose the most successful strategy based on past experience. We propose a novel criterion for strategy selection based on the directions of goal-oriented movements suggested by the different strategies. We show that the selection criterion based on this “common currency” is capable of choosing the best among TD-learning and planning strategies and can be used to solve navigational tasks in continuous state and action spaces. The model has been successfully applied to reproduce rat behavior in two water-maze tasks in which the two strategies were shown to interact. The model was used to analyze competitive and cooperative interactions between different strategies during these tasks as well as relative influence of different types of sensory cues.

International audience; Whether the central nervous system of primates uses a body-centered or an eye-centered frame for reaching is still a controversial debate in neurosciences. Does the proprioceptive information allow to represent the visual information with respect to the body or inversely, does it transform the hand position in retinal coordinates? In this paper, we implement and test two control schemes associated to each of these two hypotheses by using a computational approach from robotics. To this end, biological models of motor control are applied to a realistic dynamic model of upper body including a 4 DOF eye-neck kinematic chain and a 6DOF arm. Important characteristics, such as proprioceptive biases and sensory delay, are considered. Simulation results allow to compare the geometry of trajectories and illustrate the better robustness of the eye-centered control scheme with respect to biases and sensory delay. Applications to the control of the humanoid robot HRP2 are finally presented.

International audience; Action selection, the problem of choosing what to do next, is central to any autonomous agent architecture. We use here a multidisciplinary approach at the convergence of neuro-science, dynamical systems theory and autonomous robotics, in order to propose an efficient action selection mechanism based on a new model of the basal ganglia. We first describe new developments of contraction theory regarding locally projected dynamical systems. We exploit these results to design a stable computational model of the cortico-baso-thalamo-cortical loops. Based on recent anatomical data, we include usually neglected neu-ral projections, which participate in performing accurate selection. Finally, the efficiency of this model as an autonomous robot action selection mechanism is assessed in a standard survival task. The model exhibits valuable dithering avoidance and energy-saving properties , when compared with a simple if-then-else decision rule.

International audience; This article describes a biomimetic control architecture affording an animat both action selection and navigation functionalities. It satisfies the survival constraint of an artificial metabolism and supports several complementary navigation strategies. It builds upon an action selection model based on the basal ganglia of the vertebrate brain, using two interconnected cortico-basal ganglia-thalamo-cortical loops: a ventral one concerned with appetitive actions and a dorsal one dedicated to consummatory actions. The performances of the resulting model are evaluated in simulation. The experiments assess the prolonged survival permitted by the use of high level navigation strategies and the complementarity of navigation strategies in dynamic environments. The correctness of the behavioral choices in situations of antagonistic or synergetic internal states are also tested. Finally, the modelling choices are discussed with regard to their biomimetic plausibility, while the experimental results are estimated in terms of animat adaptivity.

International audience; The brain circuitry of saccadic eye movements, from brainstem to cortex, has been extensively studied during the last 30 years. The wealth of data gathered allowed the conception of numerous computational models. These models proposed descriptions of the putative mechanisms generating this data, and, in turn, made predictions and helped to plan new experiments. In this article, we review the computational models of the five main brain regions involved in saccade generation: reticular formation saccadic burst generators, superior colliculus, cerebellum, basal ganglia and premotor cortical areas. We present the various topics these models are concerned with: location of the feedback loop, multimodal saccades, long-term adaptation, on the fly trajectory correction, strategy and metrics selection, short-term spatial memory, transformations between retinocentric and craniocentric reference frames, sequence learning, to name the principle ones. Our objective is to provide a global view of the whole system. Indeed, narrowing too much the modelled areas while trying to explain too much data is a recurrent problem that should be avoided. Moreover, beyond the multiple research topics remaining to be solved locally, questions regarding the operation of the whole structure can now be addressed by building on the existing models.

International audience; The basal ganglia system has been proposed as a possible neural substrate for action selection in the vertebrate brain. We describe a robotic implementation of a model of the basal ganglia and demonstrate the capacity of this system to generate adaptive switching between several acts when embedded in a robot that has to ‘survive' in a laboratory environment. A comparison between this brain-inspired selection mechanism and classical `winner-takes-all\' selection highlights some adaptive properties specific to the model, such as avoidance of dithering and energy-saving. These properties derive, in part, from the capacity of simulated basal ganglia-thalamo-cortical loops to generate appropriate ‘behavioural persistence'.

The activity of dopaminergic (DA) neurons has been hypothesized to encode a reward prediction error (RPE) which corresponds to the error signal in Temporal Difference (TD) learning algorithms. This hypothesis has been reinforced by numerous studies showing the relevance of TD learning algorithms to describe the role of basal ganglia in classical conditioning. However, recent recordings of DA neurons during multi-choice tasks raised contradictory interpretations on whether DA’s RPE signal is action dependent or not. Thus the precise TD algorithm (i.e. Actor-Critic, Q-learning or SARSA) that best describes DA signals remains unknown. Here we simulate and precisely analyze these TD algorithms on a multi-choice task performed by rats. We find that DA activity previously reported in this task is best fitted by a TD error which has not fully converged, and which converged faster than observed behavioral adaptation.

In contrast with research work in Artificial Intelligence, which can test designed reasoning algorithms on specific problems in isolation during perfectly controlled discrete simulations, Robotics always faced the requirement of integrating perception, decision and action execution to work on embodied platforms interacting in and with the real world. This forced roboticists to address in various ways the question of how to efficiently coordinate different memory systems, different levels of decision-making and different learning processes within proposed artificial cognitive architectures. Such robotic proposals have often entertained strong interactions with similar questions as they are also raised in Philosophy, Neuroscience, Psychology and Cognitive Sciences in general. Here we focally review robotic work integrating different levels of action (from goal-oriented action planning to reflexes ; from action monitoring to reactive visually-guided behavior) and their associated learning mechanisms. We show that such integration is necessary and sufficient to have an artificial agent fulfill a basic level of self-performance monitoring, and as a consequence to display higher behavioral flexibility, autonomy and generalizability between different environments. We illustrate these issues with examples of experimental work on coordination of multiple learning systems in a single robot and on joint action during human-robot cooperation, finding that partly similar mechanisms for the integration of different action levels can be at work for both individual action and joint action. We highlight some of the successes and failures of these robotic approaches with the hope of feeding and contributing to similar debates on action in other fields of Cognitive Sciences., Intégration de l’action, de l’action conjointe et de l’apprentissage dans les architectures cognitives robotiques. À l’opposé des recherches en Intelligence Artificielle, où des algorithmes conçus pour des problèmes spécifiques peuvent être testés dans des conditions de simulation parfaitement contrôlée, la Robotique a toujours eu à faire face à la nécessité d’intégrer la perception, la décision et l’exécution de l’action pour pouvoir fonctionner sur des plateformes physiques en interaction dans et avec le monde réel. Cette particularité a obligé les roboticiennes et roboticiens à se poser très tôt la question de comment coordonner efficacement différents systèmes de mémoires, différents niveaux de prise de décision et différents processus d’apprentissage dans des architectures cognitives. En conséquence, les propositions d’architectures robotiques ont souvent été confrontées et nourries par les questionnements sur les architectures cognitives tels qu’abordés en Philosophie, en Psychologie, en Neurosciences, et plus généralement en Sciences Cognitives. Dans cet article, nous passons en revue les travaux robotiques qui ont abordé le problème de l’intégration de différents niveaux d’action et de leurs mécanismes d’apprentissage associés (de la planification des actions orientées vers un but aux simples réflexes ; de la surveillance de l’action aux comportements réactifs guidés par la vision). Nous montrons qu’un tel type d’intégration est nécessaire et suffisant pour permettre à un agent artificiel de montrer un niveau basique de surveillance de sa propre performance, et en conséquence de montrer de plus grandes capacités de flexibilité comportementale, d’autonomie et de généralisation à différents environnements. Nous illustrons cette problématique à travers des exemples expérimentaux sur la coordination de systèmes multiples d’apprentissage au sein d’un même robot, et sur l’application à l’action conjointe lors de la coopération homme-robot. Nous trouvons en effet que des mécanismes partiellement similaires d’intégration des niveaux d’action peuvent fonctionner à la fois pour l’action individuelle et pour l’action conjointe. Enfin, nous mettons en lumière certains des succès et des échecs de ces approches robotiques en espérant nourrir et contribuer aux débats similaires sur l’action qui se posent dans les autres champs des Sciences Cognitives., Réguigne-Khamassi Mehdi, Girard Benoît, Clodic Aurélie, Devin Sandra, Renaudo Erwan, Pacherie Elisabeth, Alami Rachid, Chatila Raja. Integration of Action, Joint Action and Learning in Robotics Cognitive Architectures. In: Intellectica. Revue de l'Association pour la Recherche Cognitive, n°65, 2016/1. Nouvelles approches en Robotique Cognitive. pp. 169-203.

International audience; For animals as well as for humans, the hypothesis of multiple memory systems involved in different navigation strategies is supported by several biological experiments. However, due to technical limitations, it remains difficult for experimentalists to elucidate how these neural systems interact. We present how a computational model of selection between navigation strategies can be used to analyse phenomena that cannot be directly observed in biological experiments. We reproduce an experiment where the rat’s behaviour is assumed to be ruled by two different navigation strategies (a cue-guided and a map-based one). Using a modelling approach, we can explain the experimental results in terms of interactions between these systems, either competing or cooperating at specific moments of the experiment. Modelling such systems can help biological investigations to explain and predict the animal behaviour