Search

Experience replay is widely used in AI to bootstrap reinforcement learning (RL) by enabling an agent to remember and reuse past experiences. Classical techniques include shuffled-, reversed-ordered- and prioritized-memory buffers, which have different properties and advantages depending on the nature of the data and problem. Interestingly, recent computational neuroscience work has shown that these techniques are relevant to model hippocampal reactivations recorded during rodent navigation. Nevertheless, the brain mechanisms for orchestrating hippocampal replay are still unclear. In this paper, we present recent neurorobotics research aiming to endow a navigating robot with a neuro-inspired RL architecture (including different learning strategies, such as model-based (MB) and model-free (MF), and different replay techniques). We illustrate through a series of numerical simulations how the specificities of robotic experimentation (e.g., autonomous state decomposition by the robot, noisy perception, state transition uncertainty, non-stationarity) can shed new lights on which replay techniques turn out to be more efficient in different situations. Finally, we close the loop by raising new hypotheses for neuroscience from such robotic models of hippocampal replay.

Reinforcement learning (RL) aims at building a policy that maximizes a task-related reward within a given domain. When the domain is known, i.e., when its states, actions and reward are defined, Markov Decision Processes (MDPs) provide a convenient theoretical framework to formalize RL. But in an open-ended learning process, an agent or robot must solve an unbounded sequence of tasks that are not known in advance and the corresponding MDPs cannot be built at design time. This defines the main challenges of open-ended learning: how can the agent learn how to behave appropriately when the adequate states, actions and rewards representations are not given? In this paper, we propose a conceptual framework to address this question. We assume an agent endowed with low-level perception and action capabilities. This agent receives an external reward when it faces a task. It must discover the state and action representations that will let it cast the tasks as MDPs in order to solve them by RL. The relevance of the action or state representation is critical for the agent to learn efficiently. Considering that the agent starts with a low level, task-agnostic state and action spaces based on its low-level perception and action capabilities, we describe open-ended learning as the challenge of building the adequate representation of states and actions, i.e., of redescribing available representations. We suggest an iterative approach to this problem based on several successive Representational Redescription processes, and highlight the corresponding challenges in which intrinsic motivations play a key role.

Despite major progress in Robotics and AI, robots are still basically “zombies” repeatedly achieving actions and tasks without understanding what they are doing. Deep-Learning AI programs classify tremendous amounts of data without grasping the meaning of their inputs or outputs. We still lack a genuine theory of the underlying principles and methods that would enable robots to understand their environment, to be cognizant of what they do, to take appropriate and timely initiatives, to learn from their own experience and to show that they know that they have learned and how. The rationale of this paper is that the understanding of its environment by an agent (the agent itself and its effects on the environment included) requires its self-awareness, which actually is itself emerging as a result of this understanding and the distinction that the agent is capable to make between its own mind-body and its environment. The paper develops along five issues: agent perception and interaction with the environment; learning actions; agent interaction with other agents–specifically humans; decision-making; and the cognitive architecture integrating these capacities.

Epileptogenesis is the gradual process responsible for converting a healthy brain into an epileptic brain. This process can be triggered by a wide range of factors, including brain injury or tumors, infections, and status epilepticus. Epileptogenesis results in aberrant synaptic plasticity, neuroinflammation and seizure-induced cell death. As Matrix Metalloproteinases (MMPs) play a crucial role in cellular plasticity by remodeling the extracellular matrix (ECM), gelatinases (MMP-2 and MMP-9) were recently highlighted as key players in epileptogenesis. In this work, we engineered a biosensor to report in situ gelatinase activity in a model of epileptogenesis. This biosensor encompasses a gelatinase-sensitive activatable cell penetrating peptide (ACPP) coupled to a TAMRA fluorophore, allowing fluorescence uptake in cells displaying endogenous gelatinase activities. In a preclinical mouse model of temporal lobe epilepsy (TLE), the intrahippocampal kainate injection, ACPPs revealed a localized distribution of gelatinase activities, refining temporal cellular changes during epileptogenesis. The activity was found particularly but not only in the ipsilateral hippocampus, starting from the CA1 area and spreading to dentate gyrus from the early stages throughout chronic epilepsy, notably in neurons and microglial cells. Thus, our work shows that ACPPs are suitable molecular imaging probes for detecting the spatiotemporal pattern of gelatinase activity during epileptogenesis, suggesting their possible use as vectors to target cellular reactive changes with treatment for epileptogenesis.