Search

Despite the vast amount of experimental findings on the role of the basal ganglia in reinforcement learning, there is still general lack of network models that use spiking neurons and plausible plasticity mechanisms to demonstrate network-level reward-based learning. In this work we extend a recent spiking actor-critic network model of the basal ganglia, aiming to create a minimal realistic model of learning from both positive and negative rewards. We hypothesize and implement in the model segregation of not only the dorsal striatum, but also of the ventral striatum into populations of medium spiny neurons (MSNs) that carry either D1 or D2 dopamine (DA) receptor type. This segregation allows explicit representation of both positive and negative expected reward within respective population. In line with recent experiments, we further assume that D1 and D2 MSN populations have distinct, opposing DA-modulated bidirectional synaptic plasticity. We implement the spiking network model in the simulator NEST and conduct experiments involving application of delayed rewards in a grid world setting, where a moving agent has to reach a goal state while maximizing the total obtained reward. We demonstrate that the network can learn not only to approach the positive rewards, but also to consequently avoid punishments as opposed to the original model. The spiking network model highlights thus functional role of D1-D2 MSN segregation within striatum and explains necessity for reversed direction of DA-dependent plasticity found at synapses converging on different types of striatal MSNs.

One of the most promising avenues for compiling connectivity data originates from the notion that individual brain regions maintain individual connectivity profiles; the functional repertoire of a cortical area ("the functional fingerprint") is closely related to its anatomical connections ("the connectional fingerprint") and, hence, a segregated cortical area may be characterized by a highly coherent connectivity pattern. Existing clustering techniques in the context of connectivity-based cortex parcellation are usually exploratory. We therefore advocate an information-theoretic framework for connectivity-based cortex parcellation which avoids many assumptions imposed by previous methods. Clustering is based upon maximizing connectivity information while allowing noise in the data to vote for the optimal number of cortical subunits. The automatic parcellation of the inferior frontal gyrus together with the precentral gyrus reveals cortical subunits consistent with previous studies.

Extending previous work, we introduce a spiking actor-critic network model of learning from reward and punishment in the basal ganglia. In the model, the striatum is taken to be segregated into populations of medium spiny neurons (MSNs) that carry either D1 or D2 dopamine receptor type. This segregation allows explicit representation of both positive and negative expected outcome within the respective population. In line with recent experiments, we further assume that D1 and D2 MSN populations have opposing dopamine-modulated bidirectional synaptic plasticity. Experiments were conducted in a grid world, where a moving agent had to reach a remote rewarded goal state. The network learned not only to approach the rewarded goal, but also to consequently avoid punishments as opposed to the previous model. The spiking network model explains functional role of D1/D2 MSN segregation within striatum, specifically the reversed direction of dopamine-dependent plasticity found at synapses converging on different MSNs.

Memory traces in the cortex are embedded into a scaffold of feed-forward and recurrent connectivity of the hierarchically organized processing pathways. Strong evidence suggests that consolidation of the memory traces in such a memory network depends on an off-line reprocessing done in the sleep state or during restful waking. It remains largely unclear, what plasticity mechanisms are involved in this consolidation process and what changes are induced at what sites in the network during memory reprocessing in the off-line regime. This study focuses on functional consequences an off-line reprocessing has in a hierarchical recurrent neuronal network that learns different person identities from natural face images in unsupervised manner [1]. Due to the inherently self-exciting, but competitive winner-take-all-like unit dynamics, the two-layered network is able to self-generate sparse activity even in the absence of external input in an off-line regime. In this regime, the network reactivates the memory traces established during preceding on-line learning. Remarkably, this off-line memory replay turns out to be highly beneficial for the network recognition performance [2]. The benefit is articulated after the off-line regime in a strong boost of identity recognition rate on the alternative face views to which the network has not been exposed during learning. Performance of both network layers is affected by the boost. Surprisingly, the positive effect is independent of synapse-specific plasticity, relying completely on a synapse-unspecific mechanism of homeostatic activity regulation. This homeostatic mechanism tunes network unit excitabilities, equalizing the excitability levels within the network layers during the off-line reprocessing and causing the performance improvement when the network is back in the on-line regime. Performing excitability equalization for the lower and the higher network layers in separate, it becomes possible to dissociate the contribution of both layers to the positive effect observed after the off-line reprocessing. Equalizing the excitability levels on only one of both layers boosts the network recognition performance, independent of whether the equalization is made on the lower or on the higher layer. The excitability equalization on the higher layer has hereby a slightly stronger effect on network performance. The full boost however is achieved only if both layers are simultaneously processed via excitability equalization. Interestingly, the full effect cannot be simply explained by adding up the separate contributions of each layer, indicating that there is a substantial synergetic interaction between both in achieving the improvement after the off-line memory reprocessing. These findings suggest that all layers of the network hierarchy contribute their distinct part to the improvement of network recognition performance if affected by the off-line reprocessing, which provides interesting hints how off-line memory reprocessing may act on the hierarchically organized pathways in the brain during the states of sleep or restful waking.

In the visual cortex, memory traces for complex objects are embedded into a scaffold of feed-forward and recurrent connectivity of the hierarchically organized visual pathway. Strong evidence suggests that consolidation of the memory traces in such a memory network depends on an off-line reprocessing done in the sleep state or during restful waking. It remains largely unclear, what plasticity mechanisms are involved in this consolidation process and what changes are induced in the network during memory reprocessing in the off-line regime. Here we focus on the functional consequences off-line reprocessing has in a hierarchical recurrent neuronal network that learns different person identities from natural face images in an unsupervised manner. Due to the inherently self-exciting, but competitive unit dynamics, the two-layered network is able to self-generate sparse activity even in the absence of external input in an off-line regime. In this regime, the network replays the memory content established during preceding on-line learning. Remarkably, this off-line memory replay turns out to be highly beneficial for the network recognition performance. The benefit is articulated after the off-line regime in a strong boost of identity recognition rate on the alternative face views to which the network has not been exposed during learning. Performance of both network layers is affected by the boost. Surprisingly, the positive effect is independent of synapse-specific plasticity, relying completely on a synapse-unspecific mechanism of homeostatic activity regulation that tunes network unit excitability. Comparing further a purely feed-forward configuration of the network with its fully recurrent original version reveals a stronger boost in recognition performance for the latter after the off-line reprocessing. These findings suggest that the off-line memory reprocessing enhances generalization capability of the hierarchical recurrent network by improving communication of contextual cues mediated via recurrent lateral and top-down connectivity.

The basal ganglia network is thought to be involved in adaptation of organism's behavior when facing its positive and negative consequences, that is, in reinforcement learning. It has been hypothesized that dopamine (DA) modulated plasticity of synapses projecting from different cortical areas to the input nuclei of the basal ganglia, the striatum, plays a central role in this form of learning, being responsible for updating future outcome expectations and action preferences. In this scheme, DA transmission is considered to convey a prediction error signal that is generated if internal expectations do not match the outcomes observed after action execution. So far, there has been no satisfying model for what neural circuits computing this signal within the basal ganglia may look like, how this computation is performed and what is the mechanistic role of DA release in adapting the system towards optimal behavior in a given task. Aiming towards a model of a canonical circuit for learning task-conform behavior from both reward and punishment, we extended a previously introduced spiking actor-critic network model of the basal ganglia [1] to contain the segregation of both the dorsal (actor) and ventral (critic) striatum into populations of D1 and D2 medium spiny neurons (MSNs). This segregation allows explicit, separate representation of both positive and negative expected outcomes by the distinct populations in the ventral striatum. The positive and negative components of expected outcome were fed to dopamine (DA) neurons in SNc/VTA region, which compute and signal reward prediction error by DA release. Based on recent experimental work [2], DA level was assumed to modulate plasticity of D1 and D2 synapses in opposing way, inducing LTP on D1 and LTD on D2 synapses if being high and vice versa if being low. Crucially, this form of opponent plasticity implements temporal-difference (TD)-like update of both positive and negative outcome expectations separately and performs appropriate action selection adaptation. We implemented the network in the NEST simulator [3] using leaky integrate-and-fire spiking neurons and designed a battery of experiments involving application of reward and punishment in various grid world tasks. In each task, an agent had to explore the states and learn to maximize the total reward obtained. Number of states, magnitudes and delays of reward and punishment were manipulated across different tasks. We demonstrate that across the tasks the network can learn both to approach the delayed rewards while consequently avoiding punishments, the latter posing severe difficulties for the previous model without D1/D2 segregation [1]. Thus, the spiking neural network model highlights the functional role of D1/D2 MSN segregation within the striatum in implementing appropriate TD-like learning from both reward and punishment and explains necessity for opponent direction of DA-dependent plasticity found at synapses converging on distinct striatal MSN types. This modeling approach can be extended in the future work to study how abnormal D1/D2 plasticity may lead to a reorganization of the basal ganglia network towards pathological, dysfunctional states, like for instance those observed in Parkinson disease under condition of progressive dopamine depletion.

Growing neuropsychological and neurophysiological evidence suggests that the visual cortex uses parts-based representations to encode, store and retrieve relevant objects. In such a scheme, objects are represented as a set of spatially distributed local features, or parts, arranged in stereotypical fashion. To encode the local appearance and to represent the relations between the constituent parts, there has to be an appropriate memory structure formed by previous experience with visual objects. Here, we propose a model how a hierarchical memory structure supporting efficient storage and rapid recall of parts-based representations can be established by an experience-driven process of self-organization. The process is based on the collaboration of slow bidirectional synaptic plasticity and homeostatic unit activity regulation, both running at the top of fast activity dynamics with winner-take-all character modulated by an oscillatory rhythm. These neural mechanisms lay down the basis for cooperation and competition between the distributed units and their synaptic connections. Choosing human face recognition as a test task, we show that, under the condition of open-ended, unsupervised incremental learning, the system is able to form memory traces for individual faces in a parts-based fashion. On a lower memory layer the synaptic structure is developed to represent local facial features and their interrelations, while the identities of different persons are captured explicitly on a higher layer. An additional property of the resulting representations is the sparseness of both the activity during the recall and the synaptic patterns comprising the memory traces., Comment: 34 pages, 12 Figures, 1 Table, published in Frontiers in Computational Neuroscience (Special Issue on Complex Systems Science and Brain Dynamics), http://www.frontiersin.org/neuroscience/computationalneuroscience/paper/10.3389/neuro.10/015.2009/

Our work deals with the self-organization [1] of a memory structure that includes multiple hierarchical levels with massive recurrent communication within and between them. Such structure has to provide a representational basis for the relevant objects to be stored and recalled in a rapid and efficient way. Assuming that the object patterns consist of many spatially distributed local features, a problem of parts-based learning is posed. We speculate on the neural mechanisms governing the process of the structure formation and demonstrate their functionality on the task of human face recognition.