Search

Reinforcement learning has been widely used in explaining animal behavior. In reinforcement learning, the agent learns the value of the states in the task, collectively constituting the task state space, and uses the knowledge to choose actions and acquire desired outcomes. It has been proposed that the orbitofrontal cortex (OFC) encodes the task state space during reinforcement learning. However, it is not well understood how the OFC acquires and stores task state information. Here, we propose a neural network model based on reservoir computing. Reservoir networks exhibit heterogeneous and dynamic activity patterns that are suitable to encode task states. The information can be extracted by a linear readout trained with reinforcement learning. We demonstrate how the network acquires and stores task structures. The network exhibits reinforcement learning behavior and its aspects resemble experimental findings of the OFC. Our study provides a theoretical explanation of how the OFC may contribute to reinforcement learning and a new approach to understanding the neural mechanism underlying reinforcement learning., Author summary Many studies employ reinforcement learning models to explain human or animal behavior, in which it is assumed that the animals know the task structure. Yet in the real life, the task structure also has to be acquired through learning. The orbitofrontal cortex has been proposed to play important roles in representing task structure, yet it is poorly understood how it does it and why it can do it. Here, we use a neural network model based on reservoir computing to approach these questions. We show that it is critical for the network to receive reward information as part of its input. Just as the orbitofrontal cortex that receives converging sensory and reward inputs, the network is able to acquire task structure and support reinforcement learning by encoding a combination of sensory and reward events. The importance of reward inputs in the model explains the sophisticate representations of reward information in the orbitofrontal cortex and provides a theoretic account of the current experimental data.

During value-based decision making, we often evaluate the value of each option sequentially by shifting our attention, even when the options are presented simultaneously. The orbitofrontal cortex (OFC) has been suggested to encode value during value-based decision making. Yet it is not known how its activity is modulated by attention shifts. We investigated this question by employing a passive viewing task that allowed us to disentangle effects of attention, value, choice and eye movement. We found that the attention modulated OFC activity through a winner-take-all mechanism. When we attracted the monkeys’ attention covertly, the OFC neuronal activity reflected the reward value of the newly attended cue. The shift of attention could be explained by a normalization model. Our results strongly argue for the hypothesis that the OFC neuronal activity represents the value of covertly attended item. They provide important insights toward the neural mechanism of value-based decision making.