Transferable traffic signal control: Reinforcement learning with graph centric state representation.

• This study identifies the restricted exploration problem encountered when training a signal control model which causes a model to obtain a partially-trained policy. • The goal of this research is to obtain a transferable policy that enhances the model's applicability on traffic states of unexperienced demand pattern using already-experienced information. • The key idea consists of two parts; 1) to represent the traffic state as graph-structured data by embedding it into a graph, 2) to process the information-condensed graph by employing message-passing graph neural network (GNN). • The experiment conducted on five unexperienced test demand scenarios which have different levels of traffic volumes and traveling patterns shows that the proposed GNN-based model obtains a transferable policy so that it adapts better to the unexperienced traffic states, while the conventional RL model fails. Reinforcement learning (RL) has emerged as an alternative approach for optimizing the traffic signal control system. However, there is a restricted exploration problem encountered when a signal control model is trained with a predefined demand scenario in the traffic simulation. With the restricted exploration, the model learns a policy based only on partial experiences in the search space, which yields a partially-trained policy. Partially-trained policy fails to adapt to some unexperienced ('unexplored', 'never-before-seen') dataset that have different distributions from the training dataset. Although this issue has critical effects on training a signal control model, it has not been considered in the literature. Therefore, this research aims to obtain a transferable policy to enhance the model's applicability on unexperienced traffic states. The key idea is to represent the state as graph-structured data, and train it using a graph neural network (GNN). Since this approach enables to learn the relationship between the features resulting from the spatial structure of the intersection, it is able to transfer the already-learned knowledge of the relationship to the unexperienced data. In order to investigate the transferability, an experiment is conducted on five unexperienced test demand scenarios. For the evaluation, the performance of the proposed GNN model is compared with the conventional DQN model that is based on vector-valued state. At first, the models are trained with only a single dataset (training demand scenario). Then, they are tested with different unexperienced dataset (test demand scenarios) without additional trainings. The results show that the proposed GNN model obtains a transferable policy so that it adapts better to the unexperienced traffic states, while the conventional DQN model fails. [ABSTRACT FROM AUTHOR]