Search

Compared to traditional traffic signal control methods, the method driven by Deep Reinforcement Learning (DRL) has shown better performance. But the problem of low sample utilization in reinforcement learning also arises. To deal with the problem, this paper presents a novel Twin Delayed Deep Deterministic Policy Gradient with Dual Buffer (TD3_DB) for traffic signal control. In the proposed framework, two experience buffers are used to store important samples and normal samples, separately, and the proportion of the two buffers is adjusted adaptively. In addition, lane pressure, describing the dynamic feature of lane traffic flow, is used for the state design of the TD3 agent, which enhances the perception of the agent toward intersections. Comprehensive experiments on different traffic flow modes has shown, the dual experience replay scheme can improve the sample utilization, and the proposed TD3_DB performs better than other methods such as original TD3, Proximal Policy Optimization (PPO), etc., effectively reducing vehicle queue length and waiting time. [ABSTRACT FROM AUTHOR]

In many real-life scenarios, multiple agents necessitate cooperation to accomplish tasks. Benefiting from the significant success of deep learning, many single-agent deep reinforcement learning algorithms have been extended to multi-agent scenarios. Overestimation in value estimation of Q-learning is a significant issue that has been studied comprehensively in the single-agent domains, but rarely in multi-agent reinforcement learning. In this paper, we first demonstrate that Q-learning-based multi-agent reinforcement learning (MARL) methods generally have notably serious overestimation issues, which cannot be alleviated by current methods. To tackle this problem, we introduce the double critic networks structure and the delayed policy update to Q-learning-based multi-agent MARL methods, which reduce the overestimation and enhance the quality of policy updating. To demonstrate the versatility of our proposed method, we select several Q-learning based MARL methods and evaluate them on several multi-agent tasks on the multi-agent particle environment and SMAC. Experimental results demonstrate that the proposed method can avoid the overestimation problem and significantly improve performance. Besides, application in the Traffic Signal Control verifies the feasibility of applying the proposed method in real-world scenarios. [ABSTRACT FROM AUTHOR]

In this study, a traffic queue model is employed to simulate traffic parameters for optimization purposes. The primary objective is to enhance the average queue length, cumulative delay, and cumulative reward for varying numbers of vehicles generated using an innovative approach. The proposed methodology integrates deep Reinforcement Q-Learning techniques implemented through a realistic simulator, Simulation of Urban MObility, based on OpenStreetMap. This simulator is used to balance the flow of traffic by monitoring traffic conditions and adjusting signals (Red/Yellow/Green) accordingly. The research extensively evaluates simulation outcomes across different scenarios involving 50, 100, 500, and 1000 vehicles, with episodes 10, 50 and 100. The results provide valuable insights into training times, simulation durations, and total rewards obtained, indicating improvements in mitigating traffic congestion. The introduction of a traffic control strategy enhances model effectiveness through the utilization of four-layer deep neural network architectures with 400 neurons in hidden layers. However, the study also highlights persistent challenges related to real-time implementation and complexities in state-action-agent relationships. The progress highlights improvements in reducing congestion and enhancing efficiency at junctions by dynamically adjusting signal phases while leveraging historical data. The results highlight the potential of dynamic traffic management strategies in effectively addressing urban traffic challenges. [ABSTRACT FROM AUTHOR]

Traffic lights are crucial for urban traffic management, as they significantly impact congestion reduction and travel safety. Traditional methods relying on hand-crafted rules and operator experience are limited in their ability to adapt to changing traffic environments. To address this challenge, we have been exploring intelligent traffic light control using deep reinforcement learning techniques. However, current approaches often suffer from inadequate training data and unstable training processes, leading to suboptimal performance and real-world consequences. In this study, we propose RELight, a novel random ensemble reinforcement learning-based traffic light control framework. RELight effectively utilizes collected empirical data, ensuring a stable and efficient training process. To evaluate the performance of our proposed framework, we conducted a comprehensive set of experiments on a variety of datasets, including four synthetic datasets and a real traffic dataset collected from surveillance cameras at an intersection in Hangzhou, China. The results show that RELight outperforms existing approaches, demonstrating its superiority and potential for practical traffic light control applications. [ABSTRACT FROM AUTHOR]

Traffic signal control (TSC) can be described as a multi-agent cooperative game. To realize cooperation, multi-agent reinforcement learning (MARL) is a significant approach, with communication being a core component. The large-scale traffic signals and the partially observable information in TSC pose a considerable challenge in finding the optimal joint control policy. This paper proposed a deep MARL model named attentional graph relations communications network (AGRCNet). Based on the Actor-Critic framework, AGRCNet designs a communication network to exchange observation information with agents to help obtain the optimal joint action, reducing the decision error caused by the partially observable condition. Specifically, through the communication network, the chain propagation of graph attention networks (GAT) and graph convolutional networks is used to expand the receptive domain of agents, improve communication efficiency and promote cooperative behavior. We simulate the traffic situation near the Nanjing Yangtze River Bridge in Simulation of Urban MObility. With a compound reward, our method performs best. Meanwhile, AGRCNet is applied to two abstract environments, and the results show that our approach can also adapt to dynamic agent relationships and is more efficient than comparison algorithms. [ABSTRACT FROM AUTHOR]

Using reinforcement learning to control traffic signal systems has been discussed in recent years, but most works focused on simple scenarios such as a single crossroads, and the methods aiming at large-scale traffic scenarios face long-time training and suboptimal results. In this work, we develop a new multi-agent reinforcement model for large-scale traffic signal control tasks, and a curriculum transfer learning method is developed to optimize the joint policy step by step. The policies for different intersections are trained in a partially observable Markov decision process with centralized training and decentralized execution mechanism, and we design transformer modules for both the policy and evaluation networks by attention mechanism. We first train policies in a simple traffic scenario, and then these policies are transferred to the next curriculum by policy reloading, while the experiences of the source task are reused selectively. With the number of agents increasing, our method can achieve satisfactory performances quickly by reusing the knowledge from previous curriculums. We conduct several experiments on the Cityflow testbed. In the case of more than 10 crossroads, our model improve the mean reward from 3.0 to 5.0. [ABSTRACT FROM AUTHOR]

Intelligent traffic light control (ITLC) algorithms are very efficient for relieving traffic congestion. Recently, many decentralized multi-agent traffic light control algorithms are proposed. These researches mainly focus on improving reinforcement learning method and coordination method. But, as all the agents need to communicate while coordinating with each other, the communication details should be improved as well. To guarantee communication effectiveness, two aspect should be considered. Firstly, a traffic condition description method need to be designed. By using this method, traffic condition can be described simply and clearly. Secondly, synchronization should be considered. As different intersections have different cycle lengths and message sending event happens at the end of each traffic signal cycle, every agent will receive messages of other agents at different time. So it is hard for an agent to decide which message is the latest one and the most valuable. Apart from communication details, reinforcement learning algorithm used for traffic signal timing should also be improved. In the traditional reinforcement learning based ITLC algorithms, either queue length of congested cars or waiting time of these cars is considered while calculating reward value. But, both of them are very important. So a new reward calculation method is needed. To solve all these problems, in this paper, a new ITLC algorithm is proposed. To improve communication efficiency, this algorithm adopts a new message sending and processing method. Besides, to measure traffic congestion in a more reasonable way, a new reward calculation method is proposed and used. This method takes both waiting time and queue length into consideration. [ABSTRACT FROM AUTHOR]

Cooperation between intersections in large-scale road networks is critical in traffic congestion. Currently, most traffic signals cooperate via pre-defined timing phases, which is extremely inefficient in real-time traffic scenarios. Most existing studies on multi-agent reinforcement learning (MARL) traffic signal control have focused on designing efficient communication methods, but have ignored the importance of how agents interact in cooperative communication. To achieve more efficient cooperation among traffic signals and alleviate urban traffic congestion, this study constructs a Graph Cooperation Q-learning Network Traffic Signal Control (GCQN-TSC) model, which is a graph cooperation network with an embedded self-attention mechanism that enables agents to adjust their attention in real time according to the dynamic traffic flow information, perceive the traffic environment quickly and effectively in a larger range, and help agents achieve more effective collaboration. Moreover, the Deep Graph Q-learning (DGQ) algorithm is proposed in this model to optimize the traffic signal control strategy according to the spatio-temporal characteristics of different traffic scenes and provide the optimal signal phase for each intersection. This study also integrates the ecological traffic concept into MARL traffic signal control, which aims to reduce traffic exhaust emissions. Finally, the proposed GCQN-TSC is experimentally validated both in a synthetic traffic grid and a real-world traffic network using the SUMO simulator. The experimental results show that GCQN-TSC outperforms other traffic signal control methods in almost all performance metrics, including average queue length and waiting time, as it can aggregate information acquired from collaborative agents and make network-level signal optimization decisions. [ABSTRACT FROM AUTHOR]

In contemporary urban, traffic signal control is still enormously difficult. Multi-agent reinforcement learning (MARL) is a promising ways to solve this problem. However, most MARL algorithms can not effectively transfer learning strategies when the agents increase or decrease. This paper proposes a new MARL algorithm called cooperative dynamic delay updating twin delayed deep deterministic policy gradient based on the exponentially weighted moving average (CoTD3-EWMA) to solve the problem. By introducing mean-field theory, the algorithm implicitly models the interaction between agents and environment. It reduces the dimension of action space and improves the scalability of the algorithm. In addition, we propose a dynamic delay updating method based on the exponentially weighted moving average (EWMA), which improves the Q value overestimation problem of the traditional TD3 algorithm. Moreover, a joint reward allocation mechanism and state sharing mechanism are proposed to improve the global strategy learning ability and robustness of the agent. The simulation results show that the performance of the new algorithm is better than the current state-of-the-art algorithms, which effectively reduces the delay time of vehicles and improves the traffic efficiency of the traffic network. [ABSTRACT FROM AUTHOR]

The paper is devoted to scheduling in multiagent systems in the framework of the Flatland 3 competition. The main aim of this competition is to develop an algorithm for the effective control of dense traffic in complex railroad networks according to a given schedule. The proposed solution is based on reinforcement learning. To adapt this method to the particular scheduling problem, a novel approach based on structuring the reward function that stimulates an agent to adhere to its schedule was developed. The architecture of the proposed model is based on a multiagent version of centralized critic with proximal policy optimization (PPO) learning. In addition, a curriculum learning strategy was developed and implemented. This allowed the agent to cope with each level of complexity on time and train the model in more difficult conditions. The proposed solution won first place in the Flatland 3 competition in the reinforcement learning track. [ABSTRACT FROM AUTHOR]

Traffic lights are an important controlling factor in traffic flows, and good policies will facilitate traffic congestion. A car's waiting time is highly related to the period in which the traffic lights are green or red; thus, a proper controlling policy reduces the waiting time and speeds up the car movement. Emergency vehicles always have a higher priority than other cars and need to reach their destination faster. In the last decade, a lot of research has been done to solve the traffic light scheduling issue using artificial intelligence techniques. Reinforcement learning with simulating human learning has the outstanding performance to solve complex problems, such as traffic lights control. In this study, a deep reinforcement learning traffic light control was implemented at a crossroad using real-time traffic information with an emphasis on emergency vehicles. The agent learns to adjust its policies to prioritize the emergency cars over the other cars. The results show less waiting time and speed up the cars' movement. It also indicates the emergency vehicles will cross the intersection with the least delay. With 200 to 600 cars and one emergency vehicle, the average delay decreases 2–16% in total using reinforcement learning. Considering more emergency vehicles, there is a 27–40% decrease in average delay for emergency vehicles. [ABSTRACT FROM AUTHOR]