Search

This paper introduces a library for cross-simulator comparison of reinforcement learning models in traffic signal control tasks. This library is developed to implement recent state-of-the-art reinforcement learning models with extensible interfaces and unified cross-simulator evaluation metrics. It supports commonly-used simulators in traffic signal control tasks, including Simulation of Urban MObility(SUMO) and CityFlow, and multiple benchmark datasets for fair comparisons. We conducted experiments to validate our implementation of the models and to calibrate the simulators so that the experiments from one simulator could be referential to the other. Based on the validated models and calibrated environments, this paper compares and reports the performance of current state-of-the-art RL algorithms across different datasets and simulators. This is the first time that these methods have been compared fairly under the same datasets with different simulators. [ABSTRACT FROM AUTHOR]

Compared to traditional traffic signal control methods, the method driven by Deep Reinforcement Learning (DRL) has shown better performance. But the problem of low sample utilization in reinforcement learning also arises. To deal with the problem, this paper presents a novel Twin Delayed Deep Deterministic Policy Gradient with Dual Buffer (TD3_DB) for traffic signal control. In the proposed framework, two experience buffers are used to store important samples and normal samples, separately, and the proportion of the two buffers is adjusted adaptively. In addition, lane pressure, describing the dynamic feature of lane traffic flow, is used for the state design of the TD3 agent, which enhances the perception of the agent toward intersections. Comprehensive experiments on different traffic flow modes has shown, the dual experience replay scheme can improve the sample utilization, and the proposed TD3_DB performs better than other methods such as original TD3, Proximal Policy Optimization (PPO), etc., effectively reducing vehicle queue length and waiting time. [ABSTRACT FROM AUTHOR]

In many real-life scenarios, multiple agents necessitate cooperation to accomplish tasks. Benefiting from the significant success of deep learning, many single-agent deep reinforcement learning algorithms have been extended to multi-agent scenarios. Overestimation in value estimation of Q-learning is a significant issue that has been studied comprehensively in the single-agent domains, but rarely in multi-agent reinforcement learning. In this paper, we first demonstrate that Q-learning-based multi-agent reinforcement learning (MARL) methods generally have notably serious overestimation issues, which cannot be alleviated by current methods. To tackle this problem, we introduce the double critic networks structure and the delayed policy update to Q-learning-based multi-agent MARL methods, which reduce the overestimation and enhance the quality of policy updating. To demonstrate the versatility of our proposed method, we select several Q-learning based MARL methods and evaluate them on several multi-agent tasks on the multi-agent particle environment and SMAC. Experimental results demonstrate that the proposed method can avoid the overestimation problem and significantly improve performance. Besides, application in the Traffic Signal Control verifies the feasibility of applying the proposed method in real-world scenarios. [ABSTRACT FROM AUTHOR]

In this study, a traffic queue model is employed to simulate traffic parameters for optimization purposes. The primary objective is to enhance the average queue length, cumulative delay, and cumulative reward for varying numbers of vehicles generated using an innovative approach. The proposed methodology integrates deep Reinforcement Q-Learning techniques implemented through a realistic simulator, Simulation of Urban MObility, based on OpenStreetMap. This simulator is used to balance the flow of traffic by monitoring traffic conditions and adjusting signals (Red/Yellow/Green) accordingly. The research extensively evaluates simulation outcomes across different scenarios involving 50, 100, 500, and 1000 vehicles, with episodes 10, 50 and 100. The results provide valuable insights into training times, simulation durations, and total rewards obtained, indicating improvements in mitigating traffic congestion. The introduction of a traffic control strategy enhances model effectiveness through the utilization of four-layer deep neural network architectures with 400 neurons in hidden layers. However, the study also highlights persistent challenges related to real-time implementation and complexities in state-action-agent relationships. The progress highlights improvements in reducing congestion and enhancing efficiency at junctions by dynamically adjusting signal phases while leveraging historical data. The results highlight the potential of dynamic traffic management strategies in effectively addressing urban traffic challenges. [ABSTRACT FROM AUTHOR]

Traffic lights are crucial for urban traffic management, as they significantly impact congestion reduction and travel safety. Traditional methods relying on hand-crafted rules and operator experience are limited in their ability to adapt to changing traffic environments. To address this challenge, we have been exploring intelligent traffic light control using deep reinforcement learning techniques. However, current approaches often suffer from inadequate training data and unstable training processes, leading to suboptimal performance and real-world consequences. In this study, we propose RELight, a novel random ensemble reinforcement learning-based traffic light control framework. RELight effectively utilizes collected empirical data, ensuring a stable and efficient training process. To evaluate the performance of our proposed framework, we conducted a comprehensive set of experiments on a variety of datasets, including four synthetic datasets and a real traffic dataset collected from surveillance cameras at an intersection in Hangzhou, China. The results show that RELight outperforms existing approaches, demonstrating its superiority and potential for practical traffic light control applications. [ABSTRACT FROM AUTHOR]

Traffic signal control (TSC) can be described as a multi-agent cooperative game. To realize cooperation, multi-agent reinforcement learning (MARL) is a significant approach, with communication being a core component. The large-scale traffic signals and the partially observable information in TSC pose a considerable challenge in finding the optimal joint control policy. This paper proposed a deep MARL model named attentional graph relations communications network (AGRCNet). Based on the Actor-Critic framework, AGRCNet designs a communication network to exchange observation information with agents to help obtain the optimal joint action, reducing the decision error caused by the partially observable condition. Specifically, through the communication network, the chain propagation of graph attention networks (GAT) and graph convolutional networks is used to expand the receptive domain of agents, improve communication efficiency and promote cooperative behavior. We simulate the traffic situation near the Nanjing Yangtze River Bridge in Simulation of Urban MObility. With a compound reward, our method performs best. Meanwhile, AGRCNet is applied to two abstract environments, and the results show that our approach can also adapt to dynamic agent relationships and is more efficient than comparison algorithms. [ABSTRACT FROM AUTHOR]

Using reinforcement learning to control traffic signal systems has been discussed in recent years, but most works focused on simple scenarios such as a single crossroads, and the methods aiming at large-scale traffic scenarios face long-time training and suboptimal results. In this work, we develop a new multi-agent reinforcement model for large-scale traffic signal control tasks, and a curriculum transfer learning method is developed to optimize the joint policy step by step. The policies for different intersections are trained in a partially observable Markov decision process with centralized training and decentralized execution mechanism, and we design transformer modules for both the policy and evaluation networks by attention mechanism. We first train policies in a simple traffic scenario, and then these policies are transferred to the next curriculum by policy reloading, while the experiences of the source task are reused selectively. With the number of agents increasing, our method can achieve satisfactory performances quickly by reusing the knowledge from previous curriculums. We conduct several experiments on the Cityflow testbed. In the case of more than 10 crossroads, our model improve the mean reward from 3.0 to 5.0. [ABSTRACT FROM AUTHOR]

Intelligent traffic light control (ITLC) algorithms are very efficient for relieving traffic congestion. Recently, many decentralized multi-agent traffic light control algorithms are proposed. These researches mainly focus on improving reinforcement learning method and coordination method. But, as all the agents need to communicate while coordinating with each other, the communication details should be improved as well. To guarantee communication effectiveness, two aspect should be considered. Firstly, a traffic condition description method need to be designed. By using this method, traffic condition can be described simply and clearly. Secondly, synchronization should be considered. As different intersections have different cycle lengths and message sending event happens at the end of each traffic signal cycle, every agent will receive messages of other agents at different time. So it is hard for an agent to decide which message is the latest one and the most valuable. Apart from communication details, reinforcement learning algorithm used for traffic signal timing should also be improved. In the traditional reinforcement learning based ITLC algorithms, either queue length of congested cars or waiting time of these cars is considered while calculating reward value. But, both of them are very important. So a new reward calculation method is needed. To solve all these problems, in this paper, a new ITLC algorithm is proposed. To improve communication efficiency, this algorithm adopts a new message sending and processing method. Besides, to measure traffic congestion in a more reasonable way, a new reward calculation method is proposed and used. This method takes both waiting time and queue length into consideration. [ABSTRACT FROM AUTHOR]

Cooperation between intersections in large-scale road networks is critical in traffic congestion. Currently, most traffic signals cooperate via pre-defined timing phases, which is extremely inefficient in real-time traffic scenarios. Most existing studies on multi-agent reinforcement learning (MARL) traffic signal control have focused on designing efficient communication methods, but have ignored the importance of how agents interact in cooperative communication. To achieve more efficient cooperation among traffic signals and alleviate urban traffic congestion, this study constructs a Graph Cooperation Q-learning Network Traffic Signal Control (GCQN-TSC) model, which is a graph cooperation network with an embedded self-attention mechanism that enables agents to adjust their attention in real time according to the dynamic traffic flow information, perceive the traffic environment quickly and effectively in a larger range, and help agents achieve more effective collaboration. Moreover, the Deep Graph Q-learning (DGQ) algorithm is proposed in this model to optimize the traffic signal control strategy according to the spatio-temporal characteristics of different traffic scenes and provide the optimal signal phase for each intersection. This study also integrates the ecological traffic concept into MARL traffic signal control, which aims to reduce traffic exhaust emissions. Finally, the proposed GCQN-TSC is experimentally validated both in a synthetic traffic grid and a real-world traffic network using the SUMO simulator. The experimental results show that GCQN-TSC outperforms other traffic signal control methods in almost all performance metrics, including average queue length and waiting time, as it can aggregate information acquired from collaborative agents and make network-level signal optimization decisions. [ABSTRACT FROM AUTHOR]

In contemporary urban, traffic signal control is still enormously difficult. Multi-agent reinforcement learning (MARL) is a promising ways to solve this problem. However, most MARL algorithms can not effectively transfer learning strategies when the agents increase or decrease. This paper proposes a new MARL algorithm called cooperative dynamic delay updating twin delayed deep deterministic policy gradient based on the exponentially weighted moving average (CoTD3-EWMA) to solve the problem. By introducing mean-field theory, the algorithm implicitly models the interaction between agents and environment. It reduces the dimension of action space and improves the scalability of the algorithm. In addition, we propose a dynamic delay updating method based on the exponentially weighted moving average (EWMA), which improves the Q value overestimation problem of the traditional TD3 algorithm. Moreover, a joint reward allocation mechanism and state sharing mechanism are proposed to improve the global strategy learning ability and robustness of the agent. The simulation results show that the performance of the new algorithm is better than the current state-of-the-art algorithms, which effectively reduces the delay time of vehicles and improves the traffic efficiency of the traffic network. [ABSTRACT FROM AUTHOR]

The paper is devoted to scheduling in multiagent systems in the framework of the Flatland 3 competition. The main aim of this competition is to develop an algorithm for the effective control of dense traffic in complex railroad networks according to a given schedule. The proposed solution is based on reinforcement learning. To adapt this method to the particular scheduling problem, a novel approach based on structuring the reward function that stimulates an agent to adhere to its schedule was developed. The architecture of the proposed model is based on a multiagent version of centralized critic with proximal policy optimization (PPO) learning. In addition, a curriculum learning strategy was developed and implemented. This allowed the agent to cope with each level of complexity on time and train the model in more difficult conditions. The proposed solution won first place in the Flatland 3 competition in the reinforcement learning track. [ABSTRACT FROM AUTHOR]

Increasing urban traffic and congestion have led to significant issues such as rising air pollution and wasted time, highlighting the need for an intelligent traffic light control (TLC) system to minimize vehicle waiting times. This paper presents a novel TLC system that leverages the Internet of Things (IoT) for data collection and employs the random forest algorithm for preprocessing and feature extraction. A deep belief network predicts future traffic conditions, and a support vector regression network is integrated to enhance prediction accuracy. Additionally, the traffic light control strategy is optimized using reinforcement learning. The proposed method is evaluated through two different scenarios. The first scenario is compared with fixed-time control and the double dueling deep neural network (3DQN) methods. The second scenario compares it with the SVM, KNN, and MAADAC approaches. Simulation results demonstrate that the proposed method significantly outperforms these alternative approaches, showing substantial improvements in average vehicle waiting times by more than 20%, 32%, and 45%, respectively. Using a deep belief network, supplemented by support vector regression, ensures high precision in forecasting traffic patterns. Furthermore, the reinforcement learning-based optimization of the traffic light control strategy effectively adapts to changing traffic conditions, providing superior traffic flow management. The results indicate that the proposed system can substantially reduce traffic congestion and improve urban traffic flow. [ABSTRACT FROM AUTHOR]

Traffic lights are an important controlling factor in traffic flows, and good policies will facilitate traffic congestion. A car's waiting time is highly related to the period in which the traffic lights are green or red; thus, a proper controlling policy reduces the waiting time and speeds up the car movement. Emergency vehicles always have a higher priority than other cars and need to reach their destination faster. In the last decade, a lot of research has been done to solve the traffic light scheduling issue using artificial intelligence techniques. Reinforcement learning with simulating human learning has the outstanding performance to solve complex problems, such as traffic lights control. In this study, a deep reinforcement learning traffic light control was implemented at a crossroad using real-time traffic information with an emphasis on emergency vehicles. The agent learns to adjust its policies to prioritize the emergency cars over the other cars. The results show less waiting time and speed up the cars' movement. It also indicates the emergency vehicles will cross the intersection with the least delay. With 200 to 600 cars and one emergency vehicle, the average delay decreases 2–16% in total using reinforcement learning. Considering more emergency vehicles, there is a 27–40% decrease in average delay for emergency vehicles. [ABSTRACT FROM AUTHOR]

As the global economy develops rapidly, traffic congestion has become a major problem for first-tier cities in various countries. In order to address the problem of failed real-time control of the traffic flow data by the traditional traffic light control as well as malicious attack and other security problems faced by the intelligent traffic light (ITL) control system, a multi-agent distributed ITL control method was proposed based on the fog computing platform and the Q learning algorithm used for the reinforcement learning in this study, and the simulation comparison was conducted by using the simulation platform jointly constructed based on the VISSIM-Excel VBA-MATLAB software. Subsequently, on the basis of puzzle difficulty of the computational Diffie–Helleman (CDH) and Hash Collision, the applicable security control scheme of ITL under the fog computing was proposed. The results reveal that the proposed intelligent control system prolongs the time of green light properly when the number of vehicles increases, thereby reducing the delay time and retention rate of vehicles; the security control scheme of ITL based on the puzzle of CDH is less efficient when the vehicle density increases, while that based on the puzzle of Hash collision is very friendly to the fog equipment. In conclusion, the proposed control method of ITL based on the fog computing and Q learning algorithm can alleviate the traffic congestion effectively, so the proposed method has high security. [ABSTRACT FROM AUTHOR]

In recent years, due to the drastic rise in the number of vehicles and the lack of sufficient infrastructure, traffic jams, air pollution, and fuel consumption have increased in cities. The optimization of timing for traffic lights is one of the solutions for the mentioned problems. Many methods have been introduced to deal with these problems, including reinforcement learning. Although a great number of learning-based methods have been used in traffic signal control, they suffer from poor performance and slow learning convergence. In this paper, a transfer learning-based method for traffic signal control has been proposed. Multi-agent system has also been used for modelling the traffic network and transfer learning has been used to make reinforcement learning agents transfer their experience to each other. Furthermore, a classifier has been utilized to classify the transferred experiences. The results show that using the proposed method leads to a significant improvement on average delay time and convergence time of the learning process. [ABSTRACT FROM AUTHOR]

Predicting traffic disturbances is a challenging problem in urban cities. Emergency vehicles (EV) is one of the biggest disturbances that affect traffic fluidity. The goal of this paper is to provide a machine learning application to deal with emergency cases in traffic networks. Particularly, we investigate the use of deep learning techniques coupled with Artificial Immune System to tackle the issue of EV guidance at signalized intersections. To accomplish this goal, we develop a traffic signal control system capable to estimate traffic status, guide EV to reach their destinations while assuming better traffic condition, control traffic signals, and adapt to new disturbances. For traffic forecasting, the suggested system inherits the advantages of convolutional neural networks, classification, and long short term memory. To control traffic signals, the suggested system uses the immune memory algorithm. To enhance and adapt control decisions to traffic disturbances, the suggested system uses a continuous learning approach assumed by an adapted reinforcement learning algorithm. Assessments using well-known algorithms from the literature are detailed in this work. The benchmarking algorithms are the preemptive longest queue first matching weight matrix system, the pre-emptive immune memory algorithm inspired case-based reasoning, and the preemptive optimized stage based fixed time algorithm. Experiments show a competitive performance of the suggested system compared to benchmarking algorithms. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL) methods learn optimal decisions in the presence of a stationary environment. However, the stationary assumption on the environment is very restrictive. In many real world problems like traffic signal control, robotic applications, etc., one often encounters situations with non-stationary environments, and in these scenarios, RL methods yield sub-optimal decisions. In this paper, we thus consider the problem of developing RL methods that obtain optimal decisions in a non-stationary environment. The goal of this problem is to maximize the long-term discounted reward accrued when the underlying model of the environment changes over time. To achieve this, we first adapt a change point algorithm to detect change in the statistics of the environment and then develop an RL algorithm that maximizes the long-run reward accrued. We illustrate that our change point method detects change in the model of the environment effectively and thus facilitates the RL algorithm in maximizing the long-run reward. We further validate the effectiveness of the proposed solution on non-stationary random Markov decision processes, a sensor energy management problem, and a traffic signal control problem. [ABSTRACT FROM AUTHOR]