Search

This paper introduces a library for cross-simulator comparison of reinforcement learning models in traffic signal control tasks. This library is developed to implement recent state-of-the-art reinforcement learning models with extensible interfaces and unified cross-simulator evaluation metrics. It supports commonly-used simulators in traffic signal control tasks, including Simulation of Urban MObility(SUMO) and CityFlow, and multiple benchmark datasets for fair comparisons. We conducted experiments to validate our implementation of the models and to calibrate the simulators so that the experiments from one simulator could be referential to the other. Based on the validated models and calibrated environments, this paper compares and reports the performance of current state-of-the-art RL algorithms across different datasets and simulators. This is the first time that these methods have been compared fairly under the same datasets with different simulators. [ABSTRACT FROM AUTHOR]

This paper presents a cooperative highway platooning strategy that integrates Multi-Agent Reinforcement Learning (MARL) with Proximal Policy Optimization (PPO) to effectively manage the complex task of merging. In modern transportation systems, platooning—where multiple vehicles travel closely together under coordinated control—promises significant improvements in traffic flow and fuel efficiency. However, the challenge of merging, which involves dynamically adjusting the formation to incorporate new vehicles, remains challenging. Our approach leverages the strengths of MARL to enable individual vehicles within a platoon to learn optimal behaviors through interactions. PPO ensures stable and efficient learning by optimizing policies balancing exploration and exploitation. Simulation results show that our method achieves merging with safety and operational efficiency. [ABSTRACT FROM AUTHOR]

Explicit signal coordination carries prior knowledge of traffic engineering and is widely accepted for global implementation. With the recent popularity of reinforcement learning, numerous researchers have turned to implicit signal coordination. However, these methods inevitably require learning coordination from scratch. To maximize the use of prior knowledge, this study proposes an explicit coordinated signal control (ECSC) method using a soft actor–critic for cycle length determination. This method can fundamentally solve the challenges encountered by traditional methods in determining the cycle length. Soft actor–critic was selected among various reinforcement learning methods. A single agent was administered to the arterials. An action is defined as the selection of a cycle length from among the candidates. The state is represented as a feature vector, including the cycle length and features of each leg at every intersection. The reward is defined as departures that indirectly minimize system vehicle delays. Simulation results indicate that ECSC significantly outperforms the baseline methods, as evident in system vehicle delay across nearly all demand scenarios and throughput in high demand scenarios. The ECSC revitalizes explicit signal coordination and introduces new perspectives on the application of reinforcement learning methods in signal coordination. [ABSTRACT FROM AUTHOR]

Recent advancements in connected automated vehicles (CAVs) and reinforcement learning (RL) hold significant promise for enhancing intelligent traffic control systems. This paper conducts a systematic review of studies on RL-based urban traffic control at signalised intersections, highlighting the significant impact of CAVs on traffic control performance improvement. We first review the fundamental concepts of RL algorithms, establishing a foundational understanding for subsequent RL-based traffic control methods. We then review recent progress in RL-based traffic signal control using CV/CAV trajectory data, RL-based CAV trajectory planning, and the cooperative control of both traffic signals and CAVs at signalised intersections. Our aim is to provide researchers with a comprehensive roadmap for future research in RL-based traffic control at signalised intersections. [ABSTRACT FROM AUTHOR]

Compared to traditional traffic signal control methods, the method driven by Deep Reinforcement Learning (DRL) has shown better performance. But the problem of low sample utilization in reinforcement learning also arises. To deal with the problem, this paper presents a novel Twin Delayed Deep Deterministic Policy Gradient with Dual Buffer (TD3_DB) for traffic signal control. In the proposed framework, two experience buffers are used to store important samples and normal samples, separately, and the proportion of the two buffers is adjusted adaptively. In addition, lane pressure, describing the dynamic feature of lane traffic flow, is used for the state design of the TD3 agent, which enhances the perception of the agent toward intersections. Comprehensive experiments on different traffic flow modes has shown, the dual experience replay scheme can improve the sample utilization, and the proposed TD3_DB performs better than other methods such as original TD3, Proximal Policy Optimization (PPO), etc., effectively reducing vehicle queue length and waiting time. [ABSTRACT FROM AUTHOR]

The automatic control of vehicle traffic in large urban networks constitutes one of the most serious challenges to modern societies, with an impact on improving the quality of human life and saving energy and time. Intersections are a special traffic structure of pivotal importance as they accumulate a large number of vehicles that should be served in an optimal manner. Constructing intelligent models that manage to automatically coordinate and steer vehicles through intersections is a key point in the fragmentation of traffic control, offering active solutions through the flexibility of automatically adapting to a variety of traffic conditions. Responding to this call, this work aims to propose an integrated active solution of automatic traffic management. We introduce a multi-agent reinforcement learning framework that effectively models traffic flow at individual unsignalized intersections. It relies on a compact agent definition, a rich information state space, and a learning process characterized not only by depth and quality, but also by substantial degrees of freedom and variability. The resulting driving profiles are further transferred to larger road networks to integrate their individual elements and compose an effective automatic traffic control platform. Experiments are conducted on simulated road networks of variable complexity, demonstrating the potential of the proposed method. [ABSTRACT FROM AUTHOR]

To improve the effectiveness of model predictive control (MPC) in dynamic traffic signal control strategies, it has been combined with graph convolutional networks (GCNs) and deep reinforcement learning (DRL) technologies. In this study, a neural-network-based traffic signal control optimization method under the MPC framework is proposed. A dynamic correlation matrix is introduced in the predictive model to adapt to the dynamic changes in correlations between nodes over time. The signal control optimization strategy is solved using DRL, where the agent explores the optimal control strategy based on pre-set constraints in the future road environment. The geometric structure and traffic flow data of a real intersection were selected as the simulation validation environment, and a joint simulation was conducted using Python and SUMO. The experimental results indicate that in low-traffic scenarios, the queue length is reduced by more than 2 vehicles compared to the selected comparison methods; in high-traffic scenarios, the queue length is reduced by an average of 17 vehicles. Under the actual traffic data of the intersection, the average speed is increased by 6.4% compared to the fixed timing method; compared to the inductive signal control method, it increases from 9.76 m/s to 11.69 m/s, an improvement of 19.7%, effectively enhancing the intersection signal control performance. [ABSTRACT FROM AUTHOR]

Traffic systems play a key role in modern society. However, these systems are increasingly suffering from problems, such as congestions. A well-known way to efficiently reduce this kind of problem is to perform traffic light control intelligently through reinforcement learning (RL) algorithms. In this context, extracting relevant features from the traffic environment to support decision-making becomes a central concern. Examples of such features include vehicle counting on each queue, identification of vehicles' origins and destinations, among others. Recently, the advent of deep learning has paved to way to efficient methods for extracting some of the aforementioned features. However, the problem of identifying vehicles and their origins and destinations within an intersection has not been fully addressed in the literature, even though such information has shown to play a role in RL-based traffic signal control. Building against this background, in this work we propose a deep learning pipeline for extracting relevant features from intersections based on traffic scenes. Our pipeline comprises three main steps: (i) a YOLO-based object detector fine-tuned using the UAVDT dataset, (ii) a tracking algorithm to keep track of vehicles along their trajectories, and (iii) an origin-destination identification algorithm. Using this pipeline, it is possible to identify vehicles as well as their origins and destinations within a given intersection. In order to assess our pipeline, we evaluated each of its modules separately as well as the pipeline as a whole. The object detector model obtained 98.2% recall and 79.5% precision, on average. The tracking algorithm obtained a MOTA of 72.6% and a MOTP of 74.4%. Finally, the complete pipeline obtained an average error rate of 3.065% in terms of origin and destination counts. [ABSTRACT FROM AUTHOR]

The increasing demand for mobility in our society poses various challenges to traffic engineering, computer science in general, and artificial intelligence in particular. Increasing the capacity of road networks is not always possible, thus a more efficient use of the available transportation infrastructure is required. Another issue is that many problems in traffic management and control are inherently decentralized and/or require adaptation to the traffic situation. Hence, there is a close relationship to multiagent reinforcement learning. However, using reinforcement learning poses the challenge that the state space is normally large and continuous, thus it is necessary to find appropriate schemes to deal with discretization of the state space. To address these issues, a multiagent system with agents learning independently via a learning algorithm was proposed, which is based on estimating Q-values from k-nearest neighbors. In the present paper, we extend this approach and include transfer of experiences among the agents, especially when an agent does not have a good set of k experiences. We deal with traffic signal control, running experiments on a traffic network in which we vary the traffic situation along time, and compare our approach to two baselines (one involving reinforcement learning and one based on fixed times). Our results show that the extended method pays off when an agent returns to an already experienced traffic situation. [ABSTRACT FROM AUTHOR]

Adaptive cruise control (ACC) is the core building block of future full autonomous driving. Numerous recent research demonstrated that Autonomous Vehicles (AVs) adopting shorter headways generally increase road capacity and may relieve congestion at bottlenecks for moderate demand scenarios. However, with high demand scenarios, bottlenecks can still be activated causing capacity breakdown. Therefore, extra control measures as dynamic traffic control near bottlenecks is necessary. The challenge is harder on urban freeways with consecutive bottlenecks which affect each other. This paper aims to improve the performance of ACC systems in a high demand scenario. A multi‐bottleneck dynamic headway control strategy based on deep reinforcement learning (DRL) that adapts headways to optimize traffic flow and minimize delay is proposed. The controller dynamically assigns an optimal headway for each controlled section, based on state measurement representing the current traffic conditions. The case study is a freeway stretch with three consecutive bottlenecks which is then extended to include eight bottlenecks. Three different RL agent configurations are presented and compared. It is quantitatively demonstrated that the proposed control strategy improves traffic and enhances the system delay by up to 22.30%, and 18.87% compared to shortest headway setting for the three‐bottleneck and the eight‐bottleneck networks, respectively. [ABSTRACT FROM AUTHOR]

In recent years, the increasing production and sales of automobiles have led to a notable rise in congestion on urban road traffic systems, particularly at ramps and intersections with traffic signals. Intelligent traffic signal control represents an effective means of addressing traffic congestion. Reinforcement learning methods have demonstrated considerable potential for addressing complex traffic signal control problems with multidimensional states and actions. In this research, the team propose Q-learning and Deep Q-Network (DQN) based signal control frameworks that use variable phase sequences and cycle times to adjust the order and the duration of signal phases to obtain a stable traffic signal control strategy. Experiments are simulated using the traffic simulator Simulation of Urban Mobility (SUMO) to test the average speed and the lane occupancy rate of vehicles entering the ramp to evaluate its safety performance and test the vehicle's traveling time to assess its stability. The simulation results show that both reinforcement learning algorithms are able to control cars in dynamic traffic environments with higher average speed and lower lane occupancy rate than the no-control method and that the DQN control model improves the average speed by about 10% and reduces the lane occupancy rate by about 30% compared to the Q-learning control model, providing a higher safety performance. [ABSTRACT FROM AUTHOR]

Traffic systems play a key role in modern society. However, these systems are increasingly suffering from problems, such as congestions. A well-known way to efficiently reduce this kind of problem is to perform traffic light control intelligently through reinforcement learning (RL) algorithms. In this context, extracting relevant features from the traffic environment to support decision-making becomes a central concern. Examples of such features include vehicle counting on each queue, identification of vehicles' origins and destinations, among others. Recently, the advent of deep learning has paved to way to efficient methods for extracting some of the aforementioned features. However, the problem of identifying vehicles and their origins and destinations within an intersection has not been fully addressed in the literature, even though such information has shown to play a role in RL-based traffic signal control. Building against this background, in this work we propose a deep learning pipeline for extracting relevant features from intersections based on traffic scenes. Our pipeline comprises three main steps: (i) a YOLO-based object detector fine-tuned using the UAVDT dataset, (ii) a tracking algorithm to keep track of vehicles along their trajectories, and (iii) an origin-destination identification algorithm. Using this pipeline, it is possible to identify vehicles as well as their origins and destinations within a given intersection. In order to assess our pipeline, we evaluated each of its modules separately as well as the pipeline as a whole. The object detector model obtained 98.2% recall and 79.5% precision, on average. The tracking algorithm obtained a MOTA of 72.6% and a MOTP of 74.4%. Finally, the complete pipeline obtained an average error rate of 3.065% in terms of origin and destination counts. [ABSTRACT FROM AUTHOR]

Multi-Agent Reinforcement Learning (MARL) has shown strong advantages in urban multi-intersection traffic signal control, but it also suffers from the problems of non-smooth environment and inter-agent coordination. However, most of the existing research on MARL traffic signal control has focused on designing efficient communication to solve the environment non-smoothness problem, while neglecting the coordination between agents. In order to coordinate among agents, this paper combines MARL and the regional mixed-strategy Nash equilibrium to construct a Deep Convolutional Nash Policy Gradient Traffic Signal Control (DCNPG-TSC) model, which enables agents to perceive the traffic environment in a wider range and achieves effective agent communication and collaboration. Additionally, a Multi-Agent Distributional Nash Policy Gradient (MADNPG) algorithm is proposed in this model, which is the first time the mixed-strategy Nash equilibrium is used for the improvement in the Multi-Agent Deep Deterministic Policy Gradient algorithm traffic signal control strategy to provide the optimal signal phase for each intersection. In addition, the eco-mobility concept is integrated into MARL traffic signal control to reduce pollutant emissions at intersections. Finally, simulation results in synthetic and real-world traffic road networks show that DCNPG-TSC outperforms other state-of-the-art MARL traffic signal control methods in almost all performance metrics, because it can aggregate the information of neighboring agents and optimize the agent's decisions through gaming to find an optimal joint equilibrium strategy for the traffic road network. [ABSTRACT FROM AUTHOR]

This research addresses multiple challenges faced by ship networks, including limited bandwidth, unstable network connections, high latency, and command priority. To solve these problems, we used reinforcement learning-based methods to simulate traffic engineering in ship networks. We focused on three aspects—traffic balance, instruction priority, and complex network structure—to evaluate reinforcement learning performance in these scenarios. Performance: We developed a reinforcement learning framework for ship network traffic engineering that treats the routing policy as the state and the network state as the environment. The agent generates routing changes and uses actions to optimize traffic services. The experimental results show that reinforcement learning optimizes network traffic balance, reasonably arranges instruction priorities, and copes with complex network structures, greatly improving the network's quality of service (QoS). Through an in-depth analysis of the experimental data, we noticed that network consumption was reduced by 9.1% under reinforcement learning. Reinforcement learning effectively implemented priority routing of high-priority instructions while reducing the occupancy rate of the edge with the highest occupancy rate in the network by 18.53%. [ABSTRACT FROM AUTHOR]

Large enterprise networks typically rely on expensive, high-speed backbone links to connect multiple campuses across diverse regions. As the volume of traffic traversing these backbone links increases, traffic engineering techniques are employed to filter or redirect traffic flows. Nevertheless, simple rerouting strategies can introduce business disruptions such as packet reordering, which significantly impact the user experience. To address this issue, we introduce an enhanced traffic scheduling algorithm named Critical Flow Rerouting with Weight- Reinforcement Learning(CFRW-RL), which builds upon the critical flow rerouting-reinforcement learning (CFR-RL) algorithm. CFRW-RL incorporates the principles of reinforcement learning, accounting for both the weights and classifications of data flows. This approach enables the algorithm to prioritize flows with lower weights for rerouting. The simulation results demonstrate that CFRW-RL significantly minimizes the rerouting of high-priority business flows and reduces business interference compared with the CFR-RL algorithm and that it maintains a similar computational complexity. [ABSTRACT FROM AUTHOR]

Machine learning (ML) methods, particularly Reinforcement Learning (RL), have gained widespread attention for optimizing traffic signal control in intelligent transportation systems. However, existing ML approaches often exhibit limitations in scalability and adaptability, particularly within large traffic networks. This paper introduces an innovative solution by integrating decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin to enhance traffic signal optimization, targeting the reduction of traffic congestion and network-wide fuel consumption associated with vehicle stops and stop delays. In this approach, DGMARL agents are employed to learn traffic state patterns and make informed decisions regarding traffic signal control. The integration with a Digital Twin module further facilitates this process by simulating and replicating the real-time asymmetric traffic behaviors of a complex traffic network. The evaluation of this proposed methodology utilized PTV-Vissim, a traffic simulation software, which also serves as the simulation engine for the Digital Twin. The study focused on the Martin Luther King (MLK) Smart Corridor in Chattanooga, Tennessee, USA, by considering symmetric and asymmetric road layouts and traffic conditions. Comparative analysis against an actuated signal control baseline approach revealed significant improvements. Experiment results demonstrate a remarkable 55.38% reduction in Eco_PI, a developed performance measure capturing the cumulative impact of stops and penalized stop delays on fuel consumption, over a 24 h scenario. In a PM-peak-hour scenario, the average reduction in Eco_PI reached 38.94%, indicating the substantial improvement achieved in optimizing traffic flow and reducing fuel consumption during high-demand periods. These findings underscore the effectiveness of the integrated DGMARL and Digital Twin approach in optimizing traffic signals, contributing to a more sustainable and efficient traffic management system. [ABSTRACT FROM AUTHOR]

In many real-life scenarios, multiple agents necessitate cooperation to accomplish tasks. Benefiting from the significant success of deep learning, many single-agent deep reinforcement learning algorithms have been extended to multi-agent scenarios. Overestimation in value estimation of Q-learning is a significant issue that has been studied comprehensively in the single-agent domains, but rarely in multi-agent reinforcement learning. In this paper, we first demonstrate that Q-learning-based multi-agent reinforcement learning (MARL) methods generally have notably serious overestimation issues, which cannot be alleviated by current methods. To tackle this problem, we introduce the double critic networks structure and the delayed policy update to Q-learning-based multi-agent MARL methods, which reduce the overestimation and enhance the quality of policy updating. To demonstrate the versatility of our proposed method, we select several Q-learning based MARL methods and evaluate them on several multi-agent tasks on the multi-agent particle environment and SMAC. Experimental results demonstrate that the proposed method can avoid the overestimation problem and significantly improve performance. Besides, application in the Traffic Signal Control verifies the feasibility of applying the proposed method in real-world scenarios. [ABSTRACT FROM AUTHOR]

This paper proposes a reinforcement learning (RL)-based traffic control strategy integrated with attention mechanism for large-scale adaptive traffic signal control (ATSC) system. The proposed attention RL integrates attention mechanism into a multiagent RL model, namely multiagent proximal policy optimization (MAPPO), so as to enable more effective, scalable, and stable learning in complex ATSC envi)ronments. In the attention RL, decentralized policies are trained using a centrally computed critic that shares an attention model, while the attention model selects relevant intersections for each agent to estimate the global critic. This framework effectively reduces the computa)tional complexity and stabilizes the training process, enhancing the ability of RL agents to control large-scale traffic networks. The proposed control strategy is tested in both a large synthetic traffic grid and a large real-world traffic network of Yangzhou city using the microscopic traffic simulation tool, SUMO. Experimental results demonstrate that the proposed approach learns stable and sustainable policies that achieve lower congestion level and faster recovery, which outperforms other state-of-art RL-based approaches, as well as a gap-based actuated controller. [ABSTRACT FROM AUTHOR]

With the development of autonomous driving, a mixed traffic flow state composed of connected automated vehicles (CAVs) and human-driven vehicles (HVs) will last for an extended period. The abundant computing resources and CAVs with high compliance in the intelligent connected environment provide a good situation for variable speed limit control on highways, which helps even the traffic flow and improves traffic efficiency and safety. In this paper, we propose a variable speed limit control method for mixed traffic flow based on deep-reinforcement learning. First, the variable speed limit control problem is abstracted into a Markov decision process and the factors of real-time CAV penetration rates and predictions are considered in the state description. Different from variable message signs (VMS), CAVs are taken as the executive objects of the controller so that the variable speed limit control for mixed traffic flow is realized indirectly through the interaction with HVs. Next, double deep Q network (DDQN) is introduced to calculate the optimal speed limit in different states. Finally, the empirical study on US101-S proves the effectiveness of the proposed model. The results show that the variable speed limit control model based on the DDQN algorithm can effectively improve the efficiency and environmental benefits of mixed traffic flow. Moreover, the multi-objective reward function can achieve a better control effect than the single objective. Besides, the proposed model outperforms other models in this paper and predictive factors can further improve proactive control performance. In addition, with the increasing penetration of CAV, the proposed model achieves a better control effect. [ABSTRACT FROM AUTHOR]

The continuous evolution of artificial intelligence and cyber–physical systems has presented promising opportunities for optimizing traffic signal control in densely populated urban areas, with the aim of alleviating traffic congestion. One area that has garnered significant interest from both researchers and practitioners is the application of deep reinforcement learning (DRL) in traffic signal control. However, DRL-based algorithms often suffer from instability due to the dynamic nature of traffic flows. Discrepancies between the environments used for training and those encountered during deployment often lead to operational failures. Moreover, conventional DRL-based traffic signal control algorithms tend to reveal vulnerabilities when faced with unforeseen events, such as sensor failure. These challenges highlight the need for innovative solutions to enhance the robustness and adaptability of such systems. To address these pertinent issues, this paper introduces StageLight, a novel two-stage multiscale learning approach, which involves learning optimal timings on a coarse time scale in stage 1, while finetuning them on a finer time scale in stage 2. Our experimental results demonstrate StageLight's remarkable capability to generalize across diverse traffic conditions and its robustness to various sensor-failure scenarios. [ABSTRACT FROM AUTHOR]

In this study, a traffic queue model is employed to simulate traffic parameters for optimization purposes. The primary objective is to enhance the average queue length, cumulative delay, and cumulative reward for varying numbers of vehicles generated using an innovative approach. The proposed methodology integrates deep Reinforcement Q-Learning techniques implemented through a realistic simulator, Simulation of Urban MObility, based on OpenStreetMap. This simulator is used to balance the flow of traffic by monitoring traffic conditions and adjusting signals (Red/Yellow/Green) accordingly. The research extensively evaluates simulation outcomes across different scenarios involving 50, 100, 500, and 1000 vehicles, with episodes 10, 50 and 100. The results provide valuable insights into training times, simulation durations, and total rewards obtained, indicating improvements in mitigating traffic congestion. The introduction of a traffic control strategy enhances model effectiveness through the utilization of four-layer deep neural network architectures with 400 neurons in hidden layers. However, the study also highlights persistent challenges related to real-time implementation and complexities in state-action-agent relationships. The progress highlights improvements in reducing congestion and enhancing efficiency at junctions by dynamically adjusting signal phases while leveraging historical data. The results highlight the potential of dynamic traffic management strategies in effectively addressing urban traffic challenges. [ABSTRACT FROM AUTHOR]

With recent technological advancements, the commercialization of autonomous vehicles (AVs) is expected to be realized soon. However, it is anticipated that a mixed traffic of AVs and human-driven vehicles (HVs) will persist for a considerable period until the Market Penetration Rate reaches 100%. During this phase, AVs and HVs will interact and coexist on the roads. Such an environment can cause unpredictable and dynamic traffic conditions due to HVs, which results in traffic problems including traffic congestion. Therefore, the routes of AVs must be controlled in a mixed traffic environment. This study proposes a multi-objective vehicle routing control method using a deep Q-network to control the driving direction at intersections in a mixed traffic environment. The objective is to distribute the traffic flow and control the routes safely and efficiently to their destination. Simulation results showed that the proposed method outperformed existing methods in terms of the driving distance, time, and waiting time of AVs, particularly in more dynamic traffic environments. Consequently, the traffic became smooth as it moved along optimal routes. [ABSTRACT FROM AUTHOR]

Traffic congestion has become a major problem in this rapidly growing world. Everyone operating a vehicle, as well as the traffic police in charge of managing the traffic, finds it difficult to become stuck in heavy traffic. For this a set, predetermined timing for traffic flow for each direction at the junction is utilized by traditional traffic light controllers. However, the concept of a fixed time traffic signal controller does not work well in places with uneven traffic. A dynamic traffic control system is therefore required, which regulates the traffic signals in accordance with the volume of traffic. This paper proposes a model that uses reinforcement learning (RL) along with deep neural networks (DNN) to manage discretions (signal status) for an environment with the help of Simulation of Urban MObility (SUMO). A simulation of real-world environment consisting a network of Four-way crossroad junction that contains 4 arriving lanes and 4 exiting lanes is used to train the agent. The main objective of this research study is to construct a model that can independently determine the best course of action and aims to provide better traffic management that will decrease the average waiting time, cause lower congestion, and provide a smooth flow of traffic. [ABSTRACT FROM AUTHOR]

the dynamic states of the road networks. In order to solve this problem, this paper proposed a deep reinforcement learning traffic light control strategy combined with fuzzy control, used SAC deep reinforcement learning to jointly optimize the phase selection and timing of traffic lights at two intersections, while considering multiple influencing factors, used fuzzy control to process the penalty function of SAC. The experimental results demonstrate that compared with the fixed cycle strategy, SAC control strategy and DDPG control strategy, the proposed traffic signal control strategy can obtain faster vehicle speed, and the fuel consumption and exhaust emissions of the vehicle are also improved. [ABSTRACT FROM AUTHOR]

In metropolitan areas, traffic jams on city streets are a major source of annoyance and financial losses. Recent advancements in data processing algorithms and the widespread availability of traffic detectors have made it possible to implement data-driven strategies for reducing traffic congestion. In order to benefit from intersection cooperation in this setting, this paper presents a distributed control strategy based on RL. In this scenario, traffic prediction software's embedding that takes into account the state of nearby junctions is used to synthesize an RL controller that controls the traffic lights. Loop detector characteristics are insufficient for precise data imputed in sophisticated traffic control systems. Most current imputation methods only use these extracted characteristics, which leads to the creation of data replicas that lack the necessary precision. The clean data are first given a statistical multi-class label, with classes ranging from C1 to Cn. Then, using a deep recurrent neural network (RNN) model, the best data model is created from the labelled spotless data and applied to the class of models in the missed-volume data. Results from simulations using TRANSYT demonstrate that the suggested strategy outperforms conventional methods in terms of waiting times and other important presentation indices. [ABSTRACT FROM AUTHOR]

The optimization and control of traffic signals is very important for logistics transportation. It not only improves the operational efficiency and safety of road traffic, but also conforms to the direction of the intelligent, green, and sustainable development of modern cities. In order to improve the optimization effect of traffic signal control, this paper proposes a traffic signal optimization method based on deep reinforcement learning and Simulation of Urban Mobility (SUMO) software for urban traffic scenarios. The intersection training scenario was established using SUMO micro traffic simulation software, and the maximum vehicle queue length and vehicle queue time were selected as performance evaluation indicators. In order to be more relevant to the real environment, the experiment uses Weibull distribution to simulate vehicle generation. Since deep reinforcement learning takes into account both perceptual and decision-making capabilities, this study proposes a traffic signal optimization control model based on the deep reinforcement learning Deep Q Network (DQN) algorithm by considering the realism and complexity of traffic intersections, and first uses the DQN algorithm to train the model in a training scenario. After that, the G-DQN (Grouping-DQN) algorithm is proposed to address the problems that the definition of states in existing studies cannot accurately represent the traffic states and the slow convergence of neural networks. Finally, the performance of the G-DQN algorithm model was compared with the original DQN algorithm model and Advantage Actor-Critic (A2C) algorithm model. The experimental results show that the improved algorithm increased the main indicators in all aspects. [ABSTRACT FROM AUTHOR]

Traffic lights are crucial for urban traffic management, as they significantly impact congestion reduction and travel safety. Traditional methods relying on hand-crafted rules and operator experience are limited in their ability to adapt to changing traffic environments. To address this challenge, we have been exploring intelligent traffic light control using deep reinforcement learning techniques. However, current approaches often suffer from inadequate training data and unstable training processes, leading to suboptimal performance and real-world consequences. In this study, we propose RELight, a novel random ensemble reinforcement learning-based traffic light control framework. RELight effectively utilizes collected empirical data, ensuring a stable and efficient training process. To evaluate the performance of our proposed framework, we conducted a comprehensive set of experiments on a variety of datasets, including four synthetic datasets and a real traffic dataset collected from surveillance cameras at an intersection in Hangzhou, China. The results show that RELight outperforms existing approaches, demonstrating its superiority and potential for practical traffic light control applications. [ABSTRACT FROM AUTHOR]

Increasing urban traffic and congestion have led to significant issues such as rising air pollution and wasted time, highlighting the need for an intelligent traffic light control (TLC) system to minimize vehicle waiting times. This paper presents a novel TLC system that leverages the Internet of Things (IoT) for data collection and employs the random forest algorithm for preprocessing and feature extraction. A deep belief network predicts future traffic conditions, and a support vector regression network is integrated to enhance prediction accuracy. Additionally, the traffic light control strategy is optimized using reinforcement learning. The proposed method is evaluated through two different scenarios. The first scenario is compared with fixed-time control and the double dueling deep neural network (3DQN) methods. The second scenario compares it with the SVM, KNN, and MAADAC approaches. Simulation results demonstrate that the proposed method significantly outperforms these alternative approaches, showing substantial improvements in average vehicle waiting times by more than 20%, 32%, and 45%, respectively. Using a deep belief network, supplemented by support vector regression, ensures high precision in forecasting traffic patterns. Furthermore, the reinforcement learning-based optimization of the traffic light control strategy effectively adapts to changing traffic conditions, providing superior traffic flow management. The results indicate that the proposed system can substantially reduce traffic congestion and improve urban traffic flow. [ABSTRACT FROM AUTHOR]

The emergence of connected and autonomous vehicles (CAVs) holds promise for fine-grained traffic control. However, due to the longevity of future mixed traffic scenarios, there is a need for an in-depth exploration of integrating the microscopic speed control of CAVs with the macroscopic variable speed limit (VSL) of human-driven vehicles (HDVs). This paper proposes a Cooperative Traffic Optimization with Multi-agent Reinforcement Learning and Evolutionary VSL (CTO-ME) framework, which combines microscopic CAV control with macroscopic VSL control. The framework incorporates a Graph Attention Mechanism (GATs) into the multi-agent reinforcement learning framework for intelligent decision-making by microscopic-level vehicles. Additionally, an evolutionary strategy is developed to design the VSL network architecture, enabling macroscopic level real-time speed limit adjustments based on infrastructure. A multi-objective reward function is proposed to optimize both micro and macro efficiency and safety, accounting for both vehicle behavior and traffic flow. Experiments on the designed Bottleneck traffic scenarios show that the proposed approach, CTO-ME, is able to achieve superior performance and outperforms other baselines in terms of traffic throughput, average speed, and safety. Specifically, CTO-ME enhances average velocity by 37%, increases overall throughput by 309%, and raises arrival ratio by 70% than traditional Intelligent Driver Model (IDM). • Macro–Micro Control Integration: We tackled the challenge of harmonizing macroscopic (Variable Speed Limit - VSL) and microscopic (Connected and Automated Vehicles - CAVs) controls to enhance both granular and overarching traffic efficiency. Our proposed method, the Cooperative Traffic Optimization with Multi-agent Reinforcement Learning and Evolutionary VSL (CTO-ME), cooperates with these control methods, paving the way for synchronized and efficient traffic management. • We integrated the Graph Attention Mechanism (GATs) into the multi-agent reinforcement learning framework to amplify the decision-making capabilities of vehicles and developed an advanced evolutionary strategy for VSL control to facilitate real-time speed limit adjustments. • We proposed a multi-objective reward function, considering both micro and macro efficiency and safety considerations, to promote optimized traffic flow and enhance overall control performance. [ABSTRACT FROM AUTHOR]

Perimeter control is a traffic management approach aimed at regulating vehicular accumulation within urban regional networks by managing flows on all border-crossing roads. Methods based on the macroscopic fundamental diagram (MFD) fall short in providing specific metering for individual roads. Recent advancements in the cell transmission model (CTM) have attempted to address this limitation but are hindered by their reliance on centralized control, which requires the availability of full information and authority over traffic generation sites. Our study proposes an innovative decentralized, game-theoretical framework for perimeter control to address these practical challenges. It is structured around two key groups of agents: perimeter agents, tasked with managing border roads, and interior agents, focused on traffic within generation sites. The framework also incorporates mechanisms for interactions between these agents and the road network, aiming to optimize their individual utilities. Additionally, we have developed a multi-agent reinforcement learning (RL) algorithm, extending the mean-field theory concept, to address the complexity of simultaneous learning by multiple agents. • Developed a perimeter control method for heterogeneous decision-makers. • Applied the macroscopic fundamental diagram concept to simplify perimeter control. • Introduced innovative reinforcement learning techniques for problem-solving. • Demonstrated superior performance using real-world traffic data. [ABSTRACT FROM AUTHOR]

A co-evolutionary traffic signal control using reinforcement learning approach (CORLA) is proposed for time-varying road networks under stochastic capacity. Classic reinforcement learning based traffic signal control cannot effectively reduce traffic congestion for large-scale road networks while standard evolutionary metaheuristics often suffer from significantly high computational cost. A co-evolutionary decomposition algorithm (CODA) is proposed to improve traffic mobility for urban road networks with time-varying traffic flow. To capture time-varying spatial evolution of traffic flow inside road links, a stochastic traffic model is presented. To fully consider road users' response, a stochastic bi-level optimization problem (SBOP) is given where road users' route choice can be fully taken into account. To efficiently implement CORLA in a large-scale road network against high-consequence realization for stochastic capacity, a coordinated co-evolutionary two phase control (CCTPC) is proposed. Numerical experiments are performed at a real-data city road network and various sizable traffic grids. As compared to state-of-the-art traffic signal control for various traffic conditions, obtained results showed that CCTPC exhibits sufficient gain of achieving road network performance and suffers from the least computational cost in all cases. • A co-evolutionary reinforcement learning-based traffic signal control (CORLA) is presented. • A novel Q-learning based Performance Index (QPI) is presented. • A stochastic bilevel optimization problem (SBOP) is proposed to minimize total travel cost. • A scalable coordinated co-evolutionary two phase control (CCTPC) is presented to reduce computational complexity. [ABSTRACT FROM AUTHOR]

In the domain of complex urban traffic networks, real-time Traffic Signal Control (TSC) serves as a pivotal strategy for mitigating congestion. Coordinating signal control across multiple intersections involves considerable complexity. Deep Reinforcement Learning (DRL) has emerged as a robust solution. In recent years, there has been rapid advancement in TSC methods, with numerous researchers employing various novel DRL methodologies. Yet, existing surveys lack timeliness and universality in capturing the latest research. There is a notable gap in current research surveys with respect to the latest developments in TSC. Therefore, the focus of this paper lies in analyzing the most recent papers from the past five years, with the aim to provide a comprehensive and multi-dimensional review of the evolution of DRL in TSC. The survey categorizes current research based on model setups, utilized algorithms, and application scenarios. Finally, this paper highlights potential directions for future TSC research. [ABSTRACT FROM AUTHOR]

Traffic Signal Control (TSC) plays a crucial role in mitigating congestion. The extensive integration of Deep Reinforcement Learning (DRL) into TSC, fueled by the advancements in artificial intelligence, has yielded remarkable efficacy. Nevertheless, prevailing DRL models are typically developed within homogeneous road networks, characterized by identical intersection types and relatively balanced traffic flow patterns. This predominant approach often results in an oversight of the impact of intersection heterogeneity, encompassing variations in intersection geometric structure, traffic demands, signal timing design, and other pertinent factors, on action selections by researchers. This paper addresses the aforementioned gap by introducing a value-decomposition-based spatiotemporal graph attention multi-agent DRL model (MARL_SGAT) that expressly accommodates intersection heterogeneity. Specifically, a heterogeneous correlation index is designed using road network structural parameters and serves to formulate a heterogeneous reward function, enabling the quantification of action execution improvements in diverse road network environments. To address the intricate spatiotemporal features of traffic flows within the heterogeneous road network, the model strategically selects and incorporates these features during the formulation by leveraging a spatiotemporal graph attention network. On these bases, double dual networks are introduced to convert the individual-global-max constraint of action selection into a value range constraint of the action advantage function, thereby facilitating model learning. Simulation results showcase the superior performance of the MARL_SGAT algorithm when compared to seven baseline algorithms. Specifically, the algorithm manifests in reduced average vehicle delays, decreased number of stops, and elevated travel speeds within the controlled road network. These advantages are particularly conspicuous in heterogeneous road network environments. • MARL method for collaborative control of heterogeneous intersections is proposed. • A correlation index is formulated to quantify intersection topology variations. • A novel reward function is designed to offer more precise evaluation method. • The double dual networks simplify the learning process for Q value function. • Performance of the proposed model is validated using actual data set. [ABSTRACT FROM AUTHOR]

Traffic signal control has always been a hot topic in the field of intelligent transportation. With the increasing complexity of urban traffic conditions due to urbanization, how to develop effective scheduling strategies to adapt to the changing traffic demands has become a key problem in current intelligent transportation. In light of this, this paper focuses on the traffic signal control problem at intersections and proposes a composite intelligent traffic signal control model based on heterogeneous graph neural networks with dual attention mechanisms and deep reinforcement learning. For the first time, the model incorporates the dual attention mechanism in graph neural networks into the traffic signal control, integrating graph neural networks with deep reinforcement learning techniques and traffic intersection scenarios. This allows for the construction of traffic condition models and the scheduling control of traffic resources, catering to the perception and decision-making needs in complex traffic environments. Firstly, the graph relationship representation of intersection resources is established, constructing the graph information structure for traffic flow and signal states. Then, a heterogeneous graph neural network is designed, incorporating both node-level and semantic-level dual attention mechanisms to characterize the traffic state and explore the relationships, enabling the extraction of explicit and implicit information in traffic intersections. Lastly, a deep reinforcement learning algorithm that combines Double Deep Q-Network (DDQN) and Dueling DQN is implemented to improve the algorithm's generalization and execution efficiency, enhancing the adaptability and stability of traffic signal scheduling in complex environments. Simulation tests are conducted on the SUMO simulation platform using real-world application datasets. Compared to four other similar traffic control model, the proposed model demonstrates performance advantages of more than 13% in terms of average reward, average delay, queue length, and waiting time. This validates the effectiveness of the proposed model. [ABSTRACT FROM AUTHOR]

Conventional platooning control algorithms that improve traffic flow require complex computations and are not well-suited for real-time operations. Moreover, prior research has primarily concentrated on idealized settings, such as isolated intersections or corridors that solely consist of fully connected automated vehicles (CAVs), making it challenging to be applied to real-world traffic systems. To overcome these challenges, this paper aims to design an innovative learning framework based on arrival timing vectors (ATVs) and traffic signal synchronization for platooning control that can improve traffic throughput and reduce fuel consumption through four basic platoon maneuvers: split, acceleration, deceleration, and no-op. We integrate reinforcement learning (RL) with neural networks (NNs) to model non-linear relationships between inputs and outputs for a complex application. The utilization of ATVs as inputs in our proposed framework furnishes an agent with ample information to facilitate a better comprehension of the actions that need to be taken for particular states. Additionally, our proposed DRL-based platooning control collaborates with traffic signal synchronization across adjacent intersections to optimize traffic efficiency in a further step. Numerical experiments have been conducted on both corridor and network scenarios, demonstrating the positive effects of synchronization and DRL-based platooning control on average delay and fuel consumption. Various DRL-based methods have been compared to demonstrate the fast convergence and improved mobility of our proposed algorithm. Furthermore, our algorithm shows robust performance in mixed-traffic environments of CAV and human-driven vehicles. • A new DRL framework using deep learning to manipulate vehicle platoon behaviors. • The approach integrates platooning control with traffic signal synchronization. • A comparison shows the 2DSARSA algorithm outperforms other popular DRL methods. • Case studies showed DRL-based control reduce delay and fuel use in different networks. [ABSTRACT FROM AUTHOR]

The shared autonomous vehicle (SAV), combining shared mobility and vehicle automation, will play a pivotal role in urban transportation. Effectively matching SAVs to stochastic customer demand poses a significant challenge, particularly when addressing microscopic multi-AV coordination to prevent conflicts and congestion within a network. To tackle this issue, we propose a novel traffic control scheme called the Integrated Operation Framework for Order Matching and AV Coordination (I-OFOMAC). This scheme introduces a third-party coordinator responsible for integrating online order matching for the SAV platform and real-time AV coordination for the road manager. Our approach utilizes an offline deep reinforcement learning method to train a state value neural network, which facilitates strategic order matching. Additionally, a neural network approximating bisimulation metrics is trained to enhance exploration and accelerate learning. To achieve scalability in network-level AV coordination, we adopt a scheduled transportation scheme. The coordinator solves the I-OFOMAC during each decision epoch, optimizing order matching and AV coordination. The I-OFOMAC aims to balance the goals of minimizing AV travel delay and maximizing SAV service revenue. We conduct extensive simulation studies in the realistic Wangjing network to evaluate the proposed I-OFOMAC scheme. Compared to two-step benchmark frameworks that implement order matching and AV coordination sequentially, the I-OFOMAC achieves an average increase of 13.28% in platform revenue (ranging from 2% to 22.53%) and a 4.48% reduction in average PAV travel delay (ranging from −6.55% to 20.17%). • Mixed AV zone. • Online operations of joint order matching and AV coordination. • Offline deep reinforcement learning framework. • Bisimulation-guided exploration. • A scalable approximate algorithm. • Numerical studies in realistic Wangjing network. [ABSTRACT FROM AUTHOR]

Single occupancy vehicles are the most attractive transportation alternative for many commuters, leading to increased traffic congestion and air pollution. Advancements in information technologies create opportunities for smart solutions that incentivize ridesharing and mode shift to higher occupancy vehicles (HOVs) to achieve the car lighter vision of cities. In this study, we present HumanLight, a novel decentralized adaptive traffic signal control algorithm designed to optimize people throughput at intersections. Our proposed controller is founded on reinforcement learning with the reward function embedding the transportation-inspired concept of pressure at the person-level. By rewarding HOV commuters with travel time savings for their efforts to merge into a single ride, HumanLight achieves equitable allocation of green times. Apart from adopting FRAP, a state-of-the-art (SOTA) base model, HumanLight introduces the concept of active vehicles, loosely defined as vehicles in proximity to the intersection within the action interval window. The proposed algorithm showcases significant headroom and scalability in different network configurations considering multimodal vehicle splits at various scenarios of HOV adoption. Improvements in person delays and queues range from 15% to over 55% compared to vehicle-level SOTA controllers. We quantify the impact of incorporating active vehicles in the formulation of our RL model for different network structures. HumanLight also enables regulation of the aggressiveness of the HOV prioritization. The impact of parameter setting on the generated phase profile is investigated as a key component of acyclic signal controllers affecting pedestrian waiting times. HumanLight's scalable, decentralized design can reshape the resolution of traffic management to be more human-centric and empower policies that incentivize ridesharing and public transit systems. [Display omitted] • HumanLight is the first scalable human-centric RL-based adaptive signal controller. • Scalability is enabled from the decentralized design and algorithmic formulation. • HumanLight can democratize urban traffic by equitably allocating green times. • Active vehicles are proposed to handle the variance in occupancy of multimodal traffic. • HumanLight offers policymakers control of the aggressiveness in HOV prioritization. [ABSTRACT FROM AUTHOR]

• A Multi-layer Graph Mask Q-Learning algorithm for traffic signals is proposed. • This study constructs a new upper and lower graph structure to extract geometric and spatial features of intersections. • This article introduces an action masking mechanism, enabling the model to be applicable to intersections of arbitrary structures when utilizing any phase as the action. • Model tests were conducted on synthetic road networks with different traffic volumes and two real urban road networks, and the test results showed that the algorithm has strong generalization ability and scalability. Due to its capability in handling complex urban intersection environments, deep reinforcement learning (DRL) has been widely applied in Adaptive Traffic Signal Control (ATSC). However, most existing algorithms are designed for specific road networks or traffic conditions, making it difficult to transfer them to new environments. Moreover, current graph-based algorithms do not fully capture the geometric and spatial features of intersections, leading to incomplete embedding of the agent's environment depiction. Additionally, the actions adopted by these algorithms are inherently based on fixed-cycle phases, limiting the flexibility of traffic signal control. To address the aforementioned issues, this paper proposes a Multi-layer Graph Mask Q-Learning (MGMQ) algorithm for multi-intersection ATSC to optimize traffic and reduce delay. Unlike previous graph-based algorithms, this paper introduces a method for computing multi-layer graphs, dividing the traffic environment into upper-level traffic network-layer graphs and lower-level intersection-layer graphs, and employs the graph attention algorithm and an improved GraphSAGE algorithm for computation. This method not only enables the generation of embedded state for intersections that include geometric and spatial features, but also allows the algorithm to adapt to different traffic conditions and road networks. Additionally, we introduce an action masking mechanism, allowing this algorithm can be adapted to different action spaces. As a result, the algorithm uses arbitrary signal phases as actions to achieve flexible traffic flow control, and can be directly applied to intersections with arbitrary geometry. The final test results demonstrate that a model trained solely on synthetic road networks can be directly transferred to other synthetic network configurations or real-world urban road networks, outperforming current state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Signal control has been effective to alleviate urban traffic congestion. Massive related works about signal timing optimization have been proposed and led to many signal control methods and systems. In recent years, reinforcement learning (RL) algorithms have attracted the increasing attention of researchers in the area of signal control optimization, since they can learn the optimal timing policy themselves by analyzing changing patterns between the traffic condition and signal timing plans. Multi-intersection traffic signal control problems face more challenges than classical single intersection problems such as the dimensionality issue as the joint action space of multiple agents grows exponentially with the number of intersections. The existing state-of-art multi-agent reinforcement learning algorithms developed for other areas may not suit well with traffic signal control given the complex spatial–temporal nature of traffic flows. In this paper, we propose a novel multi-agent counterfactual actor–critic with scheduler (MACS) framework for multiple interactions. In this method, decentralized actors control the traffic signals and the centralized critic combines recurrent policies with feed-forward critics. Additionally, a scheduler module that exchanges information among agents helps the individual agent better understand the entire environment. The proposed method was evaluated using simulation experiments based on a real-world urban street network in Shenzhen, China. Results showed that the proposed method outperforms the classic model-based method and several existing RL-based methods according to different measures such as queue length, average travel time delay, and throughput. • Develop an AI-based signal control framework to collaborate multiple intersections. • The counterfactual baseline can well assign the credit among multiple intersections. • The scheduler module helps agents better understand the entire environment. [ABSTRACT FROM AUTHOR]

To improve the performance of systems, optimization has been the prevailing approach in the past. However, the approach faces challenges when multiple goals shall be simultaneously achieved. For illustration, we study a multi-agent system, where agents have a plurality of different, and mutually inconsistent goals. We then allow agents in the system to vote on which traffic signal controllers, which were trained on different goals using deep reinforcement learning, would control the intersection. Taking decisions based on suitable voting procedures turns out to lead to favorable solutions, which perform highly for several goals rather than optimally for one goal and poorly for others. This opens up new opportunities for the management or even self-governance of complex systems that require the consideration and achievement of multiple goals, such as many systems involving humans. Here, we present results for traffic flows in urban street networks, which suggest that "democratizing traffic" would be a promising alternative to centralized control of traffic flows. • We combine voting and traffic signal control. • We use votes of agents as weights for aggregating Q-values of multiple Deep Q-Networks trained on different goals. • Voting allows users to express their preferences and choose how/which algorithm governs them. • We demonstrate the performance of different voting schemes across different voting scenarios. [ABSTRACT FROM AUTHOR]

• Develop a novel two-layer coordinated multi-agent reinforcement learning algorithm. • Synergize traffic signals through local cooperation and global coordination. • Optimization of vehicle emissions and traffic congestion as simultaneous goals. • Proposed method is performed in synthetic network and Cologne urban network on SUMO. Intersection traffic signal control considering vehicle emissions has become an important topic, however, the decision complexity of traffic signal control increases dramatically in a dynamic traffic environment with multi-intersections. It is a severe challenge to coordinate traffic signals at multi-intersections based on Internet of Things information to improve the traffic condition of the road network. This paper proposes a two-layer coordination algorithm based on multi-agent reinforcement learning—Multi-agent Coordinated Policy Optimization (MACoPO), for solving traffic signal control at multi-intersections. MACoPO consists of local cooperation, which adjusts the weights of individual rewards and neighborhood agents' rewards by using local cooperation factors (LCF), and global coordination, which updates the LCF to maximize global rewards. The state and reward functions are designed in terms of the current state of the signal, waiting queue length, vehicle density and emission concentration in the lane, vehicle delay, and vehicle emissions, thus making full use of the intersection state information. The proposed method is extensively assessed through simulation experiments using artificial and real road networks and the numerical results confirm its effectiveness in complex and dynamic real-time traffic environments with multi-intersections. [ABSTRACT FROM AUTHOR]