Search

Recent advancements in connected automated vehicles (CAVs) and reinforcement learning (RL) hold significant promise for enhancing intelligent traffic control systems. This paper conducts a systematic review of studies on RL-based urban traffic control at signalised intersections, highlighting the significant impact of CAVs on traffic control performance improvement. We first review the fundamental concepts of RL algorithms, establishing a foundational understanding for subsequent RL-based traffic control methods. We then review recent progress in RL-based traffic signal control using CV/CAV trajectory data, RL-based CAV trajectory planning, and the cooperative control of both traffic signals and CAVs at signalised intersections. Our aim is to provide researchers with a comprehensive roadmap for future research in RL-based traffic control at signalised intersections. [ABSTRACT FROM AUTHOR]

Compared to traditional traffic signal control methods, the method driven by Deep Reinforcement Learning (DRL) has shown better performance. But the problem of low sample utilization in reinforcement learning also arises. To deal with the problem, this paper presents a novel Twin Delayed Deep Deterministic Policy Gradient with Dual Buffer (TD3_DB) for traffic signal control. In the proposed framework, two experience buffers are used to store important samples and normal samples, separately, and the proportion of the two buffers is adjusted adaptively. In addition, lane pressure, describing the dynamic feature of lane traffic flow, is used for the state design of the TD3 agent, which enhances the perception of the agent toward intersections. Comprehensive experiments on different traffic flow modes has shown, the dual experience replay scheme can improve the sample utilization, and the proposed TD3_DB performs better than other methods such as original TD3, Proximal Policy Optimization (PPO), etc., effectively reducing vehicle queue length and waiting time. [ABSTRACT FROM AUTHOR]

Traffic systems play a key role in modern society. However, these systems are increasingly suffering from problems, such as congestions. A well-known way to efficiently reduce this kind of problem is to perform traffic light control intelligently through reinforcement learning (RL) algorithms. In this context, extracting relevant features from the traffic environment to support decision-making becomes a central concern. Examples of such features include vehicle counting on each queue, identification of vehicles' origins and destinations, among others. Recently, the advent of deep learning has paved to way to efficient methods for extracting some of the aforementioned features. However, the problem of identifying vehicles and their origins and destinations within an intersection has not been fully addressed in the literature, even though such information has shown to play a role in RL-based traffic signal control. Building against this background, in this work we propose a deep learning pipeline for extracting relevant features from intersections based on traffic scenes. Our pipeline comprises three main steps: (i) a YOLO-based object detector fine-tuned using the UAVDT dataset, (ii) a tracking algorithm to keep track of vehicles along their trajectories, and (iii) an origin-destination identification algorithm. Using this pipeline, it is possible to identify vehicles as well as their origins and destinations within a given intersection. In order to assess our pipeline, we evaluated each of its modules separately as well as the pipeline as a whole. The object detector model obtained 98.2% recall and 79.5% precision, on average. The tracking algorithm obtained a MOTA of 72.6% and a MOTP of 74.4%. Finally, the complete pipeline obtained an average error rate of 3.065% in terms of origin and destination counts. [ABSTRACT FROM AUTHOR]

The increasing demand for mobility in our society poses various challenges to traffic engineering, computer science in general, and artificial intelligence in particular. Increasing the capacity of road networks is not always possible, thus a more efficient use of the available transportation infrastructure is required. Another issue is that many problems in traffic management and control are inherently decentralized and/or require adaptation to the traffic situation. Hence, there is a close relationship to multiagent reinforcement learning. However, using reinforcement learning poses the challenge that the state space is normally large and continuous, thus it is necessary to find appropriate schemes to deal with discretization of the state space. To address these issues, a multiagent system with agents learning independently via a learning algorithm was proposed, which is based on estimating Q-values from k-nearest neighbors. In the present paper, we extend this approach and include transfer of experiences among the agents, especially when an agent does not have a good set of k experiences. We deal with traffic signal control, running experiments on a traffic network in which we vary the traffic situation along time, and compare our approach to two baselines (one involving reinforcement learning and one based on fixed times). Our results show that the extended method pays off when an agent returns to an already experienced traffic situation. [ABSTRACT FROM AUTHOR]

Machine learning (ML) methods, particularly Reinforcement Learning (RL), have gained widespread attention for optimizing traffic signal control in intelligent transportation systems. However, existing ML approaches often exhibit limitations in scalability and adaptability, particularly within large traffic networks. This paper introduces an innovative solution by integrating decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin to enhance traffic signal optimization, targeting the reduction of traffic congestion and network-wide fuel consumption associated with vehicle stops and stop delays. In this approach, DGMARL agents are employed to learn traffic state patterns and make informed decisions regarding traffic signal control. The integration with a Digital Twin module further facilitates this process by simulating and replicating the real-time asymmetric traffic behaviors of a complex traffic network. The evaluation of this proposed methodology utilized PTV-Vissim, a traffic simulation software, which also serves as the simulation engine for the Digital Twin. The study focused on the Martin Luther King (MLK) Smart Corridor in Chattanooga, Tennessee, USA, by considering symmetric and asymmetric road layouts and traffic conditions. Comparative analysis against an actuated signal control baseline approach revealed significant improvements. Experiment results demonstrate a remarkable 55.38% reduction in Eco_PI, a developed performance measure capturing the cumulative impact of stops and penalized stop delays on fuel consumption, over a 24 h scenario. In a PM-peak-hour scenario, the average reduction in Eco_PI reached 38.94%, indicating the substantial improvement achieved in optimizing traffic flow and reducing fuel consumption during high-demand periods. These findings underscore the effectiveness of the integrated DGMARL and Digital Twin approach in optimizing traffic signals, contributing to a more sustainable and efficient traffic management system. [ABSTRACT FROM AUTHOR]

In many real-life scenarios, multiple agents necessitate cooperation to accomplish tasks. Benefiting from the significant success of deep learning, many single-agent deep reinforcement learning algorithms have been extended to multi-agent scenarios. Overestimation in value estimation of Q-learning is a significant issue that has been studied comprehensively in the single-agent domains, but rarely in multi-agent reinforcement learning. In this paper, we first demonstrate that Q-learning-based multi-agent reinforcement learning (MARL) methods generally have notably serious overestimation issues, which cannot be alleviated by current methods. To tackle this problem, we introduce the double critic networks structure and the delayed policy update to Q-learning-based multi-agent MARL methods, which reduce the overestimation and enhance the quality of policy updating. To demonstrate the versatility of our proposed method, we select several Q-learning based MARL methods and evaluate them on several multi-agent tasks on the multi-agent particle environment and SMAC. Experimental results demonstrate that the proposed method can avoid the overestimation problem and significantly improve performance. Besides, application in the Traffic Signal Control verifies the feasibility of applying the proposed method in real-world scenarios. [ABSTRACT FROM AUTHOR]

This paper proposes a reinforcement learning (RL)-based traffic control strategy integrated with attention mechanism for large-scale adaptive traffic signal control (ATSC) system. The proposed attention RL integrates attention mechanism into a multiagent RL model, namely multiagent proximal policy optimization (MAPPO), so as to enable more effective, scalable, and stable learning in complex ATSC envi)ronments. In the attention RL, decentralized policies are trained using a centrally computed critic that shares an attention model, while the attention model selects relevant intersections for each agent to estimate the global critic. This framework effectively reduces the computa)tional complexity and stabilizes the training process, enhancing the ability of RL agents to control large-scale traffic networks. The proposed control strategy is tested in both a large synthetic traffic grid and a large real-world traffic network of Yangzhou city using the microscopic traffic simulation tool, SUMO. Experimental results demonstrate that the proposed approach learns stable and sustainable policies that achieve lower congestion level and faster recovery, which outperforms other state-of-art RL-based approaches, as well as a gap-based actuated controller. [ABSTRACT FROM AUTHOR]

With the development of autonomous driving, a mixed traffic flow state composed of connected automated vehicles (CAVs) and human-driven vehicles (HVs) will last for an extended period. The abundant computing resources and CAVs with high compliance in the intelligent connected environment provide a good situation for variable speed limit control on highways, which helps even the traffic flow and improves traffic efficiency and safety. In this paper, we propose a variable speed limit control method for mixed traffic flow based on deep-reinforcement learning. First, the variable speed limit control problem is abstracted into a Markov decision process and the factors of real-time CAV penetration rates and predictions are considered in the state description. Different from variable message signs (VMS), CAVs are taken as the executive objects of the controller so that the variable speed limit control for mixed traffic flow is realized indirectly through the interaction with HVs. Next, double deep Q network (DDQN) is introduced to calculate the optimal speed limit in different states. Finally, the empirical study on US101-S proves the effectiveness of the proposed model. The results show that the variable speed limit control model based on the DDQN algorithm can effectively improve the efficiency and environmental benefits of mixed traffic flow. Moreover, the multi-objective reward function can achieve a better control effect than the single objective. Besides, the proposed model outperforms other models in this paper and predictive factors can further improve proactive control performance. In addition, with the increasing penetration of CAV, the proposed model achieves a better control effect. [ABSTRACT FROM AUTHOR]

The continuous evolution of artificial intelligence and cyber–physical systems has presented promising opportunities for optimizing traffic signal control in densely populated urban areas, with the aim of alleviating traffic congestion. One area that has garnered significant interest from both researchers and practitioners is the application of deep reinforcement learning (DRL) in traffic signal control. However, DRL-based algorithms often suffer from instability due to the dynamic nature of traffic flows. Discrepancies between the environments used for training and those encountered during deployment often lead to operational failures. Moreover, conventional DRL-based traffic signal control algorithms tend to reveal vulnerabilities when faced with unforeseen events, such as sensor failure. These challenges highlight the need for innovative solutions to enhance the robustness and adaptability of such systems. To address these pertinent issues, this paper introduces StageLight, a novel two-stage multiscale learning approach, which involves learning optimal timings on a coarse time scale in stage 1, while finetuning them on a finer time scale in stage 2. Our experimental results demonstrate StageLight's remarkable capability to generalize across diverse traffic conditions and its robustness to various sensor-failure scenarios. [ABSTRACT FROM AUTHOR]

the dynamic states of the road networks. In order to solve this problem, this paper proposed a deep reinforcement learning traffic light control strategy combined with fuzzy control, used SAC deep reinforcement learning to jointly optimize the phase selection and timing of traffic lights at two intersections, while considering multiple influencing factors, used fuzzy control to process the penalty function of SAC. The experimental results demonstrate that compared with the fixed cycle strategy, SAC control strategy and DDPG control strategy, the proposed traffic signal control strategy can obtain faster vehicle speed, and the fuel consumption and exhaust emissions of the vehicle are also improved. [ABSTRACT FROM AUTHOR]

Increasing urban traffic and congestion have led to significant issues such as rising air pollution and wasted time, highlighting the need for an intelligent traffic light control (TLC) system to minimize vehicle waiting times. This paper presents a novel TLC system that leverages the Internet of Things (IoT) for data collection and employs the random forest algorithm for preprocessing and feature extraction. A deep belief network predicts future traffic conditions, and a support vector regression network is integrated to enhance prediction accuracy. Additionally, the traffic light control strategy is optimized using reinforcement learning. The proposed method is evaluated through two different scenarios. The first scenario is compared with fixed-time control and the double dueling deep neural network (3DQN) methods. The second scenario compares it with the SVM, KNN, and MAADAC approaches. Simulation results demonstrate that the proposed method significantly outperforms these alternative approaches, showing substantial improvements in average vehicle waiting times by more than 20%, 32%, and 45%, respectively. Using a deep belief network, supplemented by support vector regression, ensures high precision in forecasting traffic patterns. Furthermore, the reinforcement learning-based optimization of the traffic light control strategy effectively adapts to changing traffic conditions, providing superior traffic flow management. The results indicate that the proposed system can substantially reduce traffic congestion and improve urban traffic flow. [ABSTRACT FROM AUTHOR]

The emergence of connected and autonomous vehicles (CAVs) holds promise for fine-grained traffic control. However, due to the longevity of future mixed traffic scenarios, there is a need for an in-depth exploration of integrating the microscopic speed control of CAVs with the macroscopic variable speed limit (VSL) of human-driven vehicles (HDVs). This paper proposes a Cooperative Traffic Optimization with Multi-agent Reinforcement Learning and Evolutionary VSL (CTO-ME) framework, which combines microscopic CAV control with macroscopic VSL control. The framework incorporates a Graph Attention Mechanism (GATs) into the multi-agent reinforcement learning framework for intelligent decision-making by microscopic-level vehicles. Additionally, an evolutionary strategy is developed to design the VSL network architecture, enabling macroscopic level real-time speed limit adjustments based on infrastructure. A multi-objective reward function is proposed to optimize both micro and macro efficiency and safety, accounting for both vehicle behavior and traffic flow. Experiments on the designed Bottleneck traffic scenarios show that the proposed approach, CTO-ME, is able to achieve superior performance and outperforms other baselines in terms of traffic throughput, average speed, and safety. Specifically, CTO-ME enhances average velocity by 37%, increases overall throughput by 309%, and raises arrival ratio by 70% than traditional Intelligent Driver Model (IDM). • Macro–Micro Control Integration: We tackled the challenge of harmonizing macroscopic (Variable Speed Limit - VSL) and microscopic (Connected and Automated Vehicles - CAVs) controls to enhance both granular and overarching traffic efficiency. Our proposed method, the Cooperative Traffic Optimization with Multi-agent Reinforcement Learning and Evolutionary VSL (CTO-ME), cooperates with these control methods, paving the way for synchronized and efficient traffic management. • We integrated the Graph Attention Mechanism (GATs) into the multi-agent reinforcement learning framework to amplify the decision-making capabilities of vehicles and developed an advanced evolutionary strategy for VSL control to facilitate real-time speed limit adjustments. • We proposed a multi-objective reward function, considering both micro and macro efficiency and safety considerations, to promote optimized traffic flow and enhance overall control performance. [ABSTRACT FROM AUTHOR]

Perimeter control is a traffic management approach aimed at regulating vehicular accumulation within urban regional networks by managing flows on all border-crossing roads. Methods based on the macroscopic fundamental diagram (MFD) fall short in providing specific metering for individual roads. Recent advancements in the cell transmission model (CTM) have attempted to address this limitation but are hindered by their reliance on centralized control, which requires the availability of full information and authority over traffic generation sites. Our study proposes an innovative decentralized, game-theoretical framework for perimeter control to address these practical challenges. It is structured around two key groups of agents: perimeter agents, tasked with managing border roads, and interior agents, focused on traffic within generation sites. The framework also incorporates mechanisms for interactions between these agents and the road network, aiming to optimize their individual utilities. Additionally, we have developed a multi-agent reinforcement learning (RL) algorithm, extending the mean-field theory concept, to address the complexity of simultaneous learning by multiple agents. • Developed a perimeter control method for heterogeneous decision-makers. • Applied the macroscopic fundamental diagram concept to simplify perimeter control. • Introduced innovative reinforcement learning techniques for problem-solving. • Demonstrated superior performance using real-world traffic data. [ABSTRACT FROM AUTHOR]

In the domain of complex urban traffic networks, real-time Traffic Signal Control (TSC) serves as a pivotal strategy for mitigating congestion. Coordinating signal control across multiple intersections involves considerable complexity. Deep Reinforcement Learning (DRL) has emerged as a robust solution. In recent years, there has been rapid advancement in TSC methods, with numerous researchers employing various novel DRL methodologies. Yet, existing surveys lack timeliness and universality in capturing the latest research. There is a notable gap in current research surveys with respect to the latest developments in TSC. Therefore, the focus of this paper lies in analyzing the most recent papers from the past five years, with the aim to provide a comprehensive and multi-dimensional review of the evolution of DRL in TSC. The survey categorizes current research based on model setups, utilized algorithms, and application scenarios. Finally, this paper highlights potential directions for future TSC research. [ABSTRACT FROM AUTHOR]

Traffic Signal Control (TSC) plays a crucial role in mitigating congestion. The extensive integration of Deep Reinforcement Learning (DRL) into TSC, fueled by the advancements in artificial intelligence, has yielded remarkable efficacy. Nevertheless, prevailing DRL models are typically developed within homogeneous road networks, characterized by identical intersection types and relatively balanced traffic flow patterns. This predominant approach often results in an oversight of the impact of intersection heterogeneity, encompassing variations in intersection geometric structure, traffic demands, signal timing design, and other pertinent factors, on action selections by researchers. This paper addresses the aforementioned gap by introducing a value-decomposition-based spatiotemporal graph attention multi-agent DRL model (MARL_SGAT) that expressly accommodates intersection heterogeneity. Specifically, a heterogeneous correlation index is designed using road network structural parameters and serves to formulate a heterogeneous reward function, enabling the quantification of action execution improvements in diverse road network environments. To address the intricate spatiotemporal features of traffic flows within the heterogeneous road network, the model strategically selects and incorporates these features during the formulation by leveraging a spatiotemporal graph attention network. On these bases, double dual networks are introduced to convert the individual-global-max constraint of action selection into a value range constraint of the action advantage function, thereby facilitating model learning. Simulation results showcase the superior performance of the MARL_SGAT algorithm when compared to seven baseline algorithms. Specifically, the algorithm manifests in reduced average vehicle delays, decreased number of stops, and elevated travel speeds within the controlled road network. These advantages are particularly conspicuous in heterogeneous road network environments. • MARL method for collaborative control of heterogeneous intersections is proposed. • A correlation index is formulated to quantify intersection topology variations. • A novel reward function is designed to offer more precise evaluation method. • The double dual networks simplify the learning process for Q value function. • Performance of the proposed model is validated using actual data set. [ABSTRACT FROM AUTHOR]

Traffic signal control has always been a hot topic in the field of intelligent transportation. With the increasing complexity of urban traffic conditions due to urbanization, how to develop effective scheduling strategies to adapt to the changing traffic demands has become a key problem in current intelligent transportation. In light of this, this paper focuses on the traffic signal control problem at intersections and proposes a composite intelligent traffic signal control model based on heterogeneous graph neural networks with dual attention mechanisms and deep reinforcement learning. For the first time, the model incorporates the dual attention mechanism in graph neural networks into the traffic signal control, integrating graph neural networks with deep reinforcement learning techniques and traffic intersection scenarios. This allows for the construction of traffic condition models and the scheduling control of traffic resources, catering to the perception and decision-making needs in complex traffic environments. Firstly, the graph relationship representation of intersection resources is established, constructing the graph information structure for traffic flow and signal states. Then, a heterogeneous graph neural network is designed, incorporating both node-level and semantic-level dual attention mechanisms to characterize the traffic state and explore the relationships, enabling the extraction of explicit and implicit information in traffic intersections. Lastly, a deep reinforcement learning algorithm that combines Double Deep Q-Network (DDQN) and Dueling DQN is implemented to improve the algorithm's generalization and execution efficiency, enhancing the adaptability and stability of traffic signal scheduling in complex environments. Simulation tests are conducted on the SUMO simulation platform using real-world application datasets. Compared to four other similar traffic control model, the proposed model demonstrates performance advantages of more than 13% in terms of average reward, average delay, queue length, and waiting time. This validates the effectiveness of the proposed model. [ABSTRACT FROM AUTHOR]

Conventional platooning control algorithms that improve traffic flow require complex computations and are not well-suited for real-time operations. Moreover, prior research has primarily concentrated on idealized settings, such as isolated intersections or corridors that solely consist of fully connected automated vehicles (CAVs), making it challenging to be applied to real-world traffic systems. To overcome these challenges, this paper aims to design an innovative learning framework based on arrival timing vectors (ATVs) and traffic signal synchronization for platooning control that can improve traffic throughput and reduce fuel consumption through four basic platoon maneuvers: split, acceleration, deceleration, and no-op. We integrate reinforcement learning (RL) with neural networks (NNs) to model non-linear relationships between inputs and outputs for a complex application. The utilization of ATVs as inputs in our proposed framework furnishes an agent with ample information to facilitate a better comprehension of the actions that need to be taken for particular states. Additionally, our proposed DRL-based platooning control collaborates with traffic signal synchronization across adjacent intersections to optimize traffic efficiency in a further step. Numerical experiments have been conducted on both corridor and network scenarios, demonstrating the positive effects of synchronization and DRL-based platooning control on average delay and fuel consumption. Various DRL-based methods have been compared to demonstrate the fast convergence and improved mobility of our proposed algorithm. Furthermore, our algorithm shows robust performance in mixed-traffic environments of CAV and human-driven vehicles. • A new DRL framework using deep learning to manipulate vehicle platoon behaviors. • The approach integrates platooning control with traffic signal synchronization. • A comparison shows the 2DSARSA algorithm outperforms other popular DRL methods. • Case studies showed DRL-based control reduce delay and fuel use in different networks. [ABSTRACT FROM AUTHOR]

The shared autonomous vehicle (SAV), combining shared mobility and vehicle automation, will play a pivotal role in urban transportation. Effectively matching SAVs to stochastic customer demand poses a significant challenge, particularly when addressing microscopic multi-AV coordination to prevent conflicts and congestion within a network. To tackle this issue, we propose a novel traffic control scheme called the Integrated Operation Framework for Order Matching and AV Coordination (I-OFOMAC). This scheme introduces a third-party coordinator responsible for integrating online order matching for the SAV platform and real-time AV coordination for the road manager. Our approach utilizes an offline deep reinforcement learning method to train a state value neural network, which facilitates strategic order matching. Additionally, a neural network approximating bisimulation metrics is trained to enhance exploration and accelerate learning. To achieve scalability in network-level AV coordination, we adopt a scheduled transportation scheme. The coordinator solves the I-OFOMAC during each decision epoch, optimizing order matching and AV coordination. The I-OFOMAC aims to balance the goals of minimizing AV travel delay and maximizing SAV service revenue. We conduct extensive simulation studies in the realistic Wangjing network to evaluate the proposed I-OFOMAC scheme. Compared to two-step benchmark frameworks that implement order matching and AV coordination sequentially, the I-OFOMAC achieves an average increase of 13.28% in platform revenue (ranging from 2% to 22.53%) and a 4.48% reduction in average PAV travel delay (ranging from −6.55% to 20.17%). • Mixed AV zone. • Online operations of joint order matching and AV coordination. • Offline deep reinforcement learning framework. • Bisimulation-guided exploration. • A scalable approximate algorithm. • Numerical studies in realistic Wangjing network. [ABSTRACT FROM AUTHOR]

Single occupancy vehicles are the most attractive transportation alternative for many commuters, leading to increased traffic congestion and air pollution. Advancements in information technologies create opportunities for smart solutions that incentivize ridesharing and mode shift to higher occupancy vehicles (HOVs) to achieve the car lighter vision of cities. In this study, we present HumanLight, a novel decentralized adaptive traffic signal control algorithm designed to optimize people throughput at intersections. Our proposed controller is founded on reinforcement learning with the reward function embedding the transportation-inspired concept of pressure at the person-level. By rewarding HOV commuters with travel time savings for their efforts to merge into a single ride, HumanLight achieves equitable allocation of green times. Apart from adopting FRAP, a state-of-the-art (SOTA) base model, HumanLight introduces the concept of active vehicles, loosely defined as vehicles in proximity to the intersection within the action interval window. The proposed algorithm showcases significant headroom and scalability in different network configurations considering multimodal vehicle splits at various scenarios of HOV adoption. Improvements in person delays and queues range from 15% to over 55% compared to vehicle-level SOTA controllers. We quantify the impact of incorporating active vehicles in the formulation of our RL model for different network structures. HumanLight also enables regulation of the aggressiveness of the HOV prioritization. The impact of parameter setting on the generated phase profile is investigated as a key component of acyclic signal controllers affecting pedestrian waiting times. HumanLight's scalable, decentralized design can reshape the resolution of traffic management to be more human-centric and empower policies that incentivize ridesharing and public transit systems. [Display omitted] • HumanLight is the first scalable human-centric RL-based adaptive signal controller. • Scalability is enabled from the decentralized design and algorithmic formulation. • HumanLight can democratize urban traffic by equitably allocating green times. • Active vehicles are proposed to handle the variance in occupancy of multimodal traffic. • HumanLight offers policymakers control of the aggressiveness in HOV prioritization. [ABSTRACT FROM AUTHOR]

• A Multi-layer Graph Mask Q-Learning algorithm for traffic signals is proposed. • This study constructs a new upper and lower graph structure to extract geometric and spatial features of intersections. • This article introduces an action masking mechanism, enabling the model to be applicable to intersections of arbitrary structures when utilizing any phase as the action. • Model tests were conducted on synthetic road networks with different traffic volumes and two real urban road networks, and the test results showed that the algorithm has strong generalization ability and scalability. Due to its capability in handling complex urban intersection environments, deep reinforcement learning (DRL) has been widely applied in Adaptive Traffic Signal Control (ATSC). However, most existing algorithms are designed for specific road networks or traffic conditions, making it difficult to transfer them to new environments. Moreover, current graph-based algorithms do not fully capture the geometric and spatial features of intersections, leading to incomplete embedding of the agent's environment depiction. Additionally, the actions adopted by these algorithms are inherently based on fixed-cycle phases, limiting the flexibility of traffic signal control. To address the aforementioned issues, this paper proposes a Multi-layer Graph Mask Q-Learning (MGMQ) algorithm for multi-intersection ATSC to optimize traffic and reduce delay. Unlike previous graph-based algorithms, this paper introduces a method for computing multi-layer graphs, dividing the traffic environment into upper-level traffic network-layer graphs and lower-level intersection-layer graphs, and employs the graph attention algorithm and an improved GraphSAGE algorithm for computation. This method not only enables the generation of embedded state for intersections that include geometric and spatial features, but also allows the algorithm to adapt to different traffic conditions and road networks. Additionally, we introduce an action masking mechanism, allowing this algorithm can be adapted to different action spaces. As a result, the algorithm uses arbitrary signal phases as actions to achieve flexible traffic flow control, and can be directly applied to intersections with arbitrary geometry. The final test results demonstrate that a model trained solely on synthetic road networks can be directly transferred to other synthetic network configurations or real-world urban road networks, outperforming current state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Signal control has been effective to alleviate urban traffic congestion. Massive related works about signal timing optimization have been proposed and led to many signal control methods and systems. In recent years, reinforcement learning (RL) algorithms have attracted the increasing attention of researchers in the area of signal control optimization, since they can learn the optimal timing policy themselves by analyzing changing patterns between the traffic condition and signal timing plans. Multi-intersection traffic signal control problems face more challenges than classical single intersection problems such as the dimensionality issue as the joint action space of multiple agents grows exponentially with the number of intersections. The existing state-of-art multi-agent reinforcement learning algorithms developed for other areas may not suit well with traffic signal control given the complex spatial–temporal nature of traffic flows. In this paper, we propose a novel multi-agent counterfactual actor–critic with scheduler (MACS) framework for multiple interactions. In this method, decentralized actors control the traffic signals and the centralized critic combines recurrent policies with feed-forward critics. Additionally, a scheduler module that exchanges information among agents helps the individual agent better understand the entire environment. The proposed method was evaluated using simulation experiments based on a real-world urban street network in Shenzhen, China. Results showed that the proposed method outperforms the classic model-based method and several existing RL-based methods according to different measures such as queue length, average travel time delay, and throughput. • Develop an AI-based signal control framework to collaborate multiple intersections. • The counterfactual baseline can well assign the credit among multiple intersections. • The scheduler module helps agents better understand the entire environment. [ABSTRACT FROM AUTHOR]

To improve the performance of systems, optimization has been the prevailing approach in the past. However, the approach faces challenges when multiple goals shall be simultaneously achieved. For illustration, we study a multi-agent system, where agents have a plurality of different, and mutually inconsistent goals. We then allow agents in the system to vote on which traffic signal controllers, which were trained on different goals using deep reinforcement learning, would control the intersection. Taking decisions based on suitable voting procedures turns out to lead to favorable solutions, which perform highly for several goals rather than optimally for one goal and poorly for others. This opens up new opportunities for the management or even self-governance of complex systems that require the consideration and achievement of multiple goals, such as many systems involving humans. Here, we present results for traffic flows in urban street networks, which suggest that "democratizing traffic" would be a promising alternative to centralized control of traffic flows. • We combine voting and traffic signal control. • We use votes of agents as weights for aggregating Q-values of multiple Deep Q-Networks trained on different goals. • Voting allows users to express their preferences and choose how/which algorithm governs them. • We demonstrate the performance of different voting schemes across different voting scenarios. [ABSTRACT FROM AUTHOR]

• Develop a novel two-layer coordinated multi-agent reinforcement learning algorithm. • Synergize traffic signals through local cooperation and global coordination. • Optimization of vehicle emissions and traffic congestion as simultaneous goals. • Proposed method is performed in synthetic network and Cologne urban network on SUMO. Intersection traffic signal control considering vehicle emissions has become an important topic, however, the decision complexity of traffic signal control increases dramatically in a dynamic traffic environment with multi-intersections. It is a severe challenge to coordinate traffic signals at multi-intersections based on Internet of Things information to improve the traffic condition of the road network. This paper proposes a two-layer coordination algorithm based on multi-agent reinforcement learning—Multi-agent Coordinated Policy Optimization (MACoPO), for solving traffic signal control at multi-intersections. MACoPO consists of local cooperation, which adjusts the weights of individual rewards and neighborhood agents' rewards by using local cooperation factors (LCF), and global coordination, which updates the LCF to maximize global rewards. The state and reward functions are designed in terms of the current state of the signal, waiting queue length, vehicle density and emission concentration in the lane, vehicle delay, and vehicle emissions, thus making full use of the intersection state information. The proposed method is extensively assessed through simulation experiments using artificial and real road networks and the numerical results confirm its effectiveness in complex and dynamic real-time traffic environments with multi-intersections. [ABSTRACT FROM AUTHOR]