Search

The emergence of connected and autonomous vehicles (CAVs) holds promise for fine-grained traffic control. However, due to the longevity of future mixed traffic scenarios, there is a need for an in-depth exploration of integrating the microscopic speed control of CAVs with the macroscopic variable speed limit (VSL) of human-driven vehicles (HDVs). This paper proposes a Cooperative Traffic Optimization with Multi-agent Reinforcement Learning and Evolutionary VSL (CTO-ME) framework, which combines microscopic CAV control with macroscopic VSL control. The framework incorporates a Graph Attention Mechanism (GATs) into the multi-agent reinforcement learning framework for intelligent decision-making by microscopic-level vehicles. Additionally, an evolutionary strategy is developed to design the VSL network architecture, enabling macroscopic level real-time speed limit adjustments based on infrastructure. A multi-objective reward function is proposed to optimize both micro and macro efficiency and safety, accounting for both vehicle behavior and traffic flow. Experiments on the designed Bottleneck traffic scenarios show that the proposed approach, CTO-ME, is able to achieve superior performance and outperforms other baselines in terms of traffic throughput, average speed, and safety. Specifically, CTO-ME enhances average velocity by 37%, increases overall throughput by 309%, and raises arrival ratio by 70% than traditional Intelligent Driver Model (IDM). • Macro–Micro Control Integration: We tackled the challenge of harmonizing macroscopic (Variable Speed Limit - VSL) and microscopic (Connected and Automated Vehicles - CAVs) controls to enhance both granular and overarching traffic efficiency. Our proposed method, the Cooperative Traffic Optimization with Multi-agent Reinforcement Learning and Evolutionary VSL (CTO-ME), cooperates with these control methods, paving the way for synchronized and efficient traffic management. • We integrated the Graph Attention Mechanism (GATs) into the multi-agent reinforcement learning framework to amplify the decision-making capabilities of vehicles and developed an advanced evolutionary strategy for VSL control to facilitate real-time speed limit adjustments. • We proposed a multi-objective reward function, considering both micro and macro efficiency and safety considerations, to promote optimized traffic flow and enhance overall control performance. [ABSTRACT FROM AUTHOR]

Perimeter control is a traffic management approach aimed at regulating vehicular accumulation within urban regional networks by managing flows on all border-crossing roads. Methods based on the macroscopic fundamental diagram (MFD) fall short in providing specific metering for individual roads. Recent advancements in the cell transmission model (CTM) have attempted to address this limitation but are hindered by their reliance on centralized control, which requires the availability of full information and authority over traffic generation sites. Our study proposes an innovative decentralized, game-theoretical framework for perimeter control to address these practical challenges. It is structured around two key groups of agents: perimeter agents, tasked with managing border roads, and interior agents, focused on traffic within generation sites. The framework also incorporates mechanisms for interactions between these agents and the road network, aiming to optimize their individual utilities. Additionally, we have developed a multi-agent reinforcement learning (RL) algorithm, extending the mean-field theory concept, to address the complexity of simultaneous learning by multiple agents. • Developed a perimeter control method for heterogeneous decision-makers. • Applied the macroscopic fundamental diagram concept to simplify perimeter control. • Introduced innovative reinforcement learning techniques for problem-solving. • Demonstrated superior performance using real-world traffic data. [ABSTRACT FROM AUTHOR]

In the domain of complex urban traffic networks, real-time Traffic Signal Control (TSC) serves as a pivotal strategy for mitigating congestion. Coordinating signal control across multiple intersections involves considerable complexity. Deep Reinforcement Learning (DRL) has emerged as a robust solution. In recent years, there has been rapid advancement in TSC methods, with numerous researchers employing various novel DRL methodologies. Yet, existing surveys lack timeliness and universality in capturing the latest research. There is a notable gap in current research surveys with respect to the latest developments in TSC. Therefore, the focus of this paper lies in analyzing the most recent papers from the past five years, with the aim to provide a comprehensive and multi-dimensional review of the evolution of DRL in TSC. The survey categorizes current research based on model setups, utilized algorithms, and application scenarios. Finally, this paper highlights potential directions for future TSC research. [ABSTRACT FROM AUTHOR]

Traffic Signal Control (TSC) plays a crucial role in mitigating congestion. The extensive integration of Deep Reinforcement Learning (DRL) into TSC, fueled by the advancements in artificial intelligence, has yielded remarkable efficacy. Nevertheless, prevailing DRL models are typically developed within homogeneous road networks, characterized by identical intersection types and relatively balanced traffic flow patterns. This predominant approach often results in an oversight of the impact of intersection heterogeneity, encompassing variations in intersection geometric structure, traffic demands, signal timing design, and other pertinent factors, on action selections by researchers. This paper addresses the aforementioned gap by introducing a value-decomposition-based spatiotemporal graph attention multi-agent DRL model (MARL_SGAT) that expressly accommodates intersection heterogeneity. Specifically, a heterogeneous correlation index is designed using road network structural parameters and serves to formulate a heterogeneous reward function, enabling the quantification of action execution improvements in diverse road network environments. To address the intricate spatiotemporal features of traffic flows within the heterogeneous road network, the model strategically selects and incorporates these features during the formulation by leveraging a spatiotemporal graph attention network. On these bases, double dual networks are introduced to convert the individual-global-max constraint of action selection into a value range constraint of the action advantage function, thereby facilitating model learning. Simulation results showcase the superior performance of the MARL_SGAT algorithm when compared to seven baseline algorithms. Specifically, the algorithm manifests in reduced average vehicle delays, decreased number of stops, and elevated travel speeds within the controlled road network. These advantages are particularly conspicuous in heterogeneous road network environments. • MARL method for collaborative control of heterogeneous intersections is proposed. • A correlation index is formulated to quantify intersection topology variations. • A novel reward function is designed to offer more precise evaluation method. • The double dual networks simplify the learning process for Q value function. • Performance of the proposed model is validated using actual data set. [ABSTRACT FROM AUTHOR]

Traffic signal control has always been a hot topic in the field of intelligent transportation. With the increasing complexity of urban traffic conditions due to urbanization, how to develop effective scheduling strategies to adapt to the changing traffic demands has become a key problem in current intelligent transportation. In light of this, this paper focuses on the traffic signal control problem at intersections and proposes a composite intelligent traffic signal control model based on heterogeneous graph neural networks with dual attention mechanisms and deep reinforcement learning. For the first time, the model incorporates the dual attention mechanism in graph neural networks into the traffic signal control, integrating graph neural networks with deep reinforcement learning techniques and traffic intersection scenarios. This allows for the construction of traffic condition models and the scheduling control of traffic resources, catering to the perception and decision-making needs in complex traffic environments. Firstly, the graph relationship representation of intersection resources is established, constructing the graph information structure for traffic flow and signal states. Then, a heterogeneous graph neural network is designed, incorporating both node-level and semantic-level dual attention mechanisms to characterize the traffic state and explore the relationships, enabling the extraction of explicit and implicit information in traffic intersections. Lastly, a deep reinforcement learning algorithm that combines Double Deep Q-Network (DDQN) and Dueling DQN is implemented to improve the algorithm's generalization and execution efficiency, enhancing the adaptability and stability of traffic signal scheduling in complex environments. Simulation tests are conducted on the SUMO simulation platform using real-world application datasets. Compared to four other similar traffic control model, the proposed model demonstrates performance advantages of more than 13% in terms of average reward, average delay, queue length, and waiting time. This validates the effectiveness of the proposed model. [ABSTRACT FROM AUTHOR]

Conventional platooning control algorithms that improve traffic flow require complex computations and are not well-suited for real-time operations. Moreover, prior research has primarily concentrated on idealized settings, such as isolated intersections or corridors that solely consist of fully connected automated vehicles (CAVs), making it challenging to be applied to real-world traffic systems. To overcome these challenges, this paper aims to design an innovative learning framework based on arrival timing vectors (ATVs) and traffic signal synchronization for platooning control that can improve traffic throughput and reduce fuel consumption through four basic platoon maneuvers: split, acceleration, deceleration, and no-op. We integrate reinforcement learning (RL) with neural networks (NNs) to model non-linear relationships between inputs and outputs for a complex application. The utilization of ATVs as inputs in our proposed framework furnishes an agent with ample information to facilitate a better comprehension of the actions that need to be taken for particular states. Additionally, our proposed DRL-based platooning control collaborates with traffic signal synchronization across adjacent intersections to optimize traffic efficiency in a further step. Numerical experiments have been conducted on both corridor and network scenarios, demonstrating the positive effects of synchronization and DRL-based platooning control on average delay and fuel consumption. Various DRL-based methods have been compared to demonstrate the fast convergence and improved mobility of our proposed algorithm. Furthermore, our algorithm shows robust performance in mixed-traffic environments of CAV and human-driven vehicles. • A new DRL framework using deep learning to manipulate vehicle platoon behaviors. • The approach integrates platooning control with traffic signal synchronization. • A comparison shows the 2DSARSA algorithm outperforms other popular DRL methods. • Case studies showed DRL-based control reduce delay and fuel use in different networks. [ABSTRACT FROM AUTHOR]

The shared autonomous vehicle (SAV), combining shared mobility and vehicle automation, will play a pivotal role in urban transportation. Effectively matching SAVs to stochastic customer demand poses a significant challenge, particularly when addressing microscopic multi-AV coordination to prevent conflicts and congestion within a network. To tackle this issue, we propose a novel traffic control scheme called the Integrated Operation Framework for Order Matching and AV Coordination (I-OFOMAC). This scheme introduces a third-party coordinator responsible for integrating online order matching for the SAV platform and real-time AV coordination for the road manager. Our approach utilizes an offline deep reinforcement learning method to train a state value neural network, which facilitates strategic order matching. Additionally, a neural network approximating bisimulation metrics is trained to enhance exploration and accelerate learning. To achieve scalability in network-level AV coordination, we adopt a scheduled transportation scheme. The coordinator solves the I-OFOMAC during each decision epoch, optimizing order matching and AV coordination. The I-OFOMAC aims to balance the goals of minimizing AV travel delay and maximizing SAV service revenue. We conduct extensive simulation studies in the realistic Wangjing network to evaluate the proposed I-OFOMAC scheme. Compared to two-step benchmark frameworks that implement order matching and AV coordination sequentially, the I-OFOMAC achieves an average increase of 13.28% in platform revenue (ranging from 2% to 22.53%) and a 4.48% reduction in average PAV travel delay (ranging from −6.55% to 20.17%). • Mixed AV zone. • Online operations of joint order matching and AV coordination. • Offline deep reinforcement learning framework. • Bisimulation-guided exploration. • A scalable approximate algorithm. • Numerical studies in realistic Wangjing network. [ABSTRACT FROM AUTHOR]

Single occupancy vehicles are the most attractive transportation alternative for many commuters, leading to increased traffic congestion and air pollution. Advancements in information technologies create opportunities for smart solutions that incentivize ridesharing and mode shift to higher occupancy vehicles (HOVs) to achieve the car lighter vision of cities. In this study, we present HumanLight, a novel decentralized adaptive traffic signal control algorithm designed to optimize people throughput at intersections. Our proposed controller is founded on reinforcement learning with the reward function embedding the transportation-inspired concept of pressure at the person-level. By rewarding HOV commuters with travel time savings for their efforts to merge into a single ride, HumanLight achieves equitable allocation of green times. Apart from adopting FRAP, a state-of-the-art (SOTA) base model, HumanLight introduces the concept of active vehicles, loosely defined as vehicles in proximity to the intersection within the action interval window. The proposed algorithm showcases significant headroom and scalability in different network configurations considering multimodal vehicle splits at various scenarios of HOV adoption. Improvements in person delays and queues range from 15% to over 55% compared to vehicle-level SOTA controllers. We quantify the impact of incorporating active vehicles in the formulation of our RL model for different network structures. HumanLight also enables regulation of the aggressiveness of the HOV prioritization. The impact of parameter setting on the generated phase profile is investigated as a key component of acyclic signal controllers affecting pedestrian waiting times. HumanLight's scalable, decentralized design can reshape the resolution of traffic management to be more human-centric and empower policies that incentivize ridesharing and public transit systems. [Display omitted] • HumanLight is the first scalable human-centric RL-based adaptive signal controller. • Scalability is enabled from the decentralized design and algorithmic formulation. • HumanLight can democratize urban traffic by equitably allocating green times. • Active vehicles are proposed to handle the variance in occupancy of multimodal traffic. • HumanLight offers policymakers control of the aggressiveness in HOV prioritization. [ABSTRACT FROM AUTHOR]

• A Multi-layer Graph Mask Q-Learning algorithm for traffic signals is proposed. • This study constructs a new upper and lower graph structure to extract geometric and spatial features of intersections. • This article introduces an action masking mechanism, enabling the model to be applicable to intersections of arbitrary structures when utilizing any phase as the action. • Model tests were conducted on synthetic road networks with different traffic volumes and two real urban road networks, and the test results showed that the algorithm has strong generalization ability and scalability. Due to its capability in handling complex urban intersection environments, deep reinforcement learning (DRL) has been widely applied in Adaptive Traffic Signal Control (ATSC). However, most existing algorithms are designed for specific road networks or traffic conditions, making it difficult to transfer them to new environments. Moreover, current graph-based algorithms do not fully capture the geometric and spatial features of intersections, leading to incomplete embedding of the agent's environment depiction. Additionally, the actions adopted by these algorithms are inherently based on fixed-cycle phases, limiting the flexibility of traffic signal control. To address the aforementioned issues, this paper proposes a Multi-layer Graph Mask Q-Learning (MGMQ) algorithm for multi-intersection ATSC to optimize traffic and reduce delay. Unlike previous graph-based algorithms, this paper introduces a method for computing multi-layer graphs, dividing the traffic environment into upper-level traffic network-layer graphs and lower-level intersection-layer graphs, and employs the graph attention algorithm and an improved GraphSAGE algorithm for computation. This method not only enables the generation of embedded state for intersections that include geometric and spatial features, but also allows the algorithm to adapt to different traffic conditions and road networks. Additionally, we introduce an action masking mechanism, allowing this algorithm can be adapted to different action spaces. As a result, the algorithm uses arbitrary signal phases as actions to achieve flexible traffic flow control, and can be directly applied to intersections with arbitrary geometry. The final test results demonstrate that a model trained solely on synthetic road networks can be directly transferred to other synthetic network configurations or real-world urban road networks, outperforming current state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Signal control has been effective to alleviate urban traffic congestion. Massive related works about signal timing optimization have been proposed and led to many signal control methods and systems. In recent years, reinforcement learning (RL) algorithms have attracted the increasing attention of researchers in the area of signal control optimization, since they can learn the optimal timing policy themselves by analyzing changing patterns between the traffic condition and signal timing plans. Multi-intersection traffic signal control problems face more challenges than classical single intersection problems such as the dimensionality issue as the joint action space of multiple agents grows exponentially with the number of intersections. The existing state-of-art multi-agent reinforcement learning algorithms developed for other areas may not suit well with traffic signal control given the complex spatial–temporal nature of traffic flows. In this paper, we propose a novel multi-agent counterfactual actor–critic with scheduler (MACS) framework for multiple interactions. In this method, decentralized actors control the traffic signals and the centralized critic combines recurrent policies with feed-forward critics. Additionally, a scheduler module that exchanges information among agents helps the individual agent better understand the entire environment. The proposed method was evaluated using simulation experiments based on a real-world urban street network in Shenzhen, China. Results showed that the proposed method outperforms the classic model-based method and several existing RL-based methods according to different measures such as queue length, average travel time delay, and throughput. • Develop an AI-based signal control framework to collaborate multiple intersections. • The counterfactual baseline can well assign the credit among multiple intersections. • The scheduler module helps agents better understand the entire environment. [ABSTRACT FROM AUTHOR]

To improve the performance of systems, optimization has been the prevailing approach in the past. However, the approach faces challenges when multiple goals shall be simultaneously achieved. For illustration, we study a multi-agent system, where agents have a plurality of different, and mutually inconsistent goals. We then allow agents in the system to vote on which traffic signal controllers, which were trained on different goals using deep reinforcement learning, would control the intersection. Taking decisions based on suitable voting procedures turns out to lead to favorable solutions, which perform highly for several goals rather than optimally for one goal and poorly for others. This opens up new opportunities for the management or even self-governance of complex systems that require the consideration and achievement of multiple goals, such as many systems involving humans. Here, we present results for traffic flows in urban street networks, which suggest that "democratizing traffic" would be a promising alternative to centralized control of traffic flows. • We combine voting and traffic signal control. • We use votes of agents as weights for aggregating Q-values of multiple Deep Q-Networks trained on different goals. • Voting allows users to express their preferences and choose how/which algorithm governs them. • We demonstrate the performance of different voting schemes across different voting scenarios. [ABSTRACT FROM AUTHOR]

• Develop a novel two-layer coordinated multi-agent reinforcement learning algorithm. • Synergize traffic signals through local cooperation and global coordination. • Optimization of vehicle emissions and traffic congestion as simultaneous goals. • Proposed method is performed in synthetic network and Cologne urban network on SUMO. Intersection traffic signal control considering vehicle emissions has become an important topic, however, the decision complexity of traffic signal control increases dramatically in a dynamic traffic environment with multi-intersections. It is a severe challenge to coordinate traffic signals at multi-intersections based on Internet of Things information to improve the traffic condition of the road network. This paper proposes a two-layer coordination algorithm based on multi-agent reinforcement learning—Multi-agent Coordinated Policy Optimization (MACoPO), for solving traffic signal control at multi-intersections. MACoPO consists of local cooperation, which adjusts the weights of individual rewards and neighborhood agents' rewards by using local cooperation factors (LCF), and global coordination, which updates the LCF to maximize global rewards. The state and reward functions are designed in terms of the current state of the signal, waiting queue length, vehicle density and emission concentration in the lane, vehicle delay, and vehicle emissions, thus making full use of the intersection state information. The proposed method is extensively assessed through simulation experiments using artificial and real road networks and the numerical results confirm its effectiveness in complex and dynamic real-time traffic environments with multi-intersections. [ABSTRACT FROM AUTHOR]