Search

Machine learning (ML) methods, particularly Reinforcement Learning (RL), have gained widespread attention for optimizing traffic signal control in intelligent transportation systems. However, existing ML approaches often exhibit limitations in scalability and adaptability, particularly within large traffic networks. This paper introduces an innovative solution by integrating decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin to enhance traffic signal optimization, targeting the reduction of traffic congestion and network-wide fuel consumption associated with vehicle stops and stop delays. In this approach, DGMARL agents are employed to learn traffic state patterns and make informed decisions regarding traffic signal control. The integration with a Digital Twin module further facilitates this process by simulating and replicating the real-time asymmetric traffic behaviors of a complex traffic network. The evaluation of this proposed methodology utilized PTV-Vissim, a traffic simulation software, which also serves as the simulation engine for the Digital Twin. The study focused on the Martin Luther King (MLK) Smart Corridor in Chattanooga, Tennessee, USA, by considering symmetric and asymmetric road layouts and traffic conditions. Comparative analysis against an actuated signal control baseline approach revealed significant improvements. Experiment results demonstrate a remarkable 55.38% reduction in Eco_PI, a developed performance measure capturing the cumulative impact of stops and penalized stop delays on fuel consumption, over a 24 h scenario. In a PM-peak-hour scenario, the average reduction in Eco_PI reached 38.94%, indicating the substantial improvement achieved in optimizing traffic flow and reducing fuel consumption during high-demand periods. These findings underscore the effectiveness of the integrated DGMARL and Digital Twin approach in optimizing traffic signals, contributing to a more sustainable and efficient traffic management system. [ABSTRACT FROM AUTHOR]

As urban areas continue to expand, traffic congestion has emerged as a significant challenge impacting urban governance and economic development. Frequent regional traffic congestion has become a primary factor hindering urban economic growth and social activities, necessitating improved regional traffic management. Addressing regional traffic optimization and control methods based on the characteristics of regional congestion has become a crucial and complex issue in the field of traffic management and control research. This paper focuses on the macroscopic fundamental diagram (MFD) and aims to tackle the control problem without relying on traffic determination information. To address this, we introduce the Q-learning (QL) algorithm in reinforcement learning and the Deep Deterministic Policy Gradient (DDPG) algorithm in deep reinforcement learning. Subsequently, we propose the MFD-QL perimeter control model and the MFD-DDPG perimeter control model. We conduct numerical analysis and simulation experiments to verify the effectiveness of the MFD-QL and MFD-DDPG algorithms. The experimental results show that the algorithms converge rapidly to a stable state and achieve superior control effects in optimizing regional perimeter control. [ABSTRACT FROM AUTHOR]

Intelligent traffic light control (ITLC) algorithms are very efficient for relieving traffic congestion. Recently, many decentralized multi-agent traffic light control algorithms are proposed. These researches mainly focus on improving reinforcement learning method and coordination method. But, as all the agents need to communicate while coordinating with each other, the communication details should be improved as well. To guarantee communication effectiveness, two aspect should be considered. Firstly, a traffic condition description method need to be designed. By using this method, traffic condition can be described simply and clearly. Secondly, synchronization should be considered. As different intersections have different cycle lengths and message sending event happens at the end of each traffic signal cycle, every agent will receive messages of other agents at different time. So it is hard for an agent to decide which message is the latest one and the most valuable. Apart from communication details, reinforcement learning algorithm used for traffic signal timing should also be improved. In the traditional reinforcement learning based ITLC algorithms, either queue length of congested cars or waiting time of these cars is considered while calculating reward value. But, both of them are very important. So a new reward calculation method is needed. To solve all these problems, in this paper, a new ITLC algorithm is proposed. To improve communication efficiency, this algorithm adopts a new message sending and processing method. Besides, to measure traffic congestion in a more reasonable way, a new reward calculation method is proposed and used. This method takes both waiting time and queue length into consideration. [ABSTRACT FROM AUTHOR]

To contend traffic congestion on urban networks, existing studies have made great efforts to develop traffic-responsive signal timing algorithms in the last decade. More recently, as an alternative to conventional model-based algorithms, machine learning-based methods have been tested on traffic light timing problems and show promising potentials. However, many researchers and practitioners still questioned the feasibility and applicability of adopting machine learning techniques in the ATSC domain. One of the reasons is that these methods assumed flawless detectors and heavily relied on simulators for training and evaluations. To address such a critical concern, this article customizes a Deep Q-learning Learning (DQL) method to optimize traffic signal timings at urban intersections, where the partial observations from identity-based detectors are inputs, and the green splits are outputs. A simulation-free data-driven prediction model is also developed to train the DQL with reduced computational time. Then the machine learning-based methods are evaluated on a real-world case with Automatic Number-Plate Recognition (ANPR) data. Experiments show the proposed data-driven model can predict the traffic state in limited computational time, and the DQL algorithm is 3.9% better than the field experiment performance from the adaptive control system, SCOOT, and 22% better than the time-of-day plan by SYNCHRO. The results indicate the DQL methods can only yield marginal improvement with restrictive input and output settings in congested traffic flow in comparison to the conventional adaptive method. [ABSTRACT FROM AUTHOR]

The Internet of Things has a large number of terminal devices and a wide range of deployment. This feature has higher requirements for reliable transmission and parallel connection, and is more prone to network congestion resulting in the loss of power information flow and excessive delay. The traditional congestion control algorithm is not suitable for the high concurrent mass heterogeneous iot terminal access control and mass information flow storage control. In this paper, the Transformer BBR algorithm is proposed, which is a congestion control algorithm based on BBR deep reinforcement learning combined with Transformer's excellent long-term prediction ability. In the BBR algorithm bandwidth detection stage, transformer model is used as an agent to detect the network state and map it to the congestion window, so as to find the network changing state in time and make the corresponding decision action. Firstly, the congestion control strategy is learned in the simulation network environment, and then the efficiency is verified in the simulation environment. Finally, the experiment shows that the algorithm can reduce the delay while ensuring good bandwidth performance, and is superior to the main application algorithms in network throughput. [ABSTRACT FROM AUTHOR]

• An algorithm based on heterogeneous GNNs is proposed for feature fusion. • A mutual-information loss function is proposed for training. • Based on above methods, an IHA-MDGI algorithm is proposed for traffic signal control. • Experiments show the IHA-MDGI algorithm outperforms the state-of-the-art algorithms. Adaptive traffic signal control (ATSC) facilitates alleviating traffic congestion. Multi-agent deep reinforcement learning (MDRL) is a new promising algorithm for ATSC, and Graph Neural Networks (GNNs) further promote its learning ability. However, there are some drawbacks in the state-of-the-art MDRL algorithms. (1) These algorithms cannot effectively fuse diverse heterogeneous information of traffic networks due to adopting homogeneous GNNs; (2) These algorithms cannot be effectively trained due to merely adopting MDRL loss functions. In this paper, we propose an Inductive Heterogeneous graph Attention-based Multi-agent Deep Graph Infomax (IHA-MDGI) algorithm for ATSC. The proposed IHA-MDGI algorithm conducts both feature fusion via a proposed Inductive Heterogeneous graph Attention (IHA) algorithm and training via a proposed Multi-agent Deep Graph Infomax (MDGI) framework. Specifically, (1) Unlike the algorithms which adopt homogeneous GNNs, in the IHA algorithm, heterogeneous GNNs are designed to fuse both heterogeneous structural information and heterogeneous features of traffic networks, which aims to acquire heterogeneous information embeddings of traffic networks. (2) In the MDGI framework, the acquired embeddings are used to calculate the signal-control policies and Q -value for each agent, and then a mutual-information loss function is designed, which combines with the MDRL loss function to jointly train the whole algorithm. The designed mutual-information loss function focuses on maximizing mutual information between input (i.e., heterogeneous information embeddings) and output (i.e., Q -value), which can produce cooperative signal-control policies and maximize Q -value. We conduct the experiments in both real-traffic and synthetic-traffic networks under the time-varying traffic flows, and the results demonstrate that IHA-MDGI algorithm outperforms the state-of-the-art MDRL algorithms about multiple metrics. [ABSTRACT FROM AUTHOR]

The increasing traffic demand in urban areas frequently causes traffic congestion, which can be managed only through intelligent traffic signal controls. Although many recent studies have focused on reinforcement learning for traffic signal control (RL-TSC), most have focused on improving performance from an intersection perspective, targeting virtual simulation. The performance indexes from intersection perspectives are averaged by the weighted traffic flow; therefore, if the balance of each movement is not considered, the green time may be overly concentrated on the movements of heavy flow rates. Furthermore, as the ultimate purpose of traffic signal control research is to apply these controls to the real-world intersections, it is necessary to consider the real-world constraints. Hence, this study aims to design RL-TSC considering real-world applicability and confirm the appropriate design of the reward function. The limitations of the detector in the real world and the dual-ring traffic signal system are taken into account in the model design to facilitate real-world application. To design the reward for balancing traffic movements, we define the average delay weighted by traffic volume per lane and entropy of delay in the reward function. Model training is performed at the prototype intersection for ensuring scalability to multiple intersections. The model after prototype pre-training is evaluated by applying it to a network with two intersections without additional training. As a result, the reward function considering the equality of traffic movements shows the best performance. The proposed model reduces the average delay by more than 7.4% and 15.0% compared to the existing real-time adaptive signal control at two intersections, respectively. [ABSTRACT FROM AUTHOR]

Adaptive traffic signal control (ATSC) is an effective method to reduce traffic congestion in modern urban areas. Many studies adopted various approaches to adjust traffic signal plans according to real-time traffic in response to demand fluctuations to improve urban network performance (e.g., minimise delay). Recently, learning-based methods such as reinforcement learning (RL) have achieved promising results in signal plan optimisation. However, adopting these self-learning techniques in future traffic environments in the presence of connected and automated vehicles (CAVs) remains largely an open challenge. This study develops a real-time RL-based adaptive traffic signal control that optimises a signal plan to minimise the total queue length while allowing the CAVs to adjust their speed based on a fixed timing strategy to decrease total stop delays. The highlight of this work is combining a speed guidance system with a reinforcement learning-based traffic signal control. Two different performance measures are implemented to minimise total queue length and total stop delays. Results indicate that the proposed method outperforms a fixed timing plan (with optimal speed advisory in a CAV environment) and traditional actuated control, in terms of average stop delay of vehicle and queue length, particularly under saturated and oversaturated conditions. [ABSTRACT FROM AUTHOR]

Traffic congestion alleviation around intersections has been a growing challenge, and a competent traffic signal control scheme plays a pivotal role in intelligent transportation systems. Recent studies using deep reinforcement learning techniques have shown promising results for traffic signal control, but they only focus on extracting features from traffic conditions of isolated or adjacent intersections. In this work, we employed navigation information for traffic signal control, greatly enriching the features for traffic signal control with deep reinforcement learning. In addition, we are the first to propose a novel scheme DeepNavi to exploit the temporal-spatial relations from numerous navigation routes and extract dynamic real-time and future traffic features. We tested our scheme on a challenging real-world traffic dataset with 16 intersections in a residential district of Hangzhou, China. Extensive experiments were conducted and the results demonstrated that our DeepNavi scheme achieves superior performance over five popular and state-of-the-art baseline methods on different metrics, including queue length, speed, travel time and accumulative waiting time. In addition, with our method, vehicles suffer the least red lights and enjoy the most green waves, which further validates that our scheme greatly relieves the congestion and provides excellent experience for drivers. Simulations with different penetration levels of navigation routes showed that even with only part of navigation routes available in the traffic network, our scheme can obtain superior performance, further demonstrating the effectiveness and feasibility of DeepNavi. [ABSTRACT FROM AUTHOR]

Intelligent traffic light control is one of the modern approaches to solve traffic congestion, where reinforcement learning is a widely used method. Conventionally, reinforcement learning is used to determine whether to change the current phase (or choose a traffic phase) after each small interval. One major drawback of these approaches is that it makes the current traffic light phase duration uncertain before the current phase terminates. Directly determining the duration of the traffic light phase can effectively avoid this shortcoming. An adaptive traffic light timing system is proposed in this paper which can directly control the phase duration. In the proposed system, the Q-learning algorithm is employed and the action space is defined as all possible phase durations. In addition, the reward function is redefined to guide the agent to balance more traffic metrics, and the state is redefined to reduce the state space. Finally, the proposed system is evaluated by equal, unequal, and complex traffic scenarios. Results show that the proposed system has a better performance compared with other methods in controlling traffic lights, even on complex traffic scenarios. [ABSTRACT FROM AUTHOR]

Traffic lights are an important controlling factor in traffic flows, and good policies will facilitate traffic congestion. A car's waiting time is highly related to the period in which the traffic lights are green or red; thus, a proper controlling policy reduces the waiting time and speeds up the car movement. Emergency vehicles always have a higher priority than other cars and need to reach their destination faster. In the last decade, a lot of research has been done to solve the traffic light scheduling issue using artificial intelligence techniques. Reinforcement learning with simulating human learning has the outstanding performance to solve complex problems, such as traffic lights control. In this study, a deep reinforcement learning traffic light control was implemented at a crossroad using real-time traffic information with an emphasis on emergency vehicles. The agent learns to adjust its policies to prioritize the emergency cars over the other cars. The results show less waiting time and speed up the cars' movement. It also indicates the emergency vehicles will cross the intersection with the least delay. With 200 to 600 cars and one emergency vehicle, the average delay decreases 2–16% in total using reinforcement learning. Considering more emergency vehicles, there is a 27–40% decrease in average delay for emergency vehicles. [ABSTRACT FROM AUTHOR]

Improving the efficiency of traffic signal control is an effective way to alleviate traffic congestion at signalized intersections. To achieve effective management of the system-wide traffic flows, current research tends to focus on applying reinforcement learning (RL) techniques for collaborative traffic signal control in a traffic road network. However, the existing collaboration-based methods often ignore the impact of transmission delay for exchanging traffic flow information on the system. Most of the studies assume that the signal controllers can collect all instantaneous vehicular features without delay. To fill the gap, we propose an RL-based cooperative traffic signal control scheme considering the data transmission delay issue in a traffic road network. In this paper, we (1) design our new RL agents to cooperatively control the traffic signals by improving the reward and state representation based on the state-of-the-art max-pressure control theory; (2) propose a traffic state prediction method to address the data transmission delay issue by decreasing the discrepancy between the real-time and delayed traffic conditions; (3) evaluated the performance of our proposed work on both synthetic and real-world scenarios with a different range of data transmission delays. The results demonstrate that our method surpassed the performance of the previous max-pressure-based traffic signal control methods and addressed the data transmission delay issue. [ABSTRACT FROM AUTHOR]

Focusing on the problem with traffic congestion under changing environmental conditions, this paper proposed a trajectory reward light (TR-light) model by combining reinforcement learning, neural network, multi-agent and traffic simulation technology to optimize the traffic at multi-intersections. This method had considerable merits in the following aspects. The traffic organization plan was formulated based on traffic lights; multi-agent reinforcement learning was used on traffic light control; regional traffic organization was optimized through the coordination of traffic lights; the agent implemented trajectory reconstruction after the execution of each behavior so as to change the vehicle travel path without changing the OD pair, and to calculate the final reward of the agent according to the plan and reconstructed trajectory. Finally, it conducted a traffic simulation experiment through SUMO. The comparison of traffic indicators verifies that the proposed model improves the smoothness of the road network and the traffic state at the multi-intersections. Experiments show that the model is feasible and effectively mitigates the traffic congestion. [ABSTRACT FROM AUTHOR]

Cooperative Automated Driving System (ADS)-equipped Vehicles (CoAVs) can be a promising solution to traffic congestion. We present a reinforcement learning strategy using Genetic Fuzzy Systems (GFS) for cooperative merge of vehicles onto a highway in high density, mixed-autonomy traffic conditions. The CoAVs are trained to make their own decisions purely based on information available from its surrounding vehicles, thus making this a decentralized system. The GFS module in each CoAV makes recommendations on the acceleration and lane-change decisions. The CoAVs are trained on different ADS behavioral parameters, such as aggressiveness of lane change, different CoAV market penetration rates (MPRs) and traffic congestion levels. The results show the effectiveness of increasing the MPR. It is noticed that the CoAVs trained with different ADS behavioral parameters can generate completely different cooperative maneuvers. The results also show that the trained CoAVs can cooperate even with human-driven vehicles to improve the overall traffic performance. [ABSTRACT FROM AUTHOR]

Traffic congestion has become common in urban areas worldwide. To solve this problem, the method of searching a solution using artificial intelligence has recently attracted widespread attention because it can solve complex problems such as traffic signal control. This study developed two traffic signal control models using reinforcement learning and a microscopic simulation-based evaluation for an isolated intersection and two coordinated intersections. To develop these models, a deep Q-network (DQN) was used, which is a promising reinforcement learning algorithm. The performance was evaluated by comparing the developed traffic signal control models in this research with the fixed-time signal optimized by Synchro model, which is a traffic signal optimization model. The evaluation showed that the developed traffic signal control model of the isolated intersection was validated, and the coordination of intersections was superior to that of the fixed-time signal control method. [ABSTRACT FROM AUTHOR]

• Develop a novel two-layer coordinated multi-agent reinforcement learning algorithm. • Synergize traffic signals through local cooperation and global coordination. • Optimization of vehicle emissions and traffic congestion as simultaneous goals. • Proposed method is performed in synthetic network and Cologne urban network on SUMO. Intersection traffic signal control considering vehicle emissions has become an important topic, however, the decision complexity of traffic signal control increases dramatically in a dynamic traffic environment with multi-intersections. It is a severe challenge to coordinate traffic signals at multi-intersections based on Internet of Things information to improve the traffic condition of the road network. This paper proposes a two-layer coordination algorithm based on multi-agent reinforcement learning—Multi-agent Coordinated Policy Optimization (MACoPO), for solving traffic signal control at multi-intersections. MACoPO consists of local cooperation, which adjusts the weights of individual rewards and neighborhood agents' rewards by using local cooperation factors (LCF), and global coordination, which updates the LCF to maximize global rewards. The state and reward functions are designed in terms of the current state of the signal, waiting queue length, vehicle density and emission concentration in the lane, vehicle delay, and vehicle emissions, thus making full use of the intersection state information. The proposed method is extensively assessed through simulation experiments using artificial and real road networks and the numerical results confirm its effectiveness in complex and dynamic real-time traffic environments with multi-intersections. [ABSTRACT FROM AUTHOR]

Recently, with increasing urban traffic congestion, there has been an upsurge in studies on reinforcement learning for traffic signal control (RL-TSC), which enables efficient traffic management. However, most existing RL-TSC research relies on simulation, and instances of real-world application are relatively scarce. Furthermore, existing RL-TSC methods employ a step-based approach, controlling traffic signals every short step. This approach can be inefficient in oversaturated traffic conditions, where large deviations in traffic demand between movements prevent traffic signals from being effectively managed based on the overall traffic situation. In this study, we aim to transform simulation-based RL-TSC into a more practical and applicable model. We have developed a RL-TSC method capable of responding to oversaturated traffic conditions, designing an action space that enables the agent to derive an optimal signal set for every cycle length, thereby understanding the traffic situation across all movements. During each cycle length, our proposed model conducts traffic signal optimization and identifies the optimal signal through an iterative exploration. To facilitate a fast and accurate strategy search process, we developed a kinematic wave-based mesoscopic model. This model estimates the density across the entire link, based on collected traffic data and derives the state and reward values. We have validated the field applicability of the proposed RL-TSC method through a real-world demonstration at a congested intersection in Seoul, Korea. The results indicated a significant improvement in traffic congestion, with the average queue length at the intersection reduced by up to 11.4%. [ABSTRACT FROM AUTHOR]

• A decentralized DRL signal control for multi-modal traffic networks is introduced based on a microscopic environment. • Car traffic and bus performances are improved simultaneously and traded off. • The proposed method is applicable to different bus lane layouts and multiple bus lines. • The scalability and transferability of the model are verified. Managing traffic flow at intersections in a large-scale network remains challenging. Multi-modal signalized intersections integrate various objectives, including minimizing the queue length and maintaining constant bus headway. Inefficient traffic signals and bus headway control strategies may cause severe traffic jams, high delays for bus passengers, and bus bunching that harms bus line operations. To simultaneously improve the level of service for car traffic and the bus system in a multi-modal network, this paper integrates bus priority and holding with traffic signal control via decentralized controllers based on Reinforcement Learning (RL). The controller agents act and learn from a synthetic traffic environment built with the microscopic traffic simulator SUMO. Action information is shared among agents to achieve cooperation, forming a Multi-Agent Reinforcement Learning (MARL) framework. The agents simultaneously aim to minimize vehicles' total stopping time and homogenize the forward and backward space headways for buses approaching intersections at each decision step. The Deep Q-Network (DQN) algorithm is applied to manage the continuity of the state space. The tradeoff between the bus transit and car traffic objectives is discussed using various numerical experiments. The introduced method is tested in scenarios with distinct bus lane layouts and bus line deployments. The proposed controller outperforms model-based adaptive control methods and the centralized RL method regarding global traffic efficiency and bus transit stability. Furthermore, the remarkable scalability and transferability of trained models are demonstrated by applying them to several different test networks without retraining. [ABSTRACT FROM AUTHOR]

• A lane-level VSL (LVSL) approach with actor-critic framework is proposed to provide differential VSL control. • A hybrid reward function is presented to adjust relationship between the reducing crash risk and improving traffic efficiency. • A twin delayed deep deterministic policy gradient (TD3)-based LVSL approach is established for the first time. • The TD3LVSL shows better performance than baseline models in terms of improving traffic safety and traffic efficiency. Variable speed limit (VSL) control plays a vital role in the emerging connected automated vehicle highway (CAVH) system, which can alleviate recurrent traffic congestion caused by capacity drop and accidents. However, the effect of the VSL on improving traffic efficiency is still controversial. It is necessary to study how to explore the potential benefits of the VSL by balancing its influence on reducing crash risk and enhancing traffic efficiency. To fill the technological gap above, this paper proposes a reinforcement learning-based lane-level VSL (LVSL) control approach to conduct refined traffic control on the mainlines. Firstly, an actor-critic framework is developed to generate and evaluate the discrete speed limits of each lane in continuous action space. To optimize traffic control performance, a hybrid reward function is developed by synchronously considering traffic safety and traffic efficiency of the bottleneck area. Then, to solve the overestimation bias problem of the actor-critic methods caused by function approximation error, a twin delayed deep deterministic policy gradient (TD3) method is introduced to train the framework of the LVSL method. Finally, a real-world recurrent bottleneck of the State Route 91 highway in California is simulated with consideration of the connected automated vehicles to examine the performance of the TD3-based LVSL (TD3LVSL) controller. The simulation results reveal that the proposed method is capable of reducing crash risk and improving traffic efficiency synchronously. Compared with the LVSL controller based on the deep deterministic policy gradient approach, the TD3LVSL controller shows better performance in terms of traffic safety and efficiency. These findings indicate that the proposed controller could contribute to future traffic control in a CAVH environment. [ABSTRACT FROM AUTHOR]

Intelligent Transportation Systems are essential due to the increased number of traffic congestion problems and challenges nowadays. Traffic Signal Control (TSC) plays a critical role in optimizing the traffic flow and mitigating the congestion within the urban areas. Various research works have been conducted to enhance the behavior of TSCs at intersections and subsequently reduce the traffic congestion. Researchers recently leveraged Deep Learning (DL) and Reinforcement Learning (RL) techniques to optimize TSCs. In RL framework, the agent interacts with surrounding world through states, rewards and actions. The formulation of these key elements is crucial as they impact the way the RL agent behaves and optimizes its policy. However, most of existing frameworks rely on hand-crafted state and reward designs, restricting the RL agent from acting optimally. In this paper, we propose a novel approach to better formulate state and reward definitions in order to boost the performance of the traffic signal controller agent. The intuitive idea is to define both state and reward in a consistent and straightforward manner. We advocate that such a design approach helps achieving training stability and hence provides a rapid convergence to derive best policies. We consider the double deep Q-Network (DDQN) along with prioritized experience replay (PER) for the agent architecture. To evaluate the performance of our approach, we conduct series of simulations using the Simulation of Urban MObility (SUMO) environment. The statistical analysis of our results show that the performance of our proposal outperforms the state-of-the-art state and reward design approaches. [ABSTRACT FROM AUTHOR]

Recently, Adaptive Traffic Signal Control (ATSC) in the multi-intersection system is considered as one of the most critical issues in Intelligent Transportation Systems (ITS). Among the proposed AI -based approaches, Deep Reinforcement Learning (DRL) has been largely applied while showing better performances. This paper proposes a new DRL -based cooperative approach for controlling multiple intersections. The problem is modelled as a Multi-Agent Reinforcement Learning (MARL) system, while each agent is trained to select the best action to control an intersection by obtaining information about its local lanes state. The cooperation aspect is manifested in this approach by considering the effect of the state, action and reward of neighbour agents in the process of policy learning. An intersection controller applies a Deep Q-Network (DQN) method, while transferring state, action and reward received from their neighbour agents to its own loss function during the learning process. The experimental results under different scenarios shows that the proposed approach outperforms many state-of-the-art approaches in terms of three metrics: Average Waiting Time (AWT), Average Queue Length (AQL) and Average Emission CO 2 (AEC). In addition, the cooperation between the different trained DRL -based controllers allows the system to continuously learn and improve its performance by interacting with the environment, particularly when the traffic is congested. [ABSTRACT FROM AUTHOR]

Traffic signal control is of great importance to the urban transportation systems and public travel, yet it becomes challenging because of two essential factors. First, spatial–temporal correlations are crucial to an intersection scenario. However, existing works have either considered only one of these features or a simple fusion of spatial and temporal information without adequately exploiting the potential correlations. Second, some works using graph neural network treats static graph nodes among adjacent intersections, ignoring the fact that intersection traffic is changing dynamically. These dynamically changing characteristics of an intersection are likewise significant for traffic signal prediction. If these problems are not resolved, the traffic pressure will increase and people's time will be wasted. To resolve these challenges, we put forward a novel meta-learning spatial–temporal graph attention network (MetaSTGAT) for adaptive traffic signal control. Specifically, we design a graph neural framework with a graph attention network (GAT) and long short-term memory (LSTM) network to obtain spatial and temporal information. The spatial–temporal features are elaborately merged to improve its performance. Besides, to adapt the graph network to the dynamic traffic flow, i.e., the dynamics of the nodes, we propose a meta-learning method for graph neural network's weights generation. This dynamic weight generation process captures the dynamic changes of graph nodes, and thus the dynamic changes of intersections. In this way, the neighboring nodes in the graph network get new weights in advance by additional features when they influence each other. Comprehensive experiments performed in the multi-intersection scenario on synthetic and real-world datasets demonstrate the effectiveness of MetaSTGAT against other state-of-the-art methods. Our method reduces travel time by 12.23%, 19.30%, 13.84%, 10.91%, 8.24%, and 8.74% over the graph-level method CoLight on four synthetic datasets and two real-world datasets, respectively. [ABSTRACT FROM AUTHOR]

Traffic signal control (TSC) is an established yet challenging engineering solution that alleviates traffic congestion by coordinating vehicles' movements at road intersections. Theoretically, reinforcement learning (RL) is a promising method for adaptive TSC in complex urban traffic networks. However, current TSC systems still rely heavily on simplified rule-based methods in practice. In this paper, we propose: (1) two game theory-aided RL algorithms leveraging Nash Equilibrium and RL, namely Nash Advantage Actor–Critic (Nash-A2C) and Nash Asynchronous Advantage Actor–Critic (Nash-A3C); (2) a distributed computing Internet of Things (IoT) architecture for traffic simulation, which is more suitable for distributed TSC methods like the Nash-A3C deployment in its fog layer. We apply both methods in our computing architecture and obtain better performance than benchmark TSC methods by 22.1% and 9.7% reduction of congestion time and network delay, respectively. [ABSTRACT FROM AUTHOR]

The prevailing variable speed limit (VSL) systems as an effective strategy for traffic control on motorways have the disadvantage that they only work with static VSL zones. Under changing traffic conditions, VSL systems with static VSL zones may perform suboptimally. Therefore, the adaptive design of VSL zones is required in traffic scenarios where congestion characteristics vary widely over space and time. To address this problem, we propose a novel distributed spatial-temporal multi-agent VSL (DWL-ST-VSL) approach capable of dynamically adjusting the length and position of VSL zones to complement the adjustment of speed limits in current VSL control systems. To model DWL-ST-VSL, distributed W-learning (DWL), a reinforcement learning (RL)-based algorithm for collaborative agent-based self-optimization toward multiple policies, is used. Each agent uses RL to learn local policies, thereby maximizing travel speed and eliminating congestion. In addition to local policies, through the concept of remote policies, agents learn how their actions affect their immediate neighbours and which policy or action is preferred in a given situation. To assess the impact of deploying additional agents in the control loop and the different cooperation levels on the control process, DWL-ST-VSL is evaluated in a four-agent configuration (DWL4-ST-VSL). This evaluation is done via SUMO microscopic simulations using collaborative agents controlling four segments upstream of the congestion in traffic scenarios with medium and high traffic loads. DWL also allows for heterogeneity in agents' policies; cooperating agents in DWL4-ST-VSL implement two speed limit sets with different granularity. DWL4-ST-VSL outperforms all baselines (W-learning-based VSL and simple proportional speed control), which use static VSL zones. Finally, our experiments yield insights into the new concept of VSL control. This may trigger further research on using advanced learning-based technology to design a new generation of adaptive traffic control systems to meet the requirements of operating in a nonstationary environment and at the leading edge of emerging connected and autonomous vehicles in general. [ABSTRACT FROM AUTHOR]