Search

Machine learning (ML) methods, particularly Reinforcement Learning (RL), have gained widespread attention for optimizing traffic signal control in intelligent transportation systems. However, existing ML approaches often exhibit limitations in scalability and adaptability, particularly within large traffic networks. This paper introduces an innovative solution by integrating decentralized graph-based multi-agent reinforcement learning (DGMARL) with a Digital Twin to enhance traffic signal optimization, targeting the reduction of traffic congestion and network-wide fuel consumption associated with vehicle stops and stop delays. In this approach, DGMARL agents are employed to learn traffic state patterns and make informed decisions regarding traffic signal control. The integration with a Digital Twin module further facilitates this process by simulating and replicating the real-time asymmetric traffic behaviors of a complex traffic network. The evaluation of this proposed methodology utilized PTV-Vissim, a traffic simulation software, which also serves as the simulation engine for the Digital Twin. The study focused on the Martin Luther King (MLK) Smart Corridor in Chattanooga, Tennessee, USA, by considering symmetric and asymmetric road layouts and traffic conditions. Comparative analysis against an actuated signal control baseline approach revealed significant improvements. Experiment results demonstrate a remarkable 55.38% reduction in Eco_PI, a developed performance measure capturing the cumulative impact of stops and penalized stop delays on fuel consumption, over a 24 h scenario. In a PM-peak-hour scenario, the average reduction in Eco_PI reached 38.94%, indicating the substantial improvement achieved in optimizing traffic flow and reducing fuel consumption during high-demand periods. These findings underscore the effectiveness of the integrated DGMARL and Digital Twin approach in optimizing traffic signals, contributing to a more sustainable and efficient traffic management system. [ABSTRACT FROM AUTHOR]

As urban areas continue to expand, traffic congestion has emerged as a significant challenge impacting urban governance and economic development. Frequent regional traffic congestion has become a primary factor hindering urban economic growth and social activities, necessitating improved regional traffic management. Addressing regional traffic optimization and control methods based on the characteristics of regional congestion has become a crucial and complex issue in the field of traffic management and control research. This paper focuses on the macroscopic fundamental diagram (MFD) and aims to tackle the control problem without relying on traffic determination information. To address this, we introduce the Q-learning (QL) algorithm in reinforcement learning and the Deep Deterministic Policy Gradient (DDPG) algorithm in deep reinforcement learning. Subsequently, we propose the MFD-QL perimeter control model and the MFD-DDPG perimeter control model. We conduct numerical analysis and simulation experiments to verify the effectiveness of the MFD-QL and MFD-DDPG algorithms. The experimental results show that the algorithms converge rapidly to a stable state and achieve superior control effects in optimizing regional perimeter control. [ABSTRACT FROM AUTHOR]

Intelligent traffic light control (ITLC) algorithms are very efficient for relieving traffic congestion. Recently, many decentralized multi-agent traffic light control algorithms are proposed. These researches mainly focus on improving reinforcement learning method and coordination method. But, as all the agents need to communicate while coordinating with each other, the communication details should be improved as well. To guarantee communication effectiveness, two aspect should be considered. Firstly, a traffic condition description method need to be designed. By using this method, traffic condition can be described simply and clearly. Secondly, synchronization should be considered. As different intersections have different cycle lengths and message sending event happens at the end of each traffic signal cycle, every agent will receive messages of other agents at different time. So it is hard for an agent to decide which message is the latest one and the most valuable. Apart from communication details, reinforcement learning algorithm used for traffic signal timing should also be improved. In the traditional reinforcement learning based ITLC algorithms, either queue length of congested cars or waiting time of these cars is considered while calculating reward value. But, both of them are very important. So a new reward calculation method is needed. To solve all these problems, in this paper, a new ITLC algorithm is proposed. To improve communication efficiency, this algorithm adopts a new message sending and processing method. Besides, to measure traffic congestion in a more reasonable way, a new reward calculation method is proposed and used. This method takes both waiting time and queue length into consideration. [ABSTRACT FROM AUTHOR]

To contend traffic congestion on urban networks, existing studies have made great efforts to develop traffic-responsive signal timing algorithms in the last decade. More recently, as an alternative to conventional model-based algorithms, machine learning-based methods have been tested on traffic light timing problems and show promising potentials. However, many researchers and practitioners still questioned the feasibility and applicability of adopting machine learning techniques in the ATSC domain. One of the reasons is that these methods assumed flawless detectors and heavily relied on simulators for training and evaluations. To address such a critical concern, this article customizes a Deep Q-learning Learning (DQL) method to optimize traffic signal timings at urban intersections, where the partial observations from identity-based detectors are inputs, and the green splits are outputs. A simulation-free data-driven prediction model is also developed to train the DQL with reduced computational time. Then the machine learning-based methods are evaluated on a real-world case with Automatic Number-Plate Recognition (ANPR) data. Experiments show the proposed data-driven model can predict the traffic state in limited computational time, and the DQL algorithm is 3.9% better than the field experiment performance from the adaptive control system, SCOOT, and 22% better than the time-of-day plan by SYNCHRO. The results indicate the DQL methods can only yield marginal improvement with restrictive input and output settings in congested traffic flow in comparison to the conventional adaptive method. [ABSTRACT FROM AUTHOR]

The Internet of Things has a large number of terminal devices and a wide range of deployment. This feature has higher requirements for reliable transmission and parallel connection, and is more prone to network congestion resulting in the loss of power information flow and excessive delay. The traditional congestion control algorithm is not suitable for the high concurrent mass heterogeneous iot terminal access control and mass information flow storage control. In this paper, the Transformer BBR algorithm is proposed, which is a congestion control algorithm based on BBR deep reinforcement learning combined with Transformer's excellent long-term prediction ability. In the BBR algorithm bandwidth detection stage, transformer model is used as an agent to detect the network state and map it to the congestion window, so as to find the network changing state in time and make the corresponding decision action. Firstly, the congestion control strategy is learned in the simulation network environment, and then the efficiency is verified in the simulation environment. Finally, the experiment shows that the algorithm can reduce the delay while ensuring good bandwidth performance, and is superior to the main application algorithms in network throughput. [ABSTRACT FROM AUTHOR]

• An algorithm based on heterogeneous GNNs is proposed for feature fusion. • A mutual-information loss function is proposed for training. • Based on above methods, an IHA-MDGI algorithm is proposed for traffic signal control. • Experiments show the IHA-MDGI algorithm outperforms the state-of-the-art algorithms. Adaptive traffic signal control (ATSC) facilitates alleviating traffic congestion. Multi-agent deep reinforcement learning (MDRL) is a new promising algorithm for ATSC, and Graph Neural Networks (GNNs) further promote its learning ability. However, there are some drawbacks in the state-of-the-art MDRL algorithms. (1) These algorithms cannot effectively fuse diverse heterogeneous information of traffic networks due to adopting homogeneous GNNs; (2) These algorithms cannot be effectively trained due to merely adopting MDRL loss functions. In this paper, we propose an Inductive Heterogeneous graph Attention-based Multi-agent Deep Graph Infomax (IHA-MDGI) algorithm for ATSC. The proposed IHA-MDGI algorithm conducts both feature fusion via a proposed Inductive Heterogeneous graph Attention (IHA) algorithm and training via a proposed Multi-agent Deep Graph Infomax (MDGI) framework. Specifically, (1) Unlike the algorithms which adopt homogeneous GNNs, in the IHA algorithm, heterogeneous GNNs are designed to fuse both heterogeneous structural information and heterogeneous features of traffic networks, which aims to acquire heterogeneous information embeddings of traffic networks. (2) In the MDGI framework, the acquired embeddings are used to calculate the signal-control policies and Q -value for each agent, and then a mutual-information loss function is designed, which combines with the MDRL loss function to jointly train the whole algorithm. The designed mutual-information loss function focuses on maximizing mutual information between input (i.e., heterogeneous information embeddings) and output (i.e., Q -value), which can produce cooperative signal-control policies and maximize Q -value. We conduct the experiments in both real-traffic and synthetic-traffic networks under the time-varying traffic flows, and the results demonstrate that IHA-MDGI algorithm outperforms the state-of-the-art MDRL algorithms about multiple metrics. [ABSTRACT FROM AUTHOR]

The increasing traffic demand in urban areas frequently causes traffic congestion, which can be managed only through intelligent traffic signal controls. Although many recent studies have focused on reinforcement learning for traffic signal control (RL-TSC), most have focused on improving performance from an intersection perspective, targeting virtual simulation. The performance indexes from intersection perspectives are averaged by the weighted traffic flow; therefore, if the balance of each movement is not considered, the green time may be overly concentrated on the movements of heavy flow rates. Furthermore, as the ultimate purpose of traffic signal control research is to apply these controls to the real-world intersections, it is necessary to consider the real-world constraints. Hence, this study aims to design RL-TSC considering real-world applicability and confirm the appropriate design of the reward function. The limitations of the detector in the real world and the dual-ring traffic signal system are taken into account in the model design to facilitate real-world application. To design the reward for balancing traffic movements, we define the average delay weighted by traffic volume per lane and entropy of delay in the reward function. Model training is performed at the prototype intersection for ensuring scalability to multiple intersections. The model after prototype pre-training is evaluated by applying it to a network with two intersections without additional training. As a result, the reward function considering the equality of traffic movements shows the best performance. The proposed model reduces the average delay by more than 7.4% and 15.0% compared to the existing real-time adaptive signal control at two intersections, respectively. [ABSTRACT FROM AUTHOR]

Adaptive traffic signal control (ATSC) is an effective method to reduce traffic congestion in modern urban areas. Many studies adopted various approaches to adjust traffic signal plans according to real-time traffic in response to demand fluctuations to improve urban network performance (e.g., minimise delay). Recently, learning-based methods such as reinforcement learning (RL) have achieved promising results in signal plan optimisation. However, adopting these self-learning techniques in future traffic environments in the presence of connected and automated vehicles (CAVs) remains largely an open challenge. This study develops a real-time RL-based adaptive traffic signal control that optimises a signal plan to minimise the total queue length while allowing the CAVs to adjust their speed based on a fixed timing strategy to decrease total stop delays. The highlight of this work is combining a speed guidance system with a reinforcement learning-based traffic signal control. Two different performance measures are implemented to minimise total queue length and total stop delays. Results indicate that the proposed method outperforms a fixed timing plan (with optimal speed advisory in a CAV environment) and traditional actuated control, in terms of average stop delay of vehicle and queue length, particularly under saturated and oversaturated conditions. [ABSTRACT FROM AUTHOR]

Traffic congestion alleviation around intersections has been a growing challenge, and a competent traffic signal control scheme plays a pivotal role in intelligent transportation systems. Recent studies using deep reinforcement learning techniques have shown promising results for traffic signal control, but they only focus on extracting features from traffic conditions of isolated or adjacent intersections. In this work, we employed navigation information for traffic signal control, greatly enriching the features for traffic signal control with deep reinforcement learning. In addition, we are the first to propose a novel scheme DeepNavi to exploit the temporal-spatial relations from numerous navigation routes and extract dynamic real-time and future traffic features. We tested our scheme on a challenging real-world traffic dataset with 16 intersections in a residential district of Hangzhou, China. Extensive experiments were conducted and the results demonstrated that our DeepNavi scheme achieves superior performance over five popular and state-of-the-art baseline methods on different metrics, including queue length, speed, travel time and accumulative waiting time. In addition, with our method, vehicles suffer the least red lights and enjoy the most green waves, which further validates that our scheme greatly relieves the congestion and provides excellent experience for drivers. Simulations with different penetration levels of navigation routes showed that even with only part of navigation routes available in the traffic network, our scheme can obtain superior performance, further demonstrating the effectiveness and feasibility of DeepNavi. [ABSTRACT FROM AUTHOR]

Intelligent traffic light control is one of the modern approaches to solve traffic congestion, where reinforcement learning is a widely used method. Conventionally, reinforcement learning is used to determine whether to change the current phase (or choose a traffic phase) after each small interval. One major drawback of these approaches is that it makes the current traffic light phase duration uncertain before the current phase terminates. Directly determining the duration of the traffic light phase can effectively avoid this shortcoming. An adaptive traffic light timing system is proposed in this paper which can directly control the phase duration. In the proposed system, the Q-learning algorithm is employed and the action space is defined as all possible phase durations. In addition, the reward function is redefined to guide the agent to balance more traffic metrics, and the state is redefined to reduce the state space. Finally, the proposed system is evaluated by equal, unequal, and complex traffic scenarios. Results show that the proposed system has a better performance compared with other methods in controlling traffic lights, even on complex traffic scenarios. [ABSTRACT FROM AUTHOR]

Traffic lights are an important controlling factor in traffic flows, and good policies will facilitate traffic congestion. A car's waiting time is highly related to the period in which the traffic lights are green or red; thus, a proper controlling policy reduces the waiting time and speeds up the car movement. Emergency vehicles always have a higher priority than other cars and need to reach their destination faster. In the last decade, a lot of research has been done to solve the traffic light scheduling issue using artificial intelligence techniques. Reinforcement learning with simulating human learning has the outstanding performance to solve complex problems, such as traffic lights control. In this study, a deep reinforcement learning traffic light control was implemented at a crossroad using real-time traffic information with an emphasis on emergency vehicles. The agent learns to adjust its policies to prioritize the emergency cars over the other cars. The results show less waiting time and speed up the cars' movement. It also indicates the emergency vehicles will cross the intersection with the least delay. With 200 to 600 cars and one emergency vehicle, the average delay decreases 2–16% in total using reinforcement learning. Considering more emergency vehicles, there is a 27–40% decrease in average delay for emergency vehicles. [ABSTRACT FROM AUTHOR]

Improving the efficiency of traffic signal control is an effective way to alleviate traffic congestion at signalized intersections. To achieve effective management of the system-wide traffic flows, current research tends to focus on applying reinforcement learning (RL) techniques for collaborative traffic signal control in a traffic road network. However, the existing collaboration-based methods often ignore the impact of transmission delay for exchanging traffic flow information on the system. Most of the studies assume that the signal controllers can collect all instantaneous vehicular features without delay. To fill the gap, we propose an RL-based cooperative traffic signal control scheme considering the data transmission delay issue in a traffic road network. In this paper, we (1) design our new RL agents to cooperatively control the traffic signals by improving the reward and state representation based on the state-of-the-art max-pressure control theory; (2) propose a traffic state prediction method to address the data transmission delay issue by decreasing the discrepancy between the real-time and delayed traffic conditions; (3) evaluated the performance of our proposed work on both synthetic and real-world scenarios with a different range of data transmission delays. The results demonstrate that our method surpassed the performance of the previous max-pressure-based traffic signal control methods and addressed the data transmission delay issue. [ABSTRACT FROM AUTHOR]

Focusing on the problem with traffic congestion under changing environmental conditions, this paper proposed a trajectory reward light (TR-light) model by combining reinforcement learning, neural network, multi-agent and traffic simulation technology to optimize the traffic at multi-intersections. This method had considerable merits in the following aspects. The traffic organization plan was formulated based on traffic lights; multi-agent reinforcement learning was used on traffic light control; regional traffic organization was optimized through the coordination of traffic lights; the agent implemented trajectory reconstruction after the execution of each behavior so as to change the vehicle travel path without changing the OD pair, and to calculate the final reward of the agent according to the plan and reconstructed trajectory. Finally, it conducted a traffic simulation experiment through SUMO. The comparison of traffic indicators verifies that the proposed model improves the smoothness of the road network and the traffic state at the multi-intersections. Experiments show that the model is feasible and effectively mitigates the traffic congestion. [ABSTRACT FROM AUTHOR]

Cooperative Automated Driving System (ADS)-equipped Vehicles (CoAVs) can be a promising solution to traffic congestion. We present a reinforcement learning strategy using Genetic Fuzzy Systems (GFS) for cooperative merge of vehicles onto a highway in high density, mixed-autonomy traffic conditions. The CoAVs are trained to make their own decisions purely based on information available from its surrounding vehicles, thus making this a decentralized system. The GFS module in each CoAV makes recommendations on the acceleration and lane-change decisions. The CoAVs are trained on different ADS behavioral parameters, such as aggressiveness of lane change, different CoAV market penetration rates (MPRs) and traffic congestion levels. The results show the effectiveness of increasing the MPR. It is noticed that the CoAVs trained with different ADS behavioral parameters can generate completely different cooperative maneuvers. The results also show that the trained CoAVs can cooperate even with human-driven vehicles to improve the overall traffic performance. [ABSTRACT FROM AUTHOR]

Traffic congestion has become common in urban areas worldwide. To solve this problem, the method of searching a solution using artificial intelligence has recently attracted widespread attention because it can solve complex problems such as traffic signal control. This study developed two traffic signal control models using reinforcement learning and a microscopic simulation-based evaluation for an isolated intersection and two coordinated intersections. To develop these models, a deep Q-network (DQN) was used, which is a promising reinforcement learning algorithm. The performance was evaluated by comparing the developed traffic signal control models in this research with the fixed-time signal optimized by Synchro model, which is a traffic signal optimization model. The evaluation showed that the developed traffic signal control model of the isolated intersection was validated, and the coordination of intersections was superior to that of the fixed-time signal control method. [ABSTRACT FROM AUTHOR]

As the global economy develops rapidly, traffic congestion has become a major problem for first-tier cities in various countries. In order to address the problem of failed real-time control of the traffic flow data by the traditional traffic light control as well as malicious attack and other security problems faced by the intelligent traffic light (ITL) control system, a multi-agent distributed ITL control method was proposed based on the fog computing platform and the Q learning algorithm used for the reinforcement learning in this study, and the simulation comparison was conducted by using the simulation platform jointly constructed based on the VISSIM-Excel VBA-MATLAB software. Subsequently, on the basis of puzzle difficulty of the computational Diffie–Helleman (CDH) and Hash Collision, the applicable security control scheme of ITL under the fog computing was proposed. The results reveal that the proposed intelligent control system prolongs the time of green light properly when the number of vehicles increases, thereby reducing the delay time and retention rate of vehicles; the security control scheme of ITL based on the puzzle of CDH is less efficient when the vehicle density increases, while that based on the puzzle of Hash collision is very friendly to the fog equipment. In conclusion, the proposed control method of ITL based on the fog computing and Q learning algorithm can alleviate the traffic congestion effectively, so the proposed method has high security. [ABSTRACT FROM AUTHOR]

In recent years, due to the drastic rise in the number of vehicles and the lack of sufficient infrastructure, traffic jams, air pollution, and fuel consumption have increased in cities. The optimization of timing for traffic lights is one of the solutions for the mentioned problems. Many methods have been introduced to deal with these problems, including reinforcement learning. Although a great number of learning-based methods have been used in traffic signal control, they suffer from poor performance and slow learning convergence. In this paper, a transfer learning-based method for traffic signal control has been proposed. Multi-agent system has also been used for modelling the traffic network and transfer learning has been used to make reinforcement learning agents transfer their experience to each other. Furthermore, a classifier has been utilized to classify the transferred experiences. The results show that using the proposed method leads to a significant improvement on average delay time and convergence time of the learning process. [ABSTRACT FROM AUTHOR]

As urban traffic condition is diverse and complicated, applying reinforcement learning to reduce traffic congestion becomes one of the hot and promising topics. Especially, how to coordinate the traffic light controllers of multiple intersections is a key challenge for multi-agent reinforcement learning (MARL). Most existing MARL studies are based on traditional $Q$ -learning, but unstable environment leads to poor learning in the complicated and dynamic traffic scenarios. In this paper, we propose a novel multi-agent recurrent deep deterministic policy gradient (MARDDPG) algorithm based on deep deterministic policy gradient (DDPG) algorithm for traffic light control (TLC) in vehiclar networks. Specifically, the centralized learning in each critic network enables each agent to estimate the policies of other agents in the decision-making process and each agent can coordinate with each other, alleviating the problem of poor learning performance caused by environmental instability. The decentralized execution enables each agent to make decisions independently. We share parameters in actor networks to speed up the training process and reduce the memory footprint. The addition of LSTM is beneficial to alleviate the instability of the environment caused by partial observable state. We utilize surveillance cameras and vehicular networks to collect status information for each intersection. Unlike previous work, we have not only considered the vehicle but also considered the pedestrians waiting to pass through the intersection. Moreover, we also set different priorities for buses and ordinary vehicles. The experimental results in a vehicular network show that our method can run stably in various scenarios and coordinate multiple intersections, which significantly reduces vehicle congestion and pedestrian congestion. [ABSTRACT FROM AUTHOR]

Many traffic models and control methods have already been utilized in the public transportation system due to the increasing traffic congestion. Thus, an intelligent traffic model is formalized and presented to control multiple traffic light simultaneously and efficiently according to the distribution of vehicles from each incoming link (i.e. sections) in this paper. Compared with constant strategy, two methods are proposed for traffic light control, i.e., game theoretical strategy and reinforcement learning methods. Game theoretical strategy is generated in a game theoretical framework where incoming links are regarded as players and the combination of the status of traffic lights can be regarded as decisions made by these players. The cost function is evaluated and the strategy is produced with Nash equilibrium for passing maximum vehicles in an intersection. The other one is Single-Agent Reinforcement Learning (SARL), specifically with the Q-learning algorithm in this case, which is usually used in such a dynamic environment to control traffic flow so the traffic problem could be improved. Generally, the intersection is regarded as the centralized agent and controlling signal status is considered as the actions of the agent. The performance of these two methods is compared after simulated and implemented in a junction. [ABSTRACT FROM AUTHOR]

For intersection traffic congestion phenomenon, combined fog computing and reinforcement learning theory, this paper proposed a traffic light control model FRTL. The model performed intelligent coordinated control of traffic lights based on real-time traffic flow information. The fog node uploaded the collected real-time traffic flow information to the fog server, and the fog server realized information sharing on the fog platform, and the fog platform combined the processed shared data and Q learning to formulate a traffic light control algorithm. The algorithm used the detected real-time traffic data to calculate a suitable traffic light timing scheme, which was finally applied to the traffic light. The simulation results show that compared with the traditional time-phase control method and the main road control method (ATL), the FRTL control method improves the throughput of intersections, reduces the average waiting time of vehicles, and achieves the goal of properly regulating traffic lights and alleviating traffic congestion. [ABSTRACT FROM AUTHOR]

• Develop a novel two-layer coordinated multi-agent reinforcement learning algorithm. • Synergize traffic signals through local cooperation and global coordination. • Optimization of vehicle emissions and traffic congestion as simultaneous goals. • Proposed method is performed in synthetic network and Cologne urban network on SUMO. Intersection traffic signal control considering vehicle emissions has become an important topic, however, the decision complexity of traffic signal control increases dramatically in a dynamic traffic environment with multi-intersections. It is a severe challenge to coordinate traffic signals at multi-intersections based on Internet of Things information to improve the traffic condition of the road network. This paper proposes a two-layer coordination algorithm based on multi-agent reinforcement learning—Multi-agent Coordinated Policy Optimization (MACoPO), for solving traffic signal control at multi-intersections. MACoPO consists of local cooperation, which adjusts the weights of individual rewards and neighborhood agents' rewards by using local cooperation factors (LCF), and global coordination, which updates the LCF to maximize global rewards. The state and reward functions are designed in terms of the current state of the signal, waiting queue length, vehicle density and emission concentration in the lane, vehicle delay, and vehicle emissions, thus making full use of the intersection state information. The proposed method is extensively assessed through simulation experiments using artificial and real road networks and the numerical results confirm its effectiveness in complex and dynamic real-time traffic environments with multi-intersections. [ABSTRACT FROM AUTHOR]

Recently, with increasing urban traffic congestion, there has been an upsurge in studies on reinforcement learning for traffic signal control (RL-TSC), which enables efficient traffic management. However, most existing RL-TSC research relies on simulation, and instances of real-world application are relatively scarce. Furthermore, existing RL-TSC methods employ a step-based approach, controlling traffic signals every short step. This approach can be inefficient in oversaturated traffic conditions, where large deviations in traffic demand between movements prevent traffic signals from being effectively managed based on the overall traffic situation. In this study, we aim to transform simulation-based RL-TSC into a more practical and applicable model. We have developed a RL-TSC method capable of responding to oversaturated traffic conditions, designing an action space that enables the agent to derive an optimal signal set for every cycle length, thereby understanding the traffic situation across all movements. During each cycle length, our proposed model conducts traffic signal optimization and identifies the optimal signal through an iterative exploration. To facilitate a fast and accurate strategy search process, we developed a kinematic wave-based mesoscopic model. This model estimates the density across the entire link, based on collected traffic data and derives the state and reward values. We have validated the field applicability of the proposed RL-TSC method through a real-world demonstration at a congested intersection in Seoul, Korea. The results indicated a significant improvement in traffic congestion, with the average queue length at the intersection reduced by up to 11.4%. [ABSTRACT FROM AUTHOR]

Designing efficient traffic signal controllers has always been an important concern in traffic engineering. This is owing to the complex and uncertain nature of traffic environments. Within such a context, reinforcement learning has been one of the most successful methods owing to its adaptability and its online learning ability. Reinforcement learning provides traffic signals with the ability automatically to determine the ideal behaviour for achieving their objective (alleviating traffic congestion). In fact, traffic signals based on reinforcement learning are able to learn and react flexibly to different traffic situations without the need of a predefined model of the environment. In this research, the actor–critic method is used for adaptive traffic signal control (ATSC-AC). Actor–critic has the advantages of both actor-only and critic-only methods. One of the most important issues in reinforcement learning is the trade-off between exploration of the traffic environment and exploitation of the knowledge already obtained. In order to tackle this challenge, two direct exploration methods are adapted to traffic signal control and compared with two indirect exploration methods. The results reveal that ATSC-ACs based on direct exploration methods have the best performance and they consistently outperform a fixed-time controller, reducing average travel time by 21%. [ABSTRACT FROM AUTHOR]

• A decentralized DRL signal control for multi-modal traffic networks is introduced based on a microscopic environment. • Car traffic and bus performances are improved simultaneously and traded off. • The proposed method is applicable to different bus lane layouts and multiple bus lines. • The scalability and transferability of the model are verified. Managing traffic flow at intersections in a large-scale network remains challenging. Multi-modal signalized intersections integrate various objectives, including minimizing the queue length and maintaining constant bus headway. Inefficient traffic signals and bus headway control strategies may cause severe traffic jams, high delays for bus passengers, and bus bunching that harms bus line operations. To simultaneously improve the level of service for car traffic and the bus system in a multi-modal network, this paper integrates bus priority and holding with traffic signal control via decentralized controllers based on Reinforcement Learning (RL). The controller agents act and learn from a synthetic traffic environment built with the microscopic traffic simulator SUMO. Action information is shared among agents to achieve cooperation, forming a Multi-Agent Reinforcement Learning (MARL) framework. The agents simultaneously aim to minimize vehicles' total stopping time and homogenize the forward and backward space headways for buses approaching intersections at each decision step. The Deep Q-Network (DQN) algorithm is applied to manage the continuity of the state space. The tradeoff between the bus transit and car traffic objectives is discussed using various numerical experiments. The introduced method is tested in scenarios with distinct bus lane layouts and bus line deployments. The proposed controller outperforms model-based adaptive control methods and the centralized RL method regarding global traffic efficiency and bus transit stability. Furthermore, the remarkable scalability and transferability of trained models are demonstrated by applying them to several different test networks without retraining. [ABSTRACT FROM AUTHOR]

• A lane-level VSL (LVSL) approach with actor-critic framework is proposed to provide differential VSL control. • A hybrid reward function is presented to adjust relationship between the reducing crash risk and improving traffic efficiency. • A twin delayed deep deterministic policy gradient (TD3)-based LVSL approach is established for the first time. • The TD3LVSL shows better performance than baseline models in terms of improving traffic safety and traffic efficiency. Variable speed limit (VSL) control plays a vital role in the emerging connected automated vehicle highway (CAVH) system, which can alleviate recurrent traffic congestion caused by capacity drop and accidents. However, the effect of the VSL on improving traffic efficiency is still controversial. It is necessary to study how to explore the potential benefits of the VSL by balancing its influence on reducing crash risk and enhancing traffic efficiency. To fill the technological gap above, this paper proposes a reinforcement learning-based lane-level VSL (LVSL) control approach to conduct refined traffic control on the mainlines. Firstly, an actor-critic framework is developed to generate and evaluate the discrete speed limits of each lane in continuous action space. To optimize traffic control performance, a hybrid reward function is developed by synchronously considering traffic safety and traffic efficiency of the bottleneck area. Then, to solve the overestimation bias problem of the actor-critic methods caused by function approximation error, a twin delayed deep deterministic policy gradient (TD3) method is introduced to train the framework of the LVSL method. Finally, a real-world recurrent bottleneck of the State Route 91 highway in California is simulated with consideration of the connected automated vehicles to examine the performance of the TD3-based LVSL (TD3LVSL) controller. The simulation results reveal that the proposed method is capable of reducing crash risk and improving traffic efficiency synchronously. Compared with the LVSL controller based on the deep deterministic policy gradient approach, the TD3LVSL controller shows better performance in terms of traffic safety and efficiency. These findings indicate that the proposed controller could contribute to future traffic control in a CAVH environment. [ABSTRACT FROM AUTHOR]

Abstract Traffic signal control plays a pivotal role in reducing traffic congestion. Traffic signals cannot be adequately controlled with conventional methods due to the high variations and complexity in traffic environments. In recent years, reinforcement learning (RL) has shown great potential for traffic signal control because of its high adaptability, flexibility, and scalability. However, designing RL-embedded traffic signal controllers (RLTSCs) for traffic systems with a high degree of realism is faced with several challenges, among others system disturbances and large state-action spaces are considered in this research. The contribution of the present work is founded on three features: (a) evaluating the robustness of different RLTSCs against system disturbances including incidents, jaywalking, and sensor noise, (b) handling a high-dimensional state-action space by both employing different continuous state RL algorithms and reducing the state-action space in order to improve the performance and learning speed of the system, and (c) presenting a detailed empirical study of traffic signals control of downtown Tehran through seven RL algorithms: discrete state Q-learning(λ), SARSA(λ), actor-critic(λ), continuous state Q-learning(λ), SARSA(λ), actor-critic(λ), and residual actor-critic(λ). In this research, first a real-world microscopic traffic simulation of downtown Tehran is carried out, then four experiments are performed in order to find the best RLTSC with convincing robustness and strong performance. The results reveal that the RLTSC based on continuous state actor-critic(λ) has the best performance. In addition, it is found that the best RLTSC leads to saving average travel time by 22% (at the presence of high system disturbances) when it is compared with an optimized fixed-time controller. [ABSTRACT FROM AUTHOR]

Traffic congestion has become a vexing and complex issue in many urban areas. Of particular interest are the intersections where traffic bottlenecks are known to occur despite being traditionally signalized. Reinforcement learning (RL), which is an artificial intelligence approach, has been adopted in traffic signal control for monitoring and ameliorating traffic congestion. RL enables autonomous decision makers (e.g., traffic signal controllers) to observe, learn, and select the optimal action (e.g., determining the appropriate traffic phase and its timing) to manage traffic such that system performance is improved. This article reviews various RL models and algorithms applied to traffic signal control in the aspects of the representations of the RL model (i.e., state, action, and reward), performance measures, and complexity to establish a foundation for further investigation in this research field. Open issues are presented toward the end of this article to discover new research areas with the objective to spark new interest in this research field. [ABSTRACT FROM AUTHOR]

Intelligent Transportation Systems are essential due to the increased number of traffic congestion problems and challenges nowadays. Traffic Signal Control (TSC) plays a critical role in optimizing the traffic flow and mitigating the congestion within the urban areas. Various research works have been conducted to enhance the behavior of TSCs at intersections and subsequently reduce the traffic congestion. Researchers recently leveraged Deep Learning (DL) and Reinforcement Learning (RL) techniques to optimize TSCs. In RL framework, the agent interacts with surrounding world through states, rewards and actions. The formulation of these key elements is crucial as they impact the way the RL agent behaves and optimizes its policy. However, most of existing frameworks rely on hand-crafted state and reward designs, restricting the RL agent from acting optimally. In this paper, we propose a novel approach to better formulate state and reward definitions in order to boost the performance of the traffic signal controller agent. The intuitive idea is to define both state and reward in a consistent and straightforward manner. We advocate that such a design approach helps achieving training stability and hence provides a rapid convergence to derive best policies. We consider the double deep Q-Network (DDQN) along with prioritized experience replay (PER) for the agent architecture. To evaluate the performance of our approach, we conduct series of simulations using the Simulation of Urban MObility (SUMO) environment. The statistical analysis of our results show that the performance of our proposal outperforms the state-of-the-art state and reward design approaches. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL) has been applied to solve ramp-metering problems and attracted increasing attention in recent studies. However, improving traffic efficiency is the main concern of these applications, and the issue relating to user equity has not been well considered. A new RL-based system is developed in this paper to deal with equity-related problems. With the definition of three RL elements, including reward, action, and state, this system can capture the information of user equity and balance it with traffic efficiency. Simulation experiments using real traffic data collected from a real-world motorway stretch are designed to test the performance of the new system. Compared with a widely used ramp-metering algorithm ALINEA, the new system shows superior performance on improving both traffic efficiency and user equity. Specifically, with suitable parameter settings, the new system can reduce the total time spent (TTS) by motorway users by 18.5% and maintain an equally distributed total waiting time (TWT) with a low standard deviation for TWT across on-ramps close to 0. [ABSTRACT FROM AUTHOR]

Recently, Adaptive Traffic Signal Control (ATSC) in the multi-intersection system is considered as one of the most critical issues in Intelligent Transportation Systems (ITS). Among the proposed AI -based approaches, Deep Reinforcement Learning (DRL) has been largely applied while showing better performances. This paper proposes a new DRL -based cooperative approach for controlling multiple intersections. The problem is modelled as a Multi-Agent Reinforcement Learning (MARL) system, while each agent is trained to select the best action to control an intersection by obtaining information about its local lanes state. The cooperation aspect is manifested in this approach by considering the effect of the state, action and reward of neighbour agents in the process of policy learning. An intersection controller applies a Deep Q-Network (DQN) method, while transferring state, action and reward received from their neighbour agents to its own loss function during the learning process. The experimental results under different scenarios shows that the proposed approach outperforms many state-of-the-art approaches in terms of three metrics: Average Waiting Time (AWT), Average Queue Length (AQL) and Average Emission CO 2 (AEC). In addition, the cooperation between the different trained DRL -based controllers allows the system to continuously learn and improve its performance by interacting with the environment, particularly when the traffic is congested. [ABSTRACT FROM AUTHOR]

Traffic signal control is of great importance to the urban transportation systems and public travel, yet it becomes challenging because of two essential factors. First, spatial–temporal correlations are crucial to an intersection scenario. However, existing works have either considered only one of these features or a simple fusion of spatial and temporal information without adequately exploiting the potential correlations. Second, some works using graph neural network treats static graph nodes among adjacent intersections, ignoring the fact that intersection traffic is changing dynamically. These dynamically changing characteristics of an intersection are likewise significant for traffic signal prediction. If these problems are not resolved, the traffic pressure will increase and people's time will be wasted. To resolve these challenges, we put forward a novel meta-learning spatial–temporal graph attention network (MetaSTGAT) for adaptive traffic signal control. Specifically, we design a graph neural framework with a graph attention network (GAT) and long short-term memory (LSTM) network to obtain spatial and temporal information. The spatial–temporal features are elaborately merged to improve its performance. Besides, to adapt the graph network to the dynamic traffic flow, i.e., the dynamics of the nodes, we propose a meta-learning method for graph neural network's weights generation. This dynamic weight generation process captures the dynamic changes of graph nodes, and thus the dynamic changes of intersections. In this way, the neighboring nodes in the graph network get new weights in advance by additional features when they influence each other. Comprehensive experiments performed in the multi-intersection scenario on synthetic and real-world datasets demonstrate the effectiveness of MetaSTGAT against other state-of-the-art methods. Our method reduces travel time by 12.23%, 19.30%, 13.84%, 10.91%, 8.24%, and 8.74% over the graph-level method CoLight on four synthetic datasets and two real-world datasets, respectively. [ABSTRACT FROM AUTHOR]

Traffic signal control (TSC) is an established yet challenging engineering solution that alleviates traffic congestion by coordinating vehicles' movements at road intersections. Theoretically, reinforcement learning (RL) is a promising method for adaptive TSC in complex urban traffic networks. However, current TSC systems still rely heavily on simplified rule-based methods in practice. In this paper, we propose: (1) two game theory-aided RL algorithms leveraging Nash Equilibrium and RL, namely Nash Advantage Actor–Critic (Nash-A2C) and Nash Asynchronous Advantage Actor–Critic (Nash-A3C); (2) a distributed computing Internet of Things (IoT) architecture for traffic simulation, which is more suitable for distributed TSC methods like the Nash-A3C deployment in its fog layer. We apply both methods in our computing architecture and obtain better performance than benchmark TSC methods by 22.1% and 9.7% reduction of congestion time and network delay, respectively. [ABSTRACT FROM AUTHOR]

Abstract: This research presents the application of intelligent computing models in Air Traffic Flow Management (ATFM). Firstly, multi-agent system in grid computing environment is applied to deal with the problem of ATFM synchronization. The developed system consists of software agents, which are implemented in a Computational Grid platform for congestions identification, conflicts resolution and agreements negotiation among the participating airports. A metric criterion, called Agent’s Balancing Standard (ABS), is used as a basic index to measure the effectiveness of reducing both the amount of communication among agents and the delay of flights. Secondly, a brief discussion about the Meta-Level Control model is introduced in ATFM issues to improve the efficiency of communication among the agents. The system is developed to analyze the traffic flow information received, identify its importance and process it in the most adequate order. [Copyright &y& Elsevier]

The prevailing variable speed limit (VSL) systems as an effective strategy for traffic control on motorways have the disadvantage that they only work with static VSL zones. Under changing traffic conditions, VSL systems with static VSL zones may perform suboptimally. Therefore, the adaptive design of VSL zones is required in traffic scenarios where congestion characteristics vary widely over space and time. To address this problem, we propose a novel distributed spatial-temporal multi-agent VSL (DWL-ST-VSL) approach capable of dynamically adjusting the length and position of VSL zones to complement the adjustment of speed limits in current VSL control systems. To model DWL-ST-VSL, distributed W-learning (DWL), a reinforcement learning (RL)-based algorithm for collaborative agent-based self-optimization toward multiple policies, is used. Each agent uses RL to learn local policies, thereby maximizing travel speed and eliminating congestion. In addition to local policies, through the concept of remote policies, agents learn how their actions affect their immediate neighbours and which policy or action is preferred in a given situation. To assess the impact of deploying additional agents in the control loop and the different cooperation levels on the control process, DWL-ST-VSL is evaluated in a four-agent configuration (DWL4-ST-VSL). This evaluation is done via SUMO microscopic simulations using collaborative agents controlling four segments upstream of the congestion in traffic scenarios with medium and high traffic loads. DWL also allows for heterogeneity in agents' policies; cooperating agents in DWL4-ST-VSL implement two speed limit sets with different granularity. DWL4-ST-VSL outperforms all baselines (W-learning-based VSL and simple proportional speed control), which use static VSL zones. Finally, our experiments yield insights into the new concept of VSL control. This may trigger further research on using advanced learning-based technology to design a new generation of adaptive traffic control systems to meet the requirements of operating in a nonstationary environment and at the leading edge of emerging connected and autonomous vehicles in general. [ABSTRACT FROM AUTHOR]

Inefficient traffic control may cause numerous problems such as traffic congestion and energy waste. This paper proposes a novel multi-agent reinforcement learning method, named KS-DDPG (K nowledge S haring D eep D eterministic P olicy G radient) to achieve optimal control by enhancing the cooperation between traffic signals. By introducing the knowledge-sharing enabled communication protocol, each agent can access to the collective representation of the traffic environment collected by all agents. The proposed method is evaluated through two experiments respectively using synthetic and real-world datasets. The comparison with state-of-the-art reinforcement learning-based and conventional transportation methods demonstrate the proposed KS-DDPG has significant efficiency in controlling large-scale transportation networks and coping with fluctuations in traffic flow. In addition, the introduced communication mechanism has also been proven to speed up the convergence of the model without significantly increasing the computational burden. • A multi-agent reinforcement learning for adaptive signal control. • Agents can communicate with others through knowledge-sharing protocol. • Proposed algorithm achieves consistent improvements over baselines. [ABSTRACT FROM AUTHOR]

Traffic signal control helps to reduce traffic congestion and thus has been studied for a few decades. Algorithms for multi-intersection traffic signal control that adopt hierarchical deep reinforcement learning have been shown to achieve state-of-the-art performance. For these methods, a two-level hierarchical structure is generally used, in which the low-level policies employ latent goals induced by the high-level policies. Existing research either designs the latent goals manually or acquires the latent goals from the environment. However, these latent goals are not optimal, which leads to the non-optimal low-level policies. To improve this, we propose a learned-goal soft actor–critic (LSAC) algorithm, by which the optimal latent goals are automatically learned and then are used in low-level policies. We then propose a semi-decentralized feudal multi-agent (SFM) framework, which can alleviate the problem that existing multi-agent framework faces, i.e., the fast growing state space with the increase of the number of agents. Combining the above two proposed methods, an SFM-LSAC algorithm is proposed in this paper for multi-intersection traffic signal control. Other techniques are also incorporated into the SFM-LSAC algorithm such as attention mechanism. Experimental results in three real-world traffic scenarios illustrate that the SFM-LSAC algorithm outperforms other four state-of-the-art multi-intersection traffic signal control algorithms, i.e., reduces the average intersection delay, the average queue length, the average travel time, meanwhile improves the average flow and average travel speed, thus could be a new promising algorithm for multi-intersection traffic signal control. • A representation learning algorithm is proposed to learn the optimal latent goals. • A learned-goal soft actor–critic algorithm is proposed as one traffic-signal agent. • A semi-decentralized feudal multi-agent framework is proposed to lower state space. • Based on above methods, the whole algorithm is proposed for traffic signal control. • Experiments show the whole algorithm outperforms the state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

As traffic congestion in cities becomes serious, intelligent traffic signal control has been actively studied. Deep Q-Network (DQN), a representative deep reinforcement learning algorithm, is applied to various domains from fully-observable game environment to traffic signal control. Due to the effective performance of DQN, deep reinforcement learning has improved speeds and various DQN extensions have been introduced. However, most traffic signal control researches were performed at a single intersection, and because of the use of virtual simulators, there are limitations that do not take into account variables that affect actual traffic conditions. In this paper, we propose a cooperative traffic signal control with traffic flow prediction (TFP-CTSC) for a multi-intersection. A traffic flow prediction model predicts future traffic state and considers the variables that affect actual traffic conditions. In addition, for cooperative traffic signal control in multi-intersection, each intersection is modeled as an agent, and each agent is trained to take best action by receiving traffic states from the road environment. To deal with multi-intersection efficiently, agents share their traffic information with other adjacent intersections. In the experiment, TFP-CTSC is compared with existing traffic signal control algorithms in a 4 × 4 intersection environment. We verify our traffic flow prediction and cooperative method. [ABSTRACT FROM AUTHOR]

Intelligent traffic signal control helps to reduce traffic congestion and thus has been studied for a few decades. Multi-intersection cooperative traffic signal control (CTSC), which is more practical than single-intersection traffic signal control, has attracted much attention and research in recent years. Existing works on multi-intersection CTSC make responsive policies based on the sequence of agents' actions. One issue in multi-intersection CTSC is that every agent's actions are mapped from its own road information and some useful information, e.g., the distance of adjacent agents, is ignored, which may lead to suboptimal traffic signal control policies. To address this issue, in this paper a decentralized coordination graph algorithm, referred to as Multi-step return and Off-policy Asynchronous Advantage Actor-Critic Graph (MOA3CG) algorithm, is proposed. The MOA3CG algorithm is based on an asynchronous method of multiagent deep reinforcement learning and a coordination graph; the proposed algorithm makes traffic signal control policies based on current traffic states, the history of observations and other information. A new reward function and An Adjusting Matrix of Traffic Signal Phase Control (AMTSPC) are proposed, which are used by the MOA3CG algorithm in the policy-making process; the AMTSPC is to alter selection of actions by considering the distance of adjacent agents. Experimental results on real-world road scenarios show that the proposed algorithm outperforms other four state-of-the-art algorithms in terms of average delay, average traveling time of vehicles, and the throughput of vehicles, thus eventually helps to mitigate traffic congestion. • MOA3CG algorithm is proposed for multi-intersection cooperative traffic signal control. • The MOA3CG algorithm is based on multi-step return, off-policy and a graph. • An AMTSPC for selection of actions is proposed and adopted in MOA3CG. • Experiments show that MOA3CG algorithm outperforms the state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Reinforcement learning (RL)-based traffic signal control has been proven to have great potential in alleviating traffic congestion. The state definition, which is a key element in RL-based traffic signal control, plays a vital role. However, the data used for state definition in the literature are either coarse or difficult to measure directly using the prevailing detection systems for signal control. This paper proposes a deep reinforcement learning-based traffic signal control method which uses high-resolution event-based data, aiming to achieve cost-effective and efficient adaptive traffic signal control. High-resolution event-based data, which records the time when each vehicle-detector actuation/de-actuation event occurs, is informative and can be collected directly from vehicle-actuated detectors (e.g., inductive loops) with current technologies. Given the event-based data, deep learning techniques are employed to automatically extract useful features for traffic signal control. The proposed method is benchmarked with two commonly used traffic signal control strategies, i.e., the fixed-time control strategy and the actuated control strategy, and experimental results reveal that the proposed method significantly outperforms the commonly used control strategies. [ABSTRACT FROM AUTHOR]